Testing usability of ERP open source systems

Testing Usability of ERP Open Source Systems Carlos J. Costa ISCTE- IUL / Adetti-ISCTE Av. Das Forças Armadas Lisboa - PORTUGAL [email protected]

We suggest that the use of usability criteria depends on the performance evaluation. It is important identifying what are the most significant issues related to performance in order to identify the specific usability issues to identify. In the following section, we present the main usability criteria that are mostly mentioned in the literature and used in empirical and practical evaluation of usability. Then we identified the most the dependent variables that may be used to measure performance. In the next section, we identified the criteria and heuristics, in order to identify the possible independent variables of the model. Then, three models were estimated, using multivariated regression. Finally, results were analyzed.

ABSTRACT In this paper we estimated three models that may be used to identify performance of a ERP system. As dependent variables, generic performance variables were used (effectiveness, efficiency and satisfaction). As independent variables we used usability criteria. Those were derived from literature and but their inclusion in the models was estimated. In what concerns independent variables several criteria were identified like existence of online support, easiness of interpretation of output, support to navigation, help and documentation or minimize of memory load.

Categories and Subject Descriptors H.4.1 [Information System Applications]: Office Automation – Groupware. H.5.3 [Information System Applications]: Group and Organization Interfaces – Computer-supported cooperative work. K.4.3 [Computers and Society]: Organizational Impacts – Computer-supported cooperative work

2. ERP AND USABILITY According to several researchers, several categories of problems related to usability may be found in ERP systems[18][19]: - Identification of and access to the correct functionality (what may include Navigation problems or Difficulties in understanding the dependencies between the modules)

General Terms Documentation, Measurement, Performance, Experimentation, Human Factors

- Lack of Transaction execution support (corresponding to Unduly complex transactions, Need to enter data repetitively, Inconsistent behavior or Poor support for exception situations leads to system avoidance)

Keywords Usability, ERP, Organizational Systems

- System output limitations may include Inability to get required output, Need to use external tools to process the data further or Cognitive complexity of query tools

1. INTRODUCTION In this paper we analyze the main approaches that may be used to evaluate usability of a web based organizational system. Supported in the generic usability criteria (e.g. [11], [4]), we identified specific criteria. Those criteria are based in the assumption, that organizational systems are used to support a specific organizational or administrative task (e.g. entering an order, printing an invoice). Base in those criteria a evaluation tool was developed. This tool was applied to a small group of users. We also applied other tools, including Nielsen the heuristic approach. Results were analyzed statistically. A small report was also analyzed by a team of developers, in order to identify the usefulness of results obtained. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. OSDOC'10, November 8, 2010, Lisbon, Portugal. Copyright 2010 ACM 978-1-4503-0480-1/10/11..$10.00.

- Support in error situations consists of Incorrect or insufficient error messages, Lack of specificity and Missing error messages - Terminology problems are related to unfamiliar system language and the Need for a glossary - Overall system complexity may be a General feeling of overwhelming complexity leading to feelings of fear There are several ways of evaluating usability. Several techniques and methods may be employed. According to Preece [20], there are usability evaluation methods: expert evaluation, observational evaluation, survey evaluation and experimental evaluation. Each methods is implemented though different types of evaluators, different number of users, and different types of data to be collected.

25

Table 1- Usability Evaluation Methods and Techniques

Techniques Method

Expert/ heuristc Observation Survey Experimental

Q u e s t i o n n a i r e s

W a l k t h r o u g h

X

D i r e c t o b s e r v a t i o n

V i d e o r e c o r d i n g

I n t e r v i e w s

S o f t w a r e l o g g i n g

Ve r­ bal pr o­ to­ col s (th in k alo ud )

X

X

X X

In an experimental evaluation an evaluator can manipulate a number of factors associated with the interface and study their effect on user performance. It may be also used to compare the impact of several interfaces in different users.

Help users recognize, diagnose, and recover from errors

10.

Help and documentation

The system must help users recognize, diagnose, and recover from errors. Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.

Some researchers suggest the use of heuristics to analyze usability[1],[2], [3].They are called heuristics because they are more in the nature of rules of thumb than specific usability guidelines. According to Nielsen and colleagues, those heuristics may be reduced to 10:

Consistency and standards

9.

Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility The system must have an aesthetic and minimalist design -

Some issues that are considered as part of usability by some authors are considered as criteria includes not included in the evaluation of usability like utility, likeability and cost [15] or utility, .cost,. compatibility,, reliability and usefulness [11]

4.

Aesthetic and minimalist design

Accelerators (unseen by the novice user) may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions. It is why Nielsen considered that flexibility and efficiency of use are related.

A brief literature review allowed us to identify the following items to evaluate usability ([5],[9],[10],[11],[12],[13],[14],[15], [16],[17]): learnability , efficiency memorability, effectiveness, errors, effectiveness, satisfaction, ease of use , understandability, operability, usefulness, navigation, task support , screen, presentation, customization, terminology and system information, attitude , overall reaction to software, system capabilities, flexibility, content, accessibility, media use, interactivity, consistency, and attractiveness.

User control and freedom

8.

Minimize the user's memory load by making objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate. The authors, summarized: recognition rather than recall.

3. ERP CRITERIA

3.

Flexibility and efficiency of use – Accelerators

Even better than good error messages is a careful design which prevents a problem from occurring in the first place. Either eliminate error-prone conditions or check for them and present users with a confirmation option before they commit to the action.

It is necessary to plan everything very carefully: required level of user experience, hypotheses to be tested, the structure of tasks, time needed to complete the experiment. Several techniques may be used. In previous research we developed direct observation and log analysis crossed with questionnaires.[22]

Match between system and the real world

7.

Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions (Consistency and standards).

Adapted from [20],[21].

2.

Recognition rather than recall

User control and freedom mean that users often choose system functions by mistake and will need a clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialogue. Support undo and redo.

X

Visibility of system status

6.

- The system should speak the users' language, with words, phrases and concepts familiar to the user, rather than systemoriented terms. Follow real-world conventions, making information appear in a natural and logical order. It is why there must be a match between system and the real world

X X

1.

Error prevention

Visibility of system status means that the system should always keep users informed about what is going on, through appropriate feedback within reasonable time.

X X X

5.

Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large. Shneiderman’s eight golden rules for user interface design are [4]: 1. Strive for consistency;

26

2. Cater to universal usability;

-The system includes Error prevention

3. Offer informative feedback;

-Minimize the user's memory load

4. Design dialogs to yield closure;

-The system is Flexible and efficient to use

5. Prevent errors;

- The system is Aesthetic and minimalist design

6. Permit easy reversal of actions;

- The system Help users recognize, diagnose, and recover from errors

7. Support internal locus of control; and

-The system supplies help and documentation

8. Reduce short-term memory load.

In what concerns, criteria, related to ERP systems usability, we adopted the ones proposed by [5].

Other researchers proposed criteria that are specific to ERP. For example Singh and Wesson [5] suggest the following five: Navigation

A system has characteristics that improve his navigation if the following items happen:

Presentation

- Information is easy to find

Task Support

- Functionalities are easy to find.

Learnability Customization.

- There is a form of guidance within the system to aid the user when completing a business process.

4. USABILITY AND PERFORMANCE

- The user interface support a correct and effective navigation though the system - There is search functionality.

We also may analyze in what extends a system may have impact in the performance.

- Navigation does not support the different types of users.

The performance may be analyzed at individual, group or organizational level.

- The system gives information to help orienting the users Another issue very important his the presentation. It means that the

On the other hand, several metric may be used to analyze effectiveness, efficiency and even satisfaction. [6],[7], [14].

- The layout is well designed

Table 2- Performance Evaluation Individual

Group

- The information supplied by the system is useful, accurate, complete and comprehensive

Organization

Effectiveness

- Output is easy to understand and interpret.

Efficiency

- The output supplies a clear view to all the other departments

Satisfaction

- The user interface of the system is very intuitive. One characteristic that is especially important is the task support. In order that it supports tasks, the following attribute must be present in those systems:

Some research not only suggests that organizational and individual objectives may be coherent; they even suggest ways to reach this purpose [8].

-Terminology used is consistent with that of the user. - System is easy to use.

In our experience analyzing usability, it is very difficult to relating organizational and group performance with usability. So we analyzed just at the individual level.

ERP are characterized by being very complex system. Often there is a need for long period of learning. Certifications and courses are created to solve this problem. So, learnability is an issue that must be studied. We may ask if:

On the other hand, at the individual level, we may have several problems concerning what people think that is expected from them and also his own perception of efficiency and effectiveness.

5. CRITERIA DETAIL

AND

HEURISTICS

- Long introduction is not needed to learn how to use the system. - There is enough online support in the learning phase

IN

Customization is another item that is very important, specially, in this kind of systems that have an important level o customization:

From the usability heuristic [1],[2], [3], we may suggest that a system should have the following characteristics:

- The system is very customizable

-There is a Visibility of the system status -

6. METHODOLOGY

-There is a Match between system and the real world

The main purpose of the research was identifying three models that could explain performance (yi) using as independent variables usability criteria (xi) .

-The system allows User control and freedom -The system is Consistent and uses standards

27

Three linear models will be estimated:

following:

y1= a1x1+ a2x2+ a3x3 + .. + anxn

42 - There is enough online support in the learning phase

y2= a1x1+ a2x2+ a3x3 + .. + anxn

30 - Output is easy to understand and interpret

y3= a1x1+ a2x2+ a3x3 + .. + anxn

32 - The output supplies a clear view to all the other departments

Where:

11 - Match between system and the real world

y1 - Efficiency

29 - The information supplied by the system is useful, accurate, complete and comprehensive

y2 – Satisfaction

C - Constant

y3 – Effectiveness.

This model suggests that online support in the learning phase is important in order that users may feel efficient. Characteristics of the system, like simplicity of understanding and interpret, clearness of views of other departments and math between system and world are also important in order to make users feel more efficient.

Our purpose is identifying the weight (ai) associated with each one of the independent variables (xi). We used a small group of users of ERP systems. It was our purpose choosing one users that were no novices and choosing different systems (e.g. SAP, Sage, Frontaccounting) in order to have differences in what concerns usability.

Model 2

As statistical method we used multiple regression forward stepwise.

The second model has as main purpose identifying the criteria that explains satisfaction. Table 6- Model 2

7. RESULTS Model 1 The first model has as main purpose identifying the criteria that explains efficiency Table 3- Model 1 SOURCE

DF

SS

MS

F

Prob.>F

Regression

5

45.390

9.078

36.400

0

Residual

13

3.242

0.249

F 36.400

Adjusted R Squared =

SS

MS

F

Regression Residual Total

3 15 18

35.515 11.117 46.632

11.838 0.741

15.974

R 0.873

R2 0.762

F 15.974

Prob.>F 0.000

DF1 3

DF2 15

Adjusted R Squared = 0.714

Prob.>F 0.000

DF1 5

Std. Error of Estimate =

DF2 13

0.861 Table 8- Model 2

Beta

B

0.749 0.474 -0.343 0.070

0.74 0.51 -0.34

0.908

Std. Error of Estimate =

Prob.> F 0.000

Table 7- Model 2

Table 4- Model 1 R2 0.933

DF

Dependent Variable: "53. . Satisfaction - The system contribute to my satisfaction"

Total 18 48.632 Dependent Variable: 54. Efficiency - The system contribute to my efficiency (e.g. reducing time spent in a task).

R 0.966

SOURCE

23 19 15 C

0.499 Table 5- Model 1

Beta

B

Std. Error

t

Prob.>t

42

0.31

0.336

0.101

3.321

0.006

30

0.355

0.387

0.100

3.859

0.002

32

0.336

0.429

0.111

3.885

0.002

11

0.240

0.234

0.082

2.842

0.014

29

0.209

0.317

0.133

2.377

0.033

C

-2.804

Std. Error 0.151 0.149 0.153

t

Prob.>t

4.914 3.399 -2.226

0.000 0.004 0.042

The independent variables obtained from the model were the following: 23 - The user interface support a correct and effective navigation though the system 19 - The system supplies help and documentation 15 - Minimize the user's memory load C - Constant Support to navigation and help and documentation are issues that contribute to the improvement of satisfaction. Minimizing the user's memory load is not considered a positive impact in

The independent variables obtained from the model were the

28

satisfaction in this model.

system.

This result is not explained. We may hypothesize that people feel more satisfied if they know more about the system.

26 - Navigation does not support the different types of users 27 - The system gives information to help orienting the users C – Constant

Model 3

In order to analyze in what extent each one of the criteria was perceived uniformly by the users we asked a group of 29 users to use the same ERP. Then we asked them to score the system using usability criteria. It was quite intriguing verifying that there was a significant difference in perception, what may be shown by the SD (standard deviation).

The third model has as main purpose identifying the criteria that explains effectiveness. Table 9- Model 3 SOURCE

DF

SS

MS

F

Regression Residual Total

10 8 18

48.448 0.184 48.632

4.845 0.023

211.09

Prob. >F 0.000

Table 12- Criteria

Dependent Variable: "55. Effectiveness – The system contribute to my effectiveness (e.g. improving the probability of concluding a task successfully). Table 10- Model 3 R 0.998

R2 0.996

F 211.088

Prob.>F 0.000

DF1 10

DF2 8

Adjusted R Squared = 0.992 Std. Error of Estimate =

0.151

Table 11- Model 3 Beta 22 19 32 11 30 15 42 40 26 27 C

0.371 0.400 0.377 0.415 0.145 -0.203 -0.258 0.185 -0.175 0.124 -1.997

B 0.425 0.437 0.482 0.404 0.158 -0.206 -0.275 0.185 -0.170 0.143

Std. Error 0.045 0.041 0.044 0.029 0.039 0.031 0.041 0.044 0.039 0.047

t 9.426 10.690 10.959 13.953 4.069 -6.597 -6.706 4.228 -4.373 3.069

Criteria

Mean

SD

Visibility of system status

4,93

1,23

Match between system and the real world User control and freedom

5,24

1,22

5,17

1,12

Consistency and standards

4,93

1,11

Error prevention

4,761

1,04

Recognition rather than recall

4,90

1,30

Flexibility and efficiency of use

4,86

1,43

Aesthetic and minimalist design

5,38

1,32

Help users recognize, diagnose, and recover from errors Help and documentation

4,70

1,21

4,79

1,27

Then we used specific criteria and the standard deviation reduced slightly.

Prob.> t 0.000 0.000 0.000 0.000 0.004 0.000 0.000 0.003 0.002 0.015

Table 13- Criteria Criteria

Mean

SD

Navegation

4,82

0,713

Presentation

5

0,66

Support to Task

5,26

0,76

Learnability

4,37

0,72

Customization

4,75

0,73

Results presented here are very preliminary, but may suggest that using specific usability criteria may be better than using generic usability criteria.

The independent variables obtained from the model were the following: 22 - There is a form of guidance within the system to aid the user when completing a business process.

8. CONCLUSIONS

19 - The system supplies Help and documentation

In this paper we estimated three regression models that may be used to identify performance of a ERP system. As dependent variables, generic performance variables were used (effective­ ness, efficiency and satisfaction). As independent variables we used usability criteria. Those were derived from literature. Inclusion of the independent variable in each of the models was estimated using multiple regressions forward step-wise. Results suggest that some usability criteria considered as positive in a system evaluation have negative impact in a performance model.

32 - The output supplies a clear view to all the other departments 11 - There is a Match between system and the real world 30 - Output is easy to understand and interpret. 15 - Minimize the user's memory load 42 - There is enough online support in the learning phase 40 - Long introduction is not needed to learn how to use the

29

Personnel Psychology, vol. 49, 1996, pp. 1-49. [9] A. M. Lund (2001) Measuring Usability with the USE Questionnaire. STC Usability SIG Newsletter, 8:2 [10]F. Davis, . Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly, 13:3, 1989 pp 319-340 [11] J. Nielsen. Usability Engineering. Academic Press. Chapter 2.2, 1993 p. 26 [12] J. Chin, V. Diehl and K. Norman, Development of an Instrument Measuring User Satisfaction of the HumanComputer Interface. ACM CHI'88 Proceedings, 1988, pp. 213-218 [13] ISO 9126 - 1 Software egineering - Product quality - Part 1: Quality Model (2000) [14] ISO 9241-11 Ergoniomic requirements for office work with virtual display terminals (VDTs) Part 11: Guidance on Usability [15] B. Shackel, Usability - Context, framework, design and evaluation in Shackel, B & Richardsdon, S. (eds) Human Factors for Informatics Usability. Cambridge University Press, Cambridge 1991, pp. 21-38. [16] S Hassan, and F. Li, Utilising IGV approach to identify actors affecting web usability. Journal of ICT, 2 (2), 2003, pp. 25-40. [17] S. Hassan and F. Li Identifying Web Usability Criteria: The 'Scanmic' Model; Research Paper No. 2001/3; Managemenr Science: Theory, method & Practice; Strathclyde Business School. [18 ] H. Topi, W. T. Lucas, T. Babaian: Identifying Usability Issues with an ERP Implementation. ICEIS 2005 pp. 128133 [19] T. Babaian, W. Lucas, and H. Topi. Improving ERP usability through user-system collaboration. International Journal of Enterprise Information Systems, 2(3):10: 23, 2006. [20] J. Preece A Guide to Usability: human factors in computing. 1993. Addison Wesley,.. [21] M. Piteira, Contribuição para a avaliação da usabilidade da plataforma de elearning moodle, MSc Thesis, 2006, ISCTE. [22] C. Costa, "ERP Open Source," in Information Technology, Organizations and Teams, C. J. Costa, Ed. press.itml.org, 2007, pp. 159–170 [23] N. Lopes, C.. Costa: ERP localization: exploratory study in translation: European and Brazilian Portuguese. Henrique O'Neill, Manuela Aparicio, Carlos J. Costa, Aristidis Protopsaltis (Eds.): Proceedings of the 26th Annual International Conference on Design of Communication, SIGDOC 2008, Lisbon, Portugal, , 93-98.

Some those results are interesting, but additional empirical study must be performed. Behavioral tests must complement surveys. In fact, surveys prove to be insufficient in this context. So we suggest that some scenarios be designed. Supported on those scenarios experimental studies may be performed. On the other hand surveys must include more subjects and analyzed in a cross temporal period. At this moment several of the tasks involved in the use of ERP is already being analyzed. The purpose is identifying some of the most significant tasks.

9.ACKNOWLEDGMENTS This paper is partially supported by FCT – Portuguese Ministry of Research and Development and High Education

10. REFERENCES [1] R. Molich, and J. Nielsen, . Improving a human-computer dialogue, Communications of the ACM 33, 3 (March), 1990 pp. 338-348. [2] J . Nielsen,., and R. Molich, Heuristic evaluation of user interfaces, Proc. ACM CHI'90 Conf. (Seattle, WA, 1-5 April), 1990. Pp. 249-256. [3] J. Nielsen,. Heuristic evaluation. In Nielsen, J., and Mack, R.L. (Eds.), Usability Inspection Methods, John Wiley & Sons, 1994, New York, NY. [4] B. Shneiderman, Designing the user interface: strategies for effective human-computer interaction. Third Edition , . 1998 Addison Wesley Longman, [5] A. Singh and J. Wesson, “Evaluation criteria for assessing the usability of ERP systems,” Proceedings of the 2009 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists, 2009, pp. 87–95. [6] E. Frøkjær, M. Hertzum, and K. Hornbæk, “Measuring usability: are effectiveness, efficiency, and satisfaction really correlated?,” Proceedings of the SIGCHI conference on Human factors in computing systems, The Hague, The Netherlands: ACM, 2000, pp. 345-352. [7] A.Y. Lewin and J.W. Minton, “Determining Organizational Effectiveness: Another Look, and an Agenda for Research,” Management Science, vol. 32, May. 1986, pp. 514-538. [8] A.L. Kristof, “Person-organization fit: An Integrative review of its conceptualizations, Measurements and implications”

30