Using Ergonomic Rules for Evaluation by Linguistic Ergonomic Cri- teria

Using Ergonomic Rules for Evaluation by Linguistic Ergonomic Criteria François Bodart and Jean Vanderdonckt Institut d'Informatique, Facultés Universitaires Notre-Dame de la Paix, rue Grandgagnage, 21, B-5000 NAMUR (Belgium) - Tel. : +32 (0)81-72.49.75 Fax. : +32 (0)81-72.49.67 - Email : {fbodart, jvanderdonckt}@info.fundp.ac.be 1. INTRODUCTION Heuristic Evaluation is a well established method for evaluating a user interface (UI) of an interactive application [1]. This method belongs to the class of informal evaluation methods since the evaluation is performed on the basis of evaluator expertise and knowledge. One characteristic of heuristic evaluation is to guide evaluators by providing them a small set of predefined heuristics rather than a long list of general guidelines. Evaluators are given a widely applicable description of these heuristics so that they are general and largely applicable [2]. In evaluation by ergonomic criteria [3], heuristics that have been judged too general are replaced by a taxonomy of eight main ergonomic criteria (i.e., guidance, workload, explicit control, adaptability, error management, consistency, significance of code, compatibility). These criteria could be decomposed into subcriteria leading to eighteen elementary (sub-)criteria. These criteria present at least three advantages : 1. they are precisely defined, especially when distinguishing between concurrent criteria; 2. they have been experimentally tested and validated; 3. they could be directly linked to useful ergonomic rules (or guidelines). In this paper, we introduce the evaluation by linguistic ergonomic criteria which is an extension of the latter method. First, the reasons that motivated this extension are presented by showing their definition. Second, first steps toward a complete evaluation method are outlined. Third, we exemplify this approach by detailing a UI evaluation report. Finally, we discuss our experience with this work in progress. 2. EVALUATION BY LINGUISTIC ERGONOMIC CRITERIA 2.1. DEFINITION OF LINGUISTIC ERGONOMIC CRITERIA

An ergonomic criteria is a well recognized usability dimension in human-computer interaction whose reliability effectiveness and usability have been assessed [3,4]. Linguistic ergonomic criteria are criteria that have been linguistically decomposed into sub-criteria according to the seven layers of Nielsen's linguistic

model of interaction [5]. They consist of eight main criteria : compatibility, consistency, work load, adaptability, dialog control, representativeness, guidance, and error management. The idiosyncratic logic that has been followed to produce these criteria involves the steps indicated below. The eight main criteria were retained and a general wording was adopted for each to allow a full linguistic decomposition. For example, "dialog control" replaced "explicit control" since dialog control does not necessarily have to be restricted to explicit. The main criteria were sorted by decreasing rank of importance to set an explicit ordering between criteria. This absolute ordering provides a first way to a relative ordering when weighting criteria for a particular interactive task. For instance, compatibility is supposed to be more important than consistency : a compatible, but inconsistent, UI is preferable to a consistent, but incompatible, UI since a consistent UI is unuseful if not compatible with the user's task. Similarly, reducing the work load of a UI is inefficient if it is not at least consistent (consistency may reduce work load). Other criteria are ranked in the same way. The eight main criteria were subdivided into sub-criteria according to the seven layers of Nielsen's model [5] : goal, pragmatic, semantic, syntactic, lexical, alphabetic, and physic. These seven layers are highlighted for each criteria after separating important aspects : consistency is divided into consistency between applications (inter-application consistency) and consistency within a same application (intra-application consistency). Intra-application is further refined into linguistic sub-criteria as depicted in table 1. Table 1. Linguistic sub-criteria of consistency 2. Consistency 2.1 Inter-application Consistency 2.2 Intra-application Consistency 2.2.1 Pragmatic Consistency 2.2.2 Semantic Consistency 2.2.3 Syntactic Consistency 2.2.3.1 Operational Consistency 2.2.3.2 Homogeneousness 2.2.4 Lexical Consistency 2.2.4.1 Spatial Consistency 2.2.4.2 Grammatical Consistency 2.2.4.3 Linguistic Consistency 2.2.5 Alphabetical Consistency 2.2.6 Physical Consistency Compatibility, work load and adaptability have been directly refined in the same way, whereas dialog control, representativeness and guidance include a first separation between conversation (dynamic behavior) and presentation (static

appearance). Error management escapes from this rule since it follows a chronological decomposition. Criteria are consequently decomposed into sub-criteria which are sorted by decreasing linguistic level. Within a same criteria, higher-level sub-criteria are therefore considered as more important than lower-level sub-criteria. Elementary criteria which have been proved successfully assessed (i.e. sub-criteria leading to a high recognition rate in the reading matrix) have been extracted from [4] and merged into this decomposition. Finally, special criteria have been extracted from Coutaz's design criteria [6] and Ravden & Johnson's evaluation criteria [7] because practical experience showed that it was useful to identify them as a whole in recognition procedures. Moreover, managing and classifying ergonomic rules (as in [8]) has been greatly facilitated by the introduction of these fine sub-criteria They include : homogeneousness, concision, action reversibility (especially), action structure, visual clarity, explicitness,... This approach resulted in a set of eighty two elementary (sub-)criteria including the eight main criteria. A complete list of ergonomic criteria with definition, aims, decomposition and examples is electronically available through anonymous FTP at arzach.info.fundp.ac.be/~jvd/Criteria.ps and via Mosaic at http://www.info.fundp.ac.be/~jvd/Criteria.http. 2.2. Toward an evaluation method by linguistic ergonomic criteria We successfully used the following approach both in TRIDENT project [9] and Human-Computer Interaction course in a Master of Computer Science degree for four years. This approach consists in performing the following steps : 1. conduct a context analysis (i.e. a task analysis, an identification of user stereotypes, a description of work place) resulting in a series of parameters for task, user and work place (e.g. pre-requisite, objective task environment, task structure, task experience,...) 2. select appropriate ergonomic (sub-)criteria with respect to parameters listed in context analysis. Thus, specific ergonomic criteria matching the user's task are raised. This selection could be achieved manually or in a computer-aided fashion with production rules. We are currently developing a knowledge base containing such production rules (Figure 1). task profile = closed
user level = inexperienced  criteria (Explicit Actions) = very important task profile = closed
user level = intermediate  criteria (Mixed Actions) = very important task profile = closed
user level = experienced  criteria (Implicit Actions) = important task structure = high
objective environment = existing  criteria (Support Compatibility) = very important

Figure 1. Some production rules for deriving (sub-)criteria from context analysis. 3. sort all specific (sub-)criteria by rank of importance. This step is helped by existing orderings implied by criteria and by linguistic levels.

4. gather ergonomic rules for each (sub-)criteria into a list. This activity involves accessing ergonomic rules in an ergonomic knowledge base. This could be done manually by writing a checklist of ergonomic rule titles to evaluate by extracting them from style guide, standards or design guides. [9] is an example of multi-purpose design guide containing 3,700 ergonomic rules sorted by division (e.g. input, display, dialog, visual design, interaction media, interaction styles,...) by criteria and by linguistic level. We are currently implementing a hypermedia tool allowing automatic rule gathering by criteria, linguistic level, interaction style, and interaction objects. 5. define one or two task scenarios for each interactive task of the UI. for windows-based applications, this usually consists of a series of sub-tasks matching all menu item active in pull-down menus of the menu bar. For direct manipulation UI, this activity typically represent a sequence of direct actions of interaction objects to reach a task's goal. 6. experience the UI through defined task scenarios one after another ; identify, explain and report usability problems in the light of the checklist of specific ergonomic rules. Of course, problems without linkage to these rules could also be reported. 7. experience the whole UI globally and consolidate usability problems into an UI evaluation report (in particular, for intra-application consistency). 2.3. UI Evaluation report Such a report often contains steps involved in task scenarios, screen snapshots at a given time during execution and pointers of usability problems. Each problem is fully explained and argumented by one or many related ergonomic rules, themselves sorted by criteria. Figure 2 highlights a UI hard copy augmented by numbered arrows. Each number denotes a usability problem reported below. 2.4. List of usability problems (excerpt) 1. "Base line Data", "Actuals Data", "Project Trace",... are windows related to the current activity in the hierarchical project decomposition. The user always has to display these windows for each new activity. Ergonomic rule : coordinated windows should be used for dependent tasks Reference : 6.13.2.1*12, p. 433 in [8] Ergonomic criteria : minimal actions Linguistic level : lexical Possible solution : coordinated windows so that their contents automatically changes when current selection changes. 2. Four windows are displayed separately, whereas their information are all related to current selection. Ergonomic rule : multi-windowing should be used for independent tasks Reference : 6.13.2.1*1, p. 430 in [8] Ergonomic criteria : operational compatibility Linguistic level : syntactic

6. Too much windows at the same time leading to unnecessary cluttering Ergonomic rule : simultaneously displayed windows should not exceed 6 Reference : 6.13.2.2*4, p. 435 in [8] Ergonomic criteria : cognitive respect Linguistic level : semantic

Figure 2. A screen snapshot from a project management UI. ... 7. "Toolbox" window provide tools for drawing Gantt charts. This is helpful for defining project structure only, not for project information display. Ergonomic rule : windows that are not related to current interactive task should be undisplayed. Reference : 6.13.2.2*5, p. 435 in [8] Ergonomic criteria : syntactic compatibility Linguistic level : syntactic ... 12."Project Tree" window obscures the contents of another window which is placed just behind. Even the window title is no longer visible. It is therefore quite difficult to understand the aim of this window, forcing the user to click on it in order to see the information. Ergonomic rule : overlapping windows should not obscure definitively the contents of other windows which are used in the current interactive task. Reference : 6.13.2.4*4, p. 440 in [8] Ergonomic criteria : Visual guidance Linguistic level : lexical ...

3. CONCLUSION The approach mentioned in 2.2 should be interpreted only as a first step toward an empirically validated method for UI evaluation. Nevertheless, our four year experience emphasizes several advantages that were part of the main goals :  the double ordering (i.e., by criteria, by linguistic level) is a feasible method for organizing ergonomic rules;  the reliance of evaluation on context analysis (especially on task analysis) is greatly improved since specific criteria are derived from it. This strategy clearly establishes a continuum ranging from context analysis to ergonomic rules through specific ergonomic criteria. This feature is particularly interesting because general guidelines only have limited impact on usability (28% of cases) whereas specific guidelines are more influential (about 54% of cases);  the idea of giving evaluators a checklist of specific ergonomic rules provide them more guidance : having only ten heuristics sometimes seems too little, but having a ton of general rules is definitively impracticable. Conversely, a reasonable list of specific ergonomic rules may reduce the high level of expertise required by experts in Heuristic Evaluation;  the above approach does not preclude that evaluation should be carried out without users. Evaluation could be achieved with or without users. In the case of evaluating with users, experience shows that evaluators detect usability problems more quickly by referencing them with respect to ergonomic rules;  the checklist of specific ergonomic rules is intrinsically normalized so that it can be reused at both design and evaluation time. REFERENCES 1. J. Nielsen and R. Molich, Heuristic Evaluation of User Interfaces, Proc. of CHI'90 (Seattle, 12. 3. 4.

5. 6. 7. 8. 9.

5 April 1990), ACM Press, 1990, pp. 249-256. J. Nielsen, Enhancing the Explanatory Power of Usability Heuristics, Proc. of CHI'94 (Boston, 24-28 April 1994), ACM Press, 1994, pp. 152-158. J.M.C. Bastien and D.L. Scapin, Ergonomic Criteria for the Evaluation of Human-Computer Interfaces, technical Report No. 15, INRIA, Rocquencourt, May 1993. J.M.C. Bastien and D.L. Scapin, Preliminary Findings on the Effectiveness of Ergonomic Criteria for the Evaluation of Human-Computer Interfaces, Adjunct Proc. of InterCHI'93 (Amsterdam, April 24-29, 1993), pp. 187-188. J. Nielsen, A Virtual Protocol Model for Computer-Human Interaction, International Journal of Man-Machine Studies, V24N3, 1986, pp. 301-312. J. Coutaz, Interfaces homme-ordinateur - Conception et réalisation, coll. Dunod Informatique, Bordas, Paris, 1990. S. Ravden and G. Johnson, Evaluating Usability of Human-Computer Interfaces, A Practical Method, Series in Information Technology, Ellis Horwood, Chichester, 1989. J. Vanderdonckt, Guide ergonomique des interfaces homme-machine, Presses Universitaires de Namur, Namur, 1994, ISBN 2-87037-189-6. F. Bodart, A.-M. Hennebert, J.-M. Leheureux, I. Provot, J. Vanderdonckt, G. Zucchinetti, Key Activities for a Development Methodology of Interactive Applications, Chapter 4 in Critical Issues in User Interface Systems Engineering, D. Benyon and Ph. Palanque (Eds.), SpringerVerlag, 1995.