edge to be learned or used in the execution of a task. clumsy to use. Furthermore, the notation .... scribing the Goals, Operators, Methods, and Selection. Thus, it appears to be ... It is also a description sented below for doing a GOMS analysis is based on of what the user .... All of the operators in a condition have description>.
Handbook of Human-ComputerInteraction M. Helander (ed.) 0 Elsevier Science PublishersB. V. (North-Holland), 1988
Chapter 7
Towards a Practical GOMS Model Methodology for User Interface Design David E. Kieras
Making WM Use Explicit ............. 146 An Example of Using the Procedure . . 146 Description of Methods .-..-..-..---. --. 146 Modifications to Show WM Usage . . . 150 Completing the Analysis ............... 151
The University of Michigan Ann Arbor, Michigan
7.1
7.2
Introduction
135
The Cognitive Complexity Approach . Problems in Using Cognitive Complexity Model Construction A Guide to GOMS Model Construction Will the Methods Actually Work? ..... Organization of the Chapter ........
135
Overvev of the GOMS Task Anlalysis Approach What is a GOMS Analysis? ........ Definitions and a Notation for GOMS Models ...................... Goals .................... ... Operators . . . . . . . . . . . . . . . . . Methods ..................... Selection Rules:............. ... Task Descriptions and Task Instances .
Checking Sensitivity to Judgment Calls .
151
Using a GOMS Task Analysis Qualitative Evaluation of a Design . . . Predicting Human Performance with the Design .................... Method Learning Time Estimates . . . . Estimating Execution Time ........
151 151 152 152 154
Mental Workload ........... .. . .. Suggestions for Revising the Design . Using the Analysis in Documentation . .
155 155 156
7.6
Acknowledgements
156
7.7
References
156
7.1
Introduction
7.5
136 137 137 137
137 137 138 138 138 139 140 140
7.3
General Issues in GOMS Task Analysis 141 Judgment Calls............... 141 Bypassing Complex Processes ....... 141 What Tasks Should be Analyzed? ..... 142 When Can a GOMS Analysis be Done? 142
7.4
A Procedure for Constructing a GOMS Model Summary of Procedure . .... ....... Detailed Description of Procedure. . Step A. Choose the Top-Level User's Goals and Method .......... .... Step B. Do the Following Recursive Procedure: ................. ... Step C. Completing the Analysis . . Step D. Check Sensitivity to Judgment Calls and Assumptions ...........
The Cognitive Complexity Approach In several papers Kieras, Bovair, and Polson have presented the cognitive complexity approach to humancomputer interaction (HCI); the most complete current summary appears in Bovair, Kieras, and Polson (1988); see also Polson (1987) and Kieras and Polson (1985) for earlier presentations. In brief, this approach is based on constructing an explicit model of the user's procedural knowledge entailed by a particular system design. The basic content of the procedural knowledge in the simulation is described by the Card, Moran, and Newell (1983) GOMS model, described more below. But briefly, a GOMS model consists of descriptions of the Methods needed to accomplish specified Goals. The Methods are a series of steps consisting of Operators that the user performs. If there is more than one Method to accomplish a Goal, then the GOMS model
143 143 144 144 144 145 145
135
136
CHAPTER 7. A PRACTICAL GOMS MODEL METHODOLOGY FOR USER INTERFACE DESIGN
includes Selection Rules that choose the appropriate Method depending on the context. Quantitative measures defined on this explicit representation of the user's knowledge can predict important aspects of usability, that are associated with the complexity of the knowledge required to operate the system, such as the time to learn the system, amount of transfer from previous systems, and execution time. The predictions are obtained from a computer simulation model of the user's procedural knowledge that can actually execute the same tasks as the user. The procedural knowledge is represented in the form of a production system, a collection of IF-THEN rules; this representation provides a description of the knowledge in terms of units of roughly equal "size," namely the individual production rules, which can then be counted to yield quantitative estimates of the amount of knowledge to be learned or used in the execution of a task. By using this "cognitive architecture", the cognitive complexity approach to HCI has the key advantage of being able to exploit the recent work on cognitive skills in cognitive psychology (e.g., Anderson, 1982),which has been a resounding success, both theoretically and practically. The basic approach to user-interface design using the cognitive complexity approach would be that the iterative design process would be followed, but with the evaluation of a proposed design being done with simulation techniques rather than actual human user testing; only a final test of the design would require actual user testing. Additional user testing would of course be involved to develop aspects of the design, such as screen layout, that are not directly addressed by an analysis of the procedures entailed by the design. Problems in Using Cognitive Complexity Models But there are several problems in using the cognitive complexity approach as a design tool which have become clear from "technology transfer" research by Kieras, Polson, Bennett, and Karat (cf. Kieras, 1988; Karat, 1988; Bennett, Lorch, Kieras, and Polson, 1987). Two critical problems addressed by this chapter are (1) the difficulty of constructing production rule simulation models; and (2) the difficulty of doing, in a standardized and reliable way, the detailed task analysis required to construct the representation of the procedural knowledge that the user must have in order to operate the system. The essence of the simulation idea in the cognitive complexity approach is simply to describe a GOMS model for the user's knowledge in a form that is actu-
ally executable, so that running the model can verify its completeness and correctness. In the Kieras and Polson work, these models were constructed using the production rule formalism, primarily because this is a standard and current theoretical idea for the representation of procedural knowledge. However, experience in our work suggests that writing production rules is a technically difficult task, analogous in many ways to programming in assembler language. Continuing the programming analogy, what is needed is a higherlevel language for describing GOMS models, at the level that designers and HCI specialists would normally think about the user's task. The GOMS model notation in Card, Moran and Newell(1983) is an example of such a higher level language, but this source does not explain in any detail how the notation works, and it seems to be somewhat clumsy to use. Furthermore, the notation has only a weak connection to the underlying cognitive theory, meaning that there is no defined calculational base for making predictions of learning time from an explicit GOMS model. Another approach is the Task Modeling Language (TML) described by Bennett, Lorch, Kieras, and Polson (1987). TML was defined as a high-level ianguage that was directly translated to production rules, and ran on IBM PC family computers. TML made it clear that higher-level task description languages that had a clear relationship to the production rule theory were possible, and that simulation models did not require AI-workstation class machines. But TML was hard to use; the programmer had to know the underlying production rule implementation in order to use it. But our experience is that the hard part of applying the cognitive complexity approach is actually the task analysis, by which is meant a detailed and specific description of the procedures that the user has to know in order to use the system to accomplish tasks, which constitutes a GOMS model for the system. Such a description is considerably more detailed than the traditional forms of task analysis used in computer system design, which seem to have as their main target the construction of an action/object table. In contrast, a GOMS task analysis involves describing in detail the specific methods for accomplishing the goals listed in the action/object table. Thus, GOMS task analysis starts where traditional task analysis stops. The amount of analytic detail involved is considerable, making construction of these GOMS models a very lengthy and effortful task. Because of their background and training, such model building comes naturally to cognitive psychologists; however, a practi-
7.2. OVERVIEW OF THE GOMS TASK ANALYSIS APPROACH cal design methodology shouldn't require such specialization. The second problem is thus whether such analyses, which amount to constructing psychological eoonly h of how users represent the task, can be do ne
Will the Methods Actually Work?
reliably ally design computer systems.
usu-
design is to try to apply it to actual design problems.
l dThere This type of task analysis consists of first, "judgment calls" about how users view, the task, and second, a routine description of the methods entailed by the design. How to make these judgments can not be spelled out; it depends on the intuition of the person performing the analysis. However, the judgments can be recorded explicitly in the GOMS model itself, meaning that the impact they have on the design can be later accounted for, and alternative judgments considered. Once the judgment calls have been made, describing the corresponding methods seems to be quite routine, and so a standard method for generating the rest of a GOMS task analysis can be described.
is no feasible way to conduct laboratory research on this question. Due to the demands of the industrial product development process, there have been few opportunities to systematically evaluate the usefulness of any of the recently proposed design tools, even ones as straightforward as the Keystroke-Level Model (Card, Moran, & Newell, 1983, Ch. 8). Thus, at the time of this writing, the procedures presented in this chapter have to be treated as proposals that should work, given the research results, rather than as documented procedures with evidence that they do work in the design process environment.
A Guide to GOMS Model Construction
;
137
For teaching purposes, I developed a document called A Guide to GOMS TaskAnalysis, written for my course in User Interface Design and Analysis in the Coinputer Science Department at the University of Michigan. The Guide defined a language called "Natural" GOMS Language (NGOMSL) for expressing GOMS models, which like TML, has a direct relationship to the underlying production rule models and so supports quantitative predictions, but unlike TML, is relatively easy to read and write without knowledge of the production rule models. The Guide also included a procedure for constructing a GOMS model using top-down breadth-first expansion of methods, which seems to be intuitively easier than trying to describe goal structures directly, the approach usually taken in cognitive psychology and by Card, Moran, and Newell (1983, Ch. 5). My experience, together with some colleagues, is that computer scientists, such as undergraduate students, appear to be able to write GOMS models and generate predictions with reasonable facility using the methods in the Guide. Thus, it appears to be possible to conduct and use GOMS task analyses in a standard way which can be taught to people who are not specialists in Human Factors or Cognitive Psychology. This is reason for optimism, even though the current Guide is clearly just a beginning towards task analytic methodology. This chapter presents a condensed, and more scholarly, version of the Guide.
This introduction concludes with a serious caveat: The way to find out if the methodology presented in this chapter is actually an effective aid to user interface
Organization of the Chapter The remainder of this chapter is organized as follows: The first major section is about the GOMS-based approach to task analysis; it includes definitions of the parts of a GOMS model in terms of the NGOMSL notation. This section also includes a discussion of some of the general issues that underlie the approach. The second major section presents the procedure for constructing a GOMS model, along with an example. The final major section explains how to use a GOMS model evaluation of a design and for predicting human performance, and how a revised design and documentation can be based on the model.
72
Overview of the GOMS Task Analysis Approach
What is a GOMS Analysis? A GOMS analysis is a description, or model, of the knowledge that a user must have in order to carry out tasks on a device or system; it is a representation of the "how to do it" knowledge that is required by a system in order to get the intended tasks accomplished. Describing the Goals, Operators, Methods, and Selection Rules for a set of tasks in a relatively formal way is the goal of doing a GOMS task analysis. The person who is performing such an analysis is referred to as "the analyst" in this chapter. Once the GOMS model has been developed, predictions of learning and performance can be obtained as described below. A GOMS description is also a way to characterize a set of design decisions from the
138
CHAPTER 7. A PRACTICAL GOMS MODEL METHODOLOGY FOR USER INTERFACE DESIGN
point of view of the user, which can make it useful during, as well as after, design. It is also a description of what the user must learn, and so could act as a basis for training and reference documentation. Definitions and a Notation for GOMS Models This section defines each component of a GOMS model in somewhat more detail than Card, Moran, and Newell (1983). In addition, this section introduces a notation system, NGOMSL ("Natural GOMS Language"), which is an attempt to define a language that will allow GOMS models to be written down with a high degree of precision, but without the syntactic burden of ordinary formal languages, and that is also easy to read rather than cryptic and abbreviated. Despite the resulting verbosity and looseness, NGOMSL is close to a formal GOMS language that could be implemented as a running computer language. However, it is important to keep in mind that NGOMSL is not supposed to be an ordinary programming language for computers, but rather to have properties that are directly related to the underlying production rule models described by Kieras, Bovair, and Polson (Kieras & Polson, 1985; Polson, 1987; Kieras & Bovair, 1986; Bovair, Kieras, & Polson, 1988). In this chapter, the NGOMSL expressions are shown in boldface. Goals A goal is something that the user tries to accomplish. The analyst attempts to identify and represent the goals that typical users will have. A set of goals usually will have a hierarchical arrangement in which accomplishing a goal may require first accomplishing one or more subgoals. A goal description is an action-object pair in the form: , such as delete word, or moveby-find-function cursor. The verb can be complicated if necessary to distinguish between methods (see below on selection rules). Any parameters or modifiers, such as where a to-be-deleted word is located, are represented in the task description. Operators Operators are actions that the user executes. There s an IS an Important difference between goals and operators. Both take an action-object form, such as the goal of revise-document and the operator of press-key. But in a GOMS model, a goal is something to be accomplished, while an operator is just executed. This distinction is intuitively-based, and is also relative; it
depends on the level of analysis. The procedure presented below for doing a GOMS analysis is based on the idea of first describing methods using very highlevel operators, and then replacing these operators with methods that accomplish the corresponding goal by executing a series of lower-level operators. This process is repeated until the operators are all primitive operators that will not be further analyzed. The observable actions through which the user exchanges information with the system or other objects in the environment are the external operators. These include perceptual operators, which read text from a screen, scan the screen to locate the cursor and so forth, and motor operators, such as pressing a key, or moving a mouse. External operators also include interactions with other objects in the environment, such as turning a page in a marked-up manuscript, or finding the next markup on the manuscript. The analyst usually chooses or defines the external operators depending on the system or tasks. The internal actions performed by the user are the mental operators; they are non-observed and hypothetical, inferred by the theorist or analyst. In the notation system presented here, some mental operators are "built in;" these primitive operators correspond to the basic mechanisms of the cognitive processor, the "cognitive architecture," and are based on the production rule models described by Bovair, Kieras, and Polson (1988) (cf. Card, Moran, & Newell, 1983, Ch. 2). Examples are actions like making a basic decision, storing or retrieving an item in Working Memory (WM), retrieving information from Long-Term Memory (LTM), and setting up a goal to be accomplished. Other mental operators are defined by the analyst to represent complex mental activities (see below). Typical examples of such analyst-defined mental operators are determining the string to use for a find command, and determining the editing change meant by a marking on a marked-up manuscript. Below follows a brief description of the primitive mental operators; examples of their use appear later. Mental Primitivesfor Flow of Control. * Accomplish the goal of This is analogous to an ordinary CALL statement; control passes to the method for the goal, and returns hr hntega copihd here when the Foal hsbe has been accomplished. * Report goal accomplished
This is analogous to an ordinary RETURN statement, and marks the end of a method.
7.2. OVERVIEW OF THE GOMS TASK ANALYSIS APPROACH Decide: If Then
Home-hand to mouse
Decide: If Then Else
Press-key
This operator is for making a simple decision that governs the flow of control within a method. It is not supposed to be used within a selection rule set, which has its own structure (see below). The If clause typically contains operators that test some state of the environment. Notice that the complexity of a Decide operator is strictly limited; only one simple Else clause is allowed. Complex conditional situations must be handied by methods that have multiple steps, decisions, and branching, rather than single complex steps. Goto Step ingly; normally it used only with Decide operators. Memory Storage and Retrieval. Recall that Retain that Forget that Retrieve-LTM that These memory operators reflect the distinction between long-term and working memory as they are typically used in computer operation tasks. The terminology is not ideal, but there are few choices in English. Recall means to fetch from working memory; Retain means to store in working memory, while Forget means that the information is no longer needed, and so can be dropped from working memory (although counter-intuitive, this is a real phenomenon; see Bjork, 1972).The methods presented here assume that information is not lost from working memory, so Forget refers only to the deliberate dropping of information, Any problems due to "memory overload" could be identified by looking at how much has been retained relative to forgotten (see below). There is only a Retrieve operator for LTM, because in the tasks typically modelled, long-term learning and forgetting are not involved. Primitive External Operators. The analyst defines the primitive motor and perceptual operators based on the elementary actions needed by the system being analyzed. Some typical examples:
139
Type-in Move-cursor to cursor Find-menu-item Analyst-Defined Mental Operators. As discussed in some detail below, the analyst will often discover that users engage in psychological processes that are too complex to be practical to attempt to represent as methods in the GOMS model, and that often having little to do with the specifics of the system design. The analyst can em bypass these processes by defining operators that act as place holders for the mental activities that will not be further analyzed. Some frequently useful examples: Get-from-task Look in the task description for the information tagged with and put the information into working memory. Verify-result Determine whether the result of an operation is what is required. Get-next-edit-location Find out where in the text the next edit will be done. Methods A method is a sequence of steps that accomplishes a goal. A step in a method typically consists of an external operator, such a pressing a key, or a set of mental operators involved with setting up and accomplishing a subgoal. Much of the work in analyzing a user interface consists of specifying the actual steps that users carry out in order to accomplish goals, so describing the methods is the focus of the task analysis. The estimation procedures used below refer to the number of NGOMSL "Statements," each of which corresponds to a single production rule in the underlying production system models. These rules for counting statements will be noted below. The form for a method is as follows: Method to accomplish goal of Step 1. ... Step 2. ...
140
CHAPTER 7. A PRACTICAL GOMSMODEL METHODOLOGY FOR USER INTERFACE DESIGN ...
Selection Rules
Step n. Report goal accomplished. Note that more than one can appear in a step (see guidelines below), and that the last step must contain only the operator Report Goal Accomplished. The Method statement counts as one statement, and each step counts as one statement, regardless of the number of operators. However, if a Decide operator with an Else portion is present, the step counts as two statements, due to the fact that two production rules are required. As a simple example, the method for moving a file on a Macintosh would be described as:
Sometimes there is more than one method to accomplish a goal. The purpose of a selection rule is to route control to the appropriate method to accomplish the goal. The general goal should be decomposed into a set of specific goals, one for each method, and a set of mutually exclusive conditions should be described that specify which method should be used in what context. In the notation introduced here, selection rules come in sets. A set of selection rules is associated with a general goal, and consists of several If-Then rules which choose the specific goal to be accomplished. The relationship with the underlying production rule models is very direct (see Bovair, Kieras, & Polson,
Method to accomplish goal of moving a file
1988).The form for a selection rule set is:
Step 1. Select the icon for the file
Selection rule set for goal of
Step 2. Drag the icon to the destination icon Step 3. Report goal accomplished Methods often call sub-methods to accomplish subgoals. This method hierarchy takes the following form:
If Then accomplish goal of . If Then accomplish goal of .
Method to accomplish goal of Report goal accomplished.
Step 1.
Step k. Accomplish the goal of
Step n. Report goal accomplished. Method to description>
accomplish
goal
of
