Enhancing End-User Mental Models of Computer Systems through the

Proceedings of the 29th Annual Hawaii International

Conference on System Sciences -

1996

Enhancing End-User Mental Models of Computer Systems through the Use of Animation Janette Moody

J. Ellis Blanton

Assistant Professor The Citadel Charleston, SC 29409

University of South Florida Tampa, FL 33620

Associate

Matthew A. Augustine

Professor

University

of South Florida

Department of Psychology Tampa, FL 33620

presented with a non-animated model. Possible explanations for the results are discussed along with suggestion as to when various model characteristics might influence model effectiveness.

Abstract It has been suggested that the training used to teach end-users about computer systems can be enhanced by providing learners with a conceptual model of the system. A conceptual model of a system conveys the underlying structure of the device, and aids the enduser in inferring the procedures necessary for operating the system. Such models are intended to give the end-user a better understanding of how the system works, and to help the end-user formulate a more useful mental model of the system. Mixed results from the studies to date suggest that further research should focus on the characteristics of effective conceptual models, and the types of tasks for which providing a conceptual model might be helpful. In addition, given the proliferation of multimedia presentations as teaching aids, research is needed to determine if animation in the conceptual model further enhances learning. The current study examined two characteristics which have been suggested as influencing the conceptual effectiveness of a model. These characteristics are the order of presentation of the conceptual model, and the use of animation. No support was found for the hypothesis that presenting a conceptual model of computer system before a set of procedural instructions would facilitate learning and lead to improved performance. However, the results did show that subjects who were presented with an animated conceptual model interacted more effectively with the computer system and scored higher on a subsequent comprehension test than did subjects

Introduction The importance of mental models as aids for improving end-user proficiency with computer systems has been extensively researched (Santhanam and Sein, El]; Davis and Bostrom [2]: Moran, [3]). The formation of an accurate mental model of the system may depend upon whether a conceptual model (one which captures a basic overview of the components of the system) or a procedural model (one that lists “how to” commands) was utilized during the training. While conceptual mental models are seen as leading to improved performance on non-routine tasks, the goal-oriented approach of many users leads them to form procedural models, regardless of the training method used (Santhanam and Sein, [I]). Not only is the type of end-user mental model important as a predictor of end-user proficiency, but also when it is acquired has been noted as important as well. Mayer [4] found that subjects who received the conceptual model after procedural instructions (“modelafter“) did as well or better than subjects who received the conceptual model before procedural instructions

299 1060-3425196 $5.00 0 1996 IEEE

Proceedings of the 1996 Hawaii International Conference on System Sciences (HICSS-29) 1060-3425/96 $10.00 © 1996 IEEE



1996

provided with procedural instructions, with some receiving them before the conceptual model (“modelbefore”) and others after (“model-after”) the procedural instructions. Therefore, all subjects received the same treatments used the same system, and were asked to perform the same tasks. With these characteristics of the experiment held constant, the study was able to examine two concepts that may influence end-users’performance: the timing of the presentation of the conceptual model and the use of a dynamic conceptual model that incorporates animation. The study examined the effect of time on subject performance by assessing performance immediately after training and after a oneweek delay.

(“model-before”) on retention-type problems such as generating simple statements and simple programs. However, the “model-before” subjects excelled in more complex and transfer-type problems such as generating loops and interpreting statements and simple programs. The fact that the timing of the presentation of the models is an important factor in end-user outcomes is consistent with Assimilation Theory (Ausubel, [S]) which posits that meaningfY learning takes place only if the new information can be anchored to pre-existing knowledge residing in long-term memory. Mayer [4] demonstrated that providing the learner with a model would provide maximum benefits only if it were available while the information about the system was being encoded (i.e., if the conceptual model were given before the procedural instructions). ln evaluating the effectiveness of training methods in forming useful mental models, past studies have also evaluated direct manipulation interfaces (DNLI) versus command based interfaces often with contradictory results (Michard, [6]; Carroll and Mazur, [7]). Again, Assimilation Theory would lead one to expect that DMl would result in more meaningful learning given that the icons used represent familiar objects (Davis and Bostrom, [2]). While DMl provides some anchors to pre-existing knowledge (i.e., icons of files, trash cans, etc.), it does nothing to develop the user’s mental model of what action is taking place within the system as a result of his input. The user still must make further internal inferences regarding the dynamic nature of the underlying system. A computer system is a dynamic device, i.e., “things happen”. Systems accept input and produce output. Components of the system interact and perform various functions. However, in the past, studies that have provided conceptual models of such systems have either described the model in narrative form (Borgman, [S]; Foss et al. [9]), or provided a static form of a diagram/conceptual model (Mayer, [4]; Kieras and Bovair, [lo]). To date, no empirical studies have been conducted, within the framework of mental models research, to examine the effects of providing learners with a dynamic conceptual model that incorporates the use of animation. Given the proliferation of multimedia teaching aids that include animation, it would seem appropriate to evaluate the educational value of these aids. This study examined the effects of providing an animated conceptual model to novice computer users. The purpose of the animation was to provide a functional representation of how information was communicated between components of the system. In addition to the conceptual model, all subjects were

Methodology The research model for this study incorporates aspects of research models of Davis and Bostrom [2] and Santhanam and Sein [I] as depicted in Figure 1. The study investigated one “Training Method” independent variable: order of model presentation (model-before versus model-after the procedural instructions) and one “Target System” independent variable: incorporation of animation into the conceptual model (animated versus non-animated). Both of these variables were between subjects. Other “Target System” variables were held constant by having the subjects interact with the same system and accomplish the same tasks. In regard to Contextual VariabIes, the effects of Individual Differences are controlled through randomization, as in Santhanam and Sein [I], and the effects of Experience were controlled by using only subjects with very little computer experience. To assess the impact of the independent variabIes across time, all of the indices of system performance and understanding were assessed twice during the experiment, once at the end of the first session, and once at the follow-up session one week later. Therefore, the design of the study was a 2 (order of model presentation) x 2 (presence/absence of animation) x 2 (repeated measures) design (see Figure 2). For the second week, the repeated measures was conducted which involved a second systems interaction and evaluation. Hypotheses Hypothesis la. Subjects in the model-before condition will proceed through the instructions phase more rapidly, and will provide more positive ratings of the instructions than subjects in the model-after condition.

300


Proceedings of the 29th Annual Hawaii International Conference on System Sciences - 1996 Hypothesis lb. As indicated by the number of tasks completed, the time taken to complete the assigned tasks, the number of commands needed to complete the tasks, and the number of conceptual errors committed, subjects in the model-before condition will interact more effectively with the target system than subjects in the model-after condition both during the initial session and the follow-up session. No differences are predicted in terms of the number of syntax (non-conceptual) errors committed by the two groups. Hypothesis le. Subjects in the model-before and model-after conditions will perform equally well on the first (single command items) section of the system comprehension test. However, on the second (more complex, two command items) and third (system state items) sections of the system comprehension test, subjects in the model-before condition will score higher than subjects in the model-after condition. Hypothesis 2a. Subjects in the animation condition will proceed through the instruction phase more rapidly, and will provide more positive ratings of the instructions than subjects in the non-animated condition. Hypothesis 2b. As indicated by the number of tasks completed, the time taken to complete the assigned tasks, the number of commands needed to complete the tasks, and the number of conceptual errors committed, subjects in the animated condition will interact more effectively with the target system than subjects in the non-animated condition, both during the initial session and the follow-up session. No differences are predicted in terms of the number of syntax (non-conceptual) errors committed by the two groups. Hypothesis 2c. Subjects in the animated and non-animated conditions will perform equally well on the first (single command items) section of the system comprehension test. However, on the second (more complex, two command items) and third (system state items) sections of the system comprehension test, subjects in the animated condition will score higher than subjects in the non-animated condition.

classes in college or work, and no more than one class in high school, (2) no programming experience (using BASIC, FORTRAN, etc.) and no experience using application packages (e.g., word processing, spread sheets), and (3) rated their own overall experience with computers as “none” or “very little”. Training All subjects learned about and used a simplified computer operating environment called the SPEAkS system (for Simplified Program Editing And Submission system). The SPEAkS system was developed by the experimenter on an Apple Macintosh computer with HyperCard (Apple Computer, Inc., [l I]) and the associated HyperTalk programming language. The SPEAkS system was designed to be realistic enough as to simulate an actual mainframe operating environment. The system allowed users to create, save, and retrieve files (programs). The system also allowed users to see a list of programs they had saved, run programs and allowed subjects to view the output from their programs. Finally, the SPEAkS system allowed subjects to print files and output. Actually, no true printer was used. Printing consisted of subjects receiving a message that a file had been printed. Procedure The experiment was conducted in two sessions. Throughout the experiment, the software that was used to run the experiment also recorded the length of time subjects required to complete the various phases. The first session of the experiment consisted of (a) an introduction phase, (b) a system instruction phase, (c) systems interaction phase, and (d) a written comprehension test. During the follow-up session a week later, subjects repeated the system interaction phase and the written comprehensive test. First Session Upon entering the Introduction Phase: computer lab, subjects were informed of the purpose of the experiment, what they would be doing during the experiment, and what would be expected of them. Any questions about the experimental procedure were answered before continuing. Instructions Phase: This phase began with an explanation of what the system is used for, and provided detailed procedural instructions for operating the system. In addition, the instruction phase presented subjects with a conceptual model of the system. Subjects in the

Design were 64 Subjects computer naive undergraduate students enrolled in introductory psychology courses, who received extra credit for participating in psychological experiments. Subjects’ computer experience/knowledge was assessed prior to participation in the experiment, with a questionnaire addressing knowledge and prior use of computers. Individuals selected for participation in the experiment met the following criteria: (1) no previous computer

301


Proceedings of the 29th Annual Hawaii International model-before condition were presented with the conceptual model before the procedural instructions. Subjects in the model-after condition received the conceptual model after the procedural instructions. The procedural instructions explained the types of tasks subjects would perform with the system, and all of the commands necessary to perform these tasks. It explained both the function of the commands as well as the exact syntax necessary to execute the commands. The conceptual model explained the system in terms of a diagram model. The model began by introducing and explaining the function of each of the As each component was system’s components. introduced, a representation of it was added to the model. After all of the system components had been covered, each of the commands was explained. Commands were explained in terms of how they affect the model. In addition to the explanation, subjects were shown “what happens” when each of the commands is executed, For example, the explanation of the SAVE command informed subjects that using this command causes a copy of what is in the ACTIVE FILE to be sent to the LIBRARY. Following the explanation, subjects saw this action carried out on the model. Subjects in the animated condition first saw a duplicate copy of the contents of the ACTIVE FILE created on the screen, then saw this copy actually move from the ACTIVE FILE to the LIBRARY. Subjects in the non-animated condition did not see the duplicate copy of the program made, nor did they see it move across the screen. Rather, for these subjects, the copy of the contents of the ACTIVE FILE simply appeared in the LIBRARY. As part of the explanation of how the information was communicated between the various components of the model, the model presented to subjects in the animated condition described the components of the model as being connected. This is, the description of the model presented to subjects in the animated condition contained one additional frame that represented the connections as lines drawn between the various components of the model. The model presented to subjects in the non-animated condition did not show the lines drawn between the components nor did it refer to the connection. As discussed above, the connections between the components were considered to be a functional part of the animation, because they helped to provide a representation of how information was communicated between components of the system. Systems Interaction Phase: The subjects were given a set of instructions which guided them through the log-on procedure. The remainder of the instructions directed subjects to perform several tasks and informed


1996

them that two programs had already been typed in and saved for them. The tasks that the subjects were asked to perform included: entering and saving two additional programs, running all four programs, viewing the output from all four programs, printing the output from all four programs, printing a copy of all four programs, and displaying the contents of the library. The last portion of the instructions directed the subjects to log-off the system and to notify the experimenter that they had finished. Written Comprehensive Test: After the subjects completed the assigned tasks, they were given a three page written test, described below, to assess their After understanding of the SPEAkS systems. completing the test, subjects were excused. Follow-up Session. One week later, subjects then repeated the System Interaction Phase and the Written Test. The systems interaction instructions were the same as in the first session, with the exception that the programs used were different. The written test was also the same except the program names were changed. Forms aud Measures The first dependent measure assessed in the study in terms of the computer task was the amount of time taken to progress through the instruction phase. Following the instruction phase, subjects were asked to provide ratings regarding the effectiveness of the instructions given, as well as their confidence in their ability to use the system effectively. During both system interaction phases, subjects were given a set of instructions that directed them to perform several tasks using the system. Behavioral measures of subjects’ ability to use the system included the number of tasks completed, the amount of time taken to perform all tasks, the number of commands taken to complete the tasks, and the number of errors (conceptual and syntax) committed while using the system. Finally, at the end of both sessions, subjects were tested on their understanding of the SPEAkS system. This written test consisted of three groups of items. The first group described a task and asked subjects which command should be used to complete the task. The sewnd group described a more complex task and asked subjects which two commands would be necessary to complete the task. The third group described a sequence of events then asked subjects about

302


Proceedings of the 29th Annual Hawaii International Conference on System Sciences - 1996 the state of the system.

differences between subjects in the model-before and model-after conditions (F(I,60) = 2.12, p.>.lO). However, the hypothesis overall was not supported. A repeated measures MANOVA encompassing the second (two command items) and third (system state items) sections of the system comprehension test also failed to reveal a main effect for information order (W&s’ Lamba = .97, F(2,59) 4.0). Hypothesis 2a. No support was found for Hypothesis 2a. No significant differences were found between subjects in the animated versus the nonanimated conditions in terms of the time taken to progress through the instructions (F(1,63) ~1.0). Additionally, a repeated measures MANOVA encompassing the six questions on the instruction evaluation form revealed no overall effect of animation (Wilks’ Lamba = .86, F(6,55) = 1.48, p.>.20). Note again that animation refers to the explicit depiction of copies being made and moving between components of the model, and to the presence of the functional connections between the various components of the model. Hypothesis 2b. The results supported the hypothesis that subjects in the animated condition would interact more effectively with the system than subjects in the non-animated condition. A repeated measures MANOVA encompassing four measures of subject performance including number of tasks completed, time taken to complete the tasks, number of commands needed to complete the tasks, and number of conceptual errors committed revealed a significant main effect for animation (Wilks’ Lamba = .75, F(4,57) = 4.70, p..20). No difference was predicted between groups in terms of the number of syntax errors committed. However, a repeated measures ANOVA conducted on the number of syntax errors revealed that model-before (Mb) subjects committed significantly more syntax errors than model-after (Ma) subjects ((Mb = 3.66; Ma = 1.56), F(1,60)= 5.85, p.c.05). Hypothesis lc. The results were consistent with the first part of Hypothesis lc. As predicted a repeated measures ANOVA conducted on subjects’ scores on the first section of the system comprehension test (single command items) revealed no significant

303


Proceedings of the 29th Annual Hawaii International in the animated condition scored higher than subjects in the non-animated condition on all three sections of the system comprehension test. Repeated measures ANOVAs revealed a significant main effect for animation for Section 1: F(1,60) = 6.49, p.c.01; Section 2: F(1,60) = 5.42, p.c.05; and Section 3: F(1,60) = 3.92, p.c.05.


1996

the addition of animation provided subjects with a representation of the mechanism by which the characteristics of the model are manipulated. Furthermore, this addition was sufficiently salient to allow subjects in the animated condition to construct more complete, useful mental models, which in turn accounted for the improved performance. Several alternatives exist for explaining the animation effects, some of which are not incompatible with the explanation discussed above. For example, it could be argued that the animation acted as a vehicle to focus attention more effectively on specific aspects of the model, andor to emphasize key points made while the model was being presented This line of reasoning is somewhat compatible with the first explanation in that both suggest given an equal period of time studying either model (and an equal aliocation of attentional resources), studying the animated model should lead to a better understanding of the target system than the nonanimated model, which was the stated purpose of the manipulation. However, while the two lines may not be mutually exclusive, the differences between them should not be minimized. If one of the explanations, for example the attentional explanation, were found to be more appropriate, then this would lead to different lines of research and suggest different model alterations (those stressing attentional factors) than if the wnverse were true.

Discussion The data supported the premise that animation can be used to enhance a conceptual model used to train individuals about a computer system. Individuals in the animated condition interacted more effectively with the system and demonstrated a better understanding of the system than subjects in the non-animated condition. However, whether the model was presented before or after procedural instructions did not significantly influence subject performance. Implications and possible explanations of these findings are discussed below. Animation The most theoretically appealing explanation of the observed animation effects is that the addition of animation to the conceptual model allowed (stimulated) subjects in the animated condition to formulate a richer, more useful, more complete mental model of the target system, which in turn lead to enhanced performance. Much has been written about a user’s ability to “run” his or her mental model. Norman [ 121 has suggested that, at least of the modds constructed by individuals through casual use of a device, this ability is limited and can lead to inefficient behavior in using a device. If ‘Yunning” a mental model is defined as modifying the parameters of its components and/or changing its state (Williams et al., [13]), then the model presented in both conditions could be said to provide a representation of a “running” model. However, there was a critical difference between the two conditions. Both the animated and non-animated modets provided a representation of the model “at rest’, between operations. However, the animated model also provided a vivid, explicit representation of how these operations (i.e., changes in state) occurred. That is, the animated model graphically represented the wnnections between the components of the model, and depicted the dynamic transition from one state to another by showing objects moving from one model component to another. Lacking this information, subjects in the non-animated condition were left to determine for themselves how, for example, a program got from the “active file” to the library. Hence, this explanation centers on the idea that

Order of Model Presentation Subjects who were presented with the conceptual model before the procedural instructions committed more syntax errors (while interacting with the target system) than those who saw the model after the instructions. This finding is consistent with the work reported by Mayer [4], and by Mayer and Bromage [14] who found that model-after subjects tended to recall more syntactic, technical information, while modelbefore subjects retained more conceptual tiormation. However, this was the only finding of the current study which was consistent with this pattern. No differences were found between model-before and model-after subjects on the first sections (command identification) of the system wmprehension tests, which also should have tapped technicalisyntactic knowledge of the target system. Moreover, no differences were found between model-before and model-after subjects on either the advanced system comprehension tests, or on the system performance measures. Closer examination of the procedural instructions provided to subjects may help to account for these findings. First, it should be noted that the

304



procedural instructionswere written to “stand alone”as an inclusive training manual for the system. Substantial effort was investedto make the procedural instructions clear, concise and easy to understand. Elaborate explanationsof all commandswere provided as well as some examples. It may have been that the procedural instructions were too well organized for the order in which the materialwas presentedto have an effect. Tbat is, Mayer [15] suggestedthat a model is useful because it provides a framework witb which to organize otherwise abstract information. The more well organizedthe informationto be learned,the less a model needsto servein this capacity.


1996

vial, New York Holt, Rinehart,and Winston, 1968. [6]Michard, A., “Graphical Presentation of Boolean Expressionsin a DatabaseQueryLanguage:DesignNotes and an Ergonomic Evaluation”, Behavior and Information Technology, 1:3 (1982) 277-288.

[‘7]Carroll, J. and S. Mazur, “LisaLeaming”, ICCC Computer, 19:ll (1986) 35-49. [8]Borgman, CL. “The user’s mental model of an information retrieval system: An experiment on a prototype online catalog”, International Journal of Man-Machine Studies, 24 (1986) 47-64.

Summary

[9] Foss D., P. Smith-Kerker, and M. Rosson, “On comprehending a computer manual: Analysis of variables affecting pe&mnance.”International Journal of Man-Machine Studies, 26 (1987) 277-300.

The current study presented consistent evidencethat the addition of animationcan significantly enhancethe effectivenessof a conceptualmodel. The tindings are strengthenedby the fact that the differences were maintainedover tbe one week interval betweenthe initial sessionand the follow-up. Of course, additional researchis neededto determinethe precise mechanism of the effect, and to determine whether the effect generalizes to other types of tasks and different user populations.

[lO]Kieras, D. and S. Bovair, “The role of a mental model in learning to operate a device” Cognitive Science,8, (1984) 255-273. [ 1l]Apple Computer, Inc.HyperCard User’s Guide, Cupertino,CA:Apple Computer,Inc.(1987). [ 121Norman, D., “Someobservationson mentalmodels” in D. Gentner and A.L. Stevens (Eds.), Mental Models,Hillsdale, NJ: Lawrence Erlbaum Associates (1983) 7-14.

REFERENCES

[13]WiUiams, M., J. Hollan, and A. Stevens,“Human reasoningabout a simple physicalsystem”in D. Gentner and A.L. Stevens(Eds.), Mental Models, Hillsdale, NJ: LawrenceErlbaum Associates.1983 131-153.

[l] Santhanam,R. and M. Sein, “Improving End-user Proficiency: Effects of ConceptualTraining and Nature of Interaction”, Information Systems Research, 5:4 (1994) 378-399.

[ 141 Mayer, R. and R. Bromage, “Different recall protocols for technical text due to advanceorganizers” Journal of Educational Psychology, 72 (1980) 209-225.

[2] Davis, S. and R. Bostrom, “Training End-Users:An Experimental Investigation of the Roles of Computer Interface and Training Methods”, MIS Quarterly 17 (1993), 61-85.

[IS] Mayer, R., “The psychologyof how novices learn computerprogramming”Computing Surveys, 13 (1981) 121-141.

[3] Moran, T., “An Applied Psychology of the User” ComputingSurveys, 13 (1981) l-11.

[4] Mayer, R., “Some conditionsof meaningful learning for computer programming: Advance organizers and subject wntrol of frame order”Journal of Educational Psychology, 68, (1976) 143-150. [ 51 Ausubel,D.P. Educational Psychology: A Cognitive

305



Figure 1. Research mode1

306



1996

Proceedings of the 29th Annual Hawaii International Conference on System Sciences - 1996

Figure 2. Experimental model (week 1) Conceptual Model Before Instructions

Conceptual Model After Instructions

Animation

System Interaction

System Interaction

No Animation

System Interaction

System Interaction

Table 1. Results of hypotheses testing Hla Hlb Hlc(1) Hlc(2&3)

Not Supported Not Supported Supported Not Supported

F(1,63)