14276c11.pgs
6/8/06
2:33 PM
Page 361
CYBERPSYCHOLOGY & BEHAVIOR Volume 9, Number 3, 2006 © Mary Ann Liebert, Inc.
A Cognitive Approach to Game Usability and Design: Mental Model Development in Novice Real-Time Strategy Gamers JOHN GRAHAM, Ph.D.,1 LIYA ZHENG, M.S.,2 and CLEOTILDE GONZALEZ, Ph.D.3
ABSTRACT We developed a technique to observe and characterize a novice real-time-strategy (RTS) player’s mental model as it shifts with experience. We then tested this technique using an offthe-shelf RTS game, EA Games Generals. Norman defined mental models as, “an internal representation of a target system that provides predictive and explanatory power to the operator.” In the case of RTS games, the operator is the player and the target system is expressed by the relationships within the game. We studied five novice participants in laboratorycontrolled conditions playing a RTS game. They played Command and Conquer Generals for 2 h per day over the course of 5 days. A mental model analysis was generated using player dissimilarity-ratings of the game’s artificial intelligence (AI) agents analyzed using multidimensional scaling (MDS) statistical methods. We hypothesized that novices would begin with an impoverished model based on the visible physical characteristics of the game system. As they gained experience and insight, their mental models would shift and accommodate the functional characteristics of the AI agents. We found that all five of the novice participants began with the predicted physical-based mental model. However, while their models did qualitatively shift with experience, they did not necessarily change to the predicted functional-based model. This research presents an opportunity for the design of games that are guided by shifts in a player’s mental model as opposed to the typical progression through successive performance levels.
INTRODUCTION
T
takes a cognitive psychology approach to understanding how players develop concepts of game-embedded AI (artificial intelligence) agents. Traditional game design and usability has used play-testing, questionnaires, and observations.13 Our approach gathers data from novice players to determine and represent their mental model of AI agents as the player develops HIS PAPER
experience in the game. We then go on to describe a mental model approach, as opposed to a performance approach, to increase game playability. This approach holds promise as a method to increase play time without hitting the current limitations of payer physical and cognitive abilities. We use a cognitive psychology approach, as it goes one step deeper than behavioral psychology by focusing on mental processes that operate on stimuli, which contributes as to whether or not a re-
1Engineering
Psychology Laboratory, United States Military Academy, West Point, New York. Computer Interaction Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania. 3Department of Social & Decision Sciences, Carnegie Mellon University, Pittsburgh, Pennsylvania. 2Human
361
14276c11.pgs
6/8/06
2:33 PM
Page 362
362
sponse is made, when, and what it is.4 Specific to cognitive psychology, we use a construct called “mental models.” For the purposes of this paper, we will use a simplified definition of mental model: the cognitive layout that a person uses to organize information in his or her memory.10 In the case of real-time-strategy (RTS) games, the operator is the player and the target system is expressed by the relationships within the game. The relationships we specifically studied considered the various AI agents operating with and against the player. These agents, in Command & Conquer Generals, are used to control the detailed behavior of individual friendly, enemy, and civilian screen units operating at the tactical level. Laird and van Lent9 consider these game-embedded AI agents as the seedlings in the pursuit of Human-Level AI. In support of this view, the Command & Conquer Generals agents have visual qualities that give them strong relations to friendly, enemy, and civilian agents in the real-world. We are interested in how players perceive and process these AI widgets as they operate within the game. Our assumption is that the player starts with an impoverished model of the game AI agents. This impoverished model is based on the available information in the game and the player’s previous knowledge of the world.1 A novice player’s experiences will affect and alter the initial mental model.11 The two processes at work on a novice player can also be considered assimilation and accommodation.14 While the player could continue to assimilate additional details about the physical characteristics of the game, this development would not lead to success in the game. A useful mental model will have, embedded within it, the context for which it is operational.3 In the case of a RTS game, the context and experiences lead a successful player to accommodate a mental model of artificial agents that is functionally based. Players
GRAHAM ET AL.
who do not accommodate a new mental model of the game are not able to develop and implement goals that lead to success.5
METHODS Participants Five participants were drawn from the general student population of Carnegie Mellon University and the University of Pittsburgh. The two females and three males reported that this was their first RTS game experience. All of them were subjectively judged proficient in computer operation based on the first day’s training scenario. Participants were given 1 h of training to familiarize themselves with the operation of the mouse controller with respect to the game operation. On days 2–5, they engaged in repetitive 12-min scenarios, followed by a 3-min questionnaire. Data was also collected before each hourly break using a questionnaire. Lastly, performance data (win/loss) was collected following each scenario. Training All five participants engaged in a controlled training scenario using Command & Conquer Generals on day 1. The researcher-designed 40-min scenario required them to progressively apply all of the AI agents under their control as well as develop a mapping between the game and the mouse controller. They were not taught, nor did they employ, shortcut keys. This was done to reduce the potential of a large skill difference between participants. Scenario To exert some control over the player’s range of experiences and to conduct real-time data collec-
FIG. 1. Tactical-level artificial Intelligence agents operating in Command & Conquer Generals real-time-strategy game. (A) Ranger. (B) Missileman. (C) Sniper. (D) Attack helicopter. (E) Hummer. (F) Artillery. (G) Tank.
14276c11.pgs
6/8/06
2:33 PM
Page 363
COGNITIVE APPROACH TO GAME USABILITY AND DESIGN
tion, participants were required to repetitively play a limited scenario developed and written by the research team. The particular scenario was designed to reflect an actual training mission used by the U.S. Army to prepare and train future leaders. The scenario had a written description and was designed to take approximately 12 min to complete. The scenario pitted a 135-element friendly force against a computer-generated enemy seeking to capture an airfield. The enemy force was standardized and repetitively organized into four waves, but randomized in the timing of direction of movement of each wave. Success was defined as defense of friendly airfield assets, attrition of the enemy force, and preservation of the friendly force. Each day the participants were paid to play the game for 2 h, with a 1-h break. At the start of each hour-long session, the participants re-read the scenario objectives and then began play. At the completion of each hour, the participants completed the dissimilarity questionnaire (Fig. 2). This continued for 4 days, a total of 8 h. Multi-dimensional scaling This research method employs Multidimensional Scaling (MDS) methods to represent the perceived relations between game AI entities. MDS methods allow us as researchers to ask unobtrusive questions (“how similar is agent A to agent B”) and to derive from those questions underlying dimensions, while reducing the problem of experimental demand characteristics.12 This technique is not new to interface design, but it is unique as an application to game analysis and design.10 In our research, MDS is used to produce a geometrically spatial rep-
363
resentation of the game stimuli and the relationships between them. The measurement data is the psychological proximity between game stimuli.7 For our research, graphical depictions consist of the game AI agents arrayed in two-dimensional space, with distance and location used to describe the relations between them. By using Likert-scaled dissimilarity comparisons, we derive a set of relational distances of a player’s perceptions of AI agents. The specific Command & Conquer Generals Artificial Intelligence Agents used as stimuli were the ranger, missileman, sniper, tank, hummer, artillery, and attack helicopter (Fig. 1). Visualizations of seven agents were paired against one another and presented to participants to produce 21 dissimilarity ratings. Figure 3 is the first step of our application of MDS, which is to produce scatter plots of the participants’ judgments of the agents in a twodimensional plane. The representation shows one participant’s psychological distances between pairs of agents. Agents that were rated as similar are closer to one another, and the location along the x- or y-axis indicate the dimensionality of the participant’s evaluation criteria. MDS techniques can now work backwards to determine the stimuli dimensions that were used to make the participant’s judgments. For instance, Figure 4 is a MDS representation of one participant’s responses after the training session. Analyzing the results, we can observe that the Sniper, Missile, and Ranger stimuli were rated as similar by the participant, while the Sniper and Comanche were rated as dissimilar. Analyzing the dimensions of the graph against our knowledge of the game, we can further ascertain the dimensions
FIG. 2. Example response from dissimilarity questionnaire. Participants were asked to rate the stimuli on the presented scale. Stimuli, in this case, are the Ranger and Crusader Tank Artificial Intelligence Agents.
14276c11.pgs
6/8/06
2:33 PM
Page 364
364
GRAHAM ET AL.
FIG. 3. Multidimensional scaling representation of a single participant’s responses to the dissimilarity questionnaire. Items with greater proximity have a stronger relationship in the mental models of the participants.
used by the participant. In Figure 4, the participant appears to have developed readily identifiable dimensions x-axis mode of locomotion (walk vs. vehicle/aircraft) and y-axis movement medium (ground vs. fly).
RESULTS The first evaluation of participants’ mental models was conducted on data collected immediately following the first day’s training session. The initial mental models of all five of the participants were evaluated as organized based on the physical dimensions of the system (Table 1). One (1/5) participant’s initial mental model was evaluated as strong
physical, two (2/5) participants’ initial mental models were evaluated as physical, and two (2/5) participants’ initial mental models were evaluated as weak physical. These results support our hypothesis that novice players have an impoverished mental model based on the available surface features of the AI agents. The second evaluation of participants’ mental models was conducted at the completion of the fifth day of trials. One participant’s (1/5) final mental model was evaluated as physical, one participant’s (1/5) final mental model was evaluated as weak physical, one participant’s (1/5) final mental model was evaluated as functional, and two participants’ (2/5) final mental models were evaluated as other/unknown. While only one player’s mental
Vehicles Dismounts
Aircraft
FIG. 4. Example of participant’s initial mental model assessed as physical. Circles represent the dimensions assigned to relationships identified by the rater.
14276c11.pgs
6/8/06
2:33 PM
Page 365
COGNITIVE APPROACH TO GAME USABILITY AND DESIGN
TABLE 1. EXPERT’S SUBJECTIVE RATINGS OF PARTICIPANTS’ INITIAL AND FINAL MDS DISSIMILARITY GRAPHS Participant
Initial
Final
Gen1 Gen2 Gen3 Gen4 Gen5
Strong physical Weak physical Physical Physical Weak physical
Physical Other/unknown Other/unknown Functional Weak physical
The expert rated each participant’s MDS graphs based on its apparent physical or functional dimensions.
model clearly accommodated the functional relationships of the AI agents (Figure 5), a total of three of the players did shift away from their initial physical-based model. It may be that, with additional time, all would have shifted closer to the hypothesized functional-based model.
DISCUSSION As hypothesized, all of the participants’ initial mental models reflect surface feature organization. Each participants’ model was evaluated as either strong physical, physical, or weak physical. This supports the view that a player’s first experience is based on the available mental model of the AI agents’ visual features. The functional features of the AI agents suffer from opaqueness: the relation-
365
ships between the components of the system are not immediately available or represented.2 It is only with experience that the players’ mental models accommodated the functional relationships of the AI agents. One potential application of these pilot findings is to reduce the requirement for task analysis15,16 or extensive questionnaires.13 Human–computer interaction has used task analysis to produce optimized interactions, but it requires expertise and resources that may not be available to game designers. However, the simple dissimilarity questionnaire, and MDS analysis demonstrated in this paper create tremendous insight to design at a fraction of the resources of a task analysis. Furthermore, mental model monitoring offers a new opportunity for designing game progression. Traditional game progression is based on a set of levels of difficulty for all players. This progression fails when a player simply hits their physical (motor) or cognitive limitations. As a result, games can alienate a large portion of the population. Since we have shown that mental model shifts can be qualitatively assessed, game progression can be engineered by requirements to accommodate new information into the mental model. Game satisfaction and playability may therefore be based on successive “ah ha!” moments. These “aaha!” moments could consist of qualitative shifts (accommodation) in the player’s mental model driven by shifts in the AI agents’ functional relationships. This may provide interesting new challenges to players without frustrating them with the limits of their physical or cognitive abilities.
Enemy Follow on
Special Capabilities Dismounts Good against Tanks Halfway
Good against Dismounts Early in Battle
FIG. 5. Participant mental model assessed as functional. Mental model representation of participant (Gen 4) upon completion of day five trials. Circles represent the dimensions assigned to relationships identified by the rater.
14276c11.pgs
6/8/06
2:33 PM
Page 366
366
GRAHAM ET AL.
CONCLUSION This research has piloted a method for estimating and observing a novice player’s mental model within a RTS game. In support of past research, we did find that novice players initially focus on the surface characteristics of the environment and tend to move away from these surface characteristics with additional experience. While the method still requires validation, the results do show that mental models may be a powerful descriptor of a player. Future research should test how game designers can apply this information to move beyond traditional performance-based progression levels and use mental model progression to engage and hold the player’s interest. Lastly, if we can find a method to collect mental model data during game play; we can design AI agents that drive game progression.
ACKNOWLEDGMENTS Special thanks to Bill Brislin (RVMECH) for extensive scenario development and expert consultation on Command & Conquer Generals. This work was Prepared through collaborative participation in the Advanced Decision Architectures Consortium sponsored by the U. S. Army Research Laboratory under the Collaborative Technology Alliance Program, Cooperative Agreement DAAD 19-01-20009. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation thereon.
REFERENCES 1. Bransford, J. (1985). Schema activation and schema acquisition. In: Singer, H., and Ruddell, R.B. (eds.), Theoretical Models and Processes of Reading, 3rd ed. Newark, DE: International Reading Association, pp. 385–397. 2. Brehmer, B. (1992). Dynamic decision making: human control of complex systems. Acta Psychologica 81:211–241. 3. Craik, K. (1943). The Nature of Explanation. Cambridge, UK: Cambridge University Press.
4. Good, T.L., and Brophy, J.E. (1990). Educational Psychology: A Realistic Approach. New York: Longman. 5. Green, D.W. (2001). Understanding microworlds. Quarterly Journal of Experimental Psychology A 54:879–901. 6. Johnson-Laird, P.N. (1983). Mental Models. Cambridge, MA: Harvard University Press. 7. Jones, L.E. (1983). Multidimensional models of social perception, cognition, and behavior. Applied Psychological Measurement, 7:451–472. 8. Kruskal, J.B., and Wish, M. (1978). Multidimensional Scaling. Beverly Hills, CA: Sage Publications. 9. Laird, J.E., and van Lent, M. (1999). Developing an artificial intelligence engine. Proceedings of the Game Developers Conference pp. 577–588. 10. Lokuge, I., Gilbert, S.A., and Richards, W. (1996). Structuring information with mental models: a tour of Boston. Proceedings of CHI’96. New York: ACM Press. 11. Norman, D.A. (1986). Reflections on Cognition and Parallel Distributed Processing. In: McClelland, J.L., Rumelhart, D.E., and PDP Research Group (eds.), Parallel Distributed Processing,Volume 2: Psychological and Biological Models. Cambridge, MA: MIT Press, pp. 532–546. 12. Orne, M.T. (1962). On the social psychology of the psychological experiment: with particular reference to demand characteristics and their implications. American Psychologist 17:776–783. 13. Pagulayan, R.J., Keeker, K., Wixon, D., et al. (2002). User-centered design in games. In: Jacko, J., and Sears, A. (eds.), Handbook for Human-Computer Interaction in Interactive Systems. Mahwah, NJ: Lawrence Erlbaum Associates, Inc., pp. 883–906. 14. Piaget, J. (1952). The Origins of Intelligence in Children. Cook, M. (trans.). New York: International Universities Press. 15. Rasmussen, J. (1986). Information Processing and Human-Machine Interaction: An Approach to Cognitive Engineering. Amsterdam: North-Holland. 16. Woods, D.D., Johannesen, L., Cook, R.I., et al. (1994). Behind Human Error: Cognitive Systems, Computers and Hindsight. Dayton, OH: Crew Systems Ergonomic Information and Analysis Center, Wright-Patterson AFB.
Address reprint requests to: Dr. John M Graham Box 109 West Point, NY 10997 E-mail:
[email protected]
