Evaluating User Experience and Other Lies in Evaluating Games

Evaluating User Experience and Other Lies in Evaluating Games Heather Desurvire Usability/Playability Specialist Behavioristics Venice, California USA [email protected] +1 310 823 6543

Charlotte Wiberg Assistant professor Dept of informatics Umea University Umea, SWEDEN [email protected] to their hardcore counterpart suggests that casual gamers require more guidance playing video games. This in turn indicates a challenge to support the casual gamer in getting started with game play without divulging the secrets of the game itself—that is, to provide the tools to play games so casual game players have the potential to be confident in mastering the game. Therefore, inclusion of casual gamers into the mix of targeted people for whom games are designed may require specific methods and approaches in game design to better meet these needs.

ABSTRACT

Game Approachability Principles (GAP) is proposed as a set of useful guidelines for game designers to create better tutorials, or first learning levels—especially for the casual gamer. By producing more effective learning levels/tutorials, the aim of GAP is to make games more approachable to entice gamers to want to play, and play more frequently. Developing better first learning levels can be a key step to ease the casual gamer into play and to do so proactively—before it is too costly or cumbersome to restructure the tutorials to be more effective. Thus, Game Approachability in the context of game development is defined as making games initially more friendly and accessible for players who have the desire to play, yet do not always follow-through to actually play. GAP has evolved through a series of stages assessing applicability as a stand alone, heuristic based approach versus one-on-one usability testing. Outcomes suggest potential for GAP as (1) effective Heuristic Evaluation, (2) adjunct to Usability Testing, and (3) as proactive filters in beginning conceptual and first learning level tutorial design to increase Game Approachability—for all levels of gamers.

Today, game design also involves a focus on traditional usability such as creating clear terminology as well as nonintrusive, easy-to-use user interfaces. Other principles specific to games, include pace and adequate challenge (offering a game that is both not too difficult and not too easy). (Desurvire et al., 2004; Desurvire et al, 2006; Federoff 2002; Federoff, 2003; Kohnnen, et al., 2006) Both games and traditional productivity software also have the common usability assumption that users need not have a manual (or access to one) to use the software or play the game. Hardcore gamers, however, are usually more willing to seek assistance and support in order to play games and have traditionally relied on strategy guides, past expertise, cheat codes, online bulletin boards, and other players to learn to master a game. In direct contrast, casual gamers are more likely to quit playing than seek out and use any of these resources. Thus, with the growing number of casual gamers, and game developer focus on casual games, the approachability aspect of game playability is more critical today.

Author Keywords

Game Approachability, game design, method development, usability for video games ACM Classification Keywords

H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous. INTRODUCTION

GAME APPROACHABILITY

Recently, video game developers and publishers have been modifying their focus from meeting the desires of hardcore gamers to serve the less savvy and less frequent player—the casual gamer. Signaling an important shift in the game industry, this new emphasis on casual gaming indicates that the concept of “Game Approachability” may be as crucial an aspect of gaming fun and entertainment as “engagement” has been historically. Casual gamers, as their name implies, often lack extensive prior game play experience. The casual game player’s more periodic exposure to games in contrast

The concept of “Game Approachability” is closely aligned to another term “accessibility” which started to appear in HCI (human computer interaction) research around 2000 (c.f. Stefanidis, 2001). Yet the term accessibility is often used to describe the degree to which a system is usable by as many people as possible (c.f. Stefanidis, 2001; Benyon et al, 2005; Vanderheiden, 2003) and game accessibility in HCI usually relates to access for all regardless of ability or disability (c.f. www.w3.org/wai). However, in the context

1

of game development, approachability involves an adjacent although slightly different orientation—making games more accessible and easier to grasp for players who have the desire to play, yet do not always follow-through on the action to actually play. In other words, Game Approachability refers to the ease in which gamers are able to approach and avail themselves of games—for all who desire to play. In considering Game Approachability, three questions arise: (1) how can Game Approachability be best assessed and applied? (2) What may need to be revised and redesigned in methods used to assess Game Approachability? And, (3) how will these methods assist designers include better Game Approachability into games? LEARNING AS A MEANS TO GAME APPROACHABILITY

Since grasping how to play a game is essentially “learning” how to play, learning theories prove to be applicable and useful for assessing the potential for Game Approachability Principles. While there may be no one, universally accepted theory of learning most applicable, learning in the realm of electronic games can be better understood though learning theories commonly found in education and other fields. In fact, some learning theories offer potential for a high level of applicability to game design. Since theories of learning applied in education to improve student learning often involve motivation, helping behaviors, or engagement that are also central for gaming—they provide a starting point for describing how Game Approachability can improve first learning level design—particularly for casual gamers. For example, (1) Social Learning Theory (c.f. Bandera, 1977) emphasizes the importance of observation and modeling in the learning process—and for games, observing others’ play is frequently how a game is first or best introduced to a game player. Thus, “Observation and Modeling” is understandable as a principle applicable in Game Approachability. (2) Self-efficacy, another key concept and term used in education and learning (Ormond, 1999; Bandura, 1994) refers to people’s beliefs about their own capabilities or about their ability to reach a goal. In games, these types of beliefs are highly evident and support “Self Efficacy” as a component of GAP. (3) In education, good game design is used to develop principles in designing educational materials and curriculum that are both motivating and fun for students (Gee, 2003; Gee, n.d.). Thus, a subset of the elements identified by Gee that are also applicable as principles in Game Approachability include aspects of “Identity,” “Customization,” “Manipulation and Perception,” and “Information On Demand And In Time” (for a more thorough description of the points presented above see earlier publications, c.f. Desurvire et al, 2007:1) Playability, the relative quality of game play in being fun to play and usable, is also important when considering Game Approachability. In fact, Game Approachability can be understood as a subset of playability (Desurvire, H. 2004;

Pagulayan et al, 2003). Due to the linkages between Game Approachability and traditional playability, two of the frameworks for evaluating playability—(4) HEP (Heuristic Evaluation for Playability) and PLAY (game playability principles) (Desurvire, H., 2004, and Desurvire, H. and Chen, B. 2006)—are also logical inclusions for Game Approachability. While more than ten principles are under consideration, a preliminary subset of the Game Approachability Principles aimed at better engaging casual gamers is shown below. This specific “shortlist of six” Game Approachability Principles (GAP), was formulated in keeping with the learning theories and frameworks noted earlier. These developing principles were then used in a series of GAP assessment and development work, and are recommended for further research, investigation and validation (for additional description of this early version of the list, see other publications, c.f. Desurvire, 2007:1) Initial Checklist: Game Approachability Principles (GAP) for Improving Game Approachability 1. Observation and Modeling 2. Self Efficacy 3. Gee Game Based Principles (Identity, Manipulation and Perception, etc.) 4. HEP and PLAY Based Guidelines (such as players not being penalized repetitively for the same failure; varying activities and pacing during the game to minimize fatigue or boredom; etc.) 5. Demonstrate Actions and Reinforcement 6. Likeability of the Tutorial VALIDATION OF HEURISTICS

The development of the Game Approachability Principles (GAP) was next subject to a series of assessment and research study toward validating their use. The initial checklist of GAP (shown above) was first derived qualitatively from the aforementioned learning theories. In addition, a series of research activity in two stages involving two usability/playability experts, an educational learning expert, and several game designers was undertaken. Stage 1

The first stage of GAP development included conducting both Heuristic Evaluation and a Usability Test. Each method involved two video games—a Real Time Strategy game and a Shooter game. Both games evaluated were Xbox360 games in the process of development and focused on increasing Game Approachability for casual gamers. The Usability Test with GAP involved 12 players for each game. Heuristic Evaluation was used to simulate the use of GAP as a stand alone method without the aid of a Usability Test. The two methods were compared (Heuristic Evaluation versus Usability Testing) with focus on assessing what issues were similar for each method, and

what new or different issues resulted from each method. Based on these outcomes, principles which did not promote or enhance Game Approachability were dropped from the GAP list and led to the creation of a new, revised list of GAP for the next stage.

frustrated, or wanted to quit the game. Players’ comments and probing questions were used to verify any assumptions made by the evaluator. Following completion of the sessions, playability design problems were identified and alternative design solutions were generated.

Stage 2

Examples of GAP

The second initiative undertaken used a new, modified list of approachability principles resulting from Stage 1. The same two-method procedure used in Stage 1 was performed again on the Xbox360 and similarly included a Sports game and a Shooter game. This stage was conducted to provide additional assessment toward validating the usefulness of GAP with a new set of games. Three key questions addressed in this stage included: (1) did both Heuristic Evaluation and Usability Testing identify the same Game Approachability issues? (2) Did one method uncover more issues regarding Game Approachability, playability and/or usability? (3) Did the two methods complement or conflict with each other and how?

The following verbatim comments from players (some with selected accompanying screens; due to space limitations not all can be shown) provide examples of the types of Game Approachability issues identified in both Usability Testing and Heuristic Evaluation: GAP: Observation/Modeling Game Genre: Sports Racing

“Wow. I just made the car do a 180, I didn’t know that there were special moves that I could do. I wish that other cars had done these moves so that I knew they were available earlier in the game.” GAP: Self-Efficacy

Stage 3

Game Genre: Online Poker

The third initiative included data from four games to again identify differences and similarities in outcomes to Heuristic Evaluation and Usability Testing regarding Game Approachability. One of the games was a First Person Shooter game, the second an Action Adventure game in the development stage that were both played on an Xbox 360 console. The other two games included an online Poker game played using a Playstation 3 and a Real Time Strategy game in the development stage—played using a Nintendo Wii.

“I don’t feel comfortable playing this with real money. Maybe if I practiced more I would be, but right now I feel like I would just lose all my money and it wouldn’t be very fun.”

In this third stage of study, one playability evaluator performed GAP Heuristic Evaluation, focusing on how each approachability principle was supported or violated, while doing a walkthrough of the game. The evaluator also assessed and defined playability issues. After Heuristic Evaluation was performed, Usability Testing was also conducted, whereby 32 players engaged in playability sessions. The majority of the players were male and all were between the ages of 8 and 35. About half of the players were considered casual players and half were a mix of moderate and hard core players.

GAP: Demonstrate Actions and Reinforcement Game Genre: Action Adventure

“They told me the instructions but I never got a chance to try them. If you asked me now, I wouldn’t remember, I think it would have been better to try it during the tutorial.”

Each session was organized as a Usability Testing session, in an environment simulating actual game play conditions. Participants were given instructions to begin each game, asked to think aloud, and asked several probing questions while using the prototype games. All the players played a tutorial and first game levels. The players were thanked, debriefed and asked to complete a questionnaire. The evaluator recorded a log of the players’ actions, comments, failures, missteps, and coded each of these as a positive or a negative player experience. A positive experience was defined as anything that increased pleasure, immersion, and the challenge of the game. A negative experience was defined as any situation where the player was bored,

Game Genre: Sports Racing on the Wii

“I am not sure which way to hold the controller. This is really confusing, I wish that there was an animation that demonstrated to me which way to hold it.”

3

GAP: Gee Game Based Principles /Identity (Positive) Game Genre: Online Poker

“I really like the avatars and that I can choose which one I want to play with. This makes it more interesting because I feel like I really am the person playing.”

what types of issues were evident in the games. Issues identified were categorized as either a Game Approachability issue or as a playability/usability issue. Game Approachability issues were then categorized as relating to one or more of the Game Approachability heuristics in GAP. Additionally, Game Approachability issues in the games were compared to determine (1) what issues Heuristic Evaluation identified that Usability Testing did not, (2) what issues Usability Testing found that Heuristic Evaluation did not, and (3) what issues were similarly uncovered by both methods. Also, Heuristic Evaluation and Usability Testing results were compared to determine the number of Game Approachability issues identified within each method and by both methods combined. Finally, descriptions of the issues identified were compared to assess any similarities or differences in specificity resulting from each method. Examples of Issues Found Solely Using GAP in Heuristic Evaluation (Not found in Playability/Usability Testing) Scaffolding

In level two of the Action Adventure game, there is a point where the player has to cut chains from a fort. There are no clues that the player should do this. In other areas, the player is told the button to push and the object is highlighted in red—this ideally would be consistent. (GAP: HEP and PLAY Based Guidelines) Demonstration GAP: Gee Game Based Principles/Manipulation and Perception Game Genre: Action Adventure

“One thing that I liked about the game is that I was given the option to play new levels as I completed more and more of the objectives.”

Players are not provided with demonstrations of different combos to encourage them to try various ways of attacking so they may get stuck attacking in the same way. If they saw other players performing moves that were interesting, they might try to do these moves also. (GAP: Demonstrate Actions and Reinforcement; Observation and Modeling)

GAP: HEP and PLAY Based Guidelines

Examples of Issues Found Solely Using Playability/Usability Testing (Not Found in GAP Heuristic Evaluation)

Game Genre: Real Time Strategy

Player Does Not Get Move Combo

“It needed to teach you how to build a small little town or something. It never taught you how to harvest resources or anything else and you have to know this to play the game. If you are going to build up your army you have to have the resources.”

In the Shooter game, the moves are not deliberate; which can be an issue for a more moderate level player who is often very deliberate in his or her actions. It would be helpful for these types of players to start teaching the combos while still allowing mashing. (GAP: HEP and PLAY Based Guidelines)

GAP: Likeability of the Tutorial

Puzzle-Platform Stability

Game Genre: Real Time Strategy

In the Action Adventure game, the players kept jumping on the platform areas, not realizing they are unstable. The players want to fight, and not run, and the unstable platforms are not helping this. It took the players sometimes four tries, which for a moderate player is not a big issue, but for a less experienced player may well be an issue and lead to frustration and wanting to quit the game. (GAP: HEP and PLAY Based Guidelines)

“I think that this tutorial is way too long. I want to be playing the game, but instead I am still doing the tutorial. I thought the stuff in the beginning was useful, but now it just seems like too much and I am not having that much fun.” RESULTS

Following completion of Heuristic Evaluation and Usability Testing in Stage 3, the results were compared to identify

that a player had difficulty in was identified as an issue (such as unclear goals) and each separate area would also be counted as an issue. For Heuristic Evaluation, a problem encountered was counted only once and simply noted that players would continue to have problems with a certain skill because it was not learned at the point at which the designers intended.

Issues Found Using Both GAP Heuristic Evaluation and Playability/Usability Testing Too Much Help

In the Real Time Strategy Xbox360 game, the A gives clues as to which button to press (for example when walking near a chain it will say push B as soon as the player walks near the object). Help is given too quickly, even when the players’ game play doesn’t warrant it. (GAP: Self Efficacy)

The results of each stage under study, and particularly Stage 3, show that GAP Heuristic Evaluation pinpointed a different set of issues overall that were more conceptual—and gave more design clues for organizing the beginning levels for Game Approachability. In contrast, Usability Testing assessed design issues and suggestions that can be considered more granular and directly applicable to the specific game during play. In summary, while Usability Testing is more reactive in approach and can pinpoint problems in existing game design, a key differential and advantage of GAP Heuristic Evaluation is its proactive nature enabling first learning levels design to be conceptualized and planned with approachability in mind.

Unclear Goals

The yellow, triangle objective indicators might be misleading in this Real Time Strategy game when the players get near the building with the objectives. When players get to a building the arrow focuses on a point of the building that isn’t necessarily the entrance. As the player walks around the building, even toward the entrance, the arrows are pointing in a different direction, which could make them think they are headed in the wrong direction when they are in fact going the correct way. (GAP: HEP and PLAY Based Guidelines) GAP Heuristic Evaluation Counts

In comparing the two approaches in Stage 3, GAP Heuristic Evaluation identified a higher percentage of Game Approachability issues than did Usability Testing—a total of 90 issues of which 43, or 47%, related to Game Approachability (see Table 1). In contrast, Usability Testing identified 207 issues in total with 22 issues, or 11%, relating to Game Approachability. Also, GAP Heuristic Evaluation found more types of Game Approachability issues than were evident in Usability Testing—six categories of Game Approachability compared to three in Usability Testing. While understandably, Usability Testing uncovered more issues relating to playability and usability (185, or 89%); GAP Heuristic Evaluation identified 27, or 52% of issues involving playability or usability.

Table 1 Issues Identified with GAP Usability Testing vs. Heuristic Evaluation Game Approachability Usability Heuristic Principles (GAP) Testing Evaluation Observation and Modeling 0 1 Self Efficacy 0 3 Gee Game Based Principles 0 9 HEP and PLAY Based Guidelines 17 12 Demonstrate Actions and Reinforcement 2 7 Likeability of Tutorial 0 0 Other (Amount of Practice; Amount and Type of Demonstration; Information Presented in Context; Scaffolding-Failure Prevention) 3 10 Total 22 43

Level of Detail

In addition to differences in the number of approachability and playability issues identified by each method, there was also variance in the level of detail that each provided. Usability Testing referred more to (1) specific problematic areas of the games, (2) the number of players having difficulty with certain areas of the game, and (3) verbatim comments from players expressing frustration. Conversely, Heuristic Evaluation would more often (1) identify areas where a player was not given the means to master a skill and (2) indicate other areas in the game that may give players trouble since they did not learn the needed skill. These outcomes are most likely a result of Usability Testing describing problems as they occur whereas Heuristic Evaluation more often predicting problems players will likely encounter.

CONCLUSIONS

These results suggest that Game Approachability Principles (GAP) appear useful in evaluating beginning and tutorial levels of game design and offering suggestions to game designers to improve Game Approachability. Additionally, Heuristic Evaluation with GAP provides information that supplements and complements Usability Testing. GAP Heuristic Evaluation alone provided more applicable and useful information about game approachability while Usability Testing provided more information regarding game playability and usability. Thus, while Usability Testing is able to provide a level of detail not possible in Heuristic Evaluation—including specific commentary from players and the identification of areas to be improved to

Additionally, the high number of HEP and PLAY issues identified by Usability Testing may also be a result of this method’s more “in-process” versus “future play” orientation (see Table 1). For Usability Testing, each area 5

increase player enjoyment and learning—it does so through an iteration improvement process during game design. However, the Game Approachability Principles offer proactive rather than reactive opportunity by developers in creating better, more approachable tutorial, first learning levels before the design is set. Therefore, while Usability Testing may identify problematic areas for players and pinpoint ways to make the goals and game design more clear, Heuristic Evaluation using GAP would help identify necessary skill(s) not learned which will likely cause problems later in the game for a player when this skill is needed—an important and useful addendum. In summary, having game designers employ Game Approachability Principles proactively as guidelines and filters in developing the beginning levels of games and tutorials may prove useful in enhancing Game Approachability in games. GAP also offers potential advantages through combining evaluation of the user game experience during game development with Usability Testing to promote higher levels of Game Approachability. Heuristic Evaluation with GAP provides a more conceptual, proactive design plan for Game Approachability while Usability Testing provides opportunity for retroactive game design correction to improve Game Approachability. Thus, taken together, applications of GAP Heuristic Evaluation and one-on-one Usability Testing offer game designers additional and stronger pathways toward making games more approachable and accessible to the casual player—and ultimately to all players of games. ACKNOWLEDGEMENTS

We would like to give special thanks to Maria Quintero of the UCLA Graduate School of Education and Information Sciences (GSEIS) for her numerous contributions, and for useful discussions with several of the game designers at Electronic Arts Playa Vista, THQ, LucasArts, Kaos, Incinerator and Petroglyph.

Principles (PLAY) For Designing Highly Ranked Video Games (LA CHI Association Meeting Presentation). 7. Desurvire, H. (2007). List of Core and Game Approachability Principles for Good Game Design (LA CHI Association Meeting Presentation). 8. Desurvire, H. & Wiberg, C. (2007:1); Master of the Game. In the proceedings of the CMID’07 the first international conference on cross-media interaction design, held in Hemavan Sweden, April 2007. 9. Desurvire, H., Jegers, K., Wiberg, C. (2007) Evaluating Fun and Entertainment: Developing a Conceptual Framework Design of Evaluation Methods. Presented at the workshop ‘HCI and New Media Arts: Methodology and Evaluation’ at the CHI 2007 conference, San Jose CA, USA, April 10. Federoff, M. (2002). Heuristics and Usability Guidelines for the Creation and Evaluation of FUN in Video Games. Thesis at the University Graduate School of Indiana University, Dec. 2002 ([email protected]). 11. Federoff, M. (2003) User Testing for Games: Getting Better Data Earlier. Game Developer Magazine (June 2003), 35 - 40. 12. Gee, J.P. (2003) What Video Games Have to Teach Us About Learning and Literacy. New York, NY: Palgrave Macmillan. 13. Gee, J.P. (n.d.). Learning by Design: Good Video Games as Learning Machines. Accessed 12/26/05: 14. http://www.academiccolab.org/resources/documents/Ga me%20Paper.pdf. 15. Korhonen, Hannu, and Elina M. Koivisto: Playability heuristics for mobile games. Mobile HCI 2006: 9-16. 16. Ormond, J.E. (1999). Human learning (3rd ed.). Upper Saddle River, NJ: Prentice-Hall.

1. Bandura, A. (1994). Self-efficacy. In V. S. Ramachaudran (Ed.), Encyclopedia of human behavior (Vol. 4, pp. 71-81). New York: Academic Press.

17. Pagulayan, R., Keeker, K., Fuller, T., Wixon, D., & Romero, R. (2003) User-centered design in games (revision). In J. Jacko and A. Sears (Eds.), Handbook for Human-Computer Interaction in Interactive Systems. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.

2. Bandura, A. (1977). Social Learning Theory. New York: General Learning Press.

18. Stefanidis, C. (2001). Access for all. Proceedings on HCI International Conference, New Orleans, 2001.

3. Benyon, D., Turner, P., Turner, S. (2005). Designing Interactive Systems. People, Activities, Contexts, Technologies

19. Vanderheiden, G. (2003). Interaction for Diverse Users. In Jacko & Sears (eds) The Human-Computer Interaction Handbook. Fundamentals, Evolving Technologies and Emerging Applications. (pp.397-400)

REFERENCES

4. Bruer, J. (2000). Schools for Thought. Cambridge, MA: MIT Press. 5. Desurvire, H., Caplan, M., Toth, J.A (2004). Using Heuristics to Evaluate the Playability of Games. In the proceedings of the ACM conference, CHI2004. 6. Desurvire, H., Chen, B. (2006). 48 Differences Between Good and Bad Video Games: Game Playability

20. Wiberg, C. (2003). A Measure of Fun: Extending the concept of web usability. PhD Thesis. Dept of informatics, Umeå University. 21. Yee, N. (2007). Motivations of Play in Online Games. CyberPsychology and Behavior, 9, 772-775.