The Effect of Learning Mechanics Design on Learning ...

1

The Effect of Learning Mechanics Design on Learning Outcomes in a Computer-Based Geometry Game Jan L. Plass, Bruce D. Homer, Elizabeth O. Hayward, Jonathan Frye, Tsu-Ting Huang, Melissa Biles, Murphy Stein and Ken Perlin Games for Learning Institute New York University The Graduate Center of the City University of New York

Abstract. A computer-based geometry game was adapted to allow for play using a conceptual rule or an arithmetic problem-solving mechanic. Participants (n = 91) from an urban middle school were randomly assigned to experimental conditions. Results suggest that play in the number condition was more situationally interesting than play in the rule condition. Participants in the rule condition were found to perform better in the game than those in the number condition. Learning outcome results suggest that in the number condition, but not the rule condition, playing more levels in the game diminishes the gain from pretest to posttest. For the design of games for learning, results highlight the importance of choosing a game mechanic that reflects the intended learning outcomes.

1

Introduction

Over the past decade, interest in developing computer-based games and simulations for learning has been ever growing (Gee, 2007; Squire, 2003). A number of influential books and articles have argued that well-designed games embody a broad variety of learning theories and are in line with some of the “best practices” of learning (e.g., Collins & Halverson, 2009; Gee, 2003; Mayo, 2007; Plass, Homer & Hayward, 2009). However, in order to realize the potential of games to foster the acquisition of specific knowledge and skills in players, designers have to address a series of challenges. For example, how should essential elements in the game be represented through iconic information to be more easily comprehended (Plass et al., 2009)? What design patterns can be formulated that help designers apply evidence from empirical research to make games effective for learning (Plass, Perlin, & Isbister, 2010)? How does visual design affect emotion, and how do these emotions affect learning (Um, Plass, Hayward & Homer, 2011)? The question addressed in the present paper extends our previous research by asking: How does the design of the learning mechanics affect learning outcomes in a geometry game. We define learning mechanics as “patterns of behavior or building blocks of learner interactivity, which may be a single action or a set of interrelated actions that form the essential learning activity that is repeated throughout a game” (Plass et al., in press, p. 13). We argue that one of the key factors in designing effective games for learning is that the learning mechanics reflect the specific content to be learned, and that they reflect the specific user actions that will foster the acquisition of the related knowledge or skills. In our work, we employ Evidence Centered Design as the framework that allows us to design these learning mechanics and assess their impact on learning. Evidence Centered Design (ECD; Mislevy, Almond & Steinberg, 2003) provides a useful framework for developing games for learning purposes. ECD requires designers to be specific about what skills, knowledge or other traits are being assessed, and what learners need to do, say, or create in order to provide evidence of the variables being assessed. A well-developed game for learning should include specialized learning mechanics (i.e., specialized activities grounded in learning sciences) that are mapped onto corresponding game mechanics (Plass et al., in press). The current study employed a single player puzzle game that focused on geometry concepts of angles. This game was designed to allow for manipulation of the learning mechanic, which was implemented in two variants. Middleschool students were randomly assigned to play one of two versions of a math game, Noobs vs. Leets. In the first version, which we call the Rule version, players were asked to apply a geometry rule to an angle to solve an angle problem. In other words, in order to solve an angles problem, the players were asked to identify which rule they would apply but did not specify the numeric answer. Rules included the supplementary angles rule, complementary

2

angles rule, opposite angles rule, etc. In the second version, which we call the Number version, players were given the same angles problems but were asked to calculate the angle and click on the correct number value to solve for the angle. In other words, instead of identifying the rule they used to solve the problem, learners were asked to specify the numeric answer. Learning outcomes, situational interest, and game performance were dependent variables.

2

Method

Participants and Design Participants were 91 students from an urban school in a major Northeastern city. There were 33 sixth graders and 58 eighth graders. 43 participants were females (47%). Each participant was randomly assigned to one of two versions of the game: rule and number. In the rule condition, participants played the version that required them to solve problems with geometry rules, whereas in the number condition, they played the version that required them to do arithmetic calculation. Two participants were excluded as they had outlying scores on the arithmetic fluency pretest or on the pretest of geometry skills. The resulting total of 89 participants was included in the analyses. Procedure Participants were randomly assigned to conditions. First, they were administered a measure of math fluency and a pre-test of geometry knowledge that was timed for a period of 10 minutes. Once these were completed, participants engaged independently in a 25-minute game-play session of the game, Noobs vs. Leets. Participants were given the instruction: “When playing the game, do the best you can.” Participants played the game on desktop computers using a mouse. Game-play instructions were provided through cut-scenes and in-game tutorials. An experimenter was also available during this time for questions about game play. At the end of the game-play session, participants were instructed to stop all play activity. Participants then completed geometry post-test for a period of 10 minutes. After the post-test was administered, participants completed a situational interest survey and a play-testing protocol. Materials Noobs vs. Leets is a single player puzzle game that focuses on geometry concepts of angles. The game was designed to investigate the learning mechanic of rule application. The game runs in a web browser on desktop computers. The object of the game was to unlock angles, allowing the ‘noob’ character to free a fellow ‘noob’ from a cage. By unlocking angles, players open paths that the character can traverse. In the rule version of the game, angles are solved by clicking on the angle or angles to be solved and then selecting the appropriate rule. For example, a player could click on a known angle and an unknown angle that together make a right angle and then select the ‘complementary rule’ button to unlock an angle. In the number version of the game, angles are solved by clicking on an angle and selecting the correct degree of the angle. For example, a player could click on a locked right angle and then click the ‘90°’ button (Figure 1). For complex angles, players would have to calculate the correct number of degrees of the angle. The game is divided into six chapters, each consisting of 8-10 levels and focusing on a specific angle concept. These concepts include simple, complementary, supplementary, vertical, triangle, and quadrilateral. Both versions of the game contain an identical first chapter, which mixes response types, and focuses on simple angle types such as acute, obtuse, right, and straight. Each chapter begins with a cut-scene introducing and explaining the concept for that chapter. Each level within a chapter progressively increased in difficulty. Levels are complete when a player has unlocked a pathway for the noob to reach the caged noob. On some levels, immobile or mobile ‘leet’ characters block some pathways that a player could solve, requiring the player to unlock a pathway around the ‘leet’. See Figure 1 for a screen shot of each game version.

3

Fig. 1.

Two Game Mechanics in Noobs v. Angles Game: Number version (left) and Rule version (right)

Measures Participants were first given measure of math fluency. The measure consisted of 160 simple addition and subtraction arithmetic problems. Participants were provided 3 minutes to complete as many problems as possible. The measure of math fluency was adapted from the Woodcock Johnson – III Math Fluency subtest (McGrew & Woodcock, 2001), which was modified by randomizing the presentation of problems, excluding multiplication problems, and added additional two and three digit problems, so as to make the type of calculation comparable to the arithmetic employed in solving for angles. Participants also completed a pre-test and post-test of geometry knowledge. A teacher with experience teaching middle school math designed these measures. Items were adapted from questions from yearly exams based on New York state standards. The two measures consisted of nearly identical problems, with slight variations to the specific degrees used or angle being questioned. Participants were also administered the Situational Interest Survey (Linnenbrink-Garcia et al., in press). The language in the situational interest measure was simplified to ensure comprehension in our middle school sample. In addition, the measure was modified to be relevant for game play. The Situational Interest Survey asked participants to use a 7-point Likert scale to indicate their level of agreement with twelve statements, such as “The game was exciting,” “The things I learned from the game are important to me,” and “I thought the game was interesting.”

3

Results

Situational Interest In order to investigate the situational interest in the game, we conducted a between-subjects ANCOVA, controlling for number of levels completed and grade level. This analysis yielded a marginal main effect for condition, F(1, 82) = 3.64, p = .06, η2 = .05, such that the number condition (Madjusted = 5.45, SE = .20) demonstrated greater situational interest than those in the rules condition, (Madjusted = 4.90, SE = .20). Game Performance In order to examine the effect of condition on game performance, an ANCOVA was performed on the total number of levels completed in the game, with grade level added to the model as a covariate. The analysis yielded a effects for grade, F(1, 89) = 9.014, p = .004, η2 = .10, as well as condition F(1, 89) = 7.16, p = .009, η2 = .08. Participants who were in the 6th grade (M = 23.84, SD = 7.40) completed significantly fewer levels than those in the 8th grade (M = 28.52, SD = 7.32). Individuals in the rule group completed more levels (Madjusted = 28.88, SE = 1.06) than individuals in the number group (Madjusted = 24.85, SE = 1.07). This suggests that those in the rule group performed better in the game than those in the number group (Figure 2).

4

Fig. 2.

Pre- v. Post Test Results for Number vs. Rule groups

Geometry Learning Outcomes A two-way repeated-measures 2 (Time: Pretest vs. Posttest) x 2 (Question Type: Arithmetic vs. Concept) ANCOVA was conducted using average of correct responses as the dependent variable. Condition was a between-groups factor, and arithmetic fluency pretest and number of levels completed were covariates. A custom RM-ANCOVA model was run with time, question type, arithmetic fluency pretest, and levels completed entered as main effects, and time by condition, time by level, condition by level, and time by condition by level as interaction terms. Main effects were found for time, F(1, 84) = 12.61, p = .001, partial η2 = .13 and question type, F (1, 84) = 45.16, p < .001, partial η2 = .35. Therefore, there existed overall differences in the responses due to time, such that scores on the outcome measure increased from the pretest (Madjusted = .50, SE = .01) to the posttest (Madjusted = .57, SE = .01). There was also a main effect of question type, such that participants in both groups were more successful on the conceptual rule-based questions (Madjusted = .74, SE = .01) than on the arithmetic questions Madjusted = .35, SE = .01), across the pretest and posttest measures. There were also main effects for both covariates on the outcome measures: arithmetic fluency pretest, F(1, 84) = 23.51, p < .001, partial η2 = .22, and levels completed, F (1, 84) = 16.09, p < .001, partial η2 = .16. These main effects were qualified by significant interactions. There was a two-way interaction between time by condition, F(1, 84) = 6.05, p = .02, partial η2 = .07, represented in Figure 1, as well as a time by level interaction, F(1, 84) = 4.67, p = .03, partial η2 = .05. Of greater theoretical interest is the three-way interaction among time by condition by level, F(1, 84) = 5.67, p = .02, partial η2 = .06. This interaction is shown in Figure 3. When the results were decomposed by condition, there was simple interaction effect of time by level for the number group, F(1, 41) = 4.49, p = .04, partial η2 = .10, but not for the rule group. This interaction was further decomposed by running separate analyses for those who completed less than median number of levels (< 30) or more levels (≥30) for individuals in the numbers condition. For participants in the number condition who completed fewer than 30 levels, there was a significant gain from pretest to posttest, F(1, 24) = 5.55, p = .03, partial η2 = .19, whereas for those who completed more than 30 levels, there was no gain from pretest to posttest, F(1, 16) = .174, n.s. The graphic representation of this interaction suggests that for the number group, but not for the rule group, playing more levels diminishes the gain from pretest to posttest. In contrast, for the rule group, the extent of the gain from pretest to posttest was consistent, irrespective of the number of levels completed. The condition by level interaction term did not significantly account for variance in the model.

5

Fig. 3.

4

Three-way interaction among time by condition by level

Discussion

The present study investigated how variation of the learning mechanic in a geometry puzzle game affected situational interest, game performance, and learning outcomes. The number condition resulted in greater situational interest than the rule condition, suggesting that calculating the arithmetic solution for angles yields greater intrinsic interest. However, individuals in the rule condition performed better in the game. Finally, with regard to learning outcomes, results suggest that in the number condition, but not the rule condition, playing more levels in the game diminishes the gain from pretest to posttest. In contrast, for the rule group, the extent of the gain from pretest to posttest was consistent, irrespective of the number of levels completed. Overall, the results suggest that while calculating the arithmetic solution to geometry problems may be more situationally interesting, over time, this approach yields diminishing returns. These findings indicate that having an arithmetic understanding of how to solve geometry problems may promote learning outcomes in the short term, but in order to sustain gains in learning, having a conceptual rule-based understanding is necessary. Results also indicate that the choice of the game mechanic in a learning game can have significant impact on the learning outcome. Future work should examine the effects of other design factors, and also if the effects found in the current study are different for different types of games and academic areas.

6

References Collins, A., & Halverson, R. (2009). Rethinking education in the age of technology: The digital revolution and schooling in America. NY: Teachers College. Gee, J. P. (2003). What video games have to teach us about learning and literacy. New York, NY: Palgrave Macmillan. Gee, J.P. (2007). Good video games + Good learning. New York: Peter Lang. Linnenbrink-Garcia, L., Durik, A. M., Conley, A. M., Barron, K. E., Tauer, J. M., Karabenick, S. A., et al. (in press). Situational interest survey (SIS): An instrument to assess the role of situational factors in interest development. Mayo, M. J. (2007) Games for science and engineering education. Communications of the ACM, 50 (7), 30-35. McGrew, K. S. & Woodcock, R. W. (2001). Technical Manual: Woodcock-Johnson III. Mislevy, R.J., Almond, R.G., & Steinberg, L.S. (2003). On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3-67. Plass, J.L., Homer, B.D., Chang, Y. K., Frye, J., Kaczetow, W., Isbister, K. & Perlin, K. (in press). Metrics to Assess Learning and Measure Learner Variables in Simulations and Games. In Eds. El-Nasr et al., Game Telemetry and Metrics. Morgan Kaufman. Plass, J.L., Homer, B.D., Milne, C., Jordan, T., Kalyuga, S., Kim, M., & Lee, H.J. (2009). Design Factors for Effective Science Simulations: Representation of Information. International Journal of Gaming and Computer-Mediated Simulations, 1(1), 16–35. Plass, J.L., Homer, B.D., & Hayward, E. O. (2009). Design Factors for Educationally Effective Animations and Simulations. Journal of Computing in Higher Education, 21(1), 31–61. Plass, J.L., Perlin, K., & Isbister, K. (2010). The Games for Learning Institute: Research on Design Patterns for Effective Educational Games. Paper presented at the Game Developers Conference, San Francisco, March 9-13, 2010. Squire, K. (2003). Video Games in Education. International Journal of Intelligent Simulations and Gaming (2) 1. Um, E., Plass, J. L., Hayward, E. O., & Homer, B. D. (2011, December 19). Emotional Design in Multimedia Learning. Journal of Educational Psychology. Advance online publication. DOI: 10.1037/a0026609