Optimizing Motivation and Learning with Large-Scale Game Design Experiments J. Derek Lomas December 2014 CMU-HCII-14-112
Human-Computer Interaction Institute School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 Thesis Committee Ken Koedinger (Co-Chair) Jodi Forlizzi (Co-chair) Emma Brunskill Jesse Schell Submitted in partial fulfillment of the requirements for the Degree of Doctor of Philosophy in Human-Computer Interaction Copyright © 2014 J. Derek Lomas. All rights reserved. This work was supported by Carnegie Mellon University’s Program in Interdisciplinary Education Research (PIER) funded by grant number R305B090023 from the US Department of Education and by the Defense Advanced Research Projects Agency (DARPA) under Contract No. ONR N00014-12C-0284. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author.
Keywords Interaction Design Science, Funology, Science of Fun, UX optimization, games, learning, motivation, motivational design, taxonomy, fun, intrinsic motivation, enjoyment, endogenous motivation, integrity, harmony, learning curves, A/B testing, multi-armed bandits, UCB1, machine learning, game design, games for learning, serious games, task difficulty, winning and losing, metrics, online controlled experiments, integration processes, challenge, game data, completely randomized design, nomological networks, nomothetic networks,
Citation Format: Lomas, J. Derek (2014). Optimizing Motivation and Learning with Large-Scale Game Design Experiments. PhD Thesis, Human-Computer Interaction Institute, School of Computer Science, Carnegie Mellon University.
ii
Abstract Large-scale online experiments can test generalizable theories about how designs affect users. While online software companies run hundreds of thousands of experiments every day, nearly all of these experiments are simple A/B tests structured to identify which software design is better. In contrast, this thesis highlights opportunities for an “interaction design science” where online experiments can test generalizable theories explaining how and why different software designs affect user interactions. To illustrate the basic scientific opportunities inherent within large-scale online design experiments, this thesis deploys over 10,000 variations of an online educational game to more than 100,000 learners in order to test basic psychological theories of motivation. In contrast to dominant theories of motivation, which predict that a moderate level of challenge maximizes motivation, these experiments find that difficulty has a consistently negative effect on motivation, unless accompanied by specific design factors. However, a series of parallel experiments provide evidence that a moderate level of novelty maximizes motivation, while also increasing difficulty. These results suggest that previous theoretical formulations of challenge may be conflating difficulty and novelty. These experiments are conducted within Battleship Numberline, a systematically designed learning game that has been played over three million times. This thesis argues that accelerating the pace of online design experiments can accelerate basic science, particularly the scientific theory underlying interaction design. For instance, a testable taxonomy of motivational design elements is presented, which could be validated through a series of online experiments. Yet, while it may be feasible to run thousands of design experiments, analyzing and learning from this large-scale experimentation is a new and important scientific challenge. To address this issue, this thesis investigates the use of multi-armed bandit algorithms to automatically explore (and optimize) the design space of online software. To synthesize these results, this thesis provides a summary table of all 17 tested hypotheses, offers a design pattern for producing online experiments that contribute to generalizable theory and proposes a model that illustrates how online software experiments can accelerate both basic science and data-driven continuous improvement.
iii
iv
Document Summary Student motivation is a critical component of successful educational outcomes. To build a theoretical basis of human motivation, I provide a novel synthesis of Plato and modern cognitivist psychology. I show how these diverse theories converge on “integration” as a unifying process underlying different motivations and show how the concept of “intrinsic integration” is usefully applied within contemporary theories of effective learning game design. With this foundation, I synthesize existing taxonomies of motivational design in the context of educational game design. In consideration of online methods for measuring motivation, I propose using large-scale online controlled experiments to systematically test the predictions of the motivational design taxonomies. The first motivational design element in a popular taxonomy is that “moderate levels of challenge will be optimally motivating”; this hypothesis is broadly supported by multiple theories of motivation. To test the idea that online design experiments can contribute to generalizable theories of design, I design a series of online experiments to test this motivational hypothesis using Battleship Numberline, a popular educational game designed to help children develop an improved number sense (the ability to rapidly and accurately approximate the magnitude of numbers). To support the validity of the online experiments, I first run a series of classroom-based controlled experiments showing that data logs from the game can measure attributes like ability, learning, challenge and motivation. Then, 1000s of different variations of Battleship Numberline are produced to produce variation in game challenge. The results show that minimal challenge almost always maximizes motivation; this finding conflicts with modern theories of motivation, which predict that moderate levels of challenge should be optimal. Further investigations show that “novelty” (task variability) is both correlated with challenge and highly predicts motivation. When novelty is increased and challenge held constant, motivation increases. When novelty is held constant and challenge increased, motivation decreases. A further study shows that moderate levels of temporal novelty (frequency of task variation) maximizes motivation. This evidence suggests that previous theoretical formulations of challenge are conflated with novelty; a suggestion is made that challenge be considered as the combination of novelty and difficulty. Further online experiments contribute to motivational design theory by testing user choice, goals, achievement rewards and other motivational design elements. “Close games” (where a player almost won or lost) produced motivational benefits while challenge did not. A choice of difficulty resulted in a tendency for players to chose moderately challenging games, which were played for longer than easier games. Yet, if players were randomly assigned the identical game levels, they tended to play easier levels longer than moderately difficult levels. This suggests that the motivational benefits of difficulty are dependent on choice and attributions of difficulty. Together, these experiments show that large-scale online experiments can productively test basic psychological theories of motivation. In the last third of the thesis, I investigate applied research methods for “design space optimization”. Using a meta-experimental design, I evaluated three different machine learning approaches for automatically optimizing a design space based on user data. The results showed that multi-armed bandit algorithms can rapidly identify high performing design configurations within a design space and minimize player exposure to “low-value” design conditions. This result has implications on the ethics of large-scale experiments in learning software, as scientific discovery has to be balanced with the goal of maximizing student success. The results also show that automatic optimization is risky; therefore, it is recommended that designers be kept “in the loop” to ensure qualitative insight. In Table 22, I summarize and review all 17 hypotheses tested in online experiments. I then discuss of the limitations of my research, including the challenge of integrating qualitative insight. I conclude with a discussion of my contributions, an evaluation of the opportunity for a testable taxonomy of motivational design and a review the opportunities for an interaction design science.
v
Table of Experiments and Availability of Data This dissertation is based on data collected from a sequence of nine randomized controlled experiments, all conducted with variations of the game Battleship Numberline. The experimental data (along with the game and other associated files) are hosted by the Pittsburgh Science of Learning Center’s Datashop (Stamper et al., 2010) at pslcdatashop.org. While it is still challenging to archive data for long-term citation, a search for “Battleship Numberline Data” and “Datashop” should be productive. You may also contact the author at
[email protected]. Experiment Name Experiment 1: UCLA-Kentucky Classroom Pre/Post Test Experiment 2: Propel Classroom Study Experiment 3: Online Experiment (3,5,10/ship,sub) Experiment 4: Online Super Experiment Experiment 5: Online Difficulty Choice Experiment 6: Online Novelty and Frequency of Change Experiment 7: Online Winning & Losing Close Games Experiment 8: Bandit Meta-Experiment Experiment 9: Second Bandit MetaExperiment
Location and Description p40 Classroom experiment to measure transfer of game learning to paper test and reliability/validity of game as assessment p42 Classroom experiment to measure student enjoyment and ability of game to measure ability, learning & challenge p53 Online experiment to measure relationship of challenge to motivation p55 Very large online experiment to measure relationship of challenge to motivation; also, learning curves compared p66 Online experiment to investigate how choice of challenge affects motivation p73 Online experiment to measure how different frequencies of change affects motivation p79 Online experiment to investigate how winning/losing & close games affects motivation p90 Online experiment to test whether multi-armed bandit algorithms can automatically optimize game design p94 Online experiment to test whether multi-armed bandit algorithms can automatically optimize game design
Table 1: Table of Randomized Controlled Experiments
vi
Contributions This dissertation demonstrates that online games can serve as basic and applied research instruments to measure the effects of game design elements on motivation, learning and other psychological constructs. This work contributes to psychology, educational research, and HCI. 1. Psychology contribution: a. This work contributes to psychological models of achievement motivation by showing how player motivation is influenced by the interaction of constructs like ability (initial performance), challenge (probability of success), novelty (variation in activity) and other design features like goals, choice and feedforward. b. This work indicates that the dominant theory that moderate challenge maximizes motivation (i.e., “inverted U-shaped curve” hypothesis) does not generalize and provides an alternative hypothesis based on the separate consideration of difficulty factors and novelty factors. c. This work advances a testable taxonomy of motivational design elements by reviewing and synthesizing existing motivation taxonomies. d. This work contributes evidence that the pace of scientific discovery can be accelerated by an increased rate of online experimentation 2. Educational Research contribution: a. This work contributes to instructional design research by demonstrating the systematic design of a game that can develop number sense b. This work contributes to educational research and psychometrics by validating that a game can provide a reliable and valid assessment of number sense and that it can measure ability, learning, challenge and motivation at scale. 3. HCI contribution: a. This work contributes to the interface optimization literature by demonstrating how large-scale design experiments, as a form of crowdsourcing, can be used to optimize the design of a user interface. b. This work contributes to the machine learning literature by providing empirical evidence (and a public data set) that illustrates how multi-armed bandit algorithms can automatically optimize a design space while minimizing user exposure to suboptimal design configurations. c. This work contributes to the design literature by formulating design as a design space optimization problem and by evaluating new tools that support the participation of designers in a data-driven optimization process. d. This work contributes a design pattern for designing online experiments that contribute to generalizable theories about the effect of interaction designs on behavior
vii
viii
Acknowledgements Thank you to Ken Koedinger and Jodi Forlizzi for seeing me through. You are such wonderful mentors and I loved our many, many conversations. Thank you to Jesse Schell for the inspiration, support and just-in-time contracts that kept my company afloat! Emma Brunskill, I really appreciated your genuine engagement with my work. Also, a special thanks to my first-year advisor Matt Kam for his kindness and support – thank you for bringing me into the program! Thank you to my family for making this possible. Julika, I love you and appreciate all that you have done. You are always an inspiration and motivation to me. Dad, Mom, thank you for all your ceaseless support and detailed scientific feedback on my thesis! Thank you to my grandparents, who encouraged me to pursue science and become a doctor. Gabi and Norbert, thank you for being so, so kind to me. Milo Laser and Mia Pixel, you are my inspiration – and my own personal human subjects. Thank you – I hope this was fun for you. Special thanks to David Klahr and Sharon Carver for the PIER program which provided such consistent funding and intellectual stimulation. David, thank you for holding me to your high standards and putting me on double-secret probation when I was losing track of my grounding. It worked. A very special thanks to Sharon for running the amazing CMU Children’s School. I was so lucky to go to school with my son! Also, thank you Sharon for being such an influential design instructor – so much of the success of my work can be attributed to your clear instruction and thoughtful feedback. Thank you so much to Kishan Patel – how different this all would have been! You are a true friend and trusted partner. Thank you to the rest of the Playpower.org and Playpower Games team: Ankit Patel, Nirmal Patel, Rana Chandradip, Sharan Shodhan, Nikhil Poonwala, Vivek Fitkariwala, Parth Rao and Nirav Faraswami. Jeremy Galante – I’m so glad we were able to work together. What fun! A big and major thank you to Brainpop for supporting this work: Allisyn, Karina, Norman, Scott, Din and Dr. Kadar. I am deeply appreciative. Thank you to the HCII – especially Queenie. You really made this possible from the beginning to the end. Anind and Jen, thank you for helping me when I was a new father in a new place. Sara Kiesler, thank you for your patience and good advice. Bob Kraut, thank you for all the conversations, new directions… and your exceptional work. Vincent, thanks for your great intro to the field. Carolyn, thank you for an inspiring class (and final grade). Thank you Scott Stevens for the Tom Malone thesis document. Thank you to my funders: the Grable Foundation, Joan Ganz Cooney Center at Sesame Workshop, PIER (Program for Interdisciplinary Education Science) and the Institute for Educational Science (IES), AIU, Russ Shilling at DARPA, Marvell Electronics, Sprout Fund, Spark Fund, NYC Department of Education iZone, MacArthur Foundation and HASTAC. If it wasn’t for that MacArthur Foundation Digital Media and Learning Grant – don’t know what would have happened. Special thanks to Gregg Behr for all that you did in Pittsburgh to make it such a perfect place for doing this work. Also, to Michael Levine-- wow, thank you for your support, advice and encouraging smile. You were such a major impetus for this work. Jeremy Resnick, thank you for being a true mensch… you and Janera (and both of your families!) were such wonderful friends to have to Pittsburgh. Ryan Spence, Carol Wooten, and Rosanne Javorski and most especially Justin Aglio -- thank you for always being so enthusiastic and helpful in supporting the challenging integration between classrooms and research!
ix
Thank you Tanner, for always being an inspiration for fun. Thank you Jeremy Douglass and Dan Rehn for all the creative game design talk over sushi—you helped me bring a soul into the science. Thank you SO much to Albert Lin for being the best remote friend I could have imagined. Jeremy Koempel and Emily – I’m going to miss you guys so much! Thank you, Turadg Aleahmad and Kristie Boyce – you guys really brightened my time in Pittsburgh. Thank you to Eliane and Jason for being such solid friends. Verity and Alan, thanks for all the fun and excitement. Erik, Chris, Sauvik, Dan, Becca – I was really happy to have you guys around. Don Burton, thank you for my time at Techstars – I learned a lot. Bror Saxberg, thank you for being an inspiring model for practical largescale experimentation. John Behrens, thanks for all the great conversations and advice. Ilya, so glad we are working together. Thank you Iris, for my Troast (and Julia, Kelly, Becca, Jason for your fun contributions). Chris Harrison, I loved working in the lab with you – you, Julia, Robert and Girard really inspired me. Eye lasers. Luis von Ahn and Severin Hacker, thanks for doing amazing work and inviting me to all the parties. Burr Settles, I loved our conversations and hope they continue. Bren, David K., Tyler – party on. Dixie Ching, thank you for being such a great friend and collaborator! Don Miller, thanks for being so totally and perpetually rad. Lennart Nacke, thanks for doing great research and for being an all-around good guy. Thank you to my UCLA collaborators: Belinda Thompson, Jim Stigler, Keith Holyoke – you really helped me so much! Thank you to Rony Patel for finding a major error in my analysis, just in time. Thank you to Karrie Godwin and Anna Fisher for teaching me so much about developmental psych assessments. Thank you to the Microsoft User Research Team for some last minute inspiration before my defense! Also, thank you to the Latin American School for Neuroscience in Education and the McDonnell Foundation for major inspiration in Urugay. Thank you to Max Ventilla at AltSchool, Steve Ritter at Carnegie Learning, Andrea Menotti at Houghton Mifflin Harcourt, the folks at Redbird Advanced Learning, Sea World and Fingerprint Digital – I really appreciated being able to produce work in the world while working on my PhD. Also, thank you to Ann Thai for helping us launch apps on the iTunes store! Thank you to my influential professors and instructors: Gabriel Richardson, Anna Cianciolo, Sun Joo Shin, Brian Scholl. Amy Arnsten, Frank Keil, Harvey Goldblatt, Lev Manovich, Natalie Jeremijenko, Sheldon Brown, Jim Hollan, Ed Hutchins, Brian Goldfarb, Sara Keisler, Bob Kraut, Haakon Faste, Eric Paulos, Vincent Aleven, Roberta Klatzsky, Carolyn Rosé, Sharon Carver, Dick Hayes – you were all amazing professors and opened up new doors for me. Thank you to Mike Cole and Jim Levin for introducing me to the Shark Game. Thank you to Amy Smith and the IDDS opportunity – thank you Rev. George for encouraging me to stick with this! Thank you to Brian Junker, Joel Greenhouse, Trent Gougler, Beau Dabbs (and the makers of JMP) for teaching me everything I know about stats and encouraging me despite my profound lack of formal training. Thank you to Chagrin Falls Public Schools, Camp Pasquaney, Yale University, UC San Diego and Carnegie Mellon. It was a great education. Deepest appreciation! Thank you to Don Norman, Scott Klemmer and Jim Hollan for giving feedback on this document. And a very special thanks to Taza D’Oro coffeeshop. Mmm…
x
Table of Contents Keywords ............................................................................................................................................ ii Abstract ............................................................................................................................................. iii Document Summary ........................................................................................................................ v Table of Experiments and Availability of Data ................................................................................. vi Contributions .....................................................................................................................................vii Acknowledgements .......................................................................................................................... ix Table of Contents .............................................................................................................................. xi List of Figures ................................................................................................................................... xv List of Tables ................................................................................................................................... xix Chapter 1: Introduction ..................................................................................................................... 3 Model for Accelerating Learning Science ........................................................................................ 3 Measuring the Effects of Designs on Learning Outcomes ........................................................... 4 Understanding Implications for an Interaction Design Science ....................................................... 4 Model for Accelerating Interaction Design Science ......................................................................... 6 Design Space Optimization Methods .............................................................................................. 7 Basic Theories of Motivation and Learning ..................................................................................... 8 The Law of Effect ......................................................................................................................... 8 The Cognitivist View of Motivation ............................................................................................... 9 The Classical View of Motivation ................................................................................................ 11 Contrasts Between the Classical and Cognitivist View .............................................................. 11 Unifying Processes Between the Classical and Cognitivist View ............................................... 12 Integrative Theory of Value for Design and Learning ................................................................. 13 Applied Theories of Motivation and Learning ................................................................................ 14 Games for Learning: Linking Psychology of Motivation to Design ............................................. 14 Integrity as a Design Factor for Learning Game Efficacy ........................................................... 16 Taxonomy of Intrinsic Motivations for Learning ......................................................................... 17 ARCS Model of Instructional Motivation .................................................................................... 20 Taxonomy of Motivational Sources ............................................................................................ 21 Motivational Affordances ........................................................................................................... 22 Gamification ............................................................................................................................... 22 Other Motivational Taxonomies and Classifications ................................................................... 24 Towards an Empirical Taxonomy of Motivational Design Elements .............................................. 25 On the Measurement of Motivation ............................................................................................ 25 Chapter 2: The Design of Battleship Numberline ......................................................................... 27 Backwards Design Pattern ............................................................................................................ 27 Identify Instructional Goals ......................................................................................................... 28 Identify Assessments That Indicate Goals Have Been Reached ................................................ 29 Design Instruction that can “Move the Needle” on Assessments .......................................... 29 The Design of Battleship Numberline ............................................................................................ 33 The Design Space of Battleship Numberline ............................................................................. 35 Exploring the Design Space of Battleship Numberline .............................................................. 38 Chapter 3: Investigating the Effects of Battleship Numberline on Classroom Learning and Motivation ......................................................................................................................................... 39 Testing Design Hypotheses ....................................................................................................... 39 Testing Theoretical Hypotheses ................................................................................................. 39
xi
Experiment 1: UCLA-Kentucky Classroom Pre/Post Test ............................................................ 40 Experimental Design .................................................................................................................. 40 Learning Results ........................................................................................................................ 40 Assessment Validation Results .................................................................................................. 41 Experiment 2: Propel Classroom Study......................................................................................... 42 Experimental Design .................................................................................................................. 42 Learning Measurement Results ................................................................................................. 42 Challenge Measurement Results ............................................................................................... 44 Enjoyment Results ..................................................................................................................... 45 Relationship between Ability and Enjoyment ............................................................................. 46 Discussion of Outcomes ............................................................................................................ 46 Chapter 4: Using Online Experiments to Test Psychological Theory ......................................... 47 Opportunity for Online Research ................................................................................................... 47 Theoretical Constructs in Online Research ................................................................................... 47 Relationship between Challenge and Motivation .......................................................................... 49 Prior Work .................................................................................................................................. 49 Empirically Testing the Effects of Challenge on Learning and Motivation .................................... 51 Design Factors and Challenge ................................................................................................... 51 Experiment 3: Online Experiment (3,5,10/ship,sub) ...................................................................... 53 Experimental Design .................................................................................................................. 53 Measuring Challenge and Engagement ..................................................................................... 53 Participants ................................................................................................................................ 53 Results ........................................................................................................................................... 54 Discussion .................................................................................................................................. 55 Experiment 4: Online Super Experiment........................................................................................ 55 Experimental Design .................................................................................................................. 56 In-Game Pretest ......................................................................................................................... 56 Participants ................................................................................................................................ 57 Results ........................................................................................................................................... 57 Challenge as an Underlying Design Factor ................................................................................ 58 Analysis of Learning Curves ....................................................................................................... 60 Discussion of Learning Curve Analysis ...................................................................................... 61 Discussion .................................................................................................................................. 62 Theoretical Hypotheses and Design Implications ...................................................................... 63 Conclusion ..................................................................................................................................... 64 Chapter 5: Attribution Motivation and Choice of Challenge ........................................................ 65 Experiment 5: Online Difficulty Choice .......................................................................................... 66 Experimental Design .................................................................................................................. 66 Operational Measures ................................................................................................................ 66 Participants ................................................................................................................................ 67 Hypotheses ................................................................................................................................ 67 Experimental Results ................................................................................................................. 67 Discussion: Difficulty Choice Study ........................................................................................... 72 Experiment 6: Online Novelty and Frequency of Change .............................................................. 73 Procedure .................................................................................................................................. 74 Results: Emergence of the Inverted U-Shape ............................................................................... 74 Discussion: Novelty Study ......................................................................................................... 75 Theoretical Implications ............................................................................................................. 76 Chapter 6: Goals and the Inverted U Hypothesis.......................................................................... 77 Experiment 7: Online Winning and Losing Close Games .............................................................. 79 Ideal Experimental Design ......................................................................................................... 79 Manipulating Goal Difficulty ....................................................................................................... 79
xii
Description of Experimental Design Factors .............................................................................. 82 Experimental Design .................................................................................................................. 83 Results ........................................................................................................................................... 83 Analysis of Close Games and Winning and Losing .................................................................... 85 Discussion .................................................................................................................................. 86 Chapter 7: Automating Data-Driven Design and Continuous Improvement with Multi-Armed Bandits .............................................................................................................................................. 87 Multi-Armed Bandits and Design ............................................................................................... 87 Review of Common Multi-Armed Bandit Algorithms ................................................................. 88 Design Space Optimization ........................................................................................................ 89 System for Bandit-Based Optimization ...................................................................................... 90 Experiment 8: Bandit Meta-Experiment ......................................................................................... 90 Procedure................................................................................................................................... 91 Experimental System.................................................................................................................. 91 Measures .................................................................................................................................... 92 Results ........................................................................................................................................... 92 Analysis of Outcomes ................................................................................................................ 93 Experiment 9: Second Bandit Meta-Experiment ........................................................................... 94 Results ........................................................................................................................................... 95 Discussion ...................................................................................................................................... 97 Limitations .................................................................................................................................. 98 Design Lessons .......................................................................................................................... 99 Contributions .............................................................................................................................. 99 Chapter 8: Future Work and Conclusion ..................................................................................... 100 Contributions ............................................................................................................................... 100 Summary of Experimental Findings ............................................................................................. 100 Limitation: Questionable Online Measures ............................................................................... 105 Limitation: No Theory of Disengagement ................................................................................. 106 Limitation: Authentic Continuous Improvement ....................................................................... 107 Limitation: Focus on Meaning and Feeling .............................................................................. 107 Implications for an Interaction Design Science............................................................................ 107 Design Pattern for Online Theory Testing .................................................................................... 108 Implications for Motivational Design Taxonomies ....................................................................... 108 Limitation: Questionable Online Measures ............................................................................... 111 Limitation: No Theory of Disengagement ................................................................................. 113 Limitation: Authentic Continuous Improvement ....................................................................... 113 Limitation: Focus on Meaning and Feeling .............................................................................. 114 Opportunities and Barriers for Future Research .......................................................................... 114 References...................................................................................................................................... 116 Testable Taxonomy of Motivational Design Elements ................................................................. 125 Keller’s Completed ARCS Model of Motivation .................................................................... 130 Exploratory Data Analysis of Super Experiment .......................................................................... 132 Preparing Levels of Difficulty ....................................................................................................... 133 Math Planet Screen Shots ........................................................................................................... 135 Field-Based Experience in Educational Software ........................................................................ 138
xiii
xiv
List of Figures Figure 1: A positive feedback loop between theories of learning, instructional software design, large-scale use of learning products and large-scale basic/applied research in learning science. ....................................................................................................................................... 3 Figure 2: A positive feedback loop between theory, design, products at scale and large-scale applied and basic research. Online controlled experiments can optimize outcomes and support scientific research on generalizable theories of the effects of designs .......................... 7 Figure 3: Thorndike’s Learning Curve (left) of a cat escaping a puzzle box (right), where the y-axis shows duration in seconds and the x-axis shows the number of trials. For a review on learning curves, see Ritter and Schooler (2001) ....................................................................................... 9 Figure 4: The flow channel, described by Cowley ............................................................................ 15 Figure 5: Jumping times in 3rd person action games (Cowley, 2009 adapted from Cousins, 2005) 15 Figure 6: Malone’s studies of motivation involved creating multiple versions of a game by removing motivational game elements and measuring the difference in time that students would voluntarily engage, over the different designs .......................................................................... 17 Figure 7: From Deci & Ryan (2000), a spectrum describing the internalization of behaviors ........... 21 Figure 8: Different representation of numbers in Number Worlds (Griffin and Case, 1997) ............. 30 Figure 9: The image on the left shows the typing level, where a player will enter their estimate into the box in response to the position of the ship on the number line. The image on the right shows the clicking level after a player clicked to estimate the location of the presented fraction, 7/8. Prior to their click, the submarine was hidden. .................................................... 34 Figure 10: The left image shows the game’s original ship, at 10% of the number line (5% error tolerance). The ship on the right is 40% of the number line ..................................................... 35 Figure 11: Shows the change in student accuracy over the 30 different levels played. The jagged lines of condition 2 show how students went back and forth between whole number and fraction levels. Accuracy reduces over time due to the increased challenge of the game levels. ................................................................................................................................................... 41 Figure 12: The regression estimates predicting post-test score reveal that the number of trials played by the player and their game performance both significantly predict post-test scores, even after accounting for pretest, grade and condition. This provides additional evidence that the game caused the significant improvements in student test scores. ................................... 41 Figure 13: Shows the improvements in student performance from their first block of items to the second, over 4th, 5th and 6th grade. ............................................................................................ 43 Figure 14: On the left, the learning curve shows the decrease in Percent Absolute Error over opportunities playing clicking and typing (where clicking is the dotted line). On the right, the learning curve excludes the first 10 items to show that learning continues beyond a “burn-in” period. ....................................................................................................................................... 43 Figure 15: The change in “clicking” performance on individual fractions over the first and second opportunity. ............................................................................................................................... 44 Figure 16: The change in “typing” performance on individual fractions over the first and second opportunity. ............................................................................................................................... 44 Figure 17: Measures of Challenge .................................................................................................... 45
xv
Figure 18: The graph on the left shows the distribution of scores on the Intrinsic Motivation Inventory, which shows that 84% of students perceived the game neutrally or higher. On the right, the graph shows the interaction of grade and gender on intrinsic motivation. ............... 46 Figure 19: On the left, improved average success rates tended to increase intrinsic motivation for boys, but not for girls. On the right, the thick line shows the significant interaction between the quadratic term and gender, showing that the girls had an inverted U-shaped effect in response to performance, while boys did not. When grade was added as a factor, the interaction between gender and the quadratic term (i.e., shape of the curve) was still significant (p
