Measuring Flow in Educational Games and Gamified ...

Measuring Flow in Educational Games and Gamified Learning Environments David J. Shernoff Center for Math, Science, and Computer Education (CMSCE) Rutgers University, Piscataway, NJ, USA [email protected] Juho Hamari Game Research Lab, School on Information Sciences University of Tampere, Tampere, Finland [email protected] Elizabeth Rowe Educational Gaming Environments (EdGE) group @ TERC Cambridge, MA, USA [email protected]

Abstract: This paper describes how the measures of engagement used in four Finnish-US Network (FUN) studies were developed. Primary instruments utilized across the studies included a short survey of engagement that participants completed intermittently while playing educational video games based in the tradition of the Experience Sampling Method, as well as a longer psychometric survey about their experience upon completion of the game. The short form measures student engagement conceptualized as the simultaneous occurrence of high concentration, interest, and enjoyment based on flow theory and research relating each of these components to deep engagement in learning. The goal of the long form of the survey is to gauge the overall relationships between the components of flow and learning as well as to investigate which factorial structures of flow form the best fitting model. Both surveys enable investigation of the potentially moderating effects of individual factors.

Introduction Public schools are continually characterized by pervasive boredom (Steinberg, Brown, & Dornbusch 1996). Of concern to teachers for decades (Pickens 2007), boredom and apathy in class are primary reasons that many students do not become engaged in school learning (Pekrun, Goetz, Daniels, Stupnisky, & Perry 2010). Pervasive student disengagement is both a national and an international problem, with 20 to 25% of students in 28 OECD countries (i.e., those belonging to the Organisation for Economic Co-operation) classified as having low participation and/or a low sense of belonging (Willms 2003).

Increasing Engagement in Learning through Serious Educational Video Games A promising strategy for increasing engagement in meaningful forms of learning is via educational video games (Gee 2007; Steinkuehler et al. 2012) and gamification (Huotari & Hamari 2012; Hamari et al. 2014a; Hamari et al. 2014b), as observed by educational scholars for several decades now. What they have found is that the most successful games “teach” their players how to solve complex problems. The problems within a game typically start off rather easy and then progressively get more difficult as players’ skills develop. Players are motivated to learn within video games because it is clear that knowledge is powerful. The learning is situated, and occurs through a process of hypothesizing, probing, and reflecting upon the simulated world within the game. The goals are clear. Games provide players immediate and unambiguous feedback on how well they are progressing. Information becomes available to players at just the time they will be able to make

Preview version of this paper. Content and pagination may change prior to final publication.

EdMedia 2014 - Tampere, Finland, June 23-26, 2014 sense of it and use it. Within the highly engaging techniques that game designers employ to get players to “learn” the game, one finds the enactment of modern principles and theories of learning such as constructionism, inquiry-based learning, and anchored instruction. Much of the emerging scholarship on video game design is explicitly grounded in scholarship on cognition, including concepts such as Vygotsky’s zone of proximal development (e.g., Brathwaite & Schreiber 2009). Theories of what make video games fun focus on learning and problem solving (Koster 2005). According to Koster, a game becomes fun when it requires players to gain new skills at a deep level that get “chunked” and absorbed into the subconscious mind, and then requires players to apply the skills/knowledge toward some goal. Furthermore, it remains fun if it requires players to gain new skills/knowledge, or transfer their skills to new problems within the game. Ideally, this is the type of “fun” we that our Finnish-US Network (FUN) would like students to have in order to engage in deep learning. Serious games are distinct from other games in that they are designed for primary purposes other than entertainment or leisure (Davidson, 2008). They can and often are entertaining or enjoyable, but players are also engaged in the “serious” task of solving a purposeful problem, often one important to solve in real life. Educational games are developed for the primary purpose of educating or training. As a context for learning, serious games can be contrasted with traditional classroom activities that are characterized by high challenge and concentration but also low enjoyment and interest, and unstructured leisure activities that tend to be high in enjoyment but low in challenge and concentration (Larson 2000). Experiences that combine both concentration in challenging activities and enjoyment in interesting ones integrate or fuse common perceptions of play and work, and can be thought of as “serious play,” or “playful work” (Csikszentmihalyi & Schneider 2000). This conceptualization of work-play integration combines the focused, disciplined aspects of work with enjoyable aspects of leisure, a combination considered to promote learning and positive development (Larson 2000).

Theoretical Foundation: Flow Experiences and Their Relationship to Learning The integration of work and play also characterizes the psychological state which Csikszentmihalyi (1990) has called “flow.” Flow refers to a state of mind characterized by focused concentration and elevated enjoyment during intrinsically interesting activities (Shernoff, Csikszentmihalyi, Schneider, & Shernoff 2003). It represents a state of complete absorption in a challenging activity with no psychic energy left for distractions. For example, composers have described a shift in consciousness when music is “flowing” from the depth of their souls, stirred by inspiration, like being part of a river (Custodero 2005). Empirically, flow has been related to learning, talent-development, academic achievement, and creative accomplishment in a profession (Csikszentmihalyi 1996; Csikszentmihalyi, Rathunde, & Whalen, 1993). The simultaneous occurrence of elevated concentration, interest, and enjoyment encapsulates the experience of flow; and all three phenomena are related to learning (Shernoff 2013). Concentration or absorption, which is central to flow (Csikszentmihalyi 1990), is related to meaningful learning (Montessori, 1967), including depth of cognitive processing and academic performance (Corno & Mandinach, 1983). Interest directs attention, reflects intrinsic motivation, stimulates the desire to continue engagement in an activity, and is related to school achievement (Schiefele, Krapp, & Winteler 1992). Enjoyment is a positive feeling related to the demonstration of competencies, creative accomplishment, and school performance (Csikszentmihalyi, Rathunde, & Whalen 1993). In this conceptualization, engagement in learning is highest when all three components are simultaneously stimulated. Flow experiences are also characterized by a sense of complete immersion, which is also related to learning and related emotions. For example, recent experiments in neuroscience have demonstrated that when a reader is fully engrossed in a novel, the human brain is activated not only in areas responsible for attention; it also dramatically “lights up” in areas controlling affect and emotion (Thomson & Vedansom 2012). Flow theory has been a primary theoretical base for exploring the implications of learning through immersion or “being enveloped” by a virtual learning environment since the emotional composition of these experiences resemble flow and precipitate a deeper engagement with learning. Research has explicitly related the sense of “presence,” “being there,” “immersion,” or “flow” in different virtual reality interfaces with positive learning outcomes (e.g., Abrantes & Gouveia 2012; Procci, Singer, Levy, & Bowers 2012). The subjective experience of flow, according to Csikszentmihalyi’s (1990) theory, is enhanced by certain experiential conditions or properties of the task. Specifically, in most flow activities, the goals are clear, feedback with respect to meeting those goals is immediate and forthcoming, and the activity is regarded as


EdMedia 2014 - Tampere, Finland, June 23-26, 2014 meaningful, important, or autotelic (i.e., a goal in-and-of-itself, performed for the sheer experience of it). Perhaps the most central condition for flow experiences to occur is that the individual uses a high level of skill to meet a significant challenge. The activity is therefore not too easy for one’s skills, nor is it impossibly difficult. Reaching the goal is doable: one has a reasonable chance of success with sincere and concerted effort. Typically, the challenge and skill are high and in balance—individuals stretch their skills to their limits in pursuit of a challenging goal. The various combinations of high or low challenges and skills predict distinct psychological states: (a) apathy, resulting from low challenge and low skill; (b) relaxation, resulting from high skill but low challenge; (c) anxiety, resulting from high challenge but low skill; and (d) flow, resulting from high challenge combined with high skill. This model later evolved into one with eight flow channels including four intermediary or transitional states between these four quadrants (see Strati, Shernoff, & Kacker 2012). This challenge-skill dynamic introduces a growth principle that is also inherently related to learning. When learning a new skill, the challenge of even a basic task may exceed a student’s beginning level of ability, and hence one may feel overwhelmed. To reach flow, the level of skill must increase to match the challenge. Much like Vygotsky’s (1978) zone of proximal development, the level in which most learning occurs is just one step beyond the skills one has already mastered. Sufficient practice may be needed until the skill is mastered. Once it is, only one thing can restart a cycle of fresh learning: a higher level of challenge, causing one’s skill to increase yet again. Thus, the individual may progress through increasingly difficult challenges at ever-higher levels of skill. Because most video games allow the player to adjust the level of challenge as skills are increasing, the continuing cycle of new challenges resulting in the building of skills increases motivation, enhances competence, fosters growth, and extends the players’ capacities (Fullagar et al. 2012).

Measuring Flow and Engagement in Learning The most widely used methods for measuring flow has been the Experience Sampling Method (ESM; See Hektner, Schmidt, and Csikszentmihalyi 2007) and psychometric surveys (Nunally 1978) combined with Structural Equation Modelling (SEM; Nunally 1978; Hair et al., 2010). The ESM is an effective tool to measure flow because it repeatedly captures an individual’s level of engagement and affect in activity, presumed to be high during flow, as well as the conditions that are theorized to give rise to flow experiences such as high challenges and skills. The reliability and validity of ESM has been supported empirically; the reader is referred to Hektner, Schmidt, and Csikszentmihalyi (2007) for this information as well as a thorough review of the ESM. The primary instruments utilized across the FUN studies included a short survey of engagement that participants completed intermittently during the game play based in the tradition of the Experience Sampling Method (Hektner, Schmidt, & Csikszentmihalyi 2007), as well as a longer psychometric survey (Nunnally 1978) about the experience upon completion. The short survey provides a momentary account of the context and content of an individual’s experiences as they vary from one moment to the next whereas the longer survey combined with SEM is employed to gauge the overall experience. The short survey functions as an Experience Sampling Form, taking repeated snapshots of the participants’ engagement while playing educational video games. Because such video games are normally played on computers, we are embedding the survey into the game interface in order to ask the intermittent surveys. The FUN Network has adapted the these methods in order to study player’s levels of flow and learning while playing educational video games. Both the short and long survey employed in the studies are psychometric in nature. Psychometric measurement is required when measuring subjective variables, such as attitudes, enjoyment and motivations in order to increase the validity and reliability of measurement (e.g. Nunnally 1978). Both surveys enable both investigating the theoretical factorial structure of the phenomenon as well as predictive models which investigate how flow can increase learning outcomes. SEM enables multivariate analysis and, thus, permits modelling and testing of complex phenomena. This set of methodologies employs psychometric measurement instruments for latent variables and econometric measurement for the relationship of the latent variables. The short survey asks three sets of three questions (with the third set optional) at points in the game determined by the game designers. The short survey is as follows: Short Survey for Measuring the Momentary Account of the Individual’s Experiences As you were playing the game just now (or comparable phrasing by project/game):


EdMedia 2014 - Tampere, Finland, June 23-26, 2014 (Participants choose from 5 responses: “Not at all”, “A little”, “Somewhat”, “Pretty much”, “Very much”) SET 1 1. 2. 3.

Interest: How interesting was it? Enjoyment: How much did you enjoy what you were doing? Concentration: How hard were you concentrating?

SET 2 1. 2. 3.

Immersion: How immersed were you in the game Challenge: Was it challenging? Skills: How skilled were you at the game?

OPTIONAL SET 3 1. Importance: How important was this activity? 2. Did it feel more like: a) work b) play c) both d) neither 3. Learning: Did you feel you were learning? Our measure of engagement is the composite of interest, enjoyment, and concentration, the three questions asked in the first set, which has been used in a variety of ESM and SEM studies of engagement in learning in keeping with the conceptualization of engagement in learning described above (See Shernoff 2013). As also reviewed above, immersion is a central aspect of flow experiences also related to learning; and challenge and skill are the key theoretical conditions believed to precipitate flow. Importance or meaningfulness, and work/play balance asked in the optional third set are also theorized to be key conditions of flow. The third set also includes a subjective measure of learning to supplement objective gaming performance or log file data.

Long Survey for Measuring the Overall Account of the Individual’s Experiences In the long survey, shown below, respondents were given the prompt: “Think back over your entire experience with the GAME. Please answer the following questions.” Each project decided how the items would be ordered. Two scales were used for these items. Scale 1 asked respondents to choose from: ‘Not at all’, ‘A little’, ‘Somewhat’, ‘Pretty Much’, and ‘Very Much’. Scale 2 was a 5 point scale from ‘1=Strongly Disagree’ to ‘5=Strongly Agree’ with a midpoint of ‘3=Neither disagree/agree.’ Because the long survey will be used across different types of games and gamified learning environments, the survey items were written generically. “GAME” was substituted for the specific game or gamified learning environment. Similarly, the specific content included in the game or gamified learning environment was substituted for “LEARNING GOAL.” Both the ESM and SEM studies enable the consideration of demographic factors as moderator and/or predictor variables in the research model (demographic survey items not disclosed here). Previous research indicates that there are demographic differences in perceived benefits from gamification (Koivisto & Hamari 2014), and that general attitudes and orientations towards gaming can also play a role (Hamari & Tuunanen 2014) in getting engaged in games. Table 1: The survey items. Construct

Items

Interest

How interesting was the GAME? Did you feel bored while GAME? Do you wish you were doing something else? How much did you enjoy what you were doing? Interacting with it was entertaining. Interacting with it fun. How hard were you concentrating?

Enjoyment

Concentration

Scale 1 1 1 1 2 2 1

Reverse Scale? No Yes Yes No No No No


EdMedia 2014 - Tampere, Finland, June 23-26, 2014 It provided content that focused my attention. 2 No I was easily distracted while GAME. 2 Yes Challenge Was it challenging? 1 No Was it easy? 1 Yes Using it stretch my capabilities to the limits. 2 No Skill How skilled were you at the GAME? 1 No I was very skilled at GAME. 2 No I was not very good at GAME. 2 Yes Immersion How immersed were you in the GAME? 1 No I lost track of time while using it. 2 No I became very involved in the GAME forgetting about other 2 No things. Learning Did you feel you were learning? 1 No I gained a good understanding of the things GAME tried to teach. 2 No Using TASK increased my understanding of LEARNING GOAL. 2 No This TASK helped me learn 2 No Sources for items: Alavi, Marakas, & Yoo, 2002; Coller, Shernoff, & Strati, 2011; Fu, Su, & Yu, 2009; Koufaris, 2002; Van der Heijden, 2004.

Conclusions In conclusion, flow theory and both ESM and SEM methodologies provide the theoretical underpinnings and methodological framework for the Finnish-US Network (FUN) collaborative studies of engagement and learning in educational video games. This international collaboration is unique with respect to using these common conceptual frames, methods, measures across multiple contexts that span kindergarten through undergraduate education, as well as science, engineering, math, English, and foreign language. Acknowledgements This research was carried out as part of research projects (40311/12 - FUN, 40134/13 – F2P) funded by the Finnish Funding Agency for Technology and Innovation (TEKES), the U.S. National Science Foundation (EHR, NSF#1252709) as well as the Finnish Cultural Foundation. We would also like to acknowledge our colleagues within the larger project framework: Jodi Asbell-Clarke, Tapio Auvinen, Brianno Coller, Teon Edwards, Lasse Hakulinen, Petri Ihantola, Ari Korhonen, Jamie Larsen, Frans Mäyrä, Tuula Nousiainen, Jussi Okkonen, Roope Raisamo, Bill Shribman, Elisabeth Sylvan, Mikael UusiMäkelä, Mikko Vesisenaho, and Jarmo Viteli..

References Abrantes, S., & Gouveia, L. (2012). Using games for primary school: Assessing its use with flow experience. In M. M. Cruz-Cunha (Ed.), Handbook of research on serious games as educational, business and research tools (pp. 769–781). Hershey, PA: Information Science Reference. Alavi, M., Marakas, G.M., and Yoo, Y. (2002). A Comparative Study of Distributed Learning Environments on Learning Outcomes, Information Systems Research, 13, 404-415. Brathwaite, B., & Schreiber, I. (2009). Challenges for game designers: Non-digital exercises for video game designers. Boston, MA: Course Technology. Corno, L., & Mandinach, E. B. (1983). The role of cognitive engagement in classroom learning and motivation. Educational Psychologist, 18, 88–108. doi:10.1080/00461528309529266 Csikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. New York: Harper Perennial. Csikszentmihalyi, M. (1996). Creativity: Flow and the psychology of discovery and invention. New York: HarperCollins. Csikszentmihalyi, M., Rathunde, K., & Whalen, S. (1993). Talented teenagers: The roots of success and failure. New York: Cambridge University Press. Csikszentmihalyi, M., & Schneider, B. (2000). Becoming adult: How teenagers prepare for the world of work. New York: Basic Books.


EdMedia 2014 - Tampere, Finland, June 23-26, 2014 Coller, B. D., Shernoff, D. J., & Strati, A. D. (2011). Measuring engagement as students learn dynamic systems & control with a video game. Advances in Engineering Education, 2, 1-32. Custodero, L. A. (2005). Observable indicators of flow experience: A developmental perspective on musical engagement in young children from infancy to school age. Music Education Research, 7, 185–209. doi:10.1080/14613800500169431 Davidson, D. (2008). Beyond fun: Serious games and media. Pittsburgh, PA: ETC Press. Fu, F.L., Su, R.-C., & Yu, S.-C. (2009). EGameFlow: A scale to measure learners’ enjoyment of e-learning games. Computers & Education, 52 (1),101–112. doi: 10.1016/j.compedu.2008.07.004 Fullagar, C. J., Knight, P. A., & Sovern, H. S. (2013). Challenge/skill balance, flow, and performance anxiety. Applied Psychology: An International Review, 62, 236–259. doi:10.1111/j.1464–0597.2012.00494 Gee, J. P. (2007). What video games have to teach us about learning and literacy. New York: Palgrave MacMillan. Hair, J., Black, W., Babin, B., and Anderson, R. (2010). Multivariate data analysis (7th ed.): Prentice-Hall, Inc. Upper Saddle River, NJ, USA. Hamari, J., Huotari, K., & Tolvanen, J. (2014a). Gamification from the economics perspective. In S. P. Walz & S. Deterding (Eds.), The Gameful World: Approaches, Issues, Applications. Cambridge, MA: MIT Press. Hamari, J., Koivisto, J., & Sarsa, H. (2014b). Does Gamification Work? – A Literature Review of Empirical Studies on Gamification. In proceedings of the 47th Hawaii International Conference on System Sciences, Hawaii, USA, January 6-9, 2014. Hamari, J., & Tuunanen, J. (2014). Player Types: A Meta-synthesis. Transactions of the Digital Games Research Association, 1 (2), 29-53 Hektner, J. M., Schmidt, J. A., & Csikszentmihalyi, M. (2007). Experience sampling method: Measuring the quality of everyday life. Thousand Oaks, CA: Sage. Huotari K., & Hamari, J. (2012). Defining Gamification - A Service Marketing Perspective. In Proceedings of The 16th International Academic Mindtrek Conference, Tampere, Finland, October 3-5, 2012. Koivisto, J., & Hamari, J. (2014). Demographic differences in perceived benefits from gamification. Computers in Human Behavior, 35, 179-188. Koster, R. (2005). A theory of fun for game design. Scottsdale, AZ: Paraglyph Press. Koufaris, M. (2002). Applying the technology acceptance model and flow theory to online consumer behavior. Information systems research, 13 (2), 205-223. Larson, R. W. (2000). Toward a psychology of positive youth development. American Psychologist, 55, 170-183. Montessori, M. (1967). The absorbent mind (1st ed.). New York: Holt, Rinehart, and Winston. Nunnally, J. C. (1978). Psychometric theory. New York: McGraw-Hill Pekrun, R., Goetz, T., Daniels, L. M., Stupnisky, R. H., & Perry, R. P. (2010). Boredom in achievement settings: Exploring control-value antecedents and performance outcomes of a neglected emotion. Journal of Educational Psychology, 102, 531–549. doi:10.1037/a0019243 Pickens, M. T. (2007). Teacher and students perspectives on motivation within the high school science classroom., Unpublished doctoral dissertation, Auburn University, Auburn, Alabama. Procci, K., Singer, A. R., Levy, K. R., & Bowers, C. (2012). Measuring the flow experience of gamers: An evaluation of the DFS-2. Computers in Human Behavior, 28, 2306-2312. doi: 10.1016/j.chb.2012.06.039 Schiefele, U., Krapp, A., & Winteler, A. (1992). Interest as a predictor of academic achievement: A meta-analysis of research. In K. A. Renninger, S. Hidi, & A. Krapp (Eds.), The role of interest in learning and development (pp. 183– 212). Hillsdale, NJ: Erlbaum. Schmidt, J. A., Shernoff, D. J., & Csikszentmihalyi, M. (2007). Individual and situational factors related to the experience of flow in adolescence: A multilevel approach. In A. D. Ong & M. v. Dulmen (Eds.), The handbook of methods in positive psychology (pp. 542-558). Oxford: Oxford University Press. Shernoff, D. J. (2013). Optimal learning environments to promote student engagement. New York: Springer. Shernoff, D. J., Csikszentmihalyi, M., Schneider, B., & Shernoff, E. S. (2003). Student engagement in high school classrooms from the perspective of flow theory. School Psychology Quarterly, 18, 158–176. doi:10.1521/scpq.18.2.158.21860 Steinberg, L., Brown, B. B., & Dornbusch, S. M. (1996). Beyond the classroom: Why school reform has failed and what parents need to do. New York: Simon & Schuster. Steinkuehler, C., Squire, K., & Barab, S. (Eds.). (2012). Games, learning, and society: Learning and meaning in the digital age. Cambridge: Cambridge University Press. Strati, A. D., Shernoff, D. J., & Kackar, H. Z. (2012). Flow. In R. Levesque (Ed.), Encyclopedia of adolescence (pp. 1050– 1059). New York: Springer. Thompson, H., & Vedantam, S. (2012). A lively mind: your brain on Jane Austen. NPR: National Public Radio Website. Retrieved from http://www.npr.org/blogs/health/2012/10/09/162401053/a-lively-mind-your-brain-on-jane-austen. van der Heijden, H. (2004). User Acceptance of Hedonic Information Systems. MIS Quarterly, 28(4), 695–704. Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Cambridge: Harvard University Press. Willms, J. D. (2003). Student engagement at school: A sense of belonging and participation. Results from PISA 2000. Paris: OECD.
