Computer games application within alternative ... - Springer Link

Download PDF
3 downloads 8175 Views 203KB Size Report
Comment
Jan 25, 2008 - Empirical studies on the use of computer games are limited, especially. F. Ke (&). Organizational Learning and Instructional Technology ...
Education Tech Research Dev (2008) 56:539–556 DOI 10.1007/s11423-008-9086-5 DEVELOPMENT ARTICLE

Computer games application within alternative classroom goal structures: cognitive, metacognitive, and affective evaluation Fengfeng Ke

Published online: 25 January 2008  Association for Educational Communications and Technology 2008

Abstract This article reports findings on a study of educational computer games used within various classroom situations. Employing an across-stage, mixed method model, the study examined whether educational computer games, in comparison to traditional paperand-pencil drills, would be more effective in facilitating comprehensive math learning outcomes, and whether alternative classroom goal structures would enhance or reduce the effects of computer games. The findings indicated that computer games, compared with paper-and-pencil drills, were significantly more effective in promoting learning motivation but not significantly different in facilitating cognitive math test performance and metacognitive awareness. Additionally, this study established that alternative classroom goal structures mediated the effects of computer games on mathematical learning outcomes. Cooperative goal structure, as opposed to competitive and individualistic structures, significantly enhanced the effects of computer games on attitudes toward math learning. Keywords

Instructional gaming  Media in education  Classroom goal structures

For more than two decades, educationalists have discussed the potential that exists for the application of computer games to learning (e.g., Gee 2003; Gredler 1996; Malone 1981; Prensky 2001; Rieber 1996). However, in spite of a growing body of literature highlighting the educational potential of computer games, this enthusiasm is tempered by the recognition that the empirical evidence does not clearly establish improved learning outcomes for game-based activities. Major reviews on instructional computer games (Dempsey et al. 1996; Fletcher and Tobias 2006; Mitchell and Savill-Smith 2000; Randel et al. 1992; Vogel et al. 2006) indicated that the empirical findings on the learning effectiveness of computer games are often conflicting. Many game studies are either anecdotal or hypothetical. Empirical studies on the use of computer games are limited, especially F. Ke (&) Organizational Learning and Instructional Technology Program, College of Education, University of New Mexico, 388 Hokona Hall, MSC05 3040, Albuquerque, NM 87131-1231, USA e-mail: [email protected]

123

540

F. Ke

well-specified and implemented in-situ research that examines what happens in real contexts over time (Fletcher and Tobias 2006). Another common skepticism expressed about using computer games for learning lies in the lack of empirically-grounded methods of integrating computer games into classroom and lesson time. The majority of games and learning research has been concerned with learning conceptually that does not demand knowledge of subject area, hence lack connection to curriculum in school (Egenfeldt-Nielsen 2005, Unpublished doctoral dissertation). The research question remains to be studied empirically: When and how will computer games facilitate purposeful learning in school? Miller et al. (1999) and Papert (1998) suggested that the investigation into computer games for learning should focus on how games can be carefully aligned with sound pedagogical strategies to be beneficial. Kaptelinin and Cole (2002) further argued that learning outcomes achieved through educational games depend largely on the external instructional activities context that structures the way students use and interact with computer games. However, research on game-based pedagogy is especially sparse. Research on the use of computer games within different classroom goal structures— cooperative, competitive, or individualistic goal structure (Johnson and Johnson 1996) is needed and would be valuable to educationalists.

Classroom goal structures and computer games application Classroom goal structure is the specification of ‘‘the ways in which students will interact with each other and the teacher to achieve the goal’’ (Johnson et al. 1985, p. 669). In an individualistic goal structure a learner works individually to ensure his/her own learning meets a preset criterion independently from the efforts of other students. Hence learners are not concerned about what anyone else is accomplishing. A competitive goal structure focuses a learner’s effort on performing faster and more accurately than classmates. Hence learners perceive they will be rewarded based on comparison with other learners. A cooperative goal structure is the instructional use of small groups in which students work together to maximize their own and each other’s learning. Hence learners perceive that they are working together with other students to gain rewards. According to social interdependence theory, the way in which the goals in a situation are structured determines the interaction patterns among participants, which, in turn, determines the situational outcomes (Johnson and Johnson 1996). A host of research indicates that cooperation is considerably more effective than competition and individualistic efforts in promoting achievement and retention, yet the ways in which educational technology interacts with alternative classroom goal structures are relatively unexplored (Johnson and Johnson 1996, p. 804).

Theoretical framework Theoretical perspectives that shed light on the interaction between classroom goal structures and computer games are cognitive evaluation theory (Deci et al. 1999) and cognitive load theory (Pass et al. 2004). Cognitive evaluation theory predicts that when the interpersonal context for performance-contingent rewards is relatively pressuring, the rewards tend to be experienced as more controlling and lead to diminished intrinsic motivation. Conversely, when the interpersonal context is relatively noncontrolling, the rewards tend to be experienced as more informational and lead to enhancement of intrinsic motivation.

123

Computer games application

541

Therefore, it was speculated that a competitive goal structure, involving a pressuring interpersonal context for performance rewards, would diminish the motivational effects of computer games; a cooperative or individualistic goal structure, involving a relatively noncontrolling interpersonal context for performance rewards, would enhance the motivational effects of computer games. Hence a cooperative or individualistic goal structure would foster the motivational effects of computer games on learning more than a competitive goal structure. According to cognitive load theory, working memory load may be affected by the intrinsic nature of the learning tasks (intrinsic cognitive load) and the manner in which the tasks are presented (extraneous cognitive load). One’s intrinsic cognitive load is reduced through developing cognitive schemata: a large number of interacting elements for a novice might be a single element for an expert who has a schema that incorporates the elements. Extraneous cognitive load may be reduced through effective task information representation. Extraneous cognitive load and intrinsic cognitive load are additive. Therefore, it was conjectured that a combination of cooperative classroom goal structure and computer game application would facilitate cognitive learning achievement most. A cooperative goal structure would help to reduce the intrinsic load of a learning task by enabling peer discussion and cognitive schema development for both expert students (who elaborate the material to peers thus achieving cognitive restructuring) and novice ones (who replicate experts’ cognitive schema of the learning task) (Webb and Palincsar 1996). A computer game would visualize and anchor abstract concepts in a meaningful real-life context to provide just the type of external support that may reduce the extraneous cognitive load and allow students to use their precious working memory for higher-order tasks (Gredler 1996).

Empirical research on classroom goal structures for gaming Despite the large number of studies about the use of classroom goal structures alone and computer games alone, empirical studies that examine the interaction between the use of computer games and classroom goal structures are few. A recent review revealed only a few comparative studies conducted years ago (e.g., Bahr and Rieth 1989; Strommen 1993; Tanner and Lindquist 1998). Closely related to the current study is Bahr and Rieth (1989)’s investigation where 46 mildly handicapped junior high school students were assigned to work with partners on a computer-based arithmetic drill-and-practice game. Students practiced with their partners, recorded and graphed daily computer scores, and received points based on the pair’s scores (cooperative condition), comparing individual with his/her partner (competitive condition), or individual’s self-progress (individualistic condition). The study findings indicated that students gained test-based math learning achievement during the computer-based drill-andpractice game, yet there was no significant effects associated with goal conditions.

Role of learner characteristics The educational gaming literature deems learner characteristics as an essential component that moderates games’ efficacy. Gender, as noted by Bryce and Rutter (2003), influences game players’ motivation and performance. For example, De Jean et al. (1999) reported that girls had difficulty recognizing the embedded math elements in the game and more

123

542

F. Ke

boys than girls were engaged by cooperative game-playing and group problem-solving. Other learner characteristics, such as prior knowledge level and socio economic status, may also mediate the effects of computer games on learning. Moreno (2002) reported that students with low prior knowledge and low computer experience were helped most by the visual representations in the gaming situation. Paperny and Starn (1989) reported that games (as opposed to traditional instruction) produced significant knowledge gain and attitude change among students with low SES (as opposed to other students). These earlier findings on how individual difference mediates game-based learning outcomes need to be replicated by more contemporary research.

Summary Although computer games are proposed as powerful learning tools, empirical findings on how games enhance learning, especially learning in school, are still inconclusive. Research is missing in the literature on whether computer games’ efficacy can be moderated by planning/controlling external instructional activities contexts. Therefore, examining the gaming practice as a structured interaction between computer games and game-based classroom pedagogy (such as classroom goal structures) is warranted.

Research purpose Rather than using computer games as a stand-alone occurrence, this in-situ study explored computer games as a planned application and a complementary pedagogical instrument to be integrated into the school instructional system. Specifically, the study examined the interaction between alternative learning applications (computer games and paper-andpencil drills) and different classroom goal structures (individualistic, competitive, or cooperative) on fifth-graders’ mathematical learning outcomes. Student gender, socioeconomic status, and prior math ability were used as moderating learner characteristic variables. The following research questions were addressed: (a) Will computer games, in comparison to paper-and-pencil drills, be more effective in promoting math learning outcomes, (b) will alternative classroom goal structures influence the effects of computer games and conventional drills on math learning outcomes, and (c) will learner characteristics mediate the interaction between learning applications and classroom goal structures?

Method Participants The participants were 487 students recruited from eighteen 5th-grade public school classes in four rural school districts in central-Pennsylvania. All students elected to participate, although absences during the pretests or posttests contributed to attrition. Data from 358 students were included in this report. Participants varied in gender, socio economic status, and prior math ability level: 49% female, 38% economically disadvantaged; in terms of prior math ability, 23% were below basic, 20% basic, 34% proficient, and 23% advanced.

123

Computer games application

543

Research design This study used across-stage, mixed model research (Creswell 2003), with quantitative objective and data collection methods playing a dominant role. Employing a pretest-posttest quasi-experimental design, this study examined the effects of classroom goal structures (individualistic, competitive, or cooperative) and learning applications (computer games vs. paper-and-pencil drills) on criterion measures (standards-based math exam performance, attitudes questionnaire and metacognitive awareness survey responses). Students’ gender, socio economic status, and prior math ability level were moderating variables. In-field observation and selected participants’ think aloud verbal protocols provided qualitative data as secondary modes of data collection. In this study, the purpose of qualitative data collection was to corroborate and better explain the potential results from the quantitative method. Computer games used In contrast to certain game research that explored how entertainment games were customized for general learning (Squire 2003, Unpublished doctoral dissertation), this study examined computer-based educational games which aim to develop subject-related knowledge in a purposeful way and are promoted to parents and teachers as relevant to formal curriculum (Garris et al. 2002; Gredler 1996). With this purpose, ASTRA EAGLE, a series of Web-based educational games developed by one sampled school district itself, was used. The games were designed to reinforce academic standards for mathematics required by ‘‘Pennsylvania System of School Assessment (PSSA),’’ which is a standardsbased criterion-referenced assessment required of all public schools in the Commonwealth of Pennsylvania. The games were originally designed for individual play (play against computer) but can be customized for competitive (play against peers by comparing game scores) or cooperative use (collective play against computer or other teams) with the assistance of game-based classroom activity arrangements. This study used four mathematics games within the ASTRA EAGLE set that target 5th grade students. They are mainly strategy games where gameplay relies on problem-solving and decision-making skills (Crawford 1997). These mathematical games include a variety of cognitive tasks targeting math concepts comprehension and skills application, such as measurement problems, comparing whole numbers, solving simple equations, and mapping X and Y coordinates. Most tasks are contextualized in stories, characters, and actions relevant to school students. For instance, in a game called Up, Up, & Away children act as pilots who travel by balloon. A problem embedded in the game is to estimate the traveling speed (such as ‘‘If the balloon was traveling at 14 miles per hour and then sped up by a factor of 2 and then added another 1 miles per hour, how fast would it be traveling?’’). Another example is the task of locating X and Y coordinates in a game called Treasure Hunt, where game players could follow a hint (such as ‘‘Go to X15, Y3 on the map’’) to dig for treasure. Immediate feedback occurs after students’ actions. A game has multiple episodes and children must push themselves to beat the computer game for advancement to higher levels. Dependent measures and instruments In this study math learning was conceptualized as a multidimensional construct comprising all three components: ‘skill’, ‘metaskill’ and ‘will’, or in other terms, cognitive learning

123

544

F. Ke

achievement, metacognitive awareness, and motivation (Mayer, 1998 p. 51). Specifically, cognitive learning achievement was measured by a 36-item ‘‘Game Skills Arithmetic Test (GSAT)’’ that was developed based on the PSSA and focused on cognitive math skills the computer games are designed to reinforce. A panel of 5th grade math teachers from the sampled school districts vetted the test’s content validity. The KR-20 reliability of the test in this study was 0.86. Metacognitive awareness in this study referred to knowledge about cognition and regulation of cognition, measured by the Junior Metacognitive Awareness Inventory (Jr. MAI) Version A (Sperling et al. 2002). The Jr. MAI Version A, intended for use in grades three through five, is a 12-item self-report questionnaire about the way students learn. Respondents estimate, on a 3-point Likert scale (1 = never; 3 = always), the frequency with which they engaged in metacognition when learning and studying. The instrument’s reliability in this study was 0.65. In terms of motivation, the study adopted expectancy-value model in conceptualizing motivation as comprising expectancy (or perceived competence), value (goals and beliefs about the importance and interest of the task), and affective (emotional reactions to the task) (Pintrich and De Groot 1990, p. 33). Corresponding to this understanding, Tapia’s ‘‘Attitudes towards Math Inventory’’ (ATMI, Tapia and Marsh 2004) was chosen to measure motivation to math learning. This five-point Likert-scaled inventory is a 40-item survey, investigating students’ feelings toward mathematics according to four identified factors labeled: self-confidence, value, enjoyment, and motivation. The KR-20 reliability of the inventory in this study was 0.97. In addition to the three instruments that measured math learning outcomes, subjects’ think-aloud verbal protocols, based on the IMPROVE method (Mevarech and Kramarski, 1997), were examined to reveal their cognitive and metacognitive processes. Subjects were prompted to talk process out loud while solving problems in a computer-based math game or paper math drill. The prompt included brief instruction and open-ended probing questions. The instruction question was: ‘‘You will be observed during game playing (or paper drilling). Please keep talking out loud what comes to your mind while solving the problem.’’ The probing questions included: ‘‘If you were describing this problem to a friend, what would you tell him or her? How did you solve the task? Is this problem similar to any one that you have solved before?’’

Procedure and treatments Demographic data of participants, including gender, socio-economic status (normal or disadvantaged), and prior math ability levels (advanced, proficient, basic, and below basic), were collected as moderating variables prior to the treatment for later data analysis. The researcher and the school teachers administered GSAT, ATMI, and Jr. MAI as pretests. Participants in intact classes were randomly assigned to six experimental groups: individualistic paper-and-pencil drills group, competitive paper-and-pencil drills group, TGT cooperative paper-and-pencil drills group, individualistic game-playing group, competitive game-playing group, and TGT cooperative game-playing group (Fig. 1). Participants of the three game-playing groups took two orientation sessions (45 min each) during which they tried each web-based math game and became familiar with the game environment. Then, they played one math game during two 45-min sessions each week for 4 weeks. These mathematical learning games contained a variety of tasks targeting math concepts comprehension and skills application, including computation and

123

Computer games application

545

Fig. 1 Research design and procedure

estimation, adding and subtracting measurements, comparing quantities and magnitudes of numbers, and mapping X and Y coordinates. Participants of the three paper-and-pencil drills groups also took orientation sessions on drilling activities. Then, they completed paper-and-pencil math drills during two 45-min sessions each week for 4 weeks. The drill questions used for each paper-and-pencil drilling week had been retrieved from the corresponding computer math game that was played by the game-playing groups in that specific week, and printed on paper sheets.

Experimental groups Group 1—Games in TGT cooperative goal structure In this group students were rewarded based on team performance. Their interpersonal relationship was positively interdependent. Specifically, this study adopted Teams-GamesTournament, a well-established cooperative learning strategy (Slavin 1995), as the cooperative goal structure technique. Based on the TGT model, students formed heterogeneous 3–4 person teams (mixed in ability and gender). At the beginning of each game session, students collaborated with teammates for 15 min by sitting before a single computer and practicing with the games. For the remainder of the 30 min, game teams then competed with one another: each team member was assigned to a desktop computer at a tournament table to play against other teams’ representatives. At any tournament table the students were roughly comparable in achievement level. At the end of every gaming session, the players at each table compared their gaming scores to determine their rank which was then converted into points. The points that the players earned added up to a team’s score. The team scores were ranked and listed in a class newsletter, and distributed to the class at the beginning of the next treatment session. The top team received a winner’s certificate. TGT was selected because it is a cooperative technique using both group rewards and individual accountability (Slavin 1995). Researchers (Johnson and Johnson 1996; Slavin 1995) stated that cooperative learning has its greatest effect on student learning when group rewards and individual accountability exist. In addition, TGT has been applied in empirical game-based learning research and produced positive outcomes (Tanner and Lindquist 1998).

123

546

F. Ke

Group 2—Games in competitive goal structure In this group students were rewarded based on individual performance and competition against each other. Their interpersonal relationship was negatively correlated: one’s success meant the others’ letdown. Students sat before individual computers and played games independently. There was no 15-min cooperative game playing. At the end of every 45-min gaming session, individual scores were compared against others in the class. Individual ranks were announced in a class newsletter at the beginning of the next treatment session so that everyone could compare their own performance with those of other players. The top five students received winner’s certificates.

Group 3—Games in individualistic goal structure In this group students had no interpersonal interactions with others. Students simply played games individually during 45-min gaming sessions. No individual scores were compared against others in the group. There was no 15-min cooperative game playing either.

Group 4—Drills in cooperative goal structure Similar to the cooperative computer game group, students formed heterogeneous teams (mixed in ability and gender) and did teams-games-tournament activities. At the beginning, students collaborated with teammates for 15 min doing math drills. For the remainder of the 30 min, drilling teams competed against one another: each team member was assigned to a tournament table to compete against other teams’ representatives by completing math drills. Drill questions were retrieved from the four math games in ASTRA EAGLE and printed on multiple paper sheets. Students could complete as many drill sheets as they could during each experimental session. At the end of each drill session students at each table compared their total drill scores, on which teams were ranked and compared in the same manner as the set up for the cooperative computer game group.

Group 5—Drills in competitive goal structure Individually, participants did paper-and-pencil math drills. Students could complete as many drill sheets as they were able during each 45-min drilling session. At the end of every drill session individual scores were compared against others in the class. The process was similar to that of the competitive game-playing group except that students did paper drills rather than game playing.

Group 6—Drills in individualistic goal structure Individually, participants did paper-and-pencil math drills. No individual scores were compared against others, the same as the activity for individualistic game-playing group. During the 4-week experiment period, four participants from each of the six abovementioned experimental groups, comprising representatives of different gender, SES, and prior math ability, were selected for further qualitative data collection. Special efforts

123

Computer games application

547

ensured that qualitative sampling across different experimental groups was similar in composition structure and participant number. These pre-selected participants were trained in how to think-aloud in the orientation sessions. Then they were observed and requested to think aloud when they interacted with the games or paper drills. Additionally, four cooperative learning teams (two of computer games condition and two of paper-and-pencil drills condition) were closely observed during experimental sessions and their peer tutoring communications were recorded. Finally, after the 4 weeks of experiment treatments, all participants retook the GSAT math test, ATMI attitudes inventory, and Jr. MAI metacognitive awareness inventory in posttests. The questions of the GSAT in the posttest were the same as those in the pretest, but with shifted sequence.

Results Quantitative findings A preliminary analysis of the data was conducted to ensure compliance with the assumptions for the parametric statistics used in this study. The correlation analysis on the three dependent variables was also conducted for the multivariate statistics used in this study, and its result supports that the three dependent variables are significantly correlated. Table 1 summarizes the descriptive statistics for six experimental conditions. A one-way MANOVA was conducted for pre-treatment between-group comparison on three pretest measurements. The results indicated that there were no significant pre-treatment differences between experimental groups. A single Multivariate Analysis of Covariance (MANCOVA) was used to examine the main effects and the interaction between learning applications (computer games or paperand-pencil drills) and classroom goal structures (cooperative, competitive, or individualistic structure) on GSAT test performance, ATMI math learning attitudes, and Jr. Mai metacognitive awareness. Participants’ gender, socio-economic status, and prior math ability levels were considered as moderating variables and their pre-treatment scores in GSAT, JrMai, and ATMI were used as covariates. The MANCOVA results showed overall

Table 1 Descriptive statistics Adjusted posttest*

Testa Attitudesb Metacognitive awarenessc

Paper-and-pencil drills n = 181

Computer games n = 177

Individualistic n = 58 Mean

Individualistic n = 57 Mean

Competitive n = 63 Mean

Cooperative n = 60 Mean

Competitive n = 51 Mean

Cooperative n = 69 Mean

21.32

19.89

19.37

21.56

20.00

20.51

150.45

142.20

146.54

150.44

150.14

160.53

28.30

27.66

27.80

27.96

27.61

28.15

* Adjusted means using three pretest measurements (GSAT, ATMI, JrMai) as covariates a

The full score of GSAT math test is 36

b

The full score of ATMI attitudes inventory is 200

c

The full score of JrMai metacognitive awareness inventory is 36

123

548

F. Ke

a significant effect of the experimental groups on the outcome variables of mathematical learning, F(15, 789) = 2.66, p \ .01. The results also indicated a significant main effect for the learning applications, F(1, 263) = 5.16, p \ .01, and a significant main effect for the classroom goal structures, F(2, 263) = 3.07, p \ .01. Research Hypothesis 1: Computer games, as opposed to paper-and-pencil drills, would be more effective in promoting math learning outcomes. This hypothesis was partially supported by the results of MANCOVA analysis. The multivariate test indicated an overall significant effect of learning applications (games vs. drills) on comprehensive math learning outcomes. Then, within MANCOVA the univariate test for the effect of learning applications on ATMI attitudes inventory score was significant, F(1, 263) = 14.34, p \ .001. Students in the computer games groups (Mean = 153.93) scored significantly higher than those in the paper-and-pencil drills groups (Mean = 146.39). However, within MANCOVA the univariate test for the effect of learning applications on either GSAT math test performance or Jr. MAI metacognitive awareness scale result was not significant. Therefore, computer games promoted significantly more positive attitudes toward learning, but there was no significant difference between computer games and paper-and-pencil drills in facilitating cognitive math test achievement or metacognitive awareness. Research Hypothesis 2: Classroom goal structure would influence the learning effectiveness of computer games on learning; specifically, cooperative goal structure would foster the effects of computer games on learning outcomes (cognitive or affective). First, the multivariate test indicated that there was no significant interaction effect between classroom goal structures and learning applications. Therefore, no statistical evidence suggested that the effect of classroom goal structures was different across the two learning applications—computer games and paper-and-pencil drills. Hence the Hypothesis 2 is operationalized in two sub-hypotheses and discussed in the following section. Hypothesis 2.1: Cooperative goal structure, in comparison to the other two structures, would be more effective in facilitating cognitive, metacognitive, and motivational mathematical learning outcomes. This hypothesis was partially supported. The multivariate test results indicated an overall, significant main effect of classroom goal structures for mathematical learning outcomes. The univariate tests within MANCOVA indicated significant effect of classroom goal structures on both ATMI attitudes inventory score, F(2, 263) = 4.87, p \ .01, and GSAT math test performance, F(2, 263) = 3.67, p \ .05. But the univariate test within the MANCOVA for the effect of classroom goal structures on JrMAI metacognitive awareness score was not significant. The pair wise comparisons in terms of ATMI score demonstrated that students in cooperative goal structure (Mean = 153.76) did develop significantly more positive attitudes toward math than those in competitive structure (Mean = 146.04, p \ .01), but not significantly different from those in individualistic structure (Mean = 150.45, p [ .05). Then, no significant difference existed between competitive and individualistic goal structure in terms of attitudes toward math. However, the pair wise comparisons in terms of GSAT performance demonstrated that students in individualistic goal structure (Mean = 21.43) scored significantly higher than those in cooperative structure (Mean = 19.95, p \ .05) and competitive structure

123

Computer games application

549

(Mean = 19.94, p \ .05). Then, no significant difference between cooperative and competitive goal structure appeared in GSAT performance. Therefore, the conclusions are: • Individualistic goal structure facilitated math test performance significantly more than the other two goal structures (lcoop = lcomp \ lindi). • Cooperative goal structure, in comparison to competitive goal structure, was significantly more effective in facilitating attitudes toward math (lcoop [ lcomp; lcoop = lindi; lcomp = lindi). • No significant differences occurred between alternative classroom goal structures in supporting metacognitive awareness during math learning (lcoop = lcomp = lindi). Hypothesis 2.2: Computer games within cooperative goal structure, as opposed to the other five experimental treatments, will be more effective in facilitating cognitive, metacognitive, and motivational mathematical learning outcomes. The hypothesis was partially supported. First, the post hoc comparison of six experimental groups in terms of ATMI attitudes inventory score indicated that computer games within cooperative goal structure did promote significantly more positive attitudes toward math learning than the other experimental treatments, F(5, 263) = 6.16, p \ .01. The pair wise comparisons between the other five experimental groups generally indicated nonsignificant differences except that lcomp_game [ lcomp_drill, lindi_game [ lcomp_drill, and lindi_drill [ lcomp_drill. The means of the six experimental groups have been presented in Table 1. However, the post hoc comparison of the six experimental groups in terms of GSAT math test performance indicated that computer games within individualistic goal structure, instead of games within cooperative goal structure, tended to be more effective than the other five experimental groups in facilitating cognitive math test performance, although this difference was not statistically significant. The post hoc comparison of six experimental groups in terms of JrMai score indicated that there was little difference between the six experimental groups. It should be noted that most students scored very high on the JrMai pretest (grand mean is 28.21 out of 36), thus creating a ceiling effect for the measure of prior metacognitive awareness achievement.

Effects of moderating variables (learner characteristic variables) The multivariate test did not indicate a significant main effect of gender on math learning outcomes. There were no significant interactions between gender and the two treatment variables (learning applications and classroom goal structures) either. Similarly, no main effect of socio-economic status was evident, and no sufficient evidence existed to support the interaction effect between SES and the treatments. The multivariate test indicated a significant main effect of prior math ability on math learning outcomes, F(3, 263) = 2.90, p \ .01. However, there were still no significant interaction effect between prior math ability and the treatments. Therefore, insufficient statistical evidence was found to support the moderating effect of learner characteristics on the learning effectiveness of learning applications and classroom goals structures. A potential reason for not being able to identify the mediation effects of learner characteristic variables may be the small sample size of each grouping cell produced by the multi-way interaction analysis, resulting in insufficient power to detect the

123

550

F. Ke

possible interaction effects between learner characteristics and learning applications within classroom goals structures.

Qualitative findings Qualitative theme analysis was conducted with the data collected from in-field observation and selected participants’ (4 from each experimental group) think aloud verbal protocols, in an effort to corroborate the experimental study results and explain conflicts between hypotheses and findings. The researcher employed constant comparison of participants’ verbal protocols and activities with the goal of organizing the data into systematic categories of analysis by seeking recurring themes. The statements or meaning units that emerged as possible commonalities from the data were forwarded as initial themes (Creswell 2003) and coded using Nvivo software. The researcher then refined these themes by removing overlapping ones, capturing the main thrust of each theme’s meaning, and re-examining them through member checking (Guba and Lincoln 1994). Through this data coding, salient themes emerged that illuminate participants’ experience within different treatments and corroborate the quantitative findings. These themes, with supportive data as space permits, are discussed below.

Affective, cognitive, and metacognitive responses to game playing Affectively, computer-based games engaged participants. Findings suggested that computer games afforded greater retention over time than paper-and-pencil drills: gameplaying participants demonstrated focused attention and enjoyment and expressed reluctance to leave computer labs when a gaming session was ended. On the other hand, in paper-and-pencil drills setting participants demonstrated more feelings of boredom and frustration. This qualitative finding sustains the result of quantitative analysis that students in computer games groups developed significantly more positive math learning attitudes than those in paper-and-pencil drills groups. Game-playing participants were engaged in either effortful mathematical problem solving or pure entertainment pursuit. Whether these two engagements were integrated or not largely relied on a game’s gameplay design. In the games where gameplay mechanism and learning content were designed as detached segments, participants would sacrifice learning for the pursuit of pure entertainment, thus not experiencing focused learning like those in paper-and-pencil drills would. An example is a game called Up, Up, & Away where game players would not get any math questions until they crashed their balloon three times. Most participants, as observed, were reluctant to crash their balloons and they could control the balloons skillfully so it wouldn’t collide with birds, mountains, or clouds. Therefore in most situations these participants simply enjoyed the balloon tour without bothering to answer math questions. Differently, in the games where gameplay mechanism and learning content were integrated, learning and fun did team up to engage learners both affectively and cognitively. For example, in a game called Treasure Hunt, participants needed to plot coordinates on an XY graph to locate the treasure spot. When playing Treasure Hunt, participants’ think-aloud protocols indicated active cognitive thinking in question interpretation, math principles execution, and frequent self-monitoring:

123

Computer games application

551

X3 Y7, X3, Y7 (repeating the question)…where is X3…it is here…where is Y7, it should be vertical…ah, there…but I can’t get through the river…got it (he was able to use the arrow key to move the shuffle around the river and get it to the right spot) …Y7(he was checking the Y coordinate)…X3(he was checking the X coordinate)…Yeh, I found the treasure! (Protocol of David,1 Oct. 20, 2005) Therefore, not all games encouraged cognitive engagement, which explained why quantitatively computer games did not significantly surpass paper-and-pencil drills in promoting cognitive math test performance. Although there was no statistically significant difference between computer games and paper-and-pencil drills in encouraging metacognitive awareness, qualitative data indicated that game-playing participants seemed to exhibit self-regulative manners more frequently than their drilling peers. The following think-aloud verbal protocol depicted a typical game-based metacognitive regulation process: Great, with one more question I will be able to beat Fuzzy (the fictitious opponent in the game)…(read the question slowly)…’’fall down by’’…so it is a factor…multiply…24 times 4…it’s 96…Um, it’s not there (in the multiple choices), so I must be wrong…all these numbers are small…so minus?...no, no, it is divide…24 divides 4…it’s 6…B!...Is it?...yes…Yeh! I win, I win! (Protocol of Mark, Nov. 1, 2005) As the protocol implied, the game-playing participant monitored his own progress (‘‘with one more question I will be able to beat Fuzzy’’), weighted each single question toward the intention of winning, adjusted his task-oriented behavior—reading the question more slowly and conducting hypothesis generation and testing. The protocol also suggested that this participant’s metacognitive regulation process was fostered by instant feedback and goal of winning embedded in the computer game. Such kind of metacognitive regulation behaviors were commonly found in most game-playing participants’ think-aloud protocols but not in those of paper-pencil drilling participants.

Cognitive collaboration between peers was not automatic As observed, the 15-min collaborative gaming or drilling section in the cooperative gaming (Group 1) or cooperative drilling treatment (Group 4) did not lead to cognitive collaboration between peers as expected. Instead, the actual situation of peer discussion was typically like the following: (Novice): It’s subtraction... is it? (Expert): It’s 48. Just trust me on this. (Novice): I trust you on that … (followed the direction and clicked the item of ‘48’)…Uha (happily) now what should I go. (Expert): Go right here, right here. … (Novice): (reading the question) what is the next number in the following sequence…(he is still reading the question and trying to understand it) (Expert): It is 6, 6! It’s B! Just click it. (He interrupted the peer’s thinking)

1

In this paper, qualitative data was cited using pseudo names.

123

552

F. Ke

As this observation note demonstrated, the expert student became a leader rather than a peer tutor in the team. He solved questions single-handedly without communicating with peers on how he performed the cognitive tasks. Unless being prompted adeptly, many expert students did not volunteer, or in some cases did not know how, to elaborate their cognitive reasoning, thus losing the chance of cognitive restructuring (Webb and Palincsar 1996). More seriously, novice students’ cognitive processes were interrupted when expert students hastily threw in answers. As a result, peer support in cooperative groups seemed more emotional than cognitive. These qualitative findings helped to explain why cooperative structure was effective in promoting attitudes, but ineffectual in enhancing cognitive math test performance.

Goal structures on pacing in drill tasks It was observed that participants’ pacing in game-based drill tasks was dissimilar across alternative classroom goal structures. Specifically, within the individualistic goal structure, game-playing participants’ demonstrated a more self-controlled pacing when working with drill questions. As observed, they generally spent more time on a single question. Their think-aloud protocols also displayed more persistence when coping with a difficult item (e.g., multiple tests/trials of alternative problem solving strategies). On the other hand, within the competitive or cooperative goal structure game-playing participants showed a hastened pacing, possibly due to their desires to complete questions as many and as quickly as possible to surpass other individuals or teams. Comments like ‘‘Be quick!’’ ‘‘Go, go, go’’ were frequently heard among the game-playing participants within the cooperative or competitive goal structure. These participants were less patient with a single question: rather than effortful dwelling on a difficult item they would conduct wild guesses or turn to a more advanced peer for a direct answer. Their think-aloud protocols demonstrated less effort for test/trial of alternative problem solving strategies. This qualitative finding helped to clarify the advantage of individualistic structure over the other structures in promoting students’ cognitive math test performance.

Conclusions and discussions The study findings suggest that computer games, compared with paper-and-pencil drills, are significantly more effective in promoting learning motivation but not significantly different in facilitating cognitive math test performance and metacognitive awareness. Additionally, this study establishes that alternative classroom goal structures mediate the effects of computer games on mathematical learning outcomes. Cooperative goal structure, as opposed to the other two structures, significantly enhances the effects of computer games on learning motivation. Whereas individualistic goal structure seems to enhance the cognitive learning effects of computer games more than the others, although this advantage is not statistically significant. It should be emphasized that the computer-based educational games employed in this study were specifically developed for learning that is relevant to formal curriculum and specific content knowledge (mathematics). The main purpose the games fulfill is to practice already-required concepts or principles rather than to teach new knowledge and skills. Therefore, caution should be exercised when generalizing findings of this study to computer games that are developed for different purposes.

123

Computer games application

553

Efficacy and instructional design of educational game Cognitive effect and design of endogenous fantasy The finding on the insignificant difference between computer games and traditional paperand-pencil drills on cognitive math learning achievement sustains the discovery of Randel et al. (1992), who reported that among the 68 studies regarding the effects of games on student performance compared with traditional classroom instruction, the major proportion found no difference. The finding is also congruent with the conclusion of Dempsey et al. (1996) that there is no clear causal relationship between cognitive test performance and the use of computer-based games. In-field observation of students supports a common skepticism in using computer games for learning: ‘‘There is the potential for students to be distracted by the game goals and, thus, not achieve the learning goals’’ (Miller et al. 1999, p. 307). Specifically, gameplaying students may involve themselves in pursuit of pure entertainment as opposed to learning-oriented problem solving. In comparison, paper-and-pencil drilling has focused students’ effort and time to math problem solving even though it is not as enjoyable as game-based drilling. This phenomenon may explain why the study indicated a non-significant advantage of computer-based game playing in terms of cognitive learning achievement. It also confirms what educational gaming researchers (Garris et al. 2002; Rieber 1996) have proposed: using computer games for learning is more than a form of educational ‘sugar coating’. The integration of learning and fun is not automatic, but a well planned instructional design with an educational game’s endogenous fantasy—the fantasy in the game is not external but closely related to the learning content (Rieber 1996). For example, in a game representing endogenous fantasy (e.g., Treasure Hunt), it was difficult to segregate the learning section (e.g., plot XY coordinates) from the fun part hence game players could actively practice math problem solving with enjoyment. Consequently, learning became mindful or a situated understanding (Gee 2003).

Metacognitive effect and game-based self-regulated learning Certain researchers (Azevedo 2005; Pillay 2002) have reported that learners interacting with hypermedia environments tend to be more intrinsically motivated and hence engage in metacognitive regulation more actively. Such a report is not supported by quantitative data in this study. A possible reason for a lack of significant findings in the quantitative data may be a ceiling effect with the measure of metacognitive awareness before the treatment, which made it difficult to further improve the metacognitive score with a 1-month treatment. In addition, to better promote metacognitive regulation, the instant feedback in computer games should be informative elaborated feedback that offers reflective debriefing (Butler and Winne 1995). In the games used in this study, the feedback was mostly summative judgment which, according to Butler and Winne (1995), would diminish gain in metacognitive knowledge and strategies. Encouragingly, qualitative data suggests that game-playing participants were frequently engaged in metacognitive regulation processes. This finding supports Rieber’s assertion (1996) that computer games will offer a new possibility for wedding motivation and selfregulated learning within a constructivist framework.

123

554

F. Ke

Classroom goal structures for computer games Consistent with the investigation by Tanner and Lindquist (1998), this study indicates that Teams-Games-Tournament cooperative goal structure is most helpful in sustaining the motivational effects of computer games. This finding also supports the speculation based on cognitive evaluation theory (Deci et al. 1999) that the intrinsic motivation appeal of computer-based games is mediated by the external reward structures (in this case, classroom goal structures). Like Bahr and Rieth (1989), this study has not indicated significant difference among alternative classroom goal structures in influencing games’ efficacy on cognitive learning, thus not supporting the hypothesis on the advantage of cooperative goal structure based on cognitive load theory. Implied by qualitative data, this may be due to the absence of cognitive elaboration between peers in teams (hence no chance to get intrinsic cognitive load reduced) and participants’ hastened pacing (hence reduced time and effort) in cognitive task performance when they were rewarded only for outcomes rather than process of performing the cognitive task. Therefore, it is unrealistic to expect game-based collaborative learning to occur easily. First, without a reward mechanism valuing cognitive inquiry and elaboration, teamwork around game playing and winning might result in superficial emotional support. Second, not all students are capable of elaborating cognition reasoning to peers. Therefore, it is necessary to provide explicit cognitive elaboration training or scaffolding that could be internally embedded within a game or externally provided by an instructor or facilitator. On another note, the games used in this study, although customizable for collective play, are not originally multiplayer games. The game characteristics of a single-player game may influence its supremacy in serving a cooperative learning format. Future research should be conducted to interpret the interdependence between a massive multiplayer game and alternative classroom goal structures. In a summary, the study confirms that educational computer games significantly promote motivation toward learning. Computer games, when appropriately designed, also have a potential to foster players’ metacognitive regulation and engage them in active cognitive thinking. The application of external classroom goal structures influences the efficacy of educational computer games. Therefore, the key issue concerning educational computer games is not ‘‘whether or not to use computer games,’’ but ‘‘how to better design an educational computer game’’ or ‘‘how to better apply game-based classroom instructional strategies.’’ References Azevedo, R. (2005). Computer environments as metacognitive tools for enhancing learning. Educational Psychologist, 40(4), 193–197. Bahr, C., & Rieth, H. (1989). The effects of instructional computers games and drill and practice software on learning disabled students’ mathematics achievement. Computers in the Schools, 6(3–4), 87–101. Bryce, J., & Rutter, J. (2003). Gender dynamics and the social and spatial organization of computer gaming. Leisure Studies, 22(1), 1–15. Butler, D., & Winne, P. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research, 65(3), 245–281. Crawford, C. (1997). The art of computer game design. Berkeley, CA: Osborne/McGraw-Hill. Creswell, J. W. (2003). Research design: Qualitative, quantitative and mixed methods approaches (2nd ed.). Thousand Oaks, CA: Sage. Deci, E. L., Koestner, R., & Ryan, R. M. (1999). A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychological Bulletin, 125, 627–668.

123

Computer games application

555

De Jean, J., Upitis, R., Koch, C., & Young, J. (1999). The story of ‘‘Phoenix Quest’’: How girls respond to a prototype language and mathematics computer game. Gender and Education, 11(2), 207–223. Dempsey, J. V., Rasmussen, K., & Lucassen, B. (1996, October). Instructional gaming: Implications for instructional technology. Paper presented at the annual meeting of the Association for Educational Communications and Technology, Nashville, TN. Fletcher, J. D., & Tobias, S. (2006). Using games and simulations for instruction: A research review. In Proceedings of the New Learning Technologies 2006 Conference, Warrenton, VA: Society for Applied Learning Technology. Garris, R., Ahlers, R., & Driskell, J. E. (2002). Games, motivation, and learning: A research and practice model. Simulation & Gaming, 33(4), 441–467. Gee, J. P. (2003). What video games have to teach us about learning and literacy. New York: Palgrave Macmillan. Gredler, M. E. (1996). Games and simulations and their relationships to learning. In D. H. Jonassen (Ed.), Handbook of research for educational communications and technology (pp. 571–603). New York: Macmillan Library Reference. Guba, E. G., & Lincoln, Y. S. (1994). Competing paradigms in qualitative research. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research (pp. 105–117). Thousand Oaks, CA: Sage. Johnson, D. W., & Johnson, R. T. (1996). Cooperation and the use of technology. In D. H. Jonassen (Ed.), Handbook of research for educational communications and technology (pp. 785–811). New York: Macmillan Library Reference. Johnson, R., Johnson, D. W., & Stanne, M. (1985). Effects of cooperative, competitive, and individualistic goal structures on computer-assisted instruction. Journal of Educational Psychology, 77, 668–677. Kaptelinin, V., & Cole, M. (2002). Individual and collective activities in educational computer game playing. Retrieved August 15, 2004, from http://lchc.ucsd.edu/People/MCole/Activities.html. Malone, T. W. (1981). What makes computer games fun? Byte, 6, 258–277. Mayer, R. E. (1998). Cognitive, metacognitive, and motivational aspects of problem solving. Instructional Science, 26(1–2), 49–63. Mevarech, Z., & Kramarski, B. (1997). IMPROVE: A multidimensional method for teaching mathematics in heterogeneous classrooms. American Educational Research Journal, 34(2), 365–394. Miller, C. S., Lehman, J. F., & Koedinger, K. R. (1999). Goals and learning in microworlds. Cognitive Science, 23(3), 305–336. Mitchell, A., & Savill-Smith. C. (2000). The use of computer and video games for learning. London: Learning and Skill Development. Moreno, R. (2002). Who learns best with multiple representations? Cognitive theory implications for individual differences in multimedia learning. Paper presented at the World Conference on Educational Multimedia, Hypermedia, & Telecommunications. Denver, Colorado. Paperny, D. M., & Starn, J. R. (1989). Adolescent pregnancy prevention by health education computer games: Computer-assisted instruction of knowledge and attitudes. Pediatrics, 83(5), 742. Papert, S. (1998). Does easy do it? Children, games, and learning. Game Developer, 6, 87–88. Pass, F., Renkl, A., & Sweller, J. (2004). Cognitive load theory: Instructional implications of the interaction between information structures and cognitive architecture. Instructional Science, 32, 1–8. Pillay, H. (2002). An investigation of cognitive processes engaged in by recreational computer game players: Implication for skills of the future. Journal of Research on Technology in Education, 34(3), 54–56. Pintrich, P. R., & De Groot, E. V. (1990). Motivational and self-regulated learning components of classroom academic performance. Journal of Educational Psychology, 82(1), 33–40. Prensky, M. (2001). Digital game-based learning. New York: McGraw-Hill Companies. Randel, J., Morris, B., Wetzel, C. D., & Whitehall, B. (1992). The effectiveness of games for educational purposes: A review of recent research. Simulation & Gaming, 23(3), 261–276. Rieber, L. P. (1996). Seriously considering play: Designing interactive learning environments based on the blending of microworlds, simulations, and games. Educational Technology Research and Development, 44(1), 43–58. Slavin, R. (1995). Cooperative learning: Theory, research and practice. Needham Heights, MA: Simon & Schuster Company. Sperling, R. A., Howard, B. C., Miller, L. A., & Murphy, C. (2002). Measures of children’s knowledge and regulation of cognition. Contemporary Educational Psychology, 27, 51–79. Strommen, E. F. (1993). ‘‘Does yours eat leaves?’’ cooperative learning in an educational software task. Journal of Computing in Childhood Education, 4(1), 45–56. Tanner, M., & Lindquist, T. (1998). Using MONOPOLY and Teams-Games-Tournaments in accounting education: A cooperative learning teaching resource. Accounting Education, 7(2), 139–162.

123

556

F. Ke

Tapia, M., & Marsh, G. E. (2004). An instrument to measurement mathematics attitudes, Academic Exchange Quarterly, 8, 2. Retrieved August 15, 2004, from http://www.rapidintellect.com/AEQweb/ cho25344l.htm. Vogel, J. F., Vogel, D. S., Cannon-Bowers, J., Bowers, C. A., Muse, K., & Wright, M. (2006). Computer gaming and interactive simulations for learning: A meta-analysis. Journal of Educational Computing Research, 34(3), 229–243. Webb, N. M., & Palincsar, A. S. (1996). Group processes in the classroom. In D. D. B. R. Caffee (Ed.), Handbook of educational psychology (pp. 841–873). NY: Macmillan.

Dr. Fengfeng Ke is an Assistant Professor in the College of Education, University of New Mexico, MSC053040, Albuquerque, NM 87131. Her research focuses on digital game-based learning, computer-supported collaborative learning, and animations for instructional use.

123