Computers in Human Behavior 36 (2014) 365–375
Contents lists available at ScienceDirect
Computers in Human Behavior journal homepage: www.elsevier.com/locate/comphumbeh
Level of interactivity and executive functions as predictors of learning in computer-based chemistry simulations Bruce D. Homer a,⇑, Jan L. Plass b a b
The Graduate Center, City University of New York, United States CREATE Lab, New York University, United States
a r t i c l e
i n f o
Article history:
Keywords: Learning Simulations Multimedia Executive functions Cognitive load Chemistry
a b s t r a c t High school students’ learning outcomes was examined comparing exploratory vs. worked simulations. The effects of added icons and students’ executive functions were also examined. In Study 1, urban high school students (N = 84) were randomly assigned to one of four versions of a web-based simulation of kinetic molecular theory that varied in instructional format (exploratory vs. worked simulation) and representation (added icons vs. no added icons). Learning was assessed at two levels: comprehension and transfer. For transfer, a main effect was found for instructional format: the exploratory condition yielded greater levels of transfer than the worked simulation. Study 2 used the same conditions and a more complex simulation, the ideal gas law, with a similar sample of students (N = 67). For transfer, an interaction between instructional format and executive functions was found: Whereas students with higher levels of executive functions had better transfer with the exploratory condition, students with lower levels of executive functions had better transfer with the guided simulations. Results are discussed in relation to current theories of instructional design and learning. Ó 2014 Published by Elsevier Ltd.
1. Introduction In recent years there has been considerable interest in the educational effectiveness of more open-ended discovery approaches compared with more direct approaches to learning and instruction. The recent interest in this issue was set off in part by an article by Kirschner, Sweller, and Clark (2006) who criticized ‘‘minimally guided’’ instructional approaches, which they argue result in increased cognitive load and therefore reduced learning. In subsequent commentaries to their article, a number of authors point out that many instructional approaches that have a constructivist orientation, such as problem based learning, are not at all ‘‘minimally guided’’ and have been shown to be very effective (e.g., HmeloSilver, Duncan, & Clark, 2007; Schmidt, Loyens, Van Gog, & Paas, 2007). This debate has direct implications for the design of computer-based approaches to learning, such as educational games and simulation. Although many researchers have argued that the interactive, exploratory nature of many computer-based multimedia learning environments has great educational potential (e.g., Bransford, Brown, & Cocking, 1999; Duffy & Jonassen, 1992;
⇑ Corresponding author. Address: Graduate Center, CUNY, 365 5th Ave., New York, NY, 10016, United States. Tel.: +1 212 817 8292; fax: +1 212 817 1516. E-mail address:
[email protected] (B.D. Homer). http://dx.doi.org/10.1016/j.chb.2014.03.041 0747-5632/Ó 2014 Published by Elsevier Ltd.
Hannafin & Peck, 1988), others have suggested that these very exploratory features can interfere with learning. There are a number of reasons for this disagreement, including differing views on the nature of learning and the meaning of ‘‘interactivity’’ (Domagk, Schwartz, & Plass, 2010). Additionally, we propose that the effects of interactive exploration actually may be different for different learners, with certain students being better able to take advantage of the potential benefits of increased interactive exploration and others becoming overwhelmed by the cognitive load they experience as a result of the increased interactivity. This reasoning is analogous to the well-established phenomenon of expertise reversal, in which a specific educational intervention is beneficial to learners with low levels of prior knowledge, but can actually interfere with learning for individuals with higher levels of prior knowledge (Homer & Plass, 2010; Kalyuga, Ayres, Chandler, & Sweller, 2003). In a similar fashion, interactivity and exploration may have positive effects for certain learners, particularly for those who are better able to manage their cognitive resources, but have negative consequences for other learners, specifically those who have greater difficulties managing their cognitive resources. This possibility is investigated in the current paper, which examines the educational effectiveness of exploration and interactivity in computer-based multimedia learning, and how the ability to manage cognitive resources, as identified
366
B.D. Homer, J.L. Plass / Computers in Human Behavior 36 (2014) 365–375
by the individual differences factor of executive functions, interacts with exploration and interactivity to affect learning outcomes.
2. Interactivity and cognitive load in multimedia learning The considerable criticism of highly interactive exploratory educational approaches has to a large extent originated from researchers with a cognitive load theory (Sweller, 1999) orientation (e.g., Handelsman et al., 2004; Kirschner et al., 2006; Klahr & Nigam, 2004). In their review, Kirschner et al. (2006) are particularly critical of exploratory educational approaches that, they claim, offer only ‘‘minimal guidance’’ to learners. Kirschner and his colleagues argue that more direct instructional approaches, such as worked-examples, result in better learning for students. A number of studies have indeed found that appropriately structured worked-out examples can result in greater learning outcomes than a purely problem-based instruction approach (e.g., Paas, 1992; Sweller & Cooper, 1985; Zhu & Simon, 1987). For example, Tuovinen and Sweller (1999) examined learning outcomes in an interactive, exploratory vs. a worked example approach to teaching students how to use a database program with a graphical user interface. The authors hypothesized that the increased cognitive load induced by the exploratory approach would result in reduced learning outcomes for the students. In support of their hypothesis, Tuovinen and Sweller did find a negative effect for interactivity, however, only for student with no prior database experience; for students who had prior database experience, no difference was found between the two instructional approaches. Moreno and Valdez (2005) similarly examined the effects of interactivity in multimedia learning environments with students. They hypothesized that although interactivity could result in cognitive overload that would interfere with learning, interactivity could also support more active processing by the students and result in more meaningful learning. Moreno and Valdez found that students in interactive conditions had lower learning outcomes than students in non-interactive condition (Experiment 1), but that interactivity could be effective if students were provide with feedback that promoted ‘‘intentional and purposeful’’ processing of the information being presented (Experiment 3). Rather than emphasize the increased cognitive load associated with exploration and interactivity, others researchers have stressed the importance of exploration and interactivity for enabling students to become engaged with the learning materials and to ‘‘take control’’ of their own learning (e.g., Bransford et al., 1999). This more open, interactive approach to learning has been associated with increased interest and motivation (Steffe & Gale, 1995), and with better knowledge transfer and applicability (Hmelo-Silver et al., 2007). Schaffer and Hannafin (1986), for example, compared learning outcomes from educational video with different levels of interactivity. The authors systematically varied the degree of user interactivity with the video and found that the group with the most interactivity took the longest to complete the task, but also had the greatest levels of recall on the posttest. Other researchers have similarly found that interactivity in multimedia learning environments can lead to increased or deeper levels of learning. For example, Vollmeyer, Burns, and Holyoak (1994) report that students who could freely explore the effects of environmental parameters in an aquarium simulation acquired a better understanding of the simulation’s underlying properties than subjects who were given specific objectives. Similar effects have been reported in number of studies (e.g., Mayer & Chandler, 2001; Schwan & Riempp, 2004; Tung & Deng, 2006). Considering both of these two apparently disparate bodies of work, it seems as though learning approaches with more student
interactivity and exploration can result in increased cognitive load compared to less interactive, more didactic approaches, but that the added cognitive load of increased interactive exploration can also have benefits, particularly for certain learners. A task for researchers then is to determine for which students the benefits of interactivity and exploration outweigh the negative effects of increased cognitive load. The current study sought to address this question by investigating learners’ executive functions as one of the factors that predicts the effectiveness of more interactive exploratory approaches to multimedia learning. Of particular interest was the interaction between learners’ executive functions and exploratory interactivity in computer-based science simulations.
3. Executive functions and level of interactivity in multimedia learning Although precise definitions of executive function (EF) vary, EF are generally identified as being high-level abilities that influence other more basic functions, such as attention and memory, and enable the planning, monitoring and control of mental activities and behaviors (Meltzer, 2011). EF processes are typically associated with functioning in the prefrontal cortex (Bryan & Luszcz, 2000; Luria, 1966). There is no single task that serves as a direct indicator of EF (Morgan & Lilienfeld, 2000), but a number of recent efforts have attempted to create a battery of measures in order to capture a variety of different aspects of EF (e.g., Carlson, 2005; Zelazo & Bauer, 2013). When using a battery of measures is not feasible, researchers typically focus on assessing the aspect of EF that is most relevant for their task. One of the most commonly used measures for this purpose is the Stroop Task (Stroop, 1935), which assesses attention and inhibition components of EF (Homack & Riccio, 2004; MacLeod, 1991). The Stroop task is regularly included in neuropsychological assessment as a measure of selective attention, cognitive flexibility, resistance to interference, and inhibitory control (Archibald & Kerns, 1999; Homack & Riccio, 2004; Lezak, 2004; Spreen & Strauss, 1998). In a study with children and preadolescence, Brocki and Bohlin (2004) investigated the dimensionality of several measures of executive functioning, and found that Stroop-like tasks measure not only the ability to inhibit a response, but also the ability to cognitively shift to a new response. In this way, Stroop tasks capture variance in working memory as well as inhibition, two fundamental components of EF (Barkley, 1997). Because EF enable the intentional allocation of mental resources, such as attention and memory, higher levels of EF should result in a more efficient use of cognitive resources and therefore increased learning in interactive environments. In spite of these obvious implications of executive functions for learning, it is only in the past decade or so that there has been a concentrated effort to examine the implications of EF for education (Best, Miller, & Jones, 2009). Overall, a positive relation between EF and educational outcomes has been found (e.g., Bull, Espy, & Wiebe, 2008; St Clair-Thompson & Gathercole, 2006; Yeniad, Malda, Mesman, van IJzendoorn, & Pieper, 2013). For example, Blair and Razza (2007) found that measures of EF in 3- to 5-yearold children, particularly inhibitory control, predicted early math and reading abilities, independent of general intelligence. Similarly, Best, Miller & Naglieri (2011) examined data from a large sample of 5- to 17-year-olds and found that complex measures of ER were significantly correlated to specific aspects of academic achievement measures. These findings suggest that students’ EF may be of particular importance in multimedia learning environments, which can tax learners’ cognitive resources (Brünken, Plass, & Leutner, 2003; Brünken, Steinbacher, Plass, & Leutner, 2002; Mayer, 2001; Plass, Moreno, & Brünken, 2010).
B.D. Homer, J.L. Plass / Computers in Human Behavior 36 (2014) 365–375
4. The present study The present paper reports on two studies that were conducted to examine the relation between high school students’ EF and interactivity in computer-based chemistry simulations. Both studies compared two versions of a simulation, one that was a more interactive exploratory simulation requiring learners to generate and test hypotheses. The other was a less interactive, worked simulation, based loosely on the worked example approach (Paas & van Gog, 2006), in which learners clicked to play through a stepby-step procedure of actual hypotheses testing conducted by a content area expert. Study 1 utilized a simulation of Kinetic Molecular Theory (based on Plass, Homer, Milne, et al., 2009). Study 2 examined similar issues with a more complex simulation, the Ideal Gas Laws (based on Lee, Plass, & Homer, 2006). In both studies, learning outcomes were examined at two levels (comprehension and transfer), and included measures of learners’ EF. We suggest that the mixed findings on interactivity and exploration in multimedia learning may be partially explained by learners’ level of EF, which can mitigate effects of cognitive load created in a more interactive, exploratory learning situation. We therefore hypothesized that instructional format (exploratory or worked) would interact with EF to affect student learning. Specifically, we predicted that the worked simulations would be more effective for learners with lower levels of EF and exploratory simulations would be more effective for learners with higher levels of EF. 5. Study 1 The objective of Study 1 was to examine the interaction between learners’ EF (i.e., their degree of inhibition as measured by a Stroop task) and mode of instruction, specifically either a highly interactive exploration or a less interactive worked simulation version of a computer-based multimedia chemistry simulation. To examine this issue, different versions of a simulation of Kinetic Molecular Theory of Heat (Kinetic Theory) were developed that varied in their amount of interactive exploration. In the exploratory condition, participants were given an introduction that included suggestions on how best to explore the simulation and then were presented with an interactive simulation that required them to actively generate and test hypotheses about the relation between either temperature or number of molecules and pressure. In the worked simulation condition, participants were presented with a series of screen captures from the simulation with a voice-over that illustrated and explained the step-by-step procedures of hypotheses testing with the simulation. The worked simulations were similar to a worked example approach (Sweller & Cooper, 1985) in which students are shown each step required to solve a specific problem, however, with the worked simulations, the students were required to infer relations between the variable in the simulation themselves. The students would click to proceed to the next image and voice over, and could also pause, stop or replay any steps. It was hypothesized that students’ EF would interact with instructional format to affect learning. Specifically, it was hypothesized that students with lower levels of EF would have better learning outcomes in the worked simulation condition, but students with higher levels of EF would have better learning outcomes in the exploration condition. 5.1. Method 5.1.1. Participants The participants for this study came from three science classes in a New York City public high school with a racially and ethnically
367
diverse population. Of an initial 92 participants, eight were excluded do to a failure to meet the inclusion criterion of a minimum of at least 3 interactions (i.e., mouse clicks) with the simulation, yielding a final sample size of 84 students (44 female). Participants’ ages ranged from 14 to 16 years (M = 15, SD = .9), with 44% of the participants identifying as ‘‘White’’, 24% of the participants identifying as ‘‘Hispanic’’, 11% of the participants identifying as ‘‘African-American/Black’’, 3.5% of the participants identifying as ‘‘Asian’’, 3.5% identifying as ‘‘Other/mixed ethnicity’’, and 14% of the participants choosing not to identify their ethnicity. The students in this study had not previously studied any materials related to the simulation content (i.e., kinetic theory). 5.1.2. Materials For the current study, four versions of a simulation of the Kinetic Molecular Theory of Heat (Kinetic Theory) were used that had been developed as part of a larger research project. The simulations displayed a container filled with gas molecules on the left side of the screen, with sliders for adjusting temperature and number of particles. Only one slider could be adjusted at a time, with the other slider being ‘‘locked’’. On the right of the screen was an automatically updating graph illustrating the relation between pressure and either temperature or number of particles, depending on which variable was ‘‘unlocked’’ in the simulation. The simulations varied in their instructional approach. Half the simulations were exploratory in nature, which were highly interactive guided explorations that required learners to generate and test hypotheses by adjusting a slider that affected a variable (either temperature or number of molecules) and observing the effects on pressure. (See Fig. 1.) The other half were worked simulation versions of the materials that presented learners with a series of images of screen captures from the simulation. (See Fig. 2.) The specific screen captures were designed by an experienced science educator to depict a sequence of steps for manipulating variables in the simulation in a way that constituted a meaningful exploration of kinetic theory. Each step had an accompanying narration read by a professional speaker, based on a script written by the science educator. All materials were reviewed by additional science educators and a team of scientists, and then revised for scientific accuracy and optimal exploration strategy. A goal of the larger study from which the animations were drawn was to examine the effects of iconic representation, and so half of the simulations in each condition had added iconic representations and half did not. We have reported the effects of added iconic representation elsewhere (e.g., Homer & Plass, 2010; Plass, Homer, & Hayward, 2009; Plass, Homer, Milne, et al., 2009) and so although icon was included as a factor in the statistical analyses, it was not the focus of the current set of studies. Executive functions. Executive functions were assessed using a computerized version of the color-word Stroop task (Golden, 1978). For each trial of the Stroop task, a fixation dot appeared in the middle of the computer screen. Less than a second later, a word (RED, GREEN, or BLUE) appeared on the screen written in either a neutral black, or in red, green, or blue colored font. Two colored boxes appear below and participants were required to press, as quickly as possible, either the left or right arrow key, depending on which box color matches the meaning of the presented word. There were 72 trials in total, 24 in which the font colors and word names were congruent (i.e., the work ‘‘red’’ was written in red color font), 24 in which the font color was neutral (i.e., word colors written in black font), and 24 where the font colors and color names were mismatched (e.g., the word ‘‘red’’ appeared in a green color font). Response times and error rates were recorded. Response times to incorrect answers and any response time that was greater than three standard deviations from an individual’s average response time were discarded. An interference score was
368
B.D. Homer, J.L. Plass / Computers in Human Behavior 36 (2014) 365–375
Fig. 1. Exploratory version of the Kinetic Theory Simulation (with icons). In the alternative version, all icons for temperature (burners) and pressure (pressure gauge) were removed.
Fig. 2. Worked Simulation version of the Kinetic Theory Simulation (with icons). In the alternative version, all icons for temperature (burners) and pressure (pressure gauge) were removed.
B.D. Homer, J.L. Plass / Computers in Human Behavior 36 (2014) 365–375
calculated by subtracting the average time needed to complete the neutral and incongruent subtasks from the time needed to complete the congruent subtask (Valentijn et al., 2005). Higher levels of executive functions are indicated by lower interference scores. Prior knowledge was assessed though a computer-based pretest that included six multiple-choice questions and two short-answer questions about kinetic theory. The multiple choice questions were scored as pass/fail, with one point for each pass, and short answer questions were scored out of 2 by content-area experts, for a maximum total possible score of 10 for prior knowledge (maximum of 6 points for the multiple choice questions and 4 for the short answer questions). Posttest knowledge was assessed through computer-based tests that evaluated learning at two levels: comprehension and transfer. Comprehension was assessed using a 20-item multiple-choice questionnaire in which participants received one point for each correctly answered question. Transfer was assessed using a fouritem, short-answer questionnaire in which participants received up to two points for each question. All answers were coded by two content area experts who were not aware of the condition to which each participant had been assigned. Any disagreements in scores were discussed until a consensus was reached. 5.1.3. Procedure Participants were tested as in groups of approximately 15–25 students in a science classroom with individual laptop computers. Verbal directions were given to the entire group and experimenters were present to answer any questions. Testing consisted of four parts: test of prior knowledge, Stroop task, instructional phase (which include exposure to the simulation), and posttest. Before beginning, participants were randomly assigned to one of the four treatment conditions (i.e., Exploration with icons; Exploration without icons; Worked Simulation with icons; or Worked Simulation without icons). Prior to the experiment, all participants completed a demographic questionnaire, which was returned with consent letters. Participants were randomly assigned to one of the four treatment groups. At the beginning of the study, each student sat at a computer where testing and instruction took place. Headsets were distributed to students in the worked simulation group prior to beginning. All students were first presented with the test of prior knowledge and the Stroop task on the computer. Then the students interacted with the assigned version of the kinetic theory simulation. In the worked simulations, students clicked on a series of steps that illustrated a systematic exploration of the simulation. For each step, a still image of a screen shot from the simulation was displayed and an audio narration of that step was played. For example, the step displayed in Fig. 2 had the narration ‘‘Set the volume to 5 liters and lock it.’’ In the exploration condition, students were instructed to systematically manipulate either the temperature or number of particles and then observe the resulting changes in the internal pressure of the gas. General exploration guidelines were provided, but explicit directions were not. In both instructional versions of the simulation, students were advised to take all the time they needed to study the material and not to proceed to the test until they had studied (or explored) the simulations thoroughly. All students completed the procedure within a 50-min class period.
p = .08. This suggests that random assignment was successful and the students in the different treatment conditions were comparable. Furthermore, EF score was not significantly correlated with prior knowledge, X = .12, p = .30, indicating that EF and prior knowledge were separate factors. The variables were screened for statistical outliers, but none were found. To examine the question of how learning outcomes were affected by instructional format and EF, a MANOVA was conducted with the two levels of learning (comprehension and transfer) as dependent variables. For analyses, a median split was conducted on the EF measure creating two groups: ‘‘low EF’’ and ‘‘high EF’’. The MANOVA then included instructional format (exploration or worked simulation), added icons (yes or no), and EF (low or high) as categorical independent measures. Main effects for all factors, and the two-way interaction of EF and instructional format were included in the model. The only significant effect on learning outcomes was a main effect of instructional format for transfer, F (1, 79) = 4.93, p = .03, partial eta squared g2p = .06. The transfer scores were significantly higher for the exploratory group (M = 3.0, SD = .28) than for the worked simulation group (M = 2.1, SD = .27), indicating a learning advantage for the exploratory condition, see Fig. 3. Contrary to what was hypothesized, there was no significant interaction between EF and instructional format, either for transfer, F (1, 79) = .13, p = .72, or comprehension, F (1, 79) = .25, p = .62. 5.3. Discussion Study 1 examined the interaction between instructional format and learners’ EF on learning outcomes. Contrary to what was hypothesized, there was no interaction between the instructional format and EF. However, a significant main effect of instructional format was found for transfer. Specifically, students in the exploratory condition had better transfer scores than those in the worked condition. This finding indicates that for this simulation, there is no difference between the worked simulation and exploration conditions for learning at the level of comprehension, but that all students, regardless of their level of EF, showed better transfer of knowledge after interacting with the exploratory simulation. These results are consistent with instructional approaches that emphasize the importance of interactivity for deeper levels of learning, and provide empirical evidence against the suggestion that welldesigned explorations necessarily result in cognitive overload that interferes with learning. Learning from the current exploratory simulation, Kinetic Theory, which involves understanding two variables at a time (i.e.,
5.2. Results Examination of the scores from the test of prior knowledge confirmed that pretest scores did not differ significantly among the different conditions, F (3, 80) = 1.80, p = .15, with students scoring an average of 4.3 out of 10 (SD = 1.7) on this test. The groups also did not differ significantly on their EF scores, F (3, 80) = 2.32,
369
Fig. 3. Main Effect of Instructional Format on Transfer.
370
B.D. Homer, J.L. Plass / Computers in Human Behavior 36 (2014) 365–375
either the effects of temperature on pressure or the effects of number of particles on pressure), was more effective than learning from a worked simulation in which no individual exploration was possible. In the cognitive load literature, a number of effects have been found only in conditions of high cognitive load (e.g., the contiguity effect and the split attention effect). In low cognitive load conditions, the added cognitive demands imposed by non-optimal designs does not overwhelm learners’ cognitive resources and therefore does not interfere with learning. Given out findings, it is possible that the Kinetic Theory simulation was not complex enough to induce a level of cognitive load such that learners’ EF would have an effect. Therefore, a second study was conducted to expose learners to a condition of higher cognitive processing demands in order to determine if that would result in the hypothesized interaction between EF and instructional format. 6. Study 2 The objective of Study 2 was to further examine the interaction between learners’ EF (i.e., their degree of inhibition as measured by a Stroop task) and mode of instruction, which were either a highly interactive exploration, or a less interactive worked simulation version of a computer-based multimedia chemistry simulation. The learning materials were selected to be inherently more complex and therefore create conditions of higher cognitive demands than in Study 1. Specifically, intrinsic cognitive load was increased by choosing a chemistry simulation, The Ideal Gas Law, that deals with the interaction of three variable (temperature, pressure and volume) rather than just two. Additionally, for Study 2 the worked simulations used animations rather than still images that were used in Study 1. Animations can provide additional information, but also can impose greater cognitive load than still images (Homer, Plass, & Blake, 2008; Plass et al., 2009). Similar to Study 1, different versions of the Ideal Gas Law simulation were developed that varied in the amount of exploratory interactivity they required. In the exploratory condition, participants received the same instructions as in Study 1, except that they were asked to lock one variable, vary a second variable and then observe the effect on the third variable. They were asked to do this for all three variables in the simulation (i.e., volume, pressure and temperature). The worked simulation condition was similar to the worked simulation condition of Study 1 with some variation. Participants were presented with a series of animations that demonstrated step-by-step procedures of hypotheses testing conducted by a content area expert. The students would click to activate the animation depicting each step in the worked simulation, and could also pause, stop or replay any animation. Again, because the simulations were part of a larger study, they also were systematically varied to either have or not have additional iconic representation. We hypothesized that the previous mixed findings on the benefits – or costs – of interactive exploratory learning approaches may partially be explained by the individual learners’ EF. Specifically, we hypothesized that with the more complex simulation of Study 2, instructional format would interact with learners’ EF to affect learning outcomes such that learners with lower levels of EF would have better learning outcomes in the worked simulation condition and learners with higher levels of EF would have better learning outcomes in the exploratory condition. 6.1. Method 6.1.1. Participants Participants were high school chemistry students (N = 67) from a public high school in New York City in which 83% of the student are eligible for free or reduced cost lunch. Age ranged from 16 to
19 years (M = 17, SD = .72), with approximately 40% of the students being female. The students were ethnically diverse, with 52% identifying as being ‘‘Hispanic or Latino’’, 32% identifying as ‘‘Black or African American’’, 1.5% students identifying as ‘‘Arabic’’, 1.5% as ‘‘Asian’’, 1.5% as ‘‘White’’ and the remaining 10.5% of the students declining to identify an ethnicity. 6.1.2. Materials For the current study, four versions of a simulation of the Ideal Gas Law were used that had been developed as part of a larger research project. Half of the simulations were highly interactive exploratory that required learners to generate and test hypotheses. Instructions were the same as in Study 1 except that the current simulations allowed students to manipulate one of the three variables (pressure, volume or temperature) while holding a second variable constant to examine the effect on the third variable, see Fig. 4. In contrast, the worked simulations were less interactive and presented students with a series of animations that demonstrated step-by-step procedures of actual hypothesis testing with the simulation, see Fig. 5. This condition was similar to the worked simulation of Study 1 except that the students could control the animations using standard start, stop, continue, and rewind video control features. In addition, as part of the larger study, half of the simulations in both the exploratory and worked conditions had added icons. Although icon was included as a factor in the statistical analyses below, it was not the focus of the current set of studies. Prior knowledge was assessed though a pretest on gas properties related to the ideal gas law. The test consisted of 6 multiple-choice questions and 2 short answer questions. Multiple choice questions were scored as pass/fail, with one point for each pass, and short answer questions were scored out of 2 by content-area experts. This yielded a maximum total possible score of 12 for prior knowledge. Posttest knowledge was assessed, similar to Study 1, through computer-based tests that evaluated learning at two levels: comprehension and transfer. Comprehension was assessed using an eight-item multiple-choice questionnaire in which participants received one point for each correctly answered question for a maximum possible total score of eight. Transfer was assessed using a six-item, short-answer questionnaire in which participants received up to two points for each question for a total possible 12 points. All answers were coded by two content area experts who were not aware of the condition to which each participant had been assigned. Any disagreements in scores were discussed until a consensus was reached. Executive functions were assessed using the same computerbased version of the color-word Stroop task (Stroop, 1935) as described in Study 1. Again, lower interference scores on the Stroop were indicative of higher levels of EF. 6.1.3. Procedure Procedure was the same as in Study 1, with participants tested in groups of approximately 15–20 students in a science classroom with individual laptop computers. Participants were randomly assigned receive one of four versions of the Ideal Gas Law simulation (i.e., exploratory with added icons; exploratory without added icons; worked simulation with added icons; or worked simulation without added icons. Testing consisted of four parts: test of prior knowledge, Stroop task, instructional phase, and posttest. Participants returned a demographic questionnaire with their consent letters. During the instructional phase, students in the exploratory condition were instructed to systematically manipulate the temperature, volume, and pressure of the gas while one variable was locked and to then observe the resulting change in the third
B.D. Homer, J.L. Plass / Computers in Human Behavior 36 (2014) 365–375
371
Fig. 4. Exploratory Version of the Idea Gas Law Simulation without added icons.
variable. Students in the worked condition were told to click through each animation. Posttest. Once students had finished interacting with the simulations, they were given the computer-based posttests to assess learning outcomes. All students completed the procedures within a 50-min class period. 6.2. Results An inspection of the scores from the test of prior knowledge confirmed that pretest scores did not differ significantly between the different conditions, F (3, 60) = .34, p = .80, with students scoring an average of 4.6 out of 12 (SD = 2.2) on this test. The groups also did not differ significantly on their EF scores, F (3, 60) = 1.33, p = .27. This indicates that random assignment was successful and students in the different treatment conditions were comparable. Furthermore, as in Study 1, EF score was not significantly correlated with prior knowledge, X = .07, p = .58, indicating that EF and prior knowledge were separate factors. The variables were screened for statistical outliers and one participant was found to be an outlier in the EF data and was therefore omitted from subsequent analyses. To examine the question of how learning outcomes were affected by the interaction of instructional format with EF, a MANOVA was conducted with the two levels of learning (comprehension and transfer) as dependent variables. For analyses, a median split was conducted on the EF measure creating two groups:
‘‘low EF’’ and ‘‘high EF’’. The MANOVA then included instructional format (exploration or worked simulation), added icons (yes or no), and EF (low or high) as categorical independent measures. Main effects for all factors, and the two-way interaction of EF and instructional format were included in the model. None of the main effects were significant, however, the main effect of instructional format for comprehension just failed to reach significance, F (1, 58) = 3.55, p = .06, partial eta squared g2p ¼ :06. As illustrated in Fig. 6, there was a near significant trend for comprehension scores to be higher for the exploration group (M = 3.8, SD = 1.9) than for the worked simulation group (M = 3.0, SD = 1.4). As hypothesized, there was a significant interaction between EF and instructional format, but only for transfer, F (1, 58) = 5.70, p = .02, g2p ¼ :09. (See Fig. 7.) The interaction was as predicted. Students with lower EF scores had better transfer scores in the worked simulation condition (X = 2.74, SD = 2.11) compared to the exploration condition (X = 1.67, SD = 1.69), t (30) = 1.67, p = 06. In contrast, students with higher EF scores had significantly better transfer scores in the exploration condition (X = 3.57, SD = 3.02) compared to the worked simulation condition (X = 1.88, SD = 1.99), t (29) = 1.85, p = .04. An examination of the effects of EF within the different instructional format conditions indicated that for the worked simulations, although the low EF group had a higher average transfer score than the high EF group, this difference was not statistically significant, t (31) = 1.21, p = .12. For the exploratory condition, however, the high EF group scored significantly better than the low EF group, t (28) = 2.13, p = .02.
372
B.D. Homer, J.L. Plass / Computers in Human Behavior 36 (2014) 365–375
Fig. 5. Worked simulation version of Ideal Gas Law Simulation with added icons.
6.3. Discussion
Fig. 6. Main effect of instructional format on comprehension.
Study 2 was conducted to determine if learners’ EF would interact with instructional format to affect learning outcomes. Specifically, it was hypothesized that while learners with lower levels of EF would have better learning outcomes in a worked simulation condition, learners with higher levels of EF would have better learning outcomes in the exploratory condition. Learning outcomes were measured at two levels, comprehension of basic concepts presented in the simulation and transfer of the knowledge to similar problems. The hypothesized interaction between EF and instructional format was not found for comprehension, the measure of lower-level learning, where there was a trend that approached significance towards a general advantage for the more interactive exploratory condition. The hypothesized interaction between EF and instructional format was found, however, for the more complex learning outcome of transfer. Examining the different instructional methods within EF groups indicated that the students with lower levels of EF had better transfer outcomes in the worked simulation condition, and learners with higher levels of EF had better transfer outcomes in the exploratory condition. Examining the effects of level of EF within the different
B.D. Homer, J.L. Plass / Computers in Human Behavior 36 (2014) 365–375
Fig. 7. Interaction of executive functions and instructional format on transfer.
instructional methods revealed that the low EF students had higher transfer scores than the high EF students in the worked simulation, but this difference was not significant. However, for the exploratory condition, the high EF students did score significantly higher than the low EF students. Although a number of researchers have argued that exploratory approaches to learning can result in more engaged, deeper learning, other researchers from a cognitive load perspective have argued that more interactive, exploratory approaches to instruction can create too great a cognitive load that then interferes with learners being able to integrate new information into their existing mental schema. The current findings suggest that for more complex learning outcomes (e.g., transfer), learners with higher levels of EF were able to manage the increased cognitive demands of exploration and take advantage of the benefits of exploratory approaches to learning. Learners with lower levels of EF, on the other hand, may have been equally successful in managing the increased cognitive demands and may have experienced cognitive overload with exploratory approaches. These learners with lower levels of EF instead benefitted from a more the worked simulation, a more direct instructional approach.
7. General discussion The goal of the current set of studies was to compare the effects of interactive exploration of a simulation and worked simulations without exploration on learning outcomes for students with different levels of Executive Functions (EF). The question of the effectiveness of more exploratory approaches to learning is particularly relevant to educators with a more constructivist approach to learning (e.g., Problem-based learning) who argue that exploration allows for a deeper, more meaningful processing of the materials and richer learning outcomes. In contrast, instructional approaches that focus primarily on cognitive load (e.g., Kirschner et al., 2006) argue that the added cognitive resources required by such ‘‘minimally guided’’ instructional methods interfere with learning outcomes by creating situations of cognitive overload. Existing research is inconclusive, and studies exist that support both instructional approaches, suggesting that other factors may be having effects. The current paper identifies learners’ level of EF as
373
one factor that needs to be considered in a more nuanced consideration of this issue. Participants in Study 1 interacted with a simulation of Kinetic Theory. This was a simulation of relatively low complexity – only two variables need be considered at a time. The findings indicate that for the more basic learning outcome of comprehension, there were no significant differences between the two instructional approaches. For the deeper learning outcome of transfer, however, there was a significant main effect of instructional format, with the interactive exploratory group performing significantly better than the worked simulation group. This finding is in support of educators that argue for the potential of exploration to produces greater processing of materials and deeper levels of learning outcomes. Study 2, however, suggests that in situations with higher levels of inherent complexity, exploratory approaches may not be ideal for all learners. The second study examined different versions of a simulation of the idea gas law, which is more complex than the simulation in Study 1 because it involves the relation between three variables (gas, volume and pressure). The results from Study 2 found a trend for the more interactive, exploration approach to result in better comprehension outcomes, and significant interaction between learners’ level of EF and instructional format for the deeper learning outcome of transfer. As originally hypothesized, learners with higher levels of EF were better able to take advantage of the potential benefits of the exploratory instructional approach, but students with lower levels of EF had better transfer scores in the worked simulation conditions, suggesting that they experienced cognitive overload in the exploratory version of the more complex ideal gas law simulation. The current studies provide further evidence of the importance of EF in learning (Blair & Razza, 2007; Bull & Lee, 2014; Bull et al., 2008; St Clair-Thompson & Gathercole, 2006; Yeniad et al., 2013). They suggest that learners’ EF can help mitigate the effects of increased cognitive demands. This has particular implications for the design and use of educational simulations. For certain learners, such as those with lower levels of EF, simulations may need additional guidance or animations may be preferable if the content area is sufficiently complex. Similar effects have been reported for the expertise reversal effect in which scaffolds that benefit learners with lower levels of prior knowledge can actually interfere with learning for learners with higher levels of prior knowledge. The current study suggest that EF, which was not correlated with prior knowledge in the current studies, and complexity of the learning materials, should also be considered when designing interactive multimedia learning materials. There are a number of possible limitations to the current study that should be addressed in future research. First, the learners in the current study were all relatively low in prior knowledge (in Study 1, scoring 43% on the pretest and in Study 2, scoring 38% on the pretest). The findings may be different for learners with little to no prior knowledge or for learners with greater levels of prior knowledge. Another issue is that in both studies, interaction with the simulations took place only once over a relatively short period of time. The effects may be different with repeated chances to interact with the simulations. For example, perhaps repeated exposure to the simulation, or exposure to a set of similar simulations, would reduce the cognitive load induced by exploratory conditions. Finally, the current study only examined immediate learning outcomes. Future research should determine if the effects found in the current studies persist over time by including a delayed posttest. If, as suggested by the findings of the current research, the more interactive exploratory approaches result in deeper processing, then they should show learning outcome benefits that persist over time. There are a number of other areas that would benefit from additional research. For example, how does instructional approach
374
B.D. Homer, J.L. Plass / Computers in Human Behavior 36 (2014) 365–375
affect other educationally relevant factors, such as levels of engagement, interest and motivation. The current set of studies used chemistry simulations, but additional research should be conducted with simulations form other domains. Additional research is also needed to determine the extent to which the interaction between EF and instructional format found in the current study generalizes learning situations that do not involve simulations or computer-based learning environments. It is possible that this same interaction would have a similar effect for students in traditional classroom. Finally, the results from this study should be verified with other learner populations. In other research we have found, for example, that multimedia learning effects that apply a general college population could not be replicated for more specialized populations, such as medical students (Kalet et al., 2012; Song, Kalet, & Plass, 2008; Song et al., 2014). 8. Conclusions The current research provides evidence that students’ EF is one of the factors that need be considered when designing computerbased multimedia learning environments. More broadly, the current findings argue for a more complex understanding the factors that influence educational outcomes. Learners come to any instructional situation with a variety of experiences, prior knowledge, skills and abilities, all of which can influence their experiences with instructional materials. It falls on researchers to identify the most important individual factors that influence learning outcomes for specific types of instructional materials and approaches. Ideally, this knowledge can help create more effective, individualized instruction for all students. Acknowledgements The research presented in this paper was supported in part by the Institute of Education Sciences (IES), U.S. Department of Education (DoEd) R305K05014 awarded to Catherine Milne, Jan L. Plass, Bruce D. Homer, and Trace Jordan. The opinions expressed are those of the authors and do not represent views of the U.S. Department of Education. References Archibald, S. J., & Kerns, K. A. (1999). Identification and description of new tests of executive functioning in children. Child Neuropsychology, 5(2), 115–129. Barkley, R. A. (1997). Behavioral inhibition, sustained attention, and executive functions: Constructing a unifying theory of ADHD. Psychological Bulletin, 121(1), 65. Best, J. R., Miller, P. H., & Jones, L. L. (2009). Executive functions after age 5: Changes and correlates. Developmental Review, 29(3), 180–200. Best, J. R., Miller, P. H., & Naglieri, J. A. (2011). Relations between executive function and academic achievement from ages 5 to 17 in a large, representative national sample. Learning and Individual Differences, 21(4), 327–336. Blair, C., & Razza, R. P. (2007). Relating effortful control, executive function, and false belief understanding to emerging math and literacy ability in kindergarten. Child Development, 78(2), 647–663. Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.). (1999). How people learn: Brain, mind, experience, and school. Washington, DC: National Academy Press. Brocki, K. C., & Bohlin, G. (2004). Executive functions in children aged 6 to 13: A dimensional and developmental study. Developmental Neuropsychology, 26(2), 571–593. Brünken, R., Plass, J. L., & Leutner, D. (2003). Direct measurement of cognitive load in multimedia learning. Educational Psychologist, 38, 53–61. Brünken, R., Steinbacher, S., Plass, J. L., & Leutner, D. (2002). Assessment of cognitive load in multimedia learning using dual-task methodology. Experimental Psychology, 49(2), 109–119. Bryan, J., & Luszcz, M. A. (2000). Measures of fluency as predictors of incidental memory among older adults. Psychology and Aging, 15(3), 483. Bull, R., Espy, K. A., & Wiebe, S. A. (2008). Short-term memory, working memory, and executive functioning in preschoolers: Longitudinal predictors of mathematical achievement at age 7 years. Developmental Neuropsychology, 33(3), 205–228. Bull, R., & Lee, K. (2014). Executive functioning and mathematics achievement. Child Development Perspectives, 8(1), 36–41.
Carlson, S. M. (2005). Developmentally sensitive measures of executive function in preschool children. Developmental Neuropsychology, 28(2), 595–616. Domagk, S., Schwartz, R. N., & Plass, J. L. (2010). Interactivity in multimedia learning: An integrated model. Computers in Human Behavior, 26(5), 1024–1033. Duffy, T. M., & Jonassen, D. H. (Eds.). (1992). Constructivism and the technology of instruction: A conversation. Hillsdale, NJ: Lawrence Erlbaum Associates. Golden, C. J. (1978). The Stroop color and word test: A manual for clinical and experimental uses. Chicago: Stoelting Co. Handelsman, J., EbertMay, D., Beichner, R., Bruns, P., Chang, A., DeHaan, R., et al. (2004). Scientific teaching. Science Pulsars, 304, 521–522. Hannafin, M. J., & Peck, K. L. (1988). The design, development and evaluation of instructional software. New York: McMillan Publishing Company. Hmelo-Silver, C. E., Duncan, R. G., & Clark, A. C. (2007). Scaffolding and achievement in problem-based and inquiry learning: A response to Kirschner, Sweller, and Clark (2006). Educational Psychologist, 42, 99–107. Homack, S., & Riccio, C. A. (2004). A meta-analysis of the sensitivity and specificity of the Stroop Color and Word Test with children. Archives of Clinical Neuropsychology, 19, 725–743. Homer, B. D., & Plass, J. L. (2010). Expertise reversal for iconic representations in science visualizations. Instructional Science, 38, 259–276. Homer, B. D., Plass, J. L., & Blake, L. (2008). The effects of video on cognitive load and social presence in computer-based multimedia-learning. Computers in Human Behavior, 24(3), 786–797. Kalet, A., Song, H. S., Schwartz, R., Brenner, J., Ark, T., & Plass, J. L. (2012). Just enough, but not too much interactivity leads to better clinical skills performance after a computer assisted learning module. Medical Teacher, 34(10), 833–839. Kalyuga, S., Ayres, P., Chandler, P., & Sweller, J. (2003). Expertise reversal effect. Educational Psychologist, 38, 23–31. Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75–86. Klahr, D., & Nigam, M. (2004). The equivalence of learning paths in early science instruction: Effects of direct instruction and discovery learning. Psychological Science, 15(10), 661–667. Lee, H., Plass, J. L., & Homer, B. D. (2006). Optimizing cognitive load for learning from computer-based science simulations. Journal of Educational Psychology, 902–913. Lezak, M. D. (Ed.). (2004). Neuropsychological assessment. Oxford University Press. Luria, A. R. (1966). Higher cortical functions in man (2nd ed.). New York: Basic Books (original work published in 1962). MacLeod, C. M. (1991). Half a century of research on the Stroop effect: An integrative review. Psychological Bulletin, 109(2), 163. Mayer, R. E. (2001). Multimedia learning. New York: Cambridge University Press. Mayer, R. E., & Chandler, P. (2001). When learning is just a click away: Does simple user interaction foster deeper understanding of multimedia messages? Journal of Educational Psychology, 93(2), 390. Meltzer, L. (Ed.). (2011). Executive function in education: From theory to practice. New York: Guilford Press. Moreno, R., & Valdez, A. (2005). Cognitive load and learning effects of having students organize pictures and words in multimedia environments: The role of student interactivity and feedback. Educational Technology, Research and Development, 53(3), 35–45. Morgan, A. B., & Lilienfeld, S. O. (2000). A meta-analytic review of the relation between antisocial behavior and neuropsychological measures of executive function. Clinical Psychology Review, 20(1), 113–136. Paas, F. (1992). Training strategies for attaining transfer of problem solving skill in statistics: A cognitive load approach. Journal of Educational Psychology, 84, 429–434. Paas, F., & van Gog, T. (2006). Optimising worked example instruction: Different ways to increase germane cognitive load. Learning and Instruction, 16(2), 87–91. Plass, J. L., Homer, B. D., & Hayward, E. (2009). Design factors for educationally effective animations and simulations. Journal of Computing in Higher Education, 31–61. Plass, J. L., Homer, B. D., Milne, C., Jordan, T., Kalyuga, S., Kim, M., et al. (2009). Design factors for effective science simulations: Representation of information. International Journal of Gaming and Computer-Mediated Simulations, 1(1), 16–35. Plass, J. L., Moreno, R., & Brünken, R. (Eds.). (2010). Cognitive load theory. Cambridge University Press. Schaffer, L. C., & Hannafin, M. J. (1986). The effects of progressive interactivity on learning from interactive video. Educational Communication and Technology Journal, 34(2), 89–96. Schmidt, H. G., Loyens, S. M., Van Gog, T., & Paas, F. (2007). Problem-based learning is compatible with human cognitive architecture: Commentary on Kirschner, Sweller, and Clark (2006). Educational Psychologist, 42(2), 91–97. Schwan, S., & Riempp, R. (2004). The cognitive benefits of interactive videos: Learning to tie nautical knots. Learning and Instruction, 14(3), 293–305. Song, H., Kalet, A. L., & Plass, J. L. (2008). Expertise reversal effects: Can we hurt our learners with poorly designed or targeted web-module instruction? Journal of General Internal Medicine, 23, pp. 301–301. Song, H. S., Nick, M. W., Pusic, M., Sarpel, U., Plass, J. L., & Kalet, A. L. (2014). The cognitive impact of interactive design features for learning complex materials in medical education. Computers and Education, 71, 198–205. Spreen, O., & Strauss, E. (1998). A compendium of neuropsychological tests: Administration, norms, and commentary. Oxford University Press.
B.D. Homer, J.L. Plass / Computers in Human Behavior 36 (2014) 365–375 St Clair-Thompson, H. L., & Gathercole, S. E. (2006). Executive functions and achievements in school: Shifting, updating, inhibition and working memory. The Quarterly Journal of Experimental Psychology, 59, 745–759. Steffe, L., & Gale, J. (Eds.). (1995). Constructivism in education. Hillsdale, NJ: Lawrence Erlbaum Associates Inc. Stroop, J. R. (1935). Studies of interference in serial verbal reactions. Journal of Experimental Psychology, 18, 643–662. Sweller, J. (1999). Instructional design in technical areas. Camberwell, Australia: ACER Press. Sweller, J., & Cooper, G. A. (1985). The use of worked examples as a substitute for problem solving in learning algebra. Cognition and Instruction, 2, 59–89. Tung, F. W., & Deng, Y. S. (2006). Designing social presence in e-learning environments: Testing the effect of interactivity on children. Interactive Learning Environments, 14(3), 251–264. Tuovinen, J. E., & Sweller, J. (1999). A comparison of cognitive load associated with discovery learning and worked examples. Journal of Educational Psychology, 91(2), 334.
375
Valentijn, S. A., Van Boxtel, M. P., Van Hooren, S. A., Bosma, H., Beckers, H. J., Ponds, R. W., et al. (2005). Change in sensory functioning predicts change in cognitive functioning: Results from a 6-year follow-up in the Maastricht Aging Study. Journal of the American Geriatrics Society, 53(3), 374–380. Vollmeyer, R., Burns, B. D., & Holyoak, K. J. (1994). The impact of goal specificity and systematicity of strategies on the acquisition of problem structure. In A. Ram & K. Eiselt (Eds.), Proceedings of the 16th annual meeting of the cognitive science society. Hillsdale, NJ: Lawrence Erlbaum Associates. Yeniad, N., Malda, M., Mesman, J., van IJzendoorn, M. H., & Pieper, S. (2013). Shifting ability predicts math and reading performance in children: A meta-analytical study. Learning and Individual Differences, 23, 1–9. Zelazo, P. D., & Bauer, P. J. (Eds.). (2013). National institutes of health toolbox – Cognition battery (NIH Toolbox CB): Validation for children between 3 and 15 years. SRCD monograph. Ann Arbor, MI: Society for Research in Child Development. Zhu, X., & Simon, H. (1987). Learning mathematics from examples and by doing. Cognition and Instruction, 4, 137–166.
