Exploring Emergent Features of Student Interaction within an ... - MDPI

Multimodal Technologies and Interaction

Article

Exploring Emergent Features of Student Interaction within an Embodied Science Learning Simulation Jina Kang *, Robb Lindgren and James Planey Department of Curriculum and Instruction, University of Illinois at Urbana-Champaign, Champaign, IL 61820, USA; [email protected] (R.L.); [email protected] (J.P.) * Correspondence: [email protected] Received: 22 May 2018; Accepted: 21 June 2018; Published: 2 July 2018

Abstract: Theories of embodied cognition argue that human processes of thinking and reasoning are deeply connected with the actions and perceptions of the body. Recent research suggests that these theories can be successfully applied to the design of learning environments, and new technologies enable multimodal platforms that respond to students’ natural physical activity such as their gestures. This study examines how students engaged with an embodied mixed-reality science learning simulation using advanced gesture recognition techniques to support full-body interaction. The simulation environment acts as a communication platform for students to articulate their understanding of non-linear growth within different science contexts. In particular, this study investigates the different multimodal interaction metrics that were generated as students attempted to make sense of cross-cutting science concepts through using a personalized gesture scheme. Starting with video recordings of students’ full-body gestures, we examined the relationship between these embodied expressions and their subsequent success reasoning about non-linear growth. We report the patterns that we identified, and explicate our findings by detailing a few insightful cases of student interactions. Implications for the design of multimodal interaction technologies and the metrics that were used to investigate different types of students’ interactions while learning are discussed. Keywords: embodied learning; multimodal learning environments; multimodal interaction metrics; science education simulations

1. Introduction Understanding core ideas in STEM (Science, Technology, Engineering and Mathematics) domains and applying them across various content areas is critical, and is a central tenant of the Next Generation Science Standards in the United States (US) [1]. Embodied learning is an approach to educational environment design that focuses on the interplay between the knowledge creation process and learners’ physical interactions with their surroundings [2]. Body-based interactions can play a critical role in the development of both concrete and abstract understandings. With the refinement of body detection and tracking technologies arising out of the entertainment and virtual reality industries, software designed to utilize dynamic and continuous body tracking and gesture recognition can now be leveraged to facilitate embodied learning with platforms such as whole-body interactive science learning simulations. Recently, several authors have outlined basic design principles for effective embodied design [2–5]. These technology-enhanced environments can guide embodied interaction in ways that instigate new learning [6–8]. However, there is still a need to explore specific features that can inform the way that these environments support student learning, and in particular how to analyze and integrate the real-time multimodal data that is collected as students interact with the environment. High-frequency data collection and sensing technologies have the ability to capture massive amounts from multiple modes of interaction simultaneously [9,10]. This complements recent Multimodal Technologies and Interact. 2018, 2, 39; doi:10.3390/mti2030039

www.mdpi.com/journal/mti

Multimodal Technologies and Interact. 2018, 2, 39

2 of 20

perspectives on learning as multimodal (e.g., Ainsworth, Prain, & Tytler [11], Ibrahim-Didi, Hackling, Ramseger, & Sherriff [12]), but more clarity and innovation is needed to discover and articulate effective multimodal metrics within immersive and interactive learning environments. More effective metrics will allow for an increasingly sophisticated understanding of how multimodal environments facilitate student learning and provide valuable feedback on the design of specific features in these environments. In this paper, we explore the relationship between how students engage with ideas about non-linear growth within science contexts in an embodied simulation, and their subsequent reasoning following the use of simulation. We are particularly interested in investigating different multimodal interaction metrics that are generated as students attempt to make sense of the cross-cutting science concepts of scale, proportion, and quantity using a personalized gesture scheme. 1.1. Embodied Learning Environments An expanding foundation of literature on the interplay between the body and cognitive processes lends support to a body-centered, or embodied, approach to learning (e.g., Barsalou [13], Johnson [14]). Embodied cognition emphasizes the importance of the body’s physical interactions with the environment as a central anchor for a wide array of complex cognition including problem-solving, abstraction, and memory formation [15–17]. Recent studies utilizing fMRI (Functional Magnetic Resonance Imaging) imagery have strengthened the connection between mental imagery and physical actions, showing evidence that similar regions of the brain are activated while imagining as well as experiencing (e.g., Tomasino et al. [18]). Leveraging this connection within the framework of embodied learning situates the body as a critical component of the learning process, and emphasizes the importance of designing learning environments that forge and augment the connections between the imagined and the experienced [4]. Advanced technologies in gesture recognition and “mixed reality”—digital environments that embed both physical and virtual objects [19]—create new opportunities to design immersive and interactive educational environments that are responsive to the embodied nature of human learning. Lindgren and Johnson-Glenberg [5] have emphasized the importance of aligning physical actions with concepts and creating meaningful collaborative interaction. Abrahamson and Lindgren [2] have discussed challenges that are relevant to activities, materials, and facilitation for the pedagogical design of embodied environments. They argue, first, that initial activities should focus on relatively simple tasks with straightforward execution within a familiar physical context. Second, these environments should be designed such that tangible connections to the materials in the environment emerge over time through targeted feedback and gradually increasing activity complexity. Third, to facilitate productive embodied actions, learners should be guided or cued toward more expert actions and perceptions, and given time to reflect on how their body is affecting the system. More recently, DeSutter and Stieff [4] outlined three design principles for embodied learning environments, emphasizing the inclusion of instructional scaffolds that clearly connect embodied actions and spatial entities, the leveraging of embodied actions to simulate high-fidelity spatial concepts especially for unseen phenomena, and the link with innovative tools such as visualizations. Along with the established design practices, research has begun to show the benefits of embodied interactions with emerging technologies. For example, Johnson-Glenberg et al. [7] demonstrated the learning benefits of environments that they termed Embodied Mixed Reality Learning Environments (EMRELEs). The particular EMRELE that they investigated was SMALLab, which is a platform that engages students through visual, auditory, and kinesthetic modalities to learn concepts in chemistry. Compared with students in a “low” embodiment condition, students in the SMALLab condition showed higher learning gains in physics. Another study embedded middle school students in a whole-body interactive simulation of planetary astronomy called MEteor that allowed them to learn through enacting their predictions of how objects move in space. Lindgren et al. [8] examined students’ learning and attitudes about science, and found that students in the whole-body intervention showed


3 of 20

higher learning gains and engagement compared with students in a control condition that used a desktop computer version of the same simulation. A growing number of developments in embodied learning have contributed to establishing embodied design strategies and developing learning measures that so far have been significantly varied. In this study, we explore how learning occurs in a science simulation by investigating the different interaction metrics generated as students attempt to make sense of the cross-cutting science concepts of scale, proportion, and quantity. 1.2. Embodiment and Multimodal Learning Gesture has shown to play a pivotal role in how students express and even construct their ideas, with the use of gesture even more critical in content domains that rely heavily on abstract concepts and formalized representations [20–22]. Science education is one such domain, and an embodied and multimodal approach has shown to be valuable in aiding student’s synthesis and explanation of abstract concepts (e.g., Ibrahim-Didi et al. [12]). For example, Crowder [23] investigated the role that gesture played for student explanations in a classroom context on the topic of seasons, and argued that gestures are used not only for students to transmit existing knowledge, but further construct an explanation. Embodied learning draws on and enhances the synthesis of common instructional modalities with salient connections to the learners’ own motoric and perceptual systems to produce strong connections to abstract concepts and metaphors encountered during instruction [24]. One instructional approach that is aligned with the core tenets of embodied learning is multimodality. Multimodal instruction emphasizes the value of effectively integrating representations of content across multiple sensory modalities to facilitate understanding and discourse [25–28]. These modalities can take on many forms, and while classification varies, Prain and Waldrip [29] provided a broad framework: descriptive (verbal, graphic, tabular), experimental, mathematical, figurative (pictorial, analogous, or metaphoric), and kinesthetic (gestural). Early research indicates that an instructor’s verbal and non-verbal immediacy behaviors are positively related to student motivation to learn (e.g., Frymier [30], Richmond, Gorham, & McCroskey [31]). Andersen [32] provided a list of non-verbal immediacy behaviors including eye gaze, smiles, nods, relaxed body posture, forward leans, movement, gestures, vocal variety, and proximity. Recent studies of non-verbal immediacy found that there was a positive effect on students’ perception of learning and level of motivation, but only a small correlation with performed cognitive learning [33–35]. Márquez, Izquierdo, and Espinet [36] highlighted the critical role of the instructor in facilitating the integration of these modalities in a seventh grade science classroom exploring the water cycle: “Whereas speech introduces and identifies the entities, gesture locates them and dynamizes processes. Visual language, through the diagram, provides a scenario, and through arrows, facilitates the establishment of functional mechanisms that are necessary to construct an explanation of water circulation in nature.” (p. 223). Multimodal sense-making through the creation of complex representations has also been shown to enhance learning [27]. Comparative work by Ainsworth and Loizou [37] and Ainsworth and Iacovides [38] found increased learning gains when students were asked to self-explain a concept while transferring across modalities (e.g., drawing a diagram based on text or writing a text based on a diagram). Promoting a student’s self-expressive verbal representation of their learning has also shown to have benefits when combined with visual modalities such as diagrams or written text [39,40]. In this paper, we elaborate on the significance of embodied learning as a resource for sense-making in all forms of interactions that occur in interactive learning environments. We propose that the examination of multimodal interactions is of great relevance to enhance embodied learning experiences and inform the design of embodied learning environments. 1.3. ELASTIC3 S The embodied learning simulation environment that we focus on in this paper is called ELASTIC3 S (Embodied Learning Augmented through Simulation Theaters for Interacting with Cross-Cutting


4 of 20

Multimodal Technol. Interact. 2018, 2, x FOR PEER REVIEW

4 of 20

3 S platform attempts to cultivate understanding of a range of Concepts Science). The ELASTIC a range ofin science topics that are fundamentally connected to the “cross-cutting concepts”, which is science topics are fundamentally connected to the “cross-cutting concepts”, which is one ofinthe one of the corethat components described in the Next Generation Science Standards in (NGSS) thecore US components described in the Next Generation Science Standards in (NGSS) in the US [1]. These seven [1]. These seven concepts were articulated to promote a more cross-content framework, bridging concepts were articulated to promote a more cross-content framework, disciplines STEM disciplines throughout K12 education. The cross-cutting conceptbridging of “scale,STEM proportion, and throughout K12 education. The cross-cutting concept of “scale, proportion, and quantity” was the lens quantity” was the lens employed in the design of a pair of simulations focusing on scale and employed in the design of a pair of simulations focusing on scale and magnitude. Both simulations magnitude. Both simulations shared the same user interface components (graphs showing shared the same interface components (graphson showing logarithmic scales, gesture logarithmic scales,user gesture controls), but one focused earthquakes (the Richter scale) and controls), the other but one focused onand earthquakes (thepH Richter scale) and the other focused on acidity and basicity (the focused on acidity basicity (the scale). Preliminary studies of student interactions with these pH scale). Preliminary studies of student interactions with these simulations have shown promise; simulations have shown promise; students who interacted with the embodied simulations showed students interacted with thetransfer embodied simulations significant concept significantwho learning and concept gains comparedshowed with students wholearning receivedand traditional transfer gains compared students who received materials on theat same educational materials onwith the same topic [41]. In this traditional paper, we educational will be looking specifically the topic [41]. In this paper, we will be looking specifically at the interactions that students had with the interactions that students had with the earthquake simulation as they explored the cross-cutting earthquake as theyand explored thevia cross-cutting of goal scale,isproportion, and quantity concepts of simulation scale, proportion, quantity the Richterconcepts scale. The to better understand the via the Richter scale. The goal is to better understand the kinds of multimodal interactions that students kinds of multimodal interactions that students have with an embodied simulation, and develop have with ancan embodied develop metrics that kinds can inform the specific design features of metrics that inform simulation, the specific and design features of these of environments. theseAll kinds of of theenvironments. simulations developed for the ELASTIC3S platform utilize a flexible multimedia 3 S platform utilize a flexible multimedia All of the simulations for the ELASTIC laboratory space (see Figuredeveloped 1) that utilizes screens, projectors, microphones, video cameras, and laboratory spacesensors (see Figure 1) that utilizes screens, data projectors, microphones, videoare cameras, and body body tracking to facilitate unobtrusive collection while users immersed in a tracking sensors to facilitate unobtrusive data collection while users are immersed in a simulation. simulation.

(a)

(b)

Figure 1. The simulation space configured for motion tracking of a three-screen simulation: (a) ceiling Figure 1. The simulation space configured for motion tracking of a three-screen simulation: (a) ceiling mounted audio and video recording; (b) a student using the earthquake simulation. mounted audio and video recording; (b) a student using the earthquake simulation.

The earthquake simulation is designed in the Unity game engine utilizing the Microsoft Kinect The earthquake simulation is designed in the Unity game engine utilizing the Microsoft Kinect V2 body-tracking sensor (Microsoft Corporation, Seattle, WA, USA). This device allows for V2 body-tracking sensor (Microsoft Corporation, Seattle, WA, USA). This device allows for interaction interaction in three dimensions without the need for additional equipment worn by the student (see in three dimensions without the need for additional equipment worn by the student (see Figure 1b). Figure 1b). To facilitate gestural input, the simulation utilizes a hybrid hierarchical hidden Markov To facilitate gestural input, the simulation utilizes a hybrid hierarchical hidden Markov model model algorithm, which allows for robust gesture tracking and rapid training [42]. algorithm, which allows for robust gesture tracking and rapid training [42]. In the simulation, students are presented with a task that requires them to reach a certain In the simulation, students are presented with a task that requires them to reach a certain numerical numerical quantity using gestural inputs for four mathematical functions (+1, −1, ×10, and ÷10) while quantity using gestural inputs for four mathematical functions (+1, −1, ×10, and ÷10) while being being given as much time as they need to perform the quantitative operations required to reach the given as much time as they need to perform the quantitative operations required to reach the goal. goal. While the gesture recognition system can utilize any sequence of body motions to map each of While the gesture recognition system can utilize any sequence of body motions to map each of the the four mathematical functions, early pilot interviews with participants identified promising four mathematical functions, early pilot interviews with participants identified promising physical physical metaphors that resonated with students’ intuitions about mathematical operations (e.g., metaphors that resonated with students’ intuitions about mathematical operations (e.g., stacking stacking duplicates of a quantity on top of itself to represent multiplication) [43]. So, while duplicates of a quantity on top of itself to represent multiplication) [43]. So, while participants participants had the flexibility to represent them in their own way, they were prompted with the had the flexibility to represent them in their own way, they were prompted with the following following metaphors: “add one” as stacking one cube on top of a pile, “subtract one” as kicking a metaphors: “add one” as stacking one cube on top of a pile, “subtract one” as kicking a cube off the cube off the pile, “multiply by 10” as stacking a column of nine cubes on top of a single cube, and pile, “multiply by 10” as stacking a column of nine cubes on top of a single cube, and “divide by 10” “divide by 10” as splitting one’s legs and extending one leg outward as though a stack is being split as splitting one’s legs and extending one leg outward as though a stack is being split into smaller into smaller groups (see Figure 2). In addition to mapping nicely onto accessible metaphors, this groups (see Figure 2). In addition to mapping nicely onto accessible metaphors, this gesture framing gesture framing had some practical benefit as well, such as mapping each function to a different quadrant of the body, and priming gestures that we knew could be recognized well by Kinect. Before students address the earthquake-specific content, the system is trained to better recognize individual


5 of 20

had some practical benefit as well, such as mapping each function to a different quadrant of the body, 5 of 20 and priming gestures that we knew could be recognized well by Kinect. Before students address the earthquake-specific the system is trained to better individual in degree variations in degreecontent, of movement, body size, and otherrecognize idiosyncrasies of a variations participant. In the of movement, body size, and other idiosyncrasies of a participant. In the “training” phase, “training” phase, students record and practice utilizing their gestures in the context ofstudents simply record and practicenumber utilizingoftheir gestures in the context simply a certain number of cubes, creating a certain cubes, and then they laterof use thesecreating same operations and gestures to and then they later use these same operations and gestures to reach target earthquake magnitudes. reach target earthquake magnitudes. If any recognition errors are encountered during this phase, If any recognition errors within are encountered during this phase,orgestures can be re-played within the gestures can be re-played the simulation for reference, easily re-recorded at any time. When forrecognized, reference, orthe easily re-recorded at any time. When a gesture is recognized, skeletal asimulation gesture is skeletal representation in the game flashes, providing the immediate representation in the game flashes, providing immediate feedback to the student that their feedback to the student that their actions are being detected. The goal in all of the phases is actions to find are being detected.way Thetogoal in the all of the phases is to find the most efficient way to reach thein target the most efficient reach target (e.g., 234 cubes, a Richter scale magnitude of two) any (e.g., 234 cubes, a Richter scale magnitude of two) in any combination of the four gestural operations: combination of the four gestural operations: addition (+1), subtraction (−1), multiplication (×10), and addition(÷10)). (+1), subtraction (−a1), multiplication (×an 10), andindivision (÷10)). If at anythey timecan a student division If at any time student introduces error their gestural sequence, always introduces an error in their gestural sequence, they can always utilize a complementary gesture utilize a complementary gesture (such as ÷10 to counteract and erroneous ×10) to undo an action(such and as ÷ 10 to counteract and erroneous × 10) to undo an action and try again. try again. Multimodal Technol. Interact. 2018, 2, x FOR PEER REVIEW

(a)

(b)

(c)

Figure 2. The multiplication Figure 2. The student student (above) (above) and and skeleton skeleton (below) (below) showed showed the the steps steps of of making making aa multiplication gesture: (a) holding her left hand palm up in front of her stomach and extending her right arm;arm; (b) gesture: (a) holding her left hand palm up in front of her stomach and extending her right making an overhead sweeping motion withwith the right arm;arm; (c) folding the right arm such (b) making an overhead sweeping motion the right (c) folding the right arm that suchher thatright her hand meets her left hand in front of her body. right hand meets her left hand in front of her body.

In the training phase where students are creating cubes, students are initially given In the training phase where students are creating cubes, students are initially given straightforward objectives to gain familiarity with the system (i.e., create five cubes). The tasks in the straightforward objectives to gain familiarity with the system (i.e., create five cubes). The tasks training scenario gradually increase in complexity, culminating in a task that requires multiple in the training scenario gradually increase in complexity, culminating in a task that requires multiple sequenced gestural inputs (i.e., make 431 cubes). Students then transition to the science learning tasks, sequenced gestural inputs (i.e., make 431 cubes). Students then transition to the science learning tasks, where the focal quantity students that manipulate is presented through the lens of an earthquake (the where the focal quantity students that manipulate is presented through the lens of an earthquake (the amplitude of seismic waves, re-imagined as cube-based “amplitude units”). The user interface of the amplitude of seismic waves, re-imagined as cube-based “amplitude units”). The user interface of the simulation is spread across three large projector screens, with each screen attending to a different simulation is spread across three large projector screens, with each screen attending to a different element of the knowledge-building process (see Figure 3). The left screen provides real-time skeletal element of the knowledge-building process (see Figure 3). The left screen provides real-time skeletal tracking feedback to the participant, a Richter magnitude, and an earthquake wave amplitude graph, tracking feedback to the participant, a Richter magnitude, and an earthquake wave amplitude graph, as well as a “base bar” along the bottom. The base bar resizes dynamically with gestural input from as well as a “base bar” along the bottom. The base bar resizes dynamically with gestural input from the participant, and acts as an additional concrete representation of the size of a change initiated by the participant, and acts as an additional concrete representation of the size of a change initiated by a a gesture, working in concert with the amplitude graph. The center screen displays a cross-sectional gesture, working in concert with the amplitude graph. The center screen displays a cross-sectional view of a reverse tectonic fault; when triggered by the interviewer after a participant has reached a view of a reverse tectonic fault; when triggered by the interviewer after a participant has reached a goal amplitude, the plates at the fault line visibly slip and send out visualized ground waves that goal amplitude, the plates at the fault line visibly slip and send out visualized ground waves that spread across the screen. The right screen shows a detailed view of a small city, and acts as a visual spread across the screen. The right screen shows a detailed view of a small city, and acts as a visual representation of the magnitude of an earthquake and the resulting damage caused to the buildings representation of the magnitude of an earthquake and the resulting damage caused to the buildings (shaking and/or falling apart). (shaking and/or falling apart).



6 of 20

6 of 20

Figure 3. Three-screen layout of the simulation showing gesture tracking skeleton and graph Figure 3. Three-screen layout of the simulation showing gesture tracking skeleton and graph visualizations (left), earthquake mechanisms and tasks (center), and earthquake effects (right). visualizations (left), earthquake mechanisms and tasks (center), and earthquake effects (right).

Following the same gesture scheme of the cube-training scenario, students start with a Following task the same gesture scheme of2 the cube-training students with a straightforward (i.e., create a magnitude earthquake, whichscenario, corresponds to 100start amplitude straightforward task (i.e., create a magnitude 2 earthquake, which corresponds to 100 amplitude units, 102). Moving through subsequent tasks, the complexity of both their gestural inputs units, and 2 ). Moving through subsequent tasks, the complexity of both their gestural inputs and conceptual 10 conceptual understandings is progressed as they become more acclimated with the system. For understandings is progressed as task they isbecome more acclimated the to system. For example, example, a particularly challenging when the simulation asks with students set the amplitude of a particularly challenging task is when the simulation asks students to set the amplitude of the the seismic waves to create a magnitude 3.5 earthquake. To solve this task, students must estimate seismic to create magnitude3.5 3.5earthquake earthquake.(approximately To solve this task, students must the the actualwaves amplitude of aamagnitude 3160) based on theestimate graph that actual amplitude of a magnitude 3.5 earthquake (approximately 3160) based on the graph that they they have utilized in the previous two tasks (create a magnitude 2 and 3). The non-linear nature of have utilized in as thevisualized previous through two taskschanges (create in a magnitude 2 and 3). The non-linear nature of the the Richter scale the amplitude graph supports student reasoning Richter as visualized changes in the amplitude graphamplitudes supports student that that the scale amplitude of a 3.5through earthquake is not halfway between 3 and reasoning 4 due to the the amplitude of a 3.5 earthquake is not halfway between amplitudes 3 and 4 due to the exponential exponential nature of the scale. Throughout the entire task set, from a magnitude 2 to a magnitude 8, nature ofare theconfronted scale. Throughout entire tasknature set, from a magnitude 2 to a magnitude 8, students are students with thethe exponential of the Richter scale. confronted with the exponential nature of the Richter scale. This study is particularly interested in the different multimodal interaction metrics that are This as study is particularly interested different multimodal interaction metrics that are generated students attempt to make sensein of the the cross-cutting science concepts of scale, proportion, generated asusing students attempt to make sense of theThe cross-cutting science concepts of scale,the proportion, and quantity a personalized gesture scheme. purpose of the study is to explore diverse and quantity using a personalized gesture scheme. The purpose of the study is to explore the diverse types of interactions that are currently occurring in the learning environment and identify significant types of interactions are currently occurring the relevance learning environment andcollection, identify significant interaction metrics to that understand student learninginand for future data analysis, interaction metrics to understand student learning and relevance for future data collection, analysis, and the design of embodied learning environments. Our specific research questions are: and the design of embodied learning environments. Our specific research questions are: 1. What multimodal interaction metrics are available to capture student behavior as they engage cross-cutting science concepts within embodied learningstudent simulation? 1. with What multimodal interaction metrics arean available to capture behavior as they engage 2. How the multimodal allow students, not allow them, to progress withdoes cross-cutting sciencelearning conceptsenvironment within an embodied learningorsimulation? their does understanding of non-linear 2. in How the multimodal learninggrowth? environment allow students, or not allow them, to progress in their understanding of non-linear growth? 2. Materials and Methods 2. Materials and Methods 2.1. Participants 2.1. Participants Twenty-four undergraduate students (female = 18, male = 6) from a large midwestern university Twenty-four students (female = 18, male 6) these from 24 a large midwestern university participated in the undergraduate earthquake interview and simulation tasks.=Of students, 58% were white, participated in the earthquake interview and simulation tasks. Of these 24 students, 58% were white, 17% were Asian/Asian American, 20% preferred not to answer, and the rest consisted of two or more 17% groups/unreported. were Asian/Asian American, preferred to answer, andwere the rest of two or race All of the 20% interviews andnot simulation tasks bothconsisted audio and video more race groups/unreported. All of the interviews and simulation tasks were both audio and recorded for later analysis. Before participating in a simulation session, a video recorded one-on-one video recorded for later analysis. Before a simulation a video students recorded interview was conducted. Through an participating increasingly in complex series session, of questions, one-on-one interview was conducted. Through an increasingly complex of questions, students communicated their pre-simulation conceptions of earthquakes, theseries Richter scale, and the communicated their pre-simulation conceptions of earthquakes, the Richter scale, and the mathematical mathematical relationship between Richter magnitude and earthquake amplitude. During the relationshipstudents betweenwere Richter magnitude earthquake amplitude. the simulation, students simulation, presented with and a series of tasks guided by anDuring interviewer. After completing were presented with a series of tasks guided by an interviewer. After completing the simulation, the simulation, students were again interviewed and given the same opportunities to communicate students were again interviewed and given the same opportunities to communicate their conceptions. their conceptions. 2.2. Data Sources 2.2.1. Conceptual Measures During the pre-tests and post-tests, students were asked to solve problems requiring them to contrast linear and exponential growth for the earthquake topic. For example, comparing the change


7 of 20

2.2. Data Sources 2.2.1. Conceptual Measures During the pre-tests and post-tests, students were asked to solve problems requiring them to contrast linear and exponential growth for the earthquake topic. For example, comparing the change in amplitude when moving between earthquakes of varying magnitudes (e.g., evaluating the change from a 2.0 to a 5.0 compared to a 5.0 to an 8.0). Then, students were asked to contrast linear and exponential growth in a new context such as: “Which restaurant chain do you think will have the most restaurants in 12 years, assuming Restaurant A plans to open 3000 restaurants every year, and Restaurant B is going to start with one restaurant and then triple the number of restaurants every year?” Students’ answers to the pre-tests and post-tests were scored for correctness and the level of sophistication of their reasoning. Four independent raters scored each student’s understanding of exponential growth during pre-tests and post-tests with an initial 75% agreement. Any disagreements between the raters were resolved through discussion until 100% agreement was reached. 2.2.2. Simulation Metrics To analyze students’ use of the simulation, we segmented the videos according to each of seven selected tasks: two tasks from the training cube simulation—make 234 and 431 cubes (hereafter C234, C431)—and five tasks from the earthquake simulation—obtain a Richter scale of 2.0, 3.0, 3.5, 7.0, and 8.0 (hereafter R2, R3, R3.5, R7, R8). Three researchers coded the segmentation of the simulation use videos in terms of (1) time spent on gesture development (i.e., duration between when a task was given and when a student started gesturing; hereafter GD), (2) time spent on gesturing (i.e., duration between when a student started gesturing and when a target quantity (the number of cubes or the size of seismic activity) was submitted; hereafter G), (3) the number of gestures, and (4) the submitted quantity for each task. In addition, through the series of simulation tasks, students were challenged to not only manipulate their gestural inputs, but also verbally communicate their conceptual understandings. Students’ answers to the conceptual questions during the simulation were scored for correctness of their reasoning. 2.3. Methods We take a sequential explanatory design approach [44] with an initial quantitative phase followed by a qualitative phase. To address the first research question, we conducted Pearson correlation analyses to identify any relationships between the metrics that were generated as students interacted with the simulation. In reporting the results, we followed the guidelines from Evans (1996): an r value of 0.00–0.19 is “very weak”, 0.20–0.39 is “weak”, 0.40–0.59 is “moderate”, 0.60–0.79 is “strong”, and 0.80–1.0 is “very strong”. To address the second research question, we selected four students to more closely examine the ways in which student reasoning seemed to be supported by the embodied learning environment. We employed a multiple case study design [45] to draw out the potential effects of an embodied simulation by comparing the ways that their reasoning did or did not improve as a result of the simulation phase. To avoid biases from interviewer characteristics such as style and experience, four students who were interviewed by the same interviewer were selected for the case analysis [46]. The four students were identified by matching cases based on their exponential knowledge on the pre-test: Alyssa, Gabby, George, and Rhiannon. Alyssa (3) and Gabby (2) scored low on their exponential knowledge pre-tests, while George (12) and Rhiannon (11.5) scored high on their pre-tests among 24 students (M = 9.35, SD = 4.77). The matched cases allowed us to do a more in-depth, qualitative analysis of how the multimodal learning environment could support student reasoning and impact their learning of exponential growth concepts. Each of the four case videos was coded for multiple dimensions of student and interviewer interaction using Datavyu qualitative video analysis software [47]. Guided by the recent emphasis


8 of 20

on the examination of multimodal interactions (e.g., Birchfield et al. [25], Flood et al. [27], Pogue & Ahyun [33]), we initially observed the selected videos in order to characterize the types of multimodality that students experience in the embodied learning environment; that is, verbal and non-verbal interactions. For each of the cases, the verbal interactions between the student and the interviewer were Multimodal Technol. Interact. 2018, 2, x FOR PEER REVIEW 8 of 20 coded. Table 1 shows the sample code dialog. After reviewing the rough transcripts of the videos, verbal interactions were grouped into three query thetasks), student interviewer), clarification (supporting statements fromcategories: the interviewer to(questions the studentsfrom during or and interviewer), clarification (supporting statements the For interviewer to the during utterance (responses or other verbalizations from thefrom student). both “query” andstudents “clarification”, tasks), utterance (responses or other from the student). both and each and verbal interaction was also coded for verbalizations additional context, indicating that theFor code was“query” related to “content” (earthquake or exponential “simulation” design features), “gesture” “clarification”, each verbal interactioncontent), was also coded for (simulation additional context, indicating that the (gestural elements or conventions), or “task” procedures). Student utterances were also contextcode was related to “content” (earthquake or(task exponential content), “simulation” (simulation design coded as“gesture” either answers to an interviewer (answer), statements doubt or uncertainty (uncertainty), features), (gestural elements or conventions), or “task”of(task procedures). Student utterances verbalizing their thought as process verbalized(answer), indicators statements of success (success). were also context-coded either(thinking answersaloud), to anand interviewer of doubt or uncertainty (uncertainty), verbalizing their thought process (thinking aloud), and verbalized indicators Table 1. Coding scheme and sample code dialog. of success (success). Code

Definition

Sample Dialog

Questions fromscheme and sample code dialog. Table 1. Coding

“Now we want to make 300 cubes. How do you think that we interviewer (types: can make 300 with the least number of gestures?” content, simulation, Definition Dialog “Describe the damage caused Sample by this earthquake.” gesture, task) “Now we want to make 300 cubes. How do you Supporting Questionsstatements from interviewer (types: think that we can make 300 with the least number Queries from content, the interviewer to the “So, what areof you trying to do?” simulation, gesture, task) gestures?” Clarifications students during tasks “Do you want“Describe to try to the getdamage to that and we by canthis see?” caused earthquake.” (types:Supporting content, simulation, “Think about what you would start with...” statements from the “So, what are you trying to do?” gesture,totask) Clarifications interviewer the students during tasks “Do you want to try to get to that and we can see?” (types: content, Questions fromsimulation, student gesture, task) “Think about what you would start with...” Queries (types: content, simulation, “I content, should’ve added three, right?” Questions from student (types: Queries “I should’ve added three, right?” simulation, gesture, task) gesture, task) Responses or other “Oh, wait, wait...never mind, I know what I am Responses or other verbalizations from verbalizations from the “Oh, wait, wait...never mind, I know what I am going to do.” going to do.” Utterances the student (types: answer, uncertainty, think that’s it will Utterances student thinking (types: answer, “I think that’s“Iwhere it willwhere damage it.”damage it.” aloud, success) “Which Looks like J?” uncertainty, thinking “Which graph? Looksgraph? like J?” aloud, success) Queries

Code

Interviewer Interviewer

Student Student

Finally, we then coded non-verbal interactions between the student and the interviewer. We were Finally, we then coded non-verbal interactions between the student and the interviewer. We looking for different types of non-verbal interactions (e.g., facial expressions such as smiles, nods, were looking for different types of non-verbal interactions (e.g., facial expressions such as smiles, proximity [32]) during the initial review of the rough transcripts of the videos. This initial analysis nods, proximity [32]) during the initial review of the rough transcripts of the videos. This initial showed that physical proximity of the interviewer to the student was particularly notable and may analysis showed that physical proximity of the interviewer to the student was particularly notable have been related to other student behaviors. Based on this observation, we formalized a coding and may have been related to other student behaviors. Based on this observation, we formalized a scheme forscheme describing physical proximity using a color-coded reference reference video frame shown inshown Figure 4: coding for describing physical proximity using a color-coded video frame “green” (far), “yellow” (mid), or “red” (near). The(near). reference frame zones were determined based on in Figure 4: “green” (far), “yellow” (mid), or “red” The reference frame zones were determined initial video observations and were designed to keep a radius of consistent proximity from left right based on initial video observations and were designed to keep a radius of consistent proximity to from of left the student and distance from the simulation screens. All of the Datavyu codes were exported using to right of the student and distance from the simulation screens. All of the Datavyu codes were video frameusing time video codesframe for further analysis. exported time codes for further analysis.

Figure 4. Reference video frame showing interviewer (left) and student (center) with the color overlay Figure 4. Reference video frame showing interviewer (left) and student (center) with the color overlay used to code proximity. used to code proximity.

3. Results 3.1. Relationship between Embodied Interaction Metrics We investigated several interaction metrics that were generated as students used the embodied simulation such as the time spent on gesture development and time spent on gesturing. During the


9 of 20

3. Results 3.1. Relationship between Embodied Interaction Metrics Multimodal Technol. Interact. 2018, 2, x FOR PEER REVIEW

9 of 20

We investigated several interaction metrics that were generated as students used the embodied simulation such as thedevelopment, time spent on students gesture development and time spent on gesturing. the time spent on gesture may be contemplating their gesture optionsDuring and what time spent on gesture development, students may be contemplating their gesture options and what the the gestures mean, or making a selection. To better understand students’ gestural interactions within gestures mean, or making a selection. To the better understand students’ gesturaltime interactions within this this environment, we first examined overall patterns of students’ spent on gesture environment, we first examined the overall patterns of students’ time spent on gesture development development and gesturing of each task (see Figure 5). The boxplots of two tasks, R2 and R3.5, and gesturing of each task (see time Figureon 5).gesture The boxplots of two tasks, R2differently and R3.5, suggest students suggest that students spent development quite duringthat these tasks spent time on gesture development quite differently during these tasks compared with the other tasks, compared with the other tasks, which can be partly explained by task complexity: R2 as the first task which can be partly explainedand by task as the firsttask taskasofexplained the earthquake of the earthquake simulation, R3.5complexity: as the mostR2 challenging in the simulation, ELASTIC3S 3 and R3.5above. as the This mostischallenging task as explained in the ELASTIC S section above. Thistoisindividual worthy of section worthy of further investigation into the different factors leading further investigation into the different factors leading to individual student gestural time. student gestural time.

Figure 5. Boxplot of students’ time spent on gesture development and the gesturing of each task. Each Figure 5. Boxplot of students’ time spent on gesture development and the gesturing of each task. boxplot includes the information of the minimum, first quartile, median, third quartile, and maximum Each boxplot includes the information of the minimum, first quartile, median, third quartile, and (from bottom). Each circle represents a single student. maximum (from bottom). Each circle represents a single student.

We further examined the correlations of the following variables: time spent on gesture We further thetask. following variables: timecorrelation spent on analyses gesture development, andexamined time spentthe on correlations gesturing onof each The results of Pearson development, and time spent on gesturing on each task. The results of Pearson correlation analyses showed statistically significant relationships between the time spent on gesturing and the time spent showed statistically significant the2).time spent positive on gesturing and the time spent on the gesture development ofrelationships certain tasks between (see Table A strong relationship was found on the gesture development ofspent certain (see Table 2).first A strong positive relationship was found between the variables of time ontasks gesturing on the task of both the cube and earthquake between the variables of time spent on gesturing on the first task of both the cube and earthquake simulation, in that the students who tended to spend longer gestural time on the first challenging simulation, thatsimulation the students whoC234) tended to spend longer gestural timetime on the task of the in cube (i.e., also showed longer gestural on first the challenging first task oftask the of the cube simulation (i.e., C234) also showed longer gestural time on the first task of the earthquake earthquake simulation (i.e., R2), in which students are challenged to apply their knowledge acquired simulation R2),simulation in which students challenged to apply their knowledge acquired during the during the (i.e., training to a neware context. training simulation a new context. Another strongtopositive relationship was found between the time spent on gesturing and gesture Another strong positive relationship found between the time on take gesturing and time gesture development during the same task (i.e., was C431) in that students whospent tended a longer on development during the same task (i.e., C431) in that students who tended take a longer time on gesture development also took a longer time gesturing. This strong relationship suggests the students gesture also took time gesturing. This suggests theastudents seemeddevelopment not to completely plan aa longer strategy of how to reach thestrong target relationship quantity before starting gestural seemed not to completely plan a strategy of how to reach the target quantity before starting gestural input sequence, and instead spent time developing the next gesture in the sequence in thea midst of input sequence, and instead spent time developing the next gesture in the sequence in the midst of the the task. In the context of the task, this relationship is expected, as the emphasis is on students task. In the context of the task, thisofrelationship is expected, as theoperations emphasis isthat on students determining determining their own sequence gesture using different math were needed to reach a target quantity of 431 cubes.


10 of 20

their own sequence of gesture using different math operations that were needed to reach a target Multimodal Technol. Interact. 2018, 2, x FOR PEER REVIEW 10 of 20 quantity of 431 cubes. Table 2.2.Correlations Correlations of time on gesture development and gesturing during of time spentspent on gesture development (GD) and(GD) gesturing (G) during (G) simulation. Table simulation. C234 C234 GDGD G G GDGD 1 1 C234 C234 G 1 1 G 0.155 0.155 GD −0.192 −0.123 GD −0.192 −0.123 C431 C431 G −0.462 −0.028 G −0.462 −0.028 GD 0.140 0.093 R2 GD 0.161 0.140 0.7930.093 ** R2 G G 0.161 0.793 ** GD 0.227 0.443 R3.5 GD −0.190 0.227 0.315 0.443 R3.5 G −0.190 0.087 0.315 GD G −0.128 R7 G GD 0.047 −0.128 0.340 0.087 R7 G 0.047 0.340

C431 C431 GDGD GG

R2 R2 GD

R3.5 R3.5 G G

1 1 0.692 ** 1 0.692 ** 1 −0.040 −0.333 1 −0.040 −0.333 0.143 1 0.567 * 0.077 1 0.567 * 0.077 0.143 1 −0.248 −0.198 0.300 0.306 −0.248 0.251 −0.198 −0.011 0.300 0.306 0.063 0.267 0.063 0.251 −0.011 0.267 −0.269 0.286 −0.083 0.066 −0.115 0.146 0.340 −0.269 −0.082 0.286 − 0.083 0.066 −0.115 −0.082 0.146 0.340 Note: ** p < 0.01, * p < 0.05.

R7R7

GD GD

G

GD GD

GG

1 1 −0.041 −0.006 0.041 0.504 0.006 * 0.504 *

1 1 −0.134 0.238 − 0.134 0.238

1 0.174 1 0.174

1

1

Note: ** p < 0.01, * p < 0.05.

In addition, a few significant relationships were found between the gesture development time In addition, few significant were found between the gesture development time of of a task and the agesturing time ofrelationships the subsequent task. First, the time spent on gesture development aintask gesturing time of the task.aFirst, the time spent on gesture development in the and cubethe training simulation (i.e.,subsequent C431) showed significant positive relationship with the time the cube simulation (i.e., C431)task showed significant withsimulation). the time spent spent ontraining gesturing in the following (i.e., aR2; the first positive task of relationship the earthquake In on gesturing in the following (i.e., development R2; the first task of the earthquake simulation). addition, addition, for the time spent ontask gesture in the most challenging task of the In earthquake for the time spent gesture development in the was most found challenging of thespent earthquake simulation, simulation, R3.5, on a weak positive relationship with task the time on gesturing the R3.5, a weak positive relationship found with the time took spentlonger on gesturing the following task, following task, R7. That is, studentswas whose thinking process before gesturing tended to R7. That is, time students whose process took longer understand before gesturing tended to spend more spend more gesturing in thinking the following task. To further the relationship between the time gesturing in the gestural following task. further understand the relationship between the patterns patterns of students’ time andTotheir learning gains on exponential growth knowledge was of students’ and their most learning gains on exponential growth waspositioned indicated indicated in gestural Figure 6.time Interestingly, of the students who showed a lowknowledge gain score are in Figure 6. Interestingly, the students who showed low gainspent score less are positioned in the the bottom left of the most plot,ofwhich indicated that these astudents time on gesture bottom left of the which indicated that these students spent less is time on gesture development andplot, gesturing in these tasks. Further investigation needed to seedevelopment any causal and gesturing in theseembodied tasks. Further investigation needed to see metrics any causal between relationship between expressions such asisthe time spent andrelationship learning gains in the embodied expressions such as the time spent metrics and learning gains in the future study. future study.

Figure 6. Scatterplot Scatterplot of of correlation correlation between between the the ‘time ‘time spent’ spent’ variables variables of of GD GD on on C431 C431 and and G G on on R2. R2. Black circles students with low low exponential growth knowledge gain, and greyand circles represent circlesrepresent represent students with exponential growth knowledge gain, grey circles high gain students. represent high gain students.

The combined gesture development and gesturing time of each task was also examined to see the relationship with the number of gestures of each task. Noticeably, a significant positive relationship was found between the number of gestures of C234 and the combined times of the following tasks: a moderate correlation with C431 (r = 0.680, n = 23, p < 0.01), a weak correlation with R3 (r = 0.533, n = 23, p < 0.05), and a moderate correlation with R8 (r = 0.668, n = 23, p < 0.01). Since C234 is the first challenging task, the number of gestures can be an indicator of an individual student’s


11 of 20

The combined gesture development and gesturing time of each task was also examined to see the relationship with the number of gestures of each task. Noticeably, a significant positive relationship was found between the number of gestures of C234 and the combined times of the following tasks: a moderate correlation with C431 (r = 0.680, n = 23, p < 0.01), a weak correlation with R3 (r = 0.533, n = 23, p < 0.05), and a moderate correlation with R8 (r = 0.668, n = 23, p < 0.01). Since C234 is the first challenging task, the number of gestures can be an indicator of an individual student’s gestural strategy in that a lower number suggests a more efficient strategy, and vice versa. The results indicate that students who tended to make more gestures to get the target quantity (i.e., 234 cubes) had a tendency of taking more time on C431 in that they had trouble developing their own gestural strategy. The tasks in the earthquake simulation require students not only to decide the sequence of gesture using four different math operations, but also understand the relationship between a Richter magnitude and the amplitude of earthquake. In particular, the tasks of R3 and R8 are relatively straightforward. Therefore, students who made more gestures during C234 seemed to take a longer time to complete gestures during these two tasks and also have difficulties understanding how the Richter scale works overall. An accuracy score of each task was calculated based on a submitted quantity of each task: A=e

−(

Target Quantity− Actual Quantity ) Target Quantity

(1)

For example, while the correct answer for the amplitude of an earthquake that has a Richter scale magnitude of 3.5 is 3610 (i.e., target quantity), a student can submit what they think is the closest amplitude (i.e., actual quantity). The accuracy of the students’ submitted score on R3.5 (i.e., a Richter magnitude of 3.5) showed a significant correlation with their exponential growth knowledge communicated during the simulation (r = 0.543, n = 23, p < 0.01), indicating that R3.5 was a key indicator of students’ conceptual understanding. 3.2. Descriptive Cases While the descriptive and correlation analyses showed overall patterns for students who participated in the embodied simulation condition, we were interested in more closely examining the ways in which students’ reasoning seemed to be supported by the multimodal learning environment. To achieve this, we selected four students to look at in more depth based on their exponential knowledge on the pre-tests: Alyssa, Gabby, George, and Rhiannon. The matched cases allowed us to do a more in-depth, qualitative analysis of how the multimodal learning environment could support student reasoning and impact their learning gains on exponential growth knowledge. 3.2.1. Gestural Time Spent The time spent on gesture development and gesturing for each of the case students was compared with the average for the population via a normalized sigmoid function: 1

τ= 1+e

Average Timespent− Actual Timespent −( ) Average Timespent

(2)

For example, a rate close to 0 indicated a longer time spent compared with the average of all of the students, a rate close to 1 indicated a shorter time spent, and a rate close to 0.5 indicated a similar time spent with the average. After examining four students’ different time spent trends using the sigmoid function, the combined gesture development and gesturing time was shown to vary across these students (see Figure 7).


12 of 20


12 of 20


12 of 20

Figure 7. Normalized sigmoid gesture time spent rate for Alyssa, Gabby, George, and Rhiannon. A score Figure of 0.5 represents thesigmoid ‘time spent’ performance at the for theGeorge, group (N 24). 7. Normalized gesture time spent rate for average Alyssa, Gabby, and=Rhiannon. A Figure 7. Normalized sigmoid gesture time spent rate for Alyssa, Gabby, George, and Rhiannon. score of 0.5 represents the ‘time spent’ performance at the average for the group (N = 24). A score of 0.5 represents the ‘time spent’ performance at the average for the group (Nthat = 24). This high-level analysis highlights the gestural ‘time spent’ outliers invite further

This high-level analysis highlights the ‘time spent’ outliers that similar invite further exploration with additional data modalities. All gestural four students showed an overall pattern of exploration with additional data modalities. All four students showed anthat overall similar of This high-level analysis highlights the gestural ‘time spent’ outliers invite further exploration gestural time. The shared time spent patterns across tasks indicated that more time waspattern spent on R2 gestural time. The shared time spent patterns across tasks indicated that more time was spent on R2 withless additional modalities. All four students showed overall similar of gestural time. and time wasdata spent on R3.5. This pattern may have beenaninfluenced by thepattern particular interviewer and less time was spent on R3.5. This pattern may have been influenced by the particular interviewer The shared spent who patterns across tasks indicated thattomore time wastime spent on R2 and the lessmost time in these fourtime sessions, tended to allow the students spend more developing in these four sessions, who tended to allow the students to spend more time developing the most was spentgestural on R3.5.strategies. This pattern may have been influenced bythere the particular indecreased these four effective Outside of this general pattern, were twointerviewer instances of effective gestural strategies. Outside of this general pattern, there were two instances of decreased sessions, tended to the students toan spend moreinput time sequence), developing the most effective gestural time spent: Gabby on C431 (a task involving input sequence), and Alyssa on(a R2short (a short timewho spent: Gabby onallow C431 (a task involving anextended extended and Alyssa on R2 strategies. Outside of this general pattern, there were two instances of decreased time spent: Gabby sequence task and the transition to the earthquake content). This finding will be discussed below in sequence task and the transition to the earthquake content). This finding will be discussed below in on C431 (a task involving an extended input sequence), and Alyssa on R2 (a short sequence task and the Section 3.2.4.3.2.4. Section transition to the earthquake content). This finding will be discussed below in Section 3.2.4. Proximity between a Student and Interviewer 3.2.2. 3.2.2. Proximity between a Student and Interviewer 3.2.2. Proximity between a Student and Interviewer the coded proximity times,distributions distributions of of the to the student UsingUsing the coded proximity times, the interviewer’s interviewer’sproximity proximity to the student were generated using the total video frames spent in each zone of the reference frame (seeto Figure 4). Using the coded proximity times, distributions of the interviewer’s proximity student were generated using the total video frames spent in each zone of the reference frame (seethe Figure 4). As shown in Figure 8, while the type of tasks performed by each student was identical, the were generated using the total video frames spent in each zone of the reference frame (see Figure 4). As shown in Figure 8, while the type of tasks performed by each student was identical, the interviewer leveraged proximity to varying degrees among the students. When compared with the As shown in Figure 8, while the type of tasks performed by each student was identical, the interviewer interviewer leveraged proximity to varying degrees among the students. When compared with the student’s degree of growth on the exponential concept knowledge from the pre-test to the post-test, leverageddegree proximity to varying degrees among the students. When compared with the the post-test, student’s student’s of growth exponential concept knowledge from thealso pre-test the students that showedon thethe least amount of exponential concept gain were those to that had the degree of growth on the exponential concept knowledge from the pre-test to the post-test, the students the students thatinshowed the for least amount of exponential thatBoth had the interviewer proximity a larger portion of the totalconcept session gain time: were Alyssaalso andthose George. that showed the least amount ofalarger exponential concept gain were alsoproximity those hadcompared the interviewer in interviewer in proximity of the total session time: that Alyssa and George. students’ sessions alsofor hada higherportion proportion of mid and near time withBoth proximity for a larger of the total session time: Alyssa and proximity George. students’ sessions Gabby and Rhiannon. The time spent in near to eachBoth student is different students’ sessions alsoportion had a interviewer’s higher proportion of mid andproximity near time compared with also had a higher proportion of mid and near proximity time compared with Gabby and Rhiannon. between these four students; specifically, two students with a lower conceptual knowledge gain drew Gabby and Rhiannon. The interviewer’s time spent in near proximity to each student is different the interviewer’s attention most oftwo the task. This student is expected in conceptual the type of interviewer–student The interviewer’s spent specifically, inover near proximity to each is different between these four students; between these fourtime students; students with a lower knowledge gain drew relationship that was utilized in the sessions, with the interviewer providing conceptual as well as specifically, two students with a lower conceptual knowledge gain drew the interviewer’s attention the interviewer’s attention over most of the task. This is expected in the type of interviewer–student procedural over most ofthat theclarifications. task. This is expected in the type of interviewer–student relationship that was utilized relationship was utilized in the sessions, with the interviewer providing conceptual as well as in the sessions, with the interviewer providing conceptual as well as procedural clarifications. procedural clarifications.

(a)

(b)

(a)

(b) Figure 8. Cont.


13 of 20

Multimodal Technol. Interact. 2018, 2, x FOR PEER REVIEW Multimodal Technol. Interact. 2018, 2, x FOR PEER REVIEW

13 of 20 13 of 20

(c) (c)

(d) (d)

Figure 8. 8. Interviewer Interviewer proximity proximity distributions distributions during during seven seven tasks: tasks: (a) (a) Alyssa; Alyssa; (b) (b) Gabby; Gabby; (c) (c) George; George; Figure Figure 8. Interviewer proximity distributions during seven tasks: (a) Alyssa; (b) Gabby; (c) George; (d) Rhiannon. (d) Rhiannon. (d) Rhiannon.

3.2.3. Verbal Verbal Discourse Discourse 3.2.3. Verbal 3.2.3. Discourse Toexamine examinepotential potentialrelationships relationshipsbetween betweenthe thetotal totalamount amount of of interviewer interviewer support support and and student student To To examine potential relationships between the total amount of interviewer support and student learning gain, clarifying statements by the interviewer and utterances by the student were tallied. learning gain, gain, clarifying clarifying statements statements by by the the interviewer interviewer and and utterances utterances by the the student student were were tallied. tallied. learning The distribution of clarifications and utterances shown in Figure 9 by showed several noticeable The distribution of clarifications and utterances shown in Figure 9 showed several noticeable patterns. The distribution of clarifications utterances in Figuresessions 9 showed noticeable patterns. While both Alyssa and and Gabby entered shown the simulation withseveral low exponential While both Alyssa and Gabby entered the simulation sessions with low exponential concepts scores, patterns. While only bothGabby Alyssashowed and Gabby the simulation sessions Compared with low with exponential concepts scores, growthentered after completing the simulation. Alyssa, only Gabby showed growth after completing the simulation. Compared with Alyssa, Gabby had a concepts scores, only Gabby growth after completing Compared with Alyssa, Gabby had a greater numbershowed of clarification interactions withthe thesimulation. interviewer, which were primarily greater had number of clarification with the interviewer, which were primarily clarifications of Gabby a greater number task ofinteractions clarification interactions with Gabby the interviewer, which primarily clarifications of the specific and the simulation layout. also showed an were increase in the the specific task and the simulation layout. Gabby also showed an increase in the amount of “thinking clarifications of the specific taskinstances and the simulation layout.with Gabby also showed an increase in and the amount of “thinking aloud” when compared Alyssa. In contrast, George aloud” instances whenaloud” compared with Alyssa. In contrast, George and Rhiannon showed a similar amount of “thinking instances when compared with Alyssa. In contrast, George and Rhiannon showed a similar interviewer clarification distribution to Alyssa, but when examining their interviewer clarification distribution to Alyssa, but when examining their utterance theretheir was Rhiannon a similar interviewer clarification distribution to Alyssa, but whencodes, examining utterance showed codes, there was more diversity in their verbal communication. more diversity in their verbal communication. utterance codes, there was more diversity in their verbal communication.

(a) (a)

(b) (b)

Figure 9. Total frequencies of interaction codes: (a) interviewer clarification; (b) student utterances. Figure 9. Total frequencies of interaction codes: (a) interviewer clarification; (b) student utterances. Figure 9. Total frequencies of interaction codes: (a) interviewer clarification; (b) student utterances.

3.2.4. Learning as Multimodal: Alyssa versus Gabby 3.2.4. Learning as Multimodal: Alyssa versus Gabby 3.2.4. Using Learning Multimodal: Alyssametrics versus to Gabby the as gesture performance aim the spotlight at specific tasks for more in-depth Using the gesture performance metrics to aim the at specific tasks for more in-depth analysis, both Alyssa and Gabby were highlighted forspotlight their decreased gestural performance (i.e., Using the gesture performance metrics to aim the spotlight at specific tasks for more in-depth analysis, both Alyssa and Gabby were highlighted for their decreased gestural performance (i.e., longer gestural time and spent) on single tasks (see Figure In particular, these students represented analysis, both Alyssa Gabby were highlighted for their7).decreased gestural performance (i.e., longer longer gestural time spent) onconceptual single tasks (see Figure 7). In particular, these students represented different levels of exponential gain (M conceptual_gain = 4.41, SD = 3.72, N = 23). Conceptual gain gestural time spent) on single tasks (see Figure 7). In particular, these students represented different different levels of exponential conceptual gain (Mconceptual_gain = 4.41, SD = 3.72,score. N = 23). Conceptual gain scores were calculated by subtracting the pre-test score from the post-test While two students levels of exponential conceptual gain (Mconceptual_gain = 4.41, SD = 3.72, N = 23). Conceptual gain scores were calculated by subtracting the pre-test score from the gain—George post-test score.(conceptual While two students in the were high calculated pre-test group showed a the low to modest gain = in 2) scores by subtracting pre-test scoreconceptual from the post-test score. While two students in the high pre-test group showed a low to modest conceptual gain—George (conceptual gain = 2) and Rhiannon (conceptual gain = 4.5)—Alyssa and Gabby showed differences in their conceptual the high pre-test group showed a low to modest conceptual gain—George (conceptual gain = 2) and and (conceptual = 4.5)—Alyssa and Gabby showed differences in their conceptual gain.Rhiannon Alyssa scored low (3)gain her exponential knowledge pre-test, and showed no growth after Rhiannon (conceptual gain =on 4.5)—Alyssa and Gabby showed differences in their conceptual gain. gain. Alyssawith scored low (3) on her exponential knowledge pre-test, and showed no growth after interacting the simulation, scoring a 3 on the post-test (conceptual gain = 0). Gabby also scored Alyssa scored low (3) on her exponential knowledge pre-test, and showed no growth after interacting interacting with the with simulation, scoring a learning 3 on the gains post-test (conceptual gain =a 0). Gabby also scored low on pre-test a 2, abut showed in her post-test with score of 8 (conceptual with theher simulation, scoring 3 on the post-test (conceptual gain = 0). Gabby also scored low on her low her pre-test with a 2, but showed learning gains in her post-test with a score of 8 (conceptual gainon = 6). pre-test with a 2, but showed learning gains in her post-test with a score of 8 (conceptual gain = 6). gain = 6).


14 of 20

From the coded verbal interaction for Gabby during task C431, several patterns emerged. C431 asked the student to reach a quantity of 431 cubes using their gestures for math operations. While the previous task was presented with the sequence laid out by the interviewer, in this task, the student had to decide the sequence of gestures needed to reach the target quantity on their own. Gabby needed more task-centered clarifications from the interviewer. Her utterances were also task-centric and included several instances of uncertainty and thinking aloud. This pattern of utterances was not seen again until R3.5, which was another long-sequence task, suggesting that she was struggling with planning out her gestural sequences, but showed improvement by R3.5. The interviewer did not utilize proximity to a significant degree during Gabby’s session (each task had the interviewer in proximity less than 35% of the time, and C431 was less than 20%). Table 3 shows the step-by-step clarification support that the interviewer provided Gabby as she worked through the gestural sequence that was needed to reach 431 cubes. It is possible that this level of interviewer clarification resulted from her “uncertain” and “thinking aloud” verbalizations, acting as cues to the interviewer to provide more direct support and contributing to Gabby’s learning gains despite her struggles with the task sequencing. Alyssa’s R2 performance showed a different trend. R2 in this simulation represents a content shift from the cubes simulation (where the goal is to reach a specific numerical goal) to the earthquake simulation. The task of making an earthquake of magnitude 2 is procedurally simple; it only requires two multiplication gestures from the student. However, it might be challenging for some students to interact with the simulation by applying their understanding of the non-linear relationship between a Richter scale and the amplitude of an earthquake. Alyssa showed the highest number of interviewer clarifications (the most for any task during her session) as well as a significant portion (60%) of the task performed while the interviewer was in at least “green” (far) proximity. However, while Alyssa frequently provided answers to interviewer clarification, she showed few verbalizations of uncertainty or instances of thinking aloud. This hints at a larger trend for Alyssa, as across all four case students, she received the most clarifications during R2, as well as multiple instances of gestural clarifications (reminders of how to interact or what can be communicated via the gestures). This showed a potential combination of both difficulty transitioning to the earthquake content, which was perhaps exacerbated by her struggles to interact gesturally with the system, and her lower rate of thinking aloud. Table 3. C431 dialogue of a student and interviewer. Interviewer

Gabby

Think about what you would start with...

Uh...

I can help you

Um...

Ok, so you’ve got four. So, what are you adding to get to?

[mumbling]...400, and then...31 [mumbling then begins using add gesture] 31

Can you think of a way that might be quicker?

Yeah... there is probably a faster way.

I can give you a hint... What’s a quick way you can get to 403? Here, we can do it together.

Um...

Step one, I’d say make four cubes, right?

Yeah [gestures to add four]

Now, the next number we need is three... Right?

Multiply by 10 then add [gestures times 10]

So, what do we have to do to get to 43 now?

[gestures to add three]

And then... now we need to get to 431. So, what can we do to get to 430?

Multiply by 10?

Ok, yeah... Give that a shot.

[gestures times 10] And then one... [gestures to add one]

How are you doing?

Good


15 of 20

4. Discussion 4.1. Multimodal Interaction Metrics The results of correlation analyses showed students’ gestural interaction patterns particularly during certain tasks of the simulations (e.g., C431, R2) and the significant relationship between students’ gestural performances of these tasks. A tendency to spend more time gesturing does not always indicate lower performance, suggesting the need to prime students to spend an adequate amount of time constructing and reflecting upon their gestures, particularly during more challenging tasks. Another critical indicator of gestural interaction is the number of gestures performed during a training task, which was in this case C234. This emphasizes the importance of an interviewer or other agent guiding students and confirming their understanding of the meaning of their gestural actions in the early stages of the simulation experience. The significant correlation between the students’ accuracy score of R3.5 and their exponential growth knowledge communicated during the simulation demonstrated that their performance on the most challenging task was a good indication of their overall conceptual understanding. These initial findings suggest possible critical metrics that represent archetypal gestural interactions that could act as indicators of behavior requiring just-in-time feedback. In these cases, real-time feedback may allow a student to take new actions, interpret the system in a different way, and therefore better understand cross-cutting concepts in science [2]. The analysis of four cases showed that the simulation environment acts as a communication platform for students to combine multiple representations to articulate their understanding of non-linear growth within a science context. Several patterns of interaction within the simulation platform were visible that would have remained unaddressed without the close examination of the communication between a student and interviewer recorded in videos, which emphasized the need for exploration with different data modalities [9,10]. Given that most of the session events involving the close proximity of student and researcher showed low conceptual gain, a lower number of close proximity events seemed to indicate a greater learning gain. The protocol followed by interviewers contained no language regarding how to position oneself relative to the student during a simulation session, and besides being conscious of not getting too close to the Kinect sensor, this issue was not discussed by the research team. Questioning the interviewers as to their conscious use of proximity and its assumed effects may shed light on the patterns that have been observed here. Nonetheless, we can at least infer that non-verbal immediacy such as proximity does not always result in higher learning gain. Previous research investigating non-verbal immediacy found inconclusive evidence of its connection to learning gains, but some support for affective learning [33–35]. Therefore, physical proximity may be perceived as the norm and less related to student learning in this laboratory environment. Additional forms of non-verbal interactions need to be investigated in order to understand the relationship with student learning. In addition, the frequency of students’ verbalized uncertainty and acts of “thinking out loud” were more present with the students that entered the simulation with a high conceptual score or showed significant gains during simulation use. While students were prompted early in the interview to think aloud as much as possible, additional protocol reminders to consistently encourage thinking aloud were not explicit. This is consistent with the literature showing that a well-defined think-aloud protocol can lead to increased understanding between an instructor and a student, as well as elicit greater insights into the cognitive processes of learners as they interact with a system [48–50]. The analysis of the four selected cases showed that student-initiated verbalizations such as uncertainty or thinking out loud are important factors that are associated with the interviewer’s level of clarification and eventually with the students’ learning gain. This contributes to existing research showing the importance of facilitating students to articulate their gestural interaction strategies as an embodied design guidance [2]. However, this small number of cases is limited to posit their causal relationship. For instance, the physical proximity between the student and the interviewer did not show effects on learning gains; however, it may play a critical role for student engagement.

Multimodal Technologies Technol. Interact. 2018, 2,2018, x FOR REVIEW Multimodal and Interact. 2, PEER 39

16 of of 20 20 16

research is therefore needed on the effects of verbal discourse and non-verbal immediacy in More research is therefore needed on the effects of verbal discourse and non-verbal immediacy in multimodal learning environments. multimodal learning environments. In summary, the correlation analyses of interaction metrics showed overall trends of students’ In summary, thesignificant correlationrelationships analyses of interaction metrics showed trends ofof students’ simulation use and between students’ gesturaloverall performances certain simulation use and significant relationships students’ gestural performances of certain simulation tasks. However, the metrics such asbetween time spent on gesture development or gesturing may simulation tasks. However, the metrics such as time spent on gesture development or gesturing may imply students’ diverse thinking processes such as preparing gestures, reflecting their knowledge, or imply students’ diverse thinking processes such as preparing gestures, reflecting their knowledge, or communicating with an interviewer. Therefore, we closely examined how students’ reasoning communicating with an interviewer. Therefore, we closely examined how students’ reasoning seemed seemed to be supported by the multimodal learning environment using the four selected cases. One to be supported bythat the during multimodal learning environment using theafour selected One notable notable finding is gesture development and gesturing, student was cases. not only preparing finding is that during gesture development and gesturing, a student was not only preparing gestures, gestures, but also interacting with the interviewer in a variety of ways. The findings highlight the but also interacting with the interviewer in a variety of ways. The findings highlight the importance importance of multimodal data to understand how learning occurs in a complex learning 3S, and of multimodalsuch dataastoELASTIC understand how learning in types a complex learning such for as environment inform the occurs different of support thatenvironment might be needed 3 ELASTIC S, and informWe thehave different typesbegun of support that might be needed an that individual student. an individual student. currently collecting rich software logfor data was generated We currently begun rich software logonly datarecorded that wasvideo generated as a student interacts as ahave student interacts withcollecting the simulation. Since we and audio at the beginning with the simulation. Since recordedthe video and audio at the the believe data collection, of the data collection, we we didonly not include students’ log data inbeginning this study.ofWe that the we did not include the students’ log data in this study. We believe that the exploration of different exploration of different metrics using the video recordings informs what types of actions to record to metrics informs what types of actions record tohow save to thedesign time and cost for save theusing timethe andvideo cost recordings for data management and analysis, and to ultimately automated data management and analysis, and ultimately how to design automated feedback to support student feedback to support student learning in an embodied simulation environment. learning in an embodied simulation environment. 4.2. Implications for the Design of Multimodal Learning Environment 4.2. Implications for the Design of Multimodal Learning Environment The findings from this study revealed that the learning occurred was not confined to a single The findings this study revealed that the occurred was notrather, confined to a spread single factor such as thefrom characteristics and behavior oflearning an individual student; it was factor such as the characteristics and behavior of an individual student; rather, it was spread multimodally across a student, an interviewer, and the simulations in the environment. In particular, multimodally a student, an interviewer, and the simulations the environment. In particular, 3S environment the ELASTICacross consists of multimodal learning in components including different 3 S environment consists of multimodal learning components including different elements the ELASTIC elements of the knowledge-building process embedded in the simulations, a student’s gestural of the knowledge-building process embedded in the simulations, a student’s gesturaland interaction with interaction with the simulations, and communication between a student interviewer. the simulations, and communication between a student and interviewer. Interpreting the overall Interpreting the overall results, we believe we have created a platform that will allow us to results, we believe we have created learning a platform will allow us to successfully examine multimodal successfully examine multimodal inthat an environment utilizing different components and learning in an environment utilizing different components and technologies (see Figure 10). technologies (see Figure 10).

Figure Figure 10. 10. Multimodal Multimodal learning learning environment environment and and components. components.

The model model has has the the potential potential to to inform inform the the future future design design and and study study of of multimodal multimodal embodied embodied The simulation environments. environments. Recent Recent research research emphasized emphasized the the capability capability of of capturing capturing massive massive amounts amounts simulation of data in different fields of human activity in three-dimensional (3D) space and the feasibility of of data in different fields of human activity in three-dimensional (3D) space and the feasibility of novel assessment assessment by by incorporating incorporating multimodal multimodal interaction interaction data data [7,51]. [7,51]. The The practical practical significance significance of of novel this study study is is that that itit conveys conveys that that there there is is value value to to investigating investigating ways ways of of understanding understanding learning learning in in this this this dynamic environment environment and and exploring exploring different different metrics metrics generated generated as as students students attempt attempt to to make make sense sense dynamic of the cross-cutting science concepts of scale, proportion, and quantity using a personalized gesture


17 of 20

of the cross-cutting science concepts of scale, proportion, and quantity using a personalized gesture scheme. The multimodal interaction metrics that were investigated in this study can be utilized to design powerful forms of feedback and automated prompts to further enhance student learning and engagement. For example, the number of gestures tracked during the early stages of a simulation or the time spent on gesture and gesture development during any challenging task can provide hints as to whether a student is in need of additional support. Different types of visuals such as a replay of skeletal movements and graphs recorded in any previous stage can be added to support the thinking process of students who are struggling with reasoning or gestures. This suggests that such experiences can better facilitate student understanding when they are provided as explicit just-in-time guidance or personalized instruction based on their tracked progress. The findings also suggest there may be additional metrics that indicate students’ unique interaction behavior and measure their learning or engagement. Future studies will include additional groups to match log data with recorded videos using the metrics that emerged in this study. Additional data such as the volume or speed of student movement generated from Kinect will be included to investigate the role of gestures in learning in our future study. The massive amounts of dynamic data recorded when students can generate personalized artifacts could be leveraged to improve an individual’s learning experience and system performance. The ELASTIC3 S environment is in the laboratory space, which is not a typical learning environment in either a formal classroom or informal setting. However, advanced interaction technologies are now more accessible, and therefore promise to reach more populations of learners in diverse environments. Ultimately, our attempts will provide automated guidance to embed within our simulation theater and discover potential features driven from multimodal data to provide a rigid multimodal analytics framework for embodied simulations for future researchers. 5. Study Limitations Students’ participation was entire voluntarily, and the participants were randomly assigned to each group: a simulation group and traditional material group. This study only included the participants in the simulation group, in which female participants accounted for 75% of the simulation group. We did not collect the participants’ academic background such as majors and courses they have taken, but these students were recruited from a general educational psychology course in which the majority of students did not have a STEM focus. Readers are cautioned to keep these limitations in mind. 6. Conclusions Previous studies have shown that the ELASTIC3 S simulation has a positive impact on students’ understanding of cross-cutting concepts and content objectives, and suggested the needs of further investigation on critical features that facilitate student reasoning. This study explored metrics that represent multimodal interactions within the simulation platform to further understand how this multimodal learning environment supports student reasoning of exponential growth. The findings highlight the value of a multimodal approach to investigating student learning in an embodied environment. While traditional assessments are valuable metrics for gauging overall learning, relying only on assessments that occur before and after the interaction with the learning environment limits the ability to draw more nuanced conclusions about what is contributing to the change in learner performance. The multimodal data collected in this study highlights the importance of considering different dynamics such as the interviewer–student dynamic, especially in a system emphasizing physicality. The results of this study pave the way for further studies on the potential of multimodal analytics research frameworks and the design of embodied simulations in the future. Author Contributions: All authors contributed to the conceptualization of the framework and the design of the ELASTIC3 S platform. J.K. and J.P. analyzed the data and wrote the first draft. R.L. revised the manuscript and added relevant literature and interpretations of the analysis.


18 of 20

Funding: This research was funded by the National Science Foundation under Grant No. IIS-1441563. Acknowledgments: We thank the entire ELASTIC3 S team for their help with design, development, and data collections. We are especially grateful to Greg Kohlburn who gave advice regarding simulation metrics and Stephanie Sroczynski who assisted the analysis of recorded simulation sessions. Conflicts of Interest: The authors declare no conflict of interest.

References 1. 2. 3.

4.

5. 6. 7.

8. 9. 10. 11. 12.

13. 14. 15. 16. 17. 18.

19. 20. 21.

NGSS Lead States. Next Generation Science Standards; The National Academies Press: Washington, DC, USA, 2013. Abrahamson, D.; Lindrgen, R. Embodiment and embodied design. In the Cambridge Handbook of the Learning Sciences, 2nd ed.; Sawyer, R.K., Ed.; Cambridge University Press: Cambridge, UK, 2014; pp. 358–376. Black, J.B.; Segal, A.; Vitale, J.; Fadjo, C.L. Embodied cognition and learning environment design. In Theoretical Foundations of Learning Environments, 2nd ed.; Jonassen, D., Land, S., Eds.; Routledge, Taylor & Francis Group: Florence, SC, USA, 2012; pp. 198–223. DeSutter, D.; Stieff, M. Teaching students to think spatially through embodied actions: Design principles for learning environments in science, technology, engineering, and mathematics. Cogn. Res. 2017. [CrossRef] [PubMed] Lindgren, R.; Johnson-Glenberg, M. Emboldened by embodiment six precepts for research on embodied learning and mixed reality. Educ. Res. 2013, 42, 445–452. [CrossRef] Enyedy, N.; Danish, J.A.; Delacruz, G.; Kumar, M. Learning physics through play in an augmented reality environment. Int. J. Comput. Support. Collab. Learn. 2012, 7, 347–378. [CrossRef] Johnson-Glenberg, M.C.; Birchfield, D.A.; Tolentino, L.; Koziupa, T. Collaborative embodied learning in mixed reality motion-capture environments: Two science studies. J. Educ. Psychol. 2014, 106, 86–104. [CrossRef] Lindgren, R.; Tscholl, M.; Wang, S.; Johnson, E. Enhancing learning and engagement through embodied interaction within a mixed reality simulation. Comput. Educ. 2016, 95, 174–187. [CrossRef] Blikstein, P.; Worsley, M.A.B. Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks. J. Learn. Anal. 2016, 3, 220–238. [CrossRef] Smith, C.; King, B.; Gonzalez, D. Using Multimodal Learning Analytics to Identify Patterns of Interactions in a Body-Based Mathematics Activity. J. Interact. Learn. Res. 2016, 27, 355–379. Ainsworth, S.; Prain, V.; Tytler, R. Drawing to Learn in Science. Science 2011, 333, 1096–1097. [CrossRef] [PubMed] Ibrahim-Didi, K.; Hackling, M.W.; Ramseger, J.; Sherriff, B. Embodied Strategies in the Teaching and Learning of Science. In Quality Teaching in Primary Science Education; Springer: Cham, Switzerland, 2017; pp. 181–221. ISBN 978-3-319-44381-2. Barsalou, L.W. Perceptions of perceptual symbols. Behav. Brain Sci. 1999, 22, 637–660. [CrossRef] Johnson, M. The Body in the Mind: The Bodily Basis of Meaning, Imagination, and Reason; University of Chicago Press: Chicago, IL, USA, 1987. Barsalou, L.W. Grounded Cognition: Past, Present, and Future. Top. Cogn. Sci. 2010, 2, 716–724. [CrossRef] [PubMed] Glenberg, A.M. Embodiment as a unifying perspective for psychology. Wiley Interdiscip. Rev. Cogn. Sci. 2010, 1, 586–596. [CrossRef] [PubMed] Wilson, M. Six views of embodied cognition. Psychon. Bull. Rev. 2002, 9, 625–636. [CrossRef] [PubMed] Tomasino, B.; Nobile, M.; Re, M.; Bellina, M.; Garzitto, M.; Arrigoni, F.; Molteni, M.; Fabbro, F.; Brambilla, P. The mental simulation of state/psychological verbs in the adolescent brain: An fMRI study. Brain Cogn. 2018, 123, 34–46. [CrossRef] [PubMed] Milgram, P.; Kishino, F.A. Taxonomy of Mixed Reality Visual Displays. IEICE Trans. Inf. Syst. 1994, 77, 1321–1329. Alibali, M.W.; Nathan, M.J. Embodiment in Mathematics Teaching and Learning: Evidence From Learners’ and Teachers’ Gestures. J. Learn. Sci. 2012, 21, 247–286. [CrossRef] Goldin-Meadow, S.; Cook, S.W.; Mitchell, Z.A. Gesturing gives children new ideas about math. Psychol. Sci. 2009, 20, 267–272. [CrossRef] [PubMed]


22. 23. 24. 25.

26. 27.

28.

29. 30. 31.

32. 33. 34. 35. 36. 37. 38.

39. 40. 41.

42. 43.

44. 45.

19 of 20

Roth, W.M. Gestures: Their role in teaching and learning. Rev. Educ. Res. 2001, 71, 365–392. [CrossRef] Crowder, E.M. Gestures at work in sense-making science talk. J. Learn. Sci. 1996, 5, 173–208. [CrossRef] Niebert, K.; Gropengießer, H. Understanding the greenhouse effect by embodiment—Analysing and using students’ and scientists’ conceptual resources. Int. J. Sci. Educ. 2014, 36, 277–303. [CrossRef] Birchfield, D.; Thornburg, H.; Megowan-Romanowicz, M.C.; Hatton, S.; Mechtley, B.; Dolgov, I.; Burleson, W. Embodiment, multimodality, and composition: Convergent themes across HCI and education for mixed-reality learning environments. Adv. Hum. Comput. Interact. 2008. [CrossRef] Gellevij, M.; Van Der Meij, H.; De Jong, T.; Pieters, J. Multimodal versus Unimodal Instruction in a Complex Learning Context. J. Exp. Educ. 2002, 70, 215–239. [CrossRef] Flood, V.J.; Amar, F.G.; Nemirovsky, R.; Harrer, B.W.; Bruce, M.R.; Wittmann, M.C. Paying attention to gesture when students talk chemistry: Interactional resources for responsive teaching. J. Chem. Educ. 2014, 92, 11–22. [CrossRef] Sankey, M.; Birch, D.; Gardiner, M. Engaging students through multimodal learning environments: The journey continues. In Proceedings of the ASCILITE 2010: 27th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education: Curriculum, Technology and Transformation for an Unknown Future, Sydney, Australia, 5–8 December 2010; University of Queensland: Brisbane, Australia, 2010; pp. 852–863. Prain, V.; Waldrip, B. An Exploratory Study of Teachers’ and Students’ Use of Multi-modal Representations of Concepts in Primary Science. Int. J. Sci. Educ. 2006, 28, 1843–1866. [CrossRef] Frymier, A. A model of immediacy in the classroom. Commun. Q. 1994, 42, 133–144. [CrossRef] Richmond, V.; Gorham, J.; McCroskey, V. The Relationship between Selected Immediacy Behaviors and Cognitive Learning. In Communication Yearbook 10; McLaughlin, M., Ed.; Sage: Newbury Park, CA, USA, 1987; pp. 574–590. Andersen, J. Teacher immediacy as a predictor of teacher effectiveness. In Communication Yearbook 3; Nimmo, D., Ed.; Transaction Books: New Brunswick, NJ, USA, 1979; pp. 543–559. Pogue, L.L.; Ahyun, K. The effect of teacher nonverbal immediacy and credibility on student motivation and affective learning. Commun. Educ. 2006, 55, 331–344. [CrossRef] Seifert, T. Understanding student motivation. Educ. Res. 2004, 46, 137–149. [CrossRef] Witt, P.L.; Wheeless, L.R.; Allen, M. A meta-analytical review of the relationship between teacher immediacy and student learning. Commun. Monogr. 2004, 71, 184–207. [CrossRef] Márquez, C.; Izquierdo, M.; Espinet, M. Multimodal science teachers’ discourse in modeling the water cycle. Sci. Educ. 2006, 90, 202–226. [CrossRef] Ainsworth, S.; Th Loizou, A. The effects of self-explaining when learning with text or diagrams. Cogn. Sci. 2003, 27, 669–681. [CrossRef] Ainsworth, S.; Iacovides, I. Learning by constructing self-explanation diagrams. In Proceedings of the 11th Biennial Conference of European Association for Research on Learning and Instruction, Nicosia, Cyprus, 23–27 August 2005. Cooper, M.M.; Williams, L.C.; Underwood, S.M. Student understanding of intermolecular forces: A multimodal study. J. Chem. Educ. 2015, 92, 1288–1298. [CrossRef] Won, M.; Yoon, H.; Treagust, D.F. Students’ learning strategies with multiple representations: Explanations of the human breathing mechanism. Sci. Educ. 2014, 98, 840–866. [CrossRef] Morphew, J.W.; Lindgren, R.; Alameh, S. Embodied ideas of scale: Learning and engagement with a whole-body science simulation. In Proceedings of the Annual Meeting American Educational Research Association, New York, NY, USA, 13–17 April 2018. Junokas, M.J.; Lindgren, R.; Kang, J.; Morphew, J.W. Enhancing multimodal learning through personalized gesture recognition. J. Comp. Asst. Learn. 2018. [CrossRef] Alameh, S.; Morphew, J.W.; Mathayas, N.; Lindgren, R. Exploring the relationship between gesture and student reasoning regarding linear and exponential growth. In Proceedings of the International Conference of the Learning Sciences, Singapore, 20–24 June 2016; pp. 1006–1009. Creswell, J.W. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches; Sage: Thousand Oaks, CA, USA, 2013. Yin, R.K. Case Study Research: Design and Methods, 3rd ed.; Sage: Thousand Oaks, CA, USA, 2003.


46. 47. 48. 49.

50.

51.

20 of 20

Oltmann, S. Qualitative interviews: A methodological discussion of the interviewer and respondent contexts. Qual. Soc. Res. 2016, 17. [CrossRef] Datavyu Team. Datavyu: A Video Coding Tool; Databrary Project; New York University: New York, NY, USA, 2014. Cowan, J. The potential of cognitive think-aloud protocols for educational action-research. Act. Learn. High. Educ. 2017. [CrossRef] Marco-Ruiz, L.; Bønes, E.; de la Asunción, E.; Gabarron, E.; Aviles-Solis, J.C.; Lee, E.; Traver, V.; Sato, K.; Bellika, J.G. Combining multivariate statistics and the think-aloud protocol to assess Human-Computer Interaction barriers in symptom checkers. J. Biomed. Inform. 2017, 74, 104–122. [CrossRef] [PubMed] Vandevelde, S.; Van Keer, H.; Schellings, G.; Van Hout-Wolters, B. Using think-aloud protocol analysis to gain in-depth insights into upper primary school children’s self-regulated learning. Learn. Individ. Differ. 2015, 43, 11–30. [CrossRef] Worsley, M.; Blikstein, P. Leveraging Multimodal Learning Analytics to Differentiate Student Learning Strategies. In Proceedings of the Fifth International Conference on Learning Analytics and Knowledge (LAK ’15), Poughkeepsie, NY, USA, 16–20 March 2015; ACM: New York, NY, USA, 2015; pp. 360–367. © 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

Exploring Emergent Features of Student Interaction within an ... - MDPI

Exploring Emergent Features of Student Interaction within an ... - MDPI

Suggest Documents

Exploring the disconnections: Student interaction with ... - CiteSeerX

Exploring Interaction between Student Teachers During the Student ...

Exploring structural features of the interaction between the scorpion ...

Exploring Socio-Technical Features of Green Interior Design of ... - MDPI

EMERGENT ONTOLOGY IN EMBODIED INTERACTION

An initial evaluation of student withdrawals within an e-Learning ...

Exploring the tutor-student interaction in a blended university course

Exploring online interaction and online learner ... - Student Success

An Evaluation of Stacking and Tiling Features within the Traditional ...

An Evaluation of Stacking and Tiling Features within the Traditional

Exploring Gender Differences within Forest Schools as a ... - MDPI

Exploring the Use of Distributed Multiple Monitors Within an ... - IRIT

Exploring the Challenges and Opportunities of M-learning within an ...

Exploring Stable and Emergent Network Topologies

Emergent transformation games: exploring social innovation agency ...

Emergent Semantics Through Interaction in Image Databases

Quantifying Emergent Behavior of Autonomous Robots - MDPI

Exploring attributes of effective teachers - student perspectives from an ...

The Effect of Emergent Features on Judgments of ... - David Peebles

EXPLORING UNDERINSURANCE WITHIN NEW ZEALAND

EXPLORING UNDERINSURANCE WITHIN NEW ZEALAND

Exploring the Exponentially Decaying Merit of an Out-of ... - MDPI

Network Features of the EU Carbon Trade System: An ... - MDPI

Features of the fluctuation âelectromagnetic interaction

Exploring Emergent Features of Student Interaction within an ... - MDPI