An evaluation of multimodal interactions with ...

Preprint of Anastopoulou, S., Sharples, M. & Baber, C. (2011). An evaluation of multimodal interactions while learning science concepts. British Journal of Educational Technology, 42,2, 266290.

An evaluation of multimodal interactions with technology while learning science concepts Stamatina Anastopoulou, Mike Sharples, Chris Baber Stamatina Anastopoulou is a Research Fellow at the Learning Sciences Research Institute, University of Nottingham, UK. Mike Sharples is Professor of Learning Sciences and Director of the Learning Sciences Research Institute at the University of Nottingham, UK. Chris Baber is a Reader in the Department of Electronic, Electrical and Computer Engineering, University of Birmingham, Birmingham, UK. Address for correspondence: Dr Stamatina Anastopoulou, Learning Sciences Research Institute, Exchange Building, University of Nottingham, Jubilee Campus, Wollaton Road, Nottingham, NG8 1BB, UK. Email: [email protected] Abstract This paper explores the value of employing multiple modalities to facilitate science learning with technology. In particular, it is argued that when multiple modalities are employed, learners construct strong relations between physical movement and visual representations of motion. Body interactions with visual representations, enabled by interactive technologies, can encourage rhythmic cycles of engagement and reflection. A study was carried out to investigate how students interpret distance-time and velocity time graphs created through hand movement. It explored the dynamic coupling between their body movements and graphs, and their subsequent interpretation and production of graphs on paper. The results show that physical manipulation of kinematics graphs has a significantly greater effect on students’ ability to relate graphs to movement than observing the graphs being produced by someone else.

Keywords: multimodal interaction, science learning, graphical representations. Introduction Multimedia has provided new ways to represent concepts through different media formats, including animations, narratives, text and graphics and their combinations. However, evidence for the educational effectiveness of multiple media is inconclusive (Rogers, and Scaife 1996,), even though there is a plethora of empirical studies investigating the advantages and disadvantages of learning from single or multiple representations of information with the aid of computers (e.g. Ainsworth, 1999, di Sessa, 1986, Whitelock, 1991). Ainsworth (2007), for example, argues that animations may make dynamic information explicit but they introduce problems in perception and memory due to their transient nature. She also argues that animations may produce an illusion of understanding that can interfere with successful learning. Rogers and Scaife (1996) propose that learners can make mental mappings between dynamic representations (such as a pond ecosystem) if the representations are tightly coupled, so that changing a parameter in one representation causes an observable change in the other. Learning with multimedia requires engaging an appropriate mix of media to assist understanding. A combination of visual and auditory representations can enable effective learning, providing these are complementary and appropriate to the task (Gaver et al., 1991; Tabbers et al., 2000). Sweller’s Cognitive Load Theory (Sweller, 1994) addresses the

cognitive architecture of humans with the presumption that since working memory is limited then there is a need to reduce unnecessary cognitive loads for effective information processing and subsequent learning to take place. Thus, multiple media need to be integrated and coherent, to avoid increasing load and inhibiting learning (Mayer & Moreno, 2002). Where this integration can be achieved, then there may be additional benefits to learning by engaging other senses, particularly touch, to augment visual or auditory cognition. Furthermore, interacting with representations using conventional pointing devices or keyboards may not be as helpful as interacting through movement of the learner’s body. Multimodal interaction extends multimedia by considering how the human body contributes to interaction with technology. Considering learning not only as a visio-cognitive activity but also as a physical one, needs an alternative approach to computer-aided learning where an interplay among multiple sensory modalities and visual representations can be realised. Such sensory interaction is already commonplace in computer games consoles, such as the Nintendo Wii. Multimodal interaction refers to the combining of sensory and communicative modalities. A sensory modality refers to the act of perceiving the world through the senses of sight, hearing, touch, taste and smell and to the act of responding to the world through speech or movement. A communicative modality refers to symbolic representations of meaning presented in verbal, visual, aural, or kinaesthetic form. For example, playing virtual tennis with a Wii console employs sensory modalities; learning the rules of tennis from Wikipedia or YouTube employs communicative modalities. Through multimodal interactions, learners have the potential to engage with sensory and communicative modalities that are related to the subject to be learnt. These interactions augment learners’ experiences with the ability to physically manipulate symbols and see the effects of this manipulation as it occurs. Such augmentation may facilitate their understanding and enrich their experience, by incorporating a personal and authentic dimension to their learning. Incorporating body movement into the learning experience advocates the value of learning by doing (e.g. Bruner, 1966). However, if the link between the physical movement and the visual realization is weak or delayed then this might result in an inability by the learner to connect a body movement to its visual effect, or inadequate proprioceptive feedback (the relationship between actual and perceived movement). The benefits of physical manipulations of representations in understanding are in line with current trends in direct physical interactions with the aid of technologies, where manipulatives (e.g. Martin & Schwartz, 2005; Zacharia & Constantinou, 2008) or other forms of digital objects (Wyeth, 2008) are employed to facilitate learners’ understanding. This paper extends that work by investigating the benefits of employing one’s own body instead of an external object.Research into science teaching emphasises the importance of multiple modalities in classroom learning environments (Kress et al. 2001). Learners select or negotiate the meanings they derive from multiple modalities to construct conceptions about the world. Within the classroom, the use of multiple modalities offers a rich range of resources for pupils to employ while learning (Jewitt et al. 2001). Multimodal interactions can enable learners to interact with symbolic entities. Coupling physical movement to the relevant symbols and mapping changes of movement to variables can be interpreted as a type of multimodal interaction that encourages learners’ transition between concrete and abstract concepts. Creating symbols through physical movement is like writing but instead of writing words (linguistic representations), they write symbolic representations (e.g. kinematic graphs) that directly map their own movement to the features of the graph. This direct mapping can be powerful because it builds on a personal action

2

before it becomes abstract and therefore can create stronger conceptual links between the symbol and its variations. Furthermore, the de-coupling of movement and symbol should also be supported to ensure that the learner abstracts the relevant scientific concepts from the physical movement. To support such de-coupling, the emphasis is on meaningful activity where the learners can construct narratives that describe their movements in terms of the symbol, and view a symbolic representation as a historic trail of movement. For example, to understand graphs of motion, the learner should know that moving one’s body at a constant speed produces an ascending or descending straight line on a distance-time graph; or, to produce an ascending line on a velocity-time graph one needs to move increasingly fast. According to Ackermann (1996), both ‘diving in’ and ‘stepping out’ are equally needed to reach deeper understanding. Learners need to detach themselves from a situation by projecting their experience: to ‘objectify’ it and address it as if it was not theirs, to become their own observers, narrators, and critics. Then again they can reengage their previously objectified experience. They dive into it and try once more to gain intimacy (Ackermann 1996). Thus, by coupling and de-coupling activities, learners can ‘dive in’ and ‘step out’ in a rhythmic cycle, promoting deeper understanding. Including one’s own movements into the learning activity to directly change the representations enables a continuous dialogue between the concrete and the abstract. Physical movement can support the creating and testing of hypotheses so that a learner can, for example, test whether a peak on a velocity-time graph represents a speeding then slowing movement, by moving the body fast then slow. They can observe and reflect on the results, and adjusting the movement until it produces the correct visual pattern. A central issue is whether there is a difference between employing an external object (such as a mouse) to interact with external representations for learning, or to do so directly by moving a part of the body. This paper follows Papert (1980) in proposing that employing one’s own body movements to generate one or more symbolic representations can assist understanding, by enabling direct manipulation (Schneiderman, 1983) of the visual display, to give a sense of personal control and engagement and to provide continual coordination of movement with the symbols that the movement creates. Learning Physics with interactive technologies Learning physics in the classroom is presented here as an embodied activity where learners are introduced to symbols and abstract entities through the employment of multiple sensory and communication modalities. The focus is on learners’ difficulties with science, and how physical interactions with symbolic representations can help learners overcome some of these to understand abstract representations in the domain of kinematics graphs. The difficulties that are addressed in this paper relate to: 1) the inability of students to write narratives related to their practical work (Wellington and Osborne 2001), 2) interactions with data that do not successfully connect the descriptions of symbolic entities (e.g. velocity) to aspects of the real world (e.g. movement) (Wellington 1998), 3) confusions due to incorrect results not being explained to pupils, 4) inability to link one scientific concept to another. Specific difficulties with reading kinematics graphs relate to 1) ignoring the abstract concept of the graph and conceiving the graph as a picture of motion (Beichner 1990; McDermott et al. 1987), 2) confusing the meaning of the slope of the line with a particular point on the line (Brasell 1987; Mokros and Tinker 1987, Beichner 1990), 3) incorrectly superimposing upon the graph what they already know about a phenomenon (Mokros and Tinker 1987). 3

In an attempt to overcome these difficulties, we have designed a multimodal system that produces distance-time and velocity-time graphs directly from hand movements. We investigate whether the mapping of the learner’s own body directly to one or more representations can assist the learner’s engagement and ability to reflect. Involving body movements in a learning process that directly manipulates representations is considered here as a dialogue between physical and cognitive processes. When the learner moves a hand at constant velocity and sees a horizontal line being generated on the velocitytime graph on a computer screen, this provides a direct relationship between physical movement and conceptual understanding of dynamics graphs. It enables the learner to see the graphical depiction of movement and offers an opportunity to rapidly test ideas about the relation between movement and properties of the graphs. A multimodal approach to science education examines both the process and the product of science experiments. In addition to illustrating science concepts, diagrams are important tools for reasoning and model construction. Gobert & Clement (1999) suggest that studentgenerated diagramming should be evaluated as an integral part of the science curriculum. Computer technology is currently used in a variety of ways to support the creation of graphs and diagrams, connecting changes in the physical world directly to abstract representations. Data logging technologies can link learners’ actions to their effects through the employment of multiple modalities in a learning task. Data-logging refers to the process of using a computer to collect quantitative data through sensors, save and output the results of the collection. Sometimes data-logging activities include the analysis of data, such as displaying graphs. Computer-based data logging activities have been employed in science classrooms for over two decades but mainly for easy data collection and presentation (e.g. Newton, 1997, Rogers, 1997, Kennedy and Finn, 2000, Wellington, 2000, Rogers, 2002). Data logging as an interactive technology can provide learners with a mediated activity to facilitate the transition from concrete to symbolic understanding. The learning benefits of data logging activities have been well documented (see e.g. Brasell 1987, Mokros and Tinker, 1987, Nachmias and Linn, 1987). Attaching sensors to objects and interpreting the data as graphs of movement and velocity is a well-established, though not widespread, activity in the science classroom (Rogers, 2002). These activities involve attaching sensors to objects, such as a truck rolling down an incline, and then interpreting the generated graph. The difference with the current study is that first, the sensors are attached to a part of the body and, second, that the graphs are continuously plotted so that the learner can ‘write’ the graphs and immediately ‘read’ them. The current study goes beyond the studies mentioned above in that it explores the reasons why such technologies provide fruitful learning experiences. It argues that multimodal interactions are a key element of these technologies and aims to illuminate the significant role of sensory modalities when used in combination with communicative modalities. This may provide fruitful insights into designing further support for learners, either provided by technology or by learning activities. The reminder of this paper describes a study to investigate the effect of multimodal interactions with technology while learning concepts from physics. It presents an alternative approach to the design of learning technologies that focuses on studying the benefits of multimodal interactions while learning. It argues that creating symbols through physical movement directly maps the learner’s own movement to the features of the graph and therefore can create strong conceptual links to the symbol and its changes. When physical

4

manipulation is interweaved with narration and self-evaluation, learners participate in a rhythmic cycle of engagement and reflection. The study To investigate the above issues, a study was designed to explore the multimodal interactions of university students when learning to understand and connect multiple representations of motion. The students had to interact with several entities: two representations on a visual display being updated in real-time with hand movement, their own movements, and representations shown on paper. Each interaction had a learning potential from which students could benefit. As participant observers, they needed to make systematic hand movements and observe how the two representations were continuously updated. As narrators, they had to describe their movement in writing. As self-evaluators, they had to relate their movements to specific learning goals. Through physical manipulation of symbols, the learners had the opportunity to perform activities that were not only related to symbolic representations but also to real-world activities. It is suggested that physical manipulation of graphical representations can support learners in • understanding the representations and their properties, • being able to manipulate the symbolic world and interpret its changes, • drawing inferences in terms of scientific principles, and • relating among representations with similar properties. The study explores the advantages of employing multiple modalities, their effects on learners’ meaning construction and how engagement and reflection may be supported. Equipment The equipment consisted of a motion tracker to capture data and a laptop computer to process data. The motion tracker, a commercial product (Fastrak Polhemus), computes accurately the position and orientation of a tiny receiver as it moved through space. This receiver was attached to the top side of each participant’s arm, just above the wrist, with the aid of a sweatband. The data were transmitted over a high speed RS-232 interface to the computer running special-purpose software. The software was developed in LabView 6.0 by the first author and translated the data into distance-time and velocity-time graphs on the visual display. The equipment was able simultaneously to generate a d-t and the v-t graph, one above of the other. (Figure 1) These were continually updated on the screen as the participant moved the wrist from side to side. The software smoothed the data to reduce artefacts caused by noise in the sensor, producing an even graph that tracked the hand movement with no perceptible delay.

Learning context The equipment generated two line graphs in real time from the movement of a sensor attached to the hand. The students had to compare dynamic distance-time (d-t) and velocitytime (v-t) graphs produced on the display with equivalent static graphs shown on the worksheet and to relate these to their hand movements. The dynamic graphs on the display changed every millisecond, showing the position (d-t) and velocity (v-t) of the hand. Table 1 shows some typical v-t graphs produced by hand movement and the movements and equivalent graphs for Question 3 in the worksheet.

5

The learning aim was to extend the students’ understanding of kinematics graphs, demonstrated by the link between each graph and the movement, the link between the two kinematic graphs and the link between the two graphs and their (common) narrative. In particular, the study aimed to investigate: 1) how learners interact with distance-time and velocity-time graphs, 2) how they relate distance-time graphs to velocity-time graphs, 3) how they relate velocity-time graphs to distance-time graphs, 4) how they relate both graphs to linguistic descriptions of movement. Method Having the opportunity to generate d-t and v-t graphs in real time through hand movements gives students the ability, firstly, to interact physically with the graphs and, secondly, to visually relate changes between the two types of graph. The expectation is that by actively generating graphs through their own body movements, learners will engage with the task and gain an understanding of the relation between movement and equivalent graphical representations, increasing their ability to interpret and produce kinematics graphs. But it is possible that the trial and error involved in reproducing paper graphs on the screen by making hand movements could lead to confusion and incorrect learning. If that is the case, watching a teacher demonstrate the correct procedures might be a more successful way of learning. The prediction is that students who generate d-t and v-t graphs through their hand movements (Doers) will gain better scores in a paper test of the interpretation of dynamics graphs than students who watch a teacher perform the correct procedures (Watchers). The null hypothesis is that there will be no difference between the groups. Participants The participants were 18 first and second year undergraduate students (14 male and 4 female) in a University Department of Electronic, Electrical and Computer Engineering, with an average age of 20 years, standard deviation 1.75. These students had already achieved high school qualifications in Science, so were expected to have advanced knowledge of the concepts under investigation. There were nine participants in each condition. Each session lasted 50 minutes. The participants were tested separately and did not communicate with each other. Task Students were given a paper worksheet (Appendix 1) that served both as a lesson plan and as a pre- and post- intervention test of their ability to draw and interpret motion graphs. They were shown specific distance-time (d-t) graphs on the worksheet and were asked to describe how they would have to move their hand in order to generate this graph. They tried the movement with the equipment and checked if their movement was generating the same d-t graph indeed (aim 1). At the same time they could see how the equivalent velocity-time (v-t) graph was also changing (to support the learning aims 1 and 2). Therefore, on the display students could see how the two graphs corresponded to their hand movement and how each graph changed through time. Then they were asked to describe their movement in writing (to support the learning aim 4). Afterwards, they were shown a d-t graph on paper, asked to write a description of their hand movement (to support the learning aim 4) and draw on paper the velocity-time graph that corresponded to that movement and d-t graph (to support the learning aim 3). Another task was to predict and check the d-t and the v-t graph for a given description (narrative) of the hand movement that was written on the paper (to support the learning aim 1 and 4). Furthermore, they were shown a d-t graph and they had to relate 6

specific movements to the meaning of the graph’s peaks, lows and slope (to support the learning aim 1 and 2). Finally for a v-t graph they had to relate specific movements to meanings like positive, negative and constant acceleration, negative and constant velocity as well as changing direction and lack of movement (to support the learning aim 2 and 3). Procedures The students arrived in a room individually, in pre-specified time slots. They met the first author and were assigned, in order of appearance in the room, to be either a Doer or a Watcher. Each participant was asked to answer a set of initial questions about kinematics graphs (Appendix 1, task 1-4) to check their prior knowledge. These included drawing a velocity-time graph on paper equivalent to a distance-time graph shown also on paper (Table 2). It was expected that students would know about each of the graphs but their ability to translate between the two graphs would be limited. Each participant was then shown the equipment including the visual display which continued to generate both a d-t graph and a v-t graph throughout the session. Then a teaching session started, which differed for the two groups. Doers teaching session Each Doer first had to check with the aid of the technology, if the v-t graph they had drawn on the paper corresponded to the d-t graph shown on the worksheet. If it did not, they were asked to draw the corresponding one (task 5). Then the Doer generated a d-t graph with the motion tracker corresponding to the one shown on the worksheet (task 6) and wrote down the corresponding hand movements. The next task was to recall the corresponding v-t graph (task 7), then confirm that by looking at the screen. These activities aimed to strengthen their abilities to translate from graphical representations to corresponding hand movements and to provoke participants to look at both graphs on the display. The next sequence of tasks (task 8 and 9) tested their ability to draw d-t and v-t graphs corresponding to written descriptions of hand movements, then to check these by performing the appropriate actions with the aid of technology. Lastly (task 10 and 11), each Doer had to convert a d-t graph to an equivalent v-t one and vice versa, then to answer questions about the hand movements corresponding to elements in the graphs. After answering each question they could test the answer/prediction by generating the graph. There were no restrictions to how many attempts they could undertake, nor to how long it would take them to fulfil the tasks. Watchers teaching session The students in the Watchers group completed the same set of tasks as the Doers, including making predictions about graphs, but instead of generating the graphs themselves, they watched the first author of this paper generate the graph corresponding to each question, at the appropriate time, by performing a correct sequence of hand movements. After answering each question they could test the answer/prediction by watching a demonstration of the correct sequence of movements and the resulting graph. It was part of their learning task to see the movements of the researcher as well as the visual display. The intention was for each student to see clearly a correctly-performed sequence of activities, which contrasted with the Doers who sometimes performed incorrect sequences of activity corresponding to the worksheet questions, which they then had to repeat and correct. There were no restrictions regarding how many times the researcher would generate the graphs. After each ‘drawing’, the researcher asked the participants whether they wanted her to repeat the task.

7

It should be emphasised that each participant was tested separately, so the Doers did not see the experimenter perform the correct actions, nor did Watchers see the attempts of Doers. Final test After completing the tasks, each of the students was given a test, which covered the important aspects of the teaching session (Appendix 2). During the test they could not look back at the worksheets. Results The results discussed below are derived from the completed question sheets. Each sheet was blind marked by the first author, marked by the second author, and then discussed to reach agreement. The findings refer to the overall performance for each sheet. A Mann-Whitney test was used since the level of data was ordinal, the type of design unrelated and the data were not interval. The pre-intervention questions (1 to 4) were scored out of 21, producing a median score of 16 for those allocated to the Doer group, and a median score of 18, for those allocated to the Watcher group. A Mann-Whitney test showed no significant difference between the scores. Since the scores for the pre-intervention test were high, it can be inferred that the students in both groups had ability to interpret simple distance-time and velocity-time graphs. Some students had problems in translating from the d-t graph to a v-t graph. Seven students who were subsequently allocated to Doers and four Watchers drew v-t graphs that did not correspond to the d-t graph. In the final test, Doers gained a median score of 22.00 and the median for Watchers was 18.00. A Mann-Whitney test showed a significant difference between the groups (z = -1.948, p

Suggest Documents

---

An Evaluation of an Augmented Reality Multimodal ... - Semantic Scholar

Read more

An Evaluation of an Augmented Reality Multimodal ... - CiteSeerX

Read more

MULTIMODAL REVOICING INTERACTIONS Refining Mathematical ...

Read more

MULTIMODAL REVOICING INTERACTIONS Refining Mathematical ...

Read more

Multimodal Interactions for Linking Everyday

Read more

Preference dynamics with multimodal user-item interactions in ... - DElab

Read more

Multimodal Interactions with Agents in Virtual Worlds - Semantic Scholar

Read more

Design and evaluation of multimodal biometric system with fingerprint

Read more

RWC Multimodal Database for Interactions by

Read more

Multimodal Human Machine Interactions in Virtual ...

Read more

Predicting Protein–Protein Interactions from Multimodal Biological ...

Read more

MULTIMODAL SYSTEMS, RESOURCES AND EVALUATION

Read more

Performance Evaluation of Multimodal Multifeature Authentication ...

Read more

Empirical Evaluation of Interactive Multimodal Error ... - CiteSeerX

Read more

Implementation and Evaluation of multimodal input

Read more

Evaluation and Interpretation of Interactions

Read more

(Mouse) Evaluation of Drug-Drug Interactions of Repaglinide with Anti ...

Read more

Multimodal presurgical evaluation in children with focal cortical ...

Read more

An evaluation of the Mind–Body Interactions and Health Program ...

Read more

Policy Interactions at the Farm Level: An Evaluation of ... - CiteSeerX

Read more

An evaluation of the interactions among household economies ...

Read more

Multimodal interaction with BIM

Read more

Intuitive Multimodal Interaction with

Read more

Evaluation of Oxygen Interactions with Materials 3: Mission and ...

Read more

×

Report "An evaluation of multimodal interactions with ..."

Your name

Email

Reason -Select Reason- Pornographic Defamatory Illegal/Unlawful Spam Other Terms Of Service Violation File a copyright complaint

Description

Close Save changes

×

Sign In

Email

Password

Remember me Forgot password?

Sign In

Our partners will collect data and use cookies for ad personalization and measurement. Learn how we and our ad partner Google, collect and use data. Agree & close