British Journal of Educational Technology doi:10.1111/j.1467-8535.2012.01370.x
Vol 45 No 1 2014
24–35
A framework for monitoring instructional environments in a virtual world David Chodos, Eleni Stroulia, Sharla King and Mike Carbonaro David Chodos is a PhD student in Computing Science at the University of Alberta. His research is on using virtual worlds for simulation-based education. Eleni Stroulia is a professor in Computing Science at the University of Alberta, and her interests include using virtual worlds to offer innovative services. Sharla King is an assistant professor in Educational Psychology at the University of Alberta, and is interested in virtual learning communities and simulation education in the health sciences. Mike Carbonaro is a professor in Educational Psychology at the University of Alberta, and is interested in the interrelationship between learning and technology. Address for correspondence: Dr Eleni Stroulia, 2-21 Athabasca Hall, University of Alberta, Edmonton, AB, Canada, T6G 2E8. Email:
[email protected]
Abstract Virtual worlds are gaining momentum as a platform for delivering simulation-based educational experiences to students. However, a key aspect of virtual world-based education that has received little attention is recording and analyzing students’ in-world actions. This capability is essential for assessing what students have learned through their simulation experience, and engaging the students in post-simulation reflective learning. In this research, we present a framework for recording and analyzing students’ actions in a virtual world. This framework is based upon pedagogical theories of exploratory and experiential learning, and is defined in a virtual-world agnostic manner. The framework consists of two parts: (1) the Avatar Capabilities Model, which defines the educationally relevant actions that a student can take within a virtual world and (2) the Simulation Capture and Analysis toolkit that records and analyzes these actions, from an educational perspective. These analyses provide instructors with systematically collected evidence of the students’ actions during their virtual world experience. This alleviates the need for instructors to directly observe students, thereby allowing for the scaling-up of virtual worlds use in education. We have demonstrated the usefulness of the tool via a pilot study, with two students, in an emergency medical education context.
Introduction Virtual worlds (VWs) offer an appealing and cost-effective environment for synchronous distributed interactions. Recognizing their potential for education, substantial research effort is being dedicated towards developing educational programs in VWs that can support: (1) instructors, to evaluate students’ competence and provide constructive feedback on behaviour (Akyol & Garrison, 2011; Esteves, Fonseca, Morgado & Martins, 2011; Rogers, 2011); and (2) students, in performing required self-evaluation and reflection upon their own behaviour (Charles, Charles, McNeill, Bustard & Black, 2011; van der Spek, Wouters & van Oostendorp, 2011; Walker, 2009). There have been several studies on the use of educational VW scenarios to enhance, or replace traditional forms of education. Greci et al (2010) taught students patient triaging in the context of a VW pandemic-outbreak scenario; this addition to the curriculum resulted in improved team © 2012 British Educational Research Association
Framework for monitoring teaching in virtual world
25
Practitioner Notes What is already known about the topic • • • •
Virtual worlds offer a viable platform for simulation-based instruction. Virtual world simulation projects have demonstrated educational impact. Existing virtual world simulations lack comprehensive recording and analysis tools. Without these tools, student outcomes cannot be fully understood or compared.
What this paper adds • A model for describing a student’s actions in a virtual world simulation. • A comprehensive set of recording and analysis tools based on this model. • A pilot study using these tools that validates its utility. Implications for practice and/or policy • Improved understanding of student behaviour in a virtual world simulation. • Enables detailed assessment of educational impact of virtual world simulation. • Enables comparison of effectiveness of virtual world simulations with other methods.
communication and decision-making skills. Dev, Youngblood, Heinrichs and Kusumoto (2007) developed a VW scenario for treating a trauma patient, replacing a module based on a Human Patient Simulator (mannequin) with an embedded physiological model. After going through the VW scenario, their students showed significant improvement in their team-performance skills, as assessed by several evaluators. Wrzesien and Raya (2010) developed a marine biology VW to replace an in-person lecture given at an aquarium. They found that using a VW resulted in increased enjoyment and engagement, and was just as effective as the in-person lecture. There has also been substantial interest in developing new educational theories to account for how learning occurs in VWs. Arguing that VWs necessitate a new theoretical educational paradigm, Price (2009) proposed that the subject matter to be learned should be organized in (logical) concept maps mapped to “geographical” locations in the virtual environment. Therefore, at these locations, students interact with the environment in ways designed to communicate the corresponding concepts. To support the learning process of reflection, a special “history” (recording) of a student’s movements and interactions needs to be stored for viewing. An important influence on our work was the extension of Kolb’s (1984) experiential-learning theory by de Freitas and Neumann (2009) to include the construct of exploratory learning in VW environments. Kolb views the learning process as a cycle involving four activities: (1) concrete experience, (2) observation and reflection, (3) abstract-concept formation and (4) testing in new situations. The exploratory-learning model (ELM) also includes “exploration,” which involves observations on the environment, social interactions with other participants and task-specific activities that can occur in a VW. In Kolb’s (1984) model the concept of “experience” was limited to the “lived” context (real-world). De Freitas and Neumann (2009) argue that a learner’s experiences in a virtual context can be equally conclusive. They suggest, “lived and virtual interactions may have a reinforcing impact upon learning objectives, helping in pre- ‘real-life’ occupational work, and allowing for mistakes to be made in a secure environment” (de Freitas & Neumann, 2009, p. 346). ELM provides an important theoretical framework and rationale as to why learning in a virtual environment necessitates a revised perspective on the learning process and the important role reflection has in relation to this construct. Our work recognizes the subsequent need to develop mechanisms to support students and instructors to evaluate and understand the “exploration” construct that is embodied in ELM learning cycle to help support the reflective process. © 2012 British Educational Research Association
26
British Journal of Educational Technology
Vol 45 No 1 2014
There has also been some work in recording and analyzing students’ in-world actions for educational purposes. Hurst (2011) identified the issue of processing and understanding large quantities of log data as a key challenge in assessing whether skills developed in virtual-reality settings transfer to the real world. She classified recorded data into (1) reflective data, (2) machinima and (3) virtual-environment data, and identified the last category as the most relevant, covering the recording of the avatars’ in-world actions and their interactions with objects. Hurst’s (2011) work is purely theoretical; she did not create any recording or analysis tools. Nelson and Ketelhut (2008) embedded a tracking system within a multi-user virtual environment, called River City, to record relevant student activities (eg, reading signs, clicking on objects, asking questions of computer-controlled “town residents”). These logs were then used to drive a student-guidance system, which provided hints to students in the form of reflective questions. This tracking system enabled the personalization of the student learning experience, based on his activity history; however, it tracked a fairly limited number of activities and it was applicable only to River City. Girvan and Savage (2010) conducted a study delivering lessons in Second Life on the international banana industry, where students had to construct a virtual book to summarize their knowledge. To assess the students’ learning, they analyzed the chat logs of the in-world sessions, the students’ virtual books, and data collected through semi-structured interviews, thus demonstrating the importance of recording experiences to assess learning. Their method, however, was time intensive and lacked a systematic and scalable mechanism for creating an audit trail of the student’s exploration and learning experiences in the VW. In practice today, screen-capture tools are typically used for constructing a record of a student’s in-world actions. Although accurate, this method suffers from three significant challenges. First, it typically records the VW from the perspective of a single participant, thus missing the activities of other participants engaged in the simulation. Second, manually analyzing these video recordings is effort-intensive and errorprone. Finally, instructors cannot easily compare the performance of a student over time or against that of the student’s peers. To address these challenges, we constructed a systematic mechanism for supporting the recording and inspection of students’ activities in VWs. This paper describes the Avatar Capabilities Model (ACM), which we use as a foundation to construct a Simulation Capture and Analysis (SCA) tool. The ACM is a representational model of a student’s actions (exploratory movement, experiences with objects and social interactions) within a VW simulation. The SCA provides the necessary components for recording a simulation trace and analyzing it to support teachers in their assessment of student learning, and potentially to enable student’s reflection (Chodos, Stroulia & Naeimi, 2009; Chodos et al., 2010a; Chodos, Stroulia, Kuras, Carbonaro & King, 2010b). In other words, the ACM model specifies what is measured about the avatar’s behaviour, while the SCA tool collects, analyzes and presents the measurements. The development of the ACM and SCA enables instructors to see evidence of student capability without themselves observing it and thus allows for scaling up the use of VWs in education. We present and discuss a pilot study that we conducted to empirically evaluate our SCA tool. The ACM A requirement to model representation and tool creation is the design of a syntax in terms of which to record instances of the student’s learning activities in the VW. To that end, we draw from work by Schank and Abelson (1977) on codifying behaviour in terms of scripts and plans, and by Mehrabian (1972) and Verhulsdonck and Morie (2009) on non-verbal communication. Based on the theoretical work described in the preceding section, we have defined a model for describing a student’s in-world actions, executed through the student’s avatar. The model, shown in Table 1, groups these actions into three categories: (1) movement, (2) object sensing and manipulation, and (c) communication. These categories, in turn, may be considered in terms of three pedagogically © 2012 British Educational Research Association
Framework for monitoring teaching in virtual world
27
Table 1: Definition of Avatar Capabilities Model Action = < Movement | Sensing | Object Manipulation | Communication > Movement = < Move | Sit > Move = < Actor, Movement Type, Start Location, End Location > Sit/Stand = < Actor, Sit/Stand Type, Sit/Stand Location> Sensing = < Actor, Modality, Target > Object Manipulation = < Create | Hold | Transfer | Take | Interact > Create = < Actor, Created Entity > Hold = < Actor, Held Entity > Transfer = < Actor, Target, Transferred Entity > Take = < Actor, Taken Entity > Interact = < Actor, Entity, Message, Options, Choice, Response > Communication = < Speak | Write | Gesture > Speak = < Actor, Message > Write = < Actor, Message > Gesture = < Actor, Communication Type, Description >
based themes. The first theme, movement, is consistent with de Freitas and Neumann’s (2009) identification of exploration as a crucial aspect of VW activity, and Price’s (2009) linking between locations and educational concepts. The second theme, experiencing the world, includes both the sensing and object-manipulation action categories. This theme is also influenced by de Freitas and Neumann’s (2009) concept of VW exploration, as well as by Kolb’s (1984) identification of experience, observation and reflection as key elements of the learning process. Finally, the socialinteraction theme is supported by both Girvan and Savage’s (2010) use of chat logs as a means of assessing students’ learning, and also by Mehrabian (1972) and Verhulsdonck and Morie (2009) emphasis on the importance of non-verbal communication in effective social interaction. Using these categories, we have specified definitions for all actions that are possible within a VW, excluding irrelevant actions, such as, for example, changing the clothes of one’s avatar. For each action, we have defined a list of parameters that uniquely describe the action instances. The ACM defines each action from the point of view of an actor—that is, the student who has taken the action. Although this may seem reminiscent of the single-viewpoint recording problem mentioned in the Introduction, the model is used to represent the actions of all actors, thus avoiding this problem. The definitions of these actions are described in the following paragraphs. The location parameters of the move and sit/stand actions are VW coordinates. The movement type parameter indicates the way the actor is moving (eg, running, walking or flying). Note that the set of possible movement types will vary among VWs, and thus the value of this parameter will depend on the world. The sit/stand action has a similar Sit/Stand Type parameter, which can be either sit or stand. The sensing actions are described by a modality parameter, indicating the relevant sense—sight, hearing, smell, taste or touch—and a target parameter, indicating the object being sensed, based on the direction in which the actor looks, for example. Note that the various sensing actions are recorded differently, depending on a number of factors related to the sensing modality and the characteristics of the VW (discussed in detail in the SCA toolkit section). The communication actions, speak and write, are described by the message parameter, which is the content of that communication. The communication type parameter of the gesture action indicates the type of gesture, which takes a value based on the gestures available in a particular VW. Finally, all the object manipulation actions follow a similar pattern. The entity is the object which is being manipulated. For the transfer action, an additional target parameter is required, to indicate © 2012 British Educational Research Association
28
British Journal of Educational Technology
Vol 45 No 1 2014
Figure 1: Relationship between components in automatic annotation process
the recipient of the object. The interact action, which covers interactions supported by scripts attached to in-world objects, requires several additional parameters. The message parameter stores the text shown to the actor upon starting to interact with the object; options are the interaction choices that the actor is presented with; the choice parameter indicates the option that has been chosen, and the response indicates the text, action and/or media that the actor is shown upon selecting that choice option. SCA toolkit Development of SCA component The ACM lays the foundation for our SCA toolkit, by defining the syntax in terms of which the actions of the VW participants are recorded. As shown in Figure 1, given a new VW, a new ACM recorder has to be developed that will use the VW functionalities to recognize avatar actions and to record them in the ACM syntax. Once this ACM-recorder component has been developed, the SCA functionalities can be brought to bear to simulations conducted in this VW. Action traces recorded by ACM recorders for different VWs will always be similar, and consistently stored in the SCA trace database, to be analyzed by the trace-analysis component. Finally, assuming the availability of any off-the-shelf screen-casting recorder, the Actionable Video Annotation (AVA) component can automatically annotate the produced video with trace and analysis information. Recording student actions in Second Life The Second-Life ACM recorder is modelled as a small device, worn by each student’s avatar like a cell phone. The mechanism used to record a student’s actions varies according to the action type. Movement and communication actions are recorded directly by the recording tool. Object manipulation actions are recorded through communication between the relevant object and the recording device. The recording of sensing actions, finally, depends on effect of the action on nearby avatars: active sensing (eg, gaze direction) is recorded by directly tracking the acting avatar; passive sensing (eg, hearing a sound) involves coordination between the object being sensed and © 2012 British Educational Research Association
Framework for monitoring teaching in virtual world
29
Table 2: Structure for storing recorded actions Field name
Description
Actor Action Parameters Timestamp Scenario
The ID of the avatar that conducted the logged action The specific type of logged action (eg, move, sit/stand, write, create, . . .) The action parameters (eg, movement type, start location, end location for move) The time at which the action occurred An identifier for the simulation scenario in which the action occurred
all nearby avatars. The recorded actions of all participants in the VW are time-stamped and archived in a database table, whose schema is shown in Table 2. Analyzing recorded action traces The analysis component inspects the recorded-actions’ table and applies to it a variety of analyses, at the level of an individual (or a group of) student(s). Movement analysis Movement is an essential aspect of experiencing the simulated environment, and, therefore, learning. The VW coordinates of the student’s avatar are periodically recorded and are the starting point for movement analysis. To process these coordinates, the analysis component can be configured with information about locations of interest specific to each scenario. We define locations of interest in terms of a containment hierarchy, where general locations (neighbourhoods) contain more specific ones (eg, buildings), which in turn may contain rooms or specific points. This hierarchical location structure supports movement analyses at different degrees of granularity and precision. The SCA analysis component supports three different types of movement analysis: proximity to locations of interest, proximity to other students and paths (and their overall length) traversed. We have chosen these analyses because of their pedagogical relevance. The “proximity to locations of interest” analysis can provide evidence of the student’s intent to perform an activity related to a given location. Actual interactions with the objects in question indicate whether they were actually able to do so. Similarly, “proximity to other students” may provide implicit evidence of (intent for) collaboration, which can then be confirmed by instances of nearby students communicating with each other or conducting coordinated activities. Finally, analyses of the paths travelled by the students can help characterize student activity. Substantial differences in the activity characteristics of students may indicate a problem, such as unbalanced contribution to the task among a group of students. Object-manipulation analysis In the context of a learning simulation, all student interactions with in-world objects are potentially relevant to the simulated activity—although students might still choose to interact in random or unproductive ways. More interestingly, there will likely be objects in the environment that must be correctly used in order to accomplish the task at hand. For example, our medicalsimulation pilot study requires the students to manipulate a patient and to use objects, such as a stretcher, stethoscope and goggles. The SCA analysis component determines whether the expected normative interactions occurred during the scenario, and whether the “correct” order among these interactions was followed. In defining the scenario, the normative tasks of the simulation—and the correct (partial) ordering of these tasks—may be specified either using a workflow or via a record of the simulation as © 2012 British Educational Research Association
30
British Journal of Educational Technology
Vol 45 No 1 2014
conducted by an expert. Furthermore, if a student goes through the same scenario multiple times, comparing these records can help indicate the student’s progress. Overall trace analyses Several more analyses of the overall trace are of interest. • Time: There are several time-related metrics, relevant to learning and assessment. On one hand, time-on-task is an indicator of the student’s engagement with the learning activity itself (Garris, Ahlers & Driskell, 2003). On the other, in computerized testing, the relationship between response times and response accuracies is examined both at the level of an individual learner, usually as a trade-off, and for a population of test takers, where a positive correlation may be found (Fox, Klein Entink & van der Linden 2007). • Conversation and social interactions: Examining in-world chat messages, our analysis component computes the numbers of phrases contributed by each student to identify the most active student. Further inspection of the messages content for keywords or phrases can help determine the tone of an individual student, or of the conversation as a whole. These interaction measures, when aggregated and cross-referenced with co-location, may be used to analyze collaboration and social interaction among groups of students. Textual interactions between learners in VWs have been identified as an important component in a student’s learning experience (Jarmon, Traphagan, Mayrath & Trivedi, 2009) and in the development of collaboration skills and the construction of knowledge (Jonassen & Reeves, 1996). For example, Hew and Cheung (2010) in their research review identified such key questions as: “Do learners make use of the communication features of their avatars?”; “Do the use of avatar-based virtual worlds facilitate interaction among students?; “What are some of these communicative features?” (p. 43). • Attention: By analyzing each student’s gaze, movement and interactions, the instructor can obtain a sense of whether the student is focused on the simulation tasks. A distracted student might wander around the virtual environment, zoom in on numerous objects, and have many extraneous conversations. This is, clearly, a subjective analysis and not a rigorous method for analyzing a student’s attention; however, it has long been known that attention plays an important role in learning (Trabasso, Bower & Gelman, 1968). SCA provides a mechanism for the instructor to obtain some indication of the students’ attention; such analytics were not previous available in a VW context. • Patterns and sequences: Finally, our analysis component includes two different knowledgeextraction methods: pattern and sequence mining. Once the simulation trace has been summarized into a sequence of action types (and their associated parameters), the Apriori algorithm (Agrawal & Srikant, 1994) can be used to recognize interesting and frequently occurring action sequences. On a per-student level, recurring attempts at the same activity may provide evidence of lack of a particular piece of knowledge, in which case the instructor may provide appropriate scaffold feedback to the student (Vygotsky, 1978). On a scenario level, consistency across students on desired activity patterns will constitute evidence of skill learning. Video annotation The last component of our SCA tool automatically adds annotations of all recorded actions to a video recording of the in-world activity during a simulation, using the AVA service (Chodos et al, 2009). For each parsed action, a text caption is added to the video at the time that the action occurred. Thus, the video is automatically annotated with all the actions of all students, and when the video is viewed, a description of each parsed action will be superimposed on the video at the appropriate moment. This functionality overcomes the single-person-perspective shortcoming of current screen-capture tools, which was described in the Introduction. © 2012 British Educational Research Association
Framework for monitoring teaching in virtual world
31
This automatic video-annotation capability provides the instructor with a solid starting point for analyzing the video recording of a simulation session. Using the video-annotation interface, the instructor can further augment the video recording of the student’s actions, with more relevant annotations, for example, to indicate where missing actions should have been, or to ask questions of the student at certain points. Pilot study: Emergency Medical Technician (EMT) rescue and hand-off We have used our SCA tool in the simulation of a scenario where EMTs rescue a car-crash victim and hand him off to hospital personnel. The EMT students must treat the victim at the accident scene, load him into an ambulance and transport him to the hospital, while communicating with each other and with staff at the hospital emergency room (ER). During the hospital hand-off, they must convey the patient’s status, the treatment they have administered, and any other relevant information. We developed this scenario in collaboration with paramedic and emergency medicine instructors to ensure its clinical relevance and plausibility and its appropriateness, from a communications-skills perspective. Screenshots of this scenario scenes are shown in Figures 2 and 3. Sample Based on the knowledge and experience required by the scenario roles, we recruited participants among 15 first-year EMT students at the Northern Alberta Institute of Technology. Two students enacted the scenario, with a paramedic instructor supervising the rescue process, and an emergency medicine instructor playing the role of an ER nurse and evaluating the hand-off conversation. Methodology First, each student was provided with a brief (15-minute) one-on-one training session on the basics of using the VW environment—controlling their avatar, communicating with other users and using the required equipment (eg, spine board, gurney). This session was not recorded. Next,
Figure 2: Emergency rescue screenshot © 2012 British Educational Research Association
32
British Journal of Educational Technology
Vol 45 No 1 2014
Figure 3: Emergency room screenshot Table 3: Summary of analysis for Emergency Medical Technicians rescue and hand-off scenario Participant Participant 1 Participant 2
# actions
Time (minutes)
Pace (actions/ minute)
Most frequent action
Distance travelled (m)
35 111
60 60
0.58 1.85
Palpate chest Call hospital
446.67 694.89
the students enacted the EMT-ER hand-off scenario, discussed above. The students’ activities were tracked using the ACM recorder and they were later analyzed with relevant tools from the SCA toolkit. Finally, a post-scenario debriefing session was conducted by the paramedic and emergency medicine instructors. This conversation was recorded using screen-capture software (Camtasia, TechSmith Corporation, Okemos, Michigan, USA); the participants’ conversation was then transcribed and analyzed in order to elicit key themes from the debriefing. Findings Some of our findings, based on our analysis of the recorded simulation trace, are summarized in Table 3. Participant 2 was asked to take on a “lead” role in the trauma team, and we therefore expected to see a greater level of activity. Indeed, he was by far the more active of the two, having executed nearly three times as many actions as Participant 1. He also travelled much further, which also indicates a greater level of activity. These numbers are consistent with the participants’ roles, and confirm the expected activity levels. In order to provide a more detailed analysis of the participants’ behaviour, we tracked their movements between landmark locations. These sequences are shown alongside the “correct” sequence (as described by our domain experts) in Table 4. From this table, it is clear that Participant 2 was much closer to the correct sequence than Participant 1. Taking the analysis of Tables 3 and 4 together, one can see that Participant 2 was both more active, and more successful at visiting the “correct” sequence of locations than Participant 1. The © 2012 British Educational Research Association
Framework for monitoring teaching in virtual world
33
Table 4: Landmarks visited during Emergency Medical Technicians scenario Participant 1 Start position (fire hall) Ambulance at accident Behind ambulance at accident Victim Behind ambulance at ER
Participant 2
“Correct” sequence
Start position (fire hall) Ambulance at accident Behind ambulance at accident Gurney Victim Gurney Ambulance at accident Behind ambulance at accident Hospital Behind ambulance at ER Hospital Behind ambulance at ER
Start position (fire hall) Ambulance at accident Behind ambulance at accident Gurney Victim Gurney Behind ambulance at accident Ambulance at accident Behind ambulance at accident Hospital Behind ambulance at ER
ER, emergency room.
first conclusion is consistent with the roles assumed by Participants 1 and 2 (“support” and “lead” paramedic respectively), and provides evidence that these roles were adopted successfully. The second conclusion, while a positive result for Participant 2 in terms of that student’s individual performance, indicates a low level of coordination among the participants. That is, the degree to which the students’ visit sequences diverge indicates that they were not able to coordinate their movement effectively. The hand-off from the EMT students to the instructor playing the role of an ER nurse was followed by a debriefing among the students and instructors. The primary focus of this debriefing was an evaluation of the students’ performance by the instructors, although the simulation experience as a whole was also discussed. The instructors offered students feedback about improving their communication skills, and the whole team discussed the pros and cons of the simulation environment. Specifically, students appreciated having the opportunity to practice the patient handoff process and valued the feedback that they received from the instructor playing the role of an ER nurse. The students also commented on some technical challenges that could be improved and suggested the inclusion of more participants (acting as bystanders, for instance) to increase the realism and complexity of the simulation. Debriefing is a regular part of the simulation process in the health sciences (Rudolph, Simon, Dufresne & Raemer, 2006). Our tool can enhance the debriefing activity by presenting the instructor with the analysis results immediately after the completion of the simulation. However, due to timing constraints, in this particular pilot, the instructors did not have a chance to incorporate the simulation-analysis results into their feedback. Limitations Our study is limited in three ways that could impact the validity of our results. The first limitation involves the design of the study itself. As it was a pilot study, we only have a small number of participants, due to our limited pool of potential users. Second, we did not conduct a preintervention assessment, which limits the validity of our inferences. However, the feedback of the participating instructors was very positive and the collected data—in spite of their small size— convincingly demonstrates the utility of the tools, and the validity of the underlying framework and ACM model. The second concern refers to the relative lack of fidelity of the VW simulation setting, as compared with the real-world environment. Aiming to make the VW environment, places and objects “close enough” to the real thing, we consulted extensively with domain experts and pilot-tested the scenario before the study. However, this process is not foolproof, and © 2012 British Educational Research Association
34
British Journal of Educational Technology
Vol 45 No 1 2014
seemingly small details may have been perceived as errors by the students. Finally, it is possible that some inherent property of the VW experience—the user interaction for example—may be detrimental to learning. By conducting further simulation studies we hope to either dispel this concern or, at worst, to understand and mitigate it. Conclusions and future work VWs are increasingly being adopted as simulation platforms for competency-based educational programs. Recent work (Girvan & Savage, 2010; Hurst, 2011; Nelson & Ketelhut, 2008) has established the lack of (and need for) student-assessment tools, to support instructors that deliver educational experiences in VWs. To that end, we have developed the SCA framework, which enables the systematic recording and analysis of the students’ actions as they go through VW simulation sessions. The SCA makes three significant contributions to the state of the art. First, drawing on research from educational psychology, the framework identifies the in-world activities that are relevant to assessing the students’ educational progress. Second, the framework proposes the ACM, a generic model of in-world student activity that describes, in an implementationagnostic manner, the educationally relevant actions that can be taken within a VW, and provides the foundation for the comprehensive SCA recording and analysis toolkit. The only requirement for using the SCA tool in the context of a different VW is to develop a VW-specific ACM recorder; in Figure 1, only the dashed components are VW specific. Clearly, this process assumes either access to the source code of the VW or, as in our work, the ability to create a component with user-tracking functionality using the VW’s development tools. Third, the SCA analyses (Analyzing recorded action traces section) support instructors with the task of understanding the actions of their students in a VW and answering a range of questions relevant to assessing the students’ competence. We have demonstrated the utility of the framework with a pilot study in a healthcare domain, which convincingly demonstrates that the recording tool produces comprehensive records of the student actions and that the subsequent analyses can help instructors to assess student performance. In the future, we plan to expand the analysis toolkit, to integrate additional VWs with our recording and analyses tools and to conduct further empirical studies to more precisely evaluate the usefulness of our tool. Acknowledgements We would like to thank Dr Patricia Boechler, Andrew Reid, Lisa Torres and Norbert Werner for their assistance and input while conducting the pilot study. This work was supported by NSERC (through a PGS scholarship for the first author, and through the GRAND NCE for the other three authors), AITF, IBM and the Access to the Future Innovation Fund (Advanced Education and Technology, Government of Alberta). References
Agrawal, R. & Srikant, R. (1994). Fast algorithms for mining association rules. Proceedings of the 20th International Conference on Very Large Data Bases, 487–499. Akyol, Z. & Garrison, D. R. (2011). Understanding cognitive presence in an online and blended community of inquiry: assessing outcomes and processes for deep approaches to learning. British Journal of Educational Technology, 42, 2, 233–250. Charles, D., Charles, T., McNeill, M., Bustard, D. & Black, M. (2011). Game-based feedback for educational multi-user virtual environments. British Journal of Educational Technology, 42, 4, 638–654. Chodos, D., Stroulia, E., Boechler, P., King, S., Kuras, P., Carbonaro, M. et al (2010a). Healthcare education with virtual-world simulations. Software Engineering for Health Care 2010, May 3–4, Cape Town, South Africa. Chodos, D., Stroulia, E., Kuras, P., Carbonaro, M. & King, S. (2010b). MERITS training system: using virtual worlds for simulation-based training. Computing Science and Education (CSEDU) 2010, April 7–10, Valencia, Spain. © 2012 British Educational Research Association
Framework for monitoring teaching in virtual world
35
Chodos, D., Stroulia, E. & Naeimi, P. (2009). An integrated framework for simulation-based training on video and in a virtual world. Journal of Virtual Worlds Research, 2, 1, 1–28. de Freitas, S. & Neumann, T. (2009). The use of “exploratory learning” for supporting immersive learning in virtual environments. Computers & Education, 52, 343–352. Dev, P., Youngblood, P., Heinrichs, W. L. & Kusumoto, L. (2007). Virtual worlds and team training. Anaesthesiology Clinics, 25, 321–336. Esteves, M., Fonseca, B., Morgado, L. & Martins, P. (2011). Improving teaching and learning of computer programming through the use of the Second Life virtual world. British Journal of Educational Technology, 42, 4, 624–637. Fox, J.-P., Klein Entink, R. H. & van der Linden, W. J. (2007). Modeling of responses and response times with the package CIRT. Journal of Statistical Software, 20, 7, 1–14. Garris, R., Ahlers, R. & Driskell, J. E. (2003). Games, motivation, and learning: a research and practice model. Simulation & Gaming December 2002, 33, 441–467. Girvan, C. & Savage, T. (2010). Identifying an appropriate pedagogy for virtual worlds: a communal constructivism case study. Computers and Education, 55, 342–349. Greci, L. S., Ramloll, R., Hurst, S., Garman, K., Beedasy, J., Pieper, E. B. et al (2010). Pandemic flu patient surge planning and practice in a virtual environment. Presentation given at Innovations in Emergency Medicine Education, Society for Academic Emergency Medicine Annual Meeting, Phoenix, AZ, USA, June 3–6, 2010. Hew, K. F. & Cheung, W. S. (2010). Use of three-dimensional (3-D) immersive virtual worlds in K-12 and higher educational settings: a review of the research. British Journal of Educational Technology, 41, 1, 33–55. Hurst, S. (2011). Towards high-resolution ethnography for evaluation of team-oriented virtual reality training for medicine. Presentation given at Medicine Meets Virtual Reality 18, Newport Beach, CA, USA, February 9–12, 2011. Jarmon, L., Traphagan, T., Mayrath, M. & Trivedi, A. (2009). Virtual world teaching, experiential learning, and assessment: an interdisciplinary communication course in Second Life. Computers and Education, 53, 169–182. Jonassen, D. H. & Reeves, T. C. (1996). Learning with technology: using computers as cognitive tools. In D. H. Jonassen (Ed.), Handbook of research for educational communications and technology (pp. 693–719). New York: Macmillan. Kolb, D. A. (1984). Experiential learning. Englewood Cliffs, NJ: Prentice Hall. Mehrabian, A. (1972). Nonverbal communication. Chicago, IL: Aldine-Atherton. Nelson, B. C. & Ketelhut, D. J. (2008). Exploring embedded guidance and self-efficacy in educational and multi-user virtual environments. Computer Supported Collaborative Work, 3, 413–427. Price, C. (2009). The path between pedagogy and technology: establishing a theoretical basis for the development of educational game environments. In T. Connolly, M. Stansfield and L. Boyle (Eds.), Game-based learning: advancements for multi-sensory human computer interfaces (pp. 191–213). Hershey, PA: IGI Global. Rogers, L. (2011). Developing simulations in multi-user virtual environments to enhance healthcare education. British Journal of Educational Technology, 42, 4, 608–615. Rudolph, J. W., Simon, R., Dufresne, R. L. & Raemer, D. B. (2006). There’s no such thing as “nonjudgmental” debriefing: a theory and method for debriefing with good judgment. Simulation and Healthcare, 1, 40–55. Schank, R. & Abelson, R. P. (1977). Scripts, plans, goals and understanding; an inquiry into human knowledge structures. Hillsdale, NJ: Lawrence Erlbaum. Trabasso, T., Bower, G. H. & Gelman, R. (1968). Attention in learning: theory and research. New York: Wiley. van der Spek, E. D., Wouters, P. & van Oostendorp, H. (2011). Code red: triage or cognition-based design rules enhancing decision making training in a game environment. British Journal of Educational Technology, 42, 3, 441–455. Verhulsdonck, G. & Morie, J. F. (2009). Virtual chironomia: developing non-verbal communication standards in virtual worlds. Journal of Virtual Worlds Research, 2, 3, 1–10. Vygotsky, L. S. (1978). Mind in society: the development of higher psychological processes. Cambridge, MA: Harvard University Press. Walker, V. L. (2009). 3D virtual learning in counselor education: using second life in counselor skill development. Journal of Virtual Worlds Research, 2, 1, 1–14. Wrzesien, M. & Raya, M. A. (2010). Learning in serious virtual worlds: evaluation of learning effectiveness and appeal to students in the E-Junior project. Computers and Education, 55, 178–187.
© 2012 British Educational Research Association
