Education and Information Technologies 5:4 (2000): 291±304 # 2000 Kluwer Academic Publishers, Manufactured in The Netherlands
Design of Interactivity in Virtual Reality Applications with Emphasis on Educational Software Using Formal Interaction Speci®cation COSTAS N. DIPLAS* Educational Software Development Laboratory, Department of Mathematics, University of Patra, Greece. Mailing address: ESDLab, P.O. BOX 1399, zip code: 265 00, University of Patra, Patra, Greece. * Corresponding author. E-mail:
[email protected] PANAYOTIS E. PINTELAS Educational Software Development Laboratory, Department of Mathematics, University of Patra, Greece. E-mail:
[email protected] Virtual Reality (VR) technology has already entered into the area of the educational software and delivers systems where the trainees can use interactive virtual microworlds and bene®t by transfer of experience, interacting directly with the learning domain. This paper describes the Virtual Multi Flow Graph (Virtual-MFG) graphical formal model and the Interaction Speci®cation Workspace (ISW) software architecture for the interaction speci®cation and design of VR applications with emphasis on educational software. The interaction designer speci®es the interaction issues of the ®nal system formally, using the tools of ISW. The virtual microworld's objects database is updated with these interaction speci®cations which include both the virtual objects' dynamic properties and their tutoring capabilities. The model is validated by applying it on an existing VR educational software (EIKON). The Virtual-MFG graphs specifying a learning scenario of EIKON along with the application of ISW on EIKON are also presented. Keywords: virtual reality; interaction speci®cation; interaction design; educational software; formal speci®cation.
Towards a Framework for the Interaction Speci®cation and Interaction Design of Educational Virtual Reality Software Virtual Reality (VR) is related with a range of applications which focus on the visualization of physical systems, using interactive three-dimensional (3D) graphics. The prime effort is more on the high quality graphics of the virtual objects and scenes rather than the establishment of software design issues. A range of systems for virtual worlds building, objects modeling for various application domains, graphical representation of the information retrieval process and commercial VR development platforms have been presented (Superscape 1998), (Sense8 1996), (Card et al., 1991). These systems provide editors for the description of the objects' geometry and a set of subroutine libraries, device drivers and support programs and usually a high-level programming language for the description of the virtual objects' dynamic properties and
292
DIPLAS AND PINTELAS
behavior. The developers of a VR application use these systems and tools during the implementation phase. The most well known VR development platforms regard the interactive dialogue between the ®nal user and the VR microworld as part of the application's functionality but they do not include dedicated tools for the speci®cation of this dialogue. The software designers usually extract the speci®cations and produce the speci®cation documents in informal text mode or use general-purpose software for the de®nition of the speci®cations. This leads to speci®cations which are dif®cult to be revised and consequently leads to ®nal systems which may not meet the initial requirements and users needs. Prototyping is critical for the design and development of conventional software applications, but it is more critical especially for the design and development of Educational Virtual Reality (EVR) software, because the structuring of the application content and the micro-worlds may prove too complex, depending on the learning scenario and the instructional strategy to be followed. The functional properties of the produced EVR objects are easier to specify, revise and supervise than the behavioral ones. So, the designers and developers of such complex and large EVRs need a great degree of supervision over the ®nal EVR's objects behavioral properties. These behavioral properties will determine the success of the ®nal EVR as a training tool. This fact forces the presence of a strong prototyping mechanism during the design time. The designers and developers use tools with conventional user interface which is either text or windows based and produce applications with three-dimensional and immersive interfaces. But the user-trainee of the ®nal VR application interacts more with 3D graphics rather than conventional user interface widgets. This dissimilarity poses a semantic gap between the interaction metaphors the designers and developers of VR software use, and the interaction metaphors the ®nal user uses. The problem which arises is that the designers can not easily determine possible problems (such as kinesthetic) the ®nal user meets inside the VR microworld. Our work attempts to provide the authors-designers with a framework for the speci®cation and design of the target EVR applications. The authors have not been able to trace any similar framework in the literature. Thus comparison with such a framework is not possible. The proposed framework includes the Interaction Speci®cation Workspace (ISW) architecture for the design of EVR applications and the Virtual Multi Flow Graph (Virtual-MFG) as the underlying interaction speci®cation formal model. The designer uses a 3D editor included in ISW and constructs 3D graphs which are composed of Virtual-MFG components. These graphs are substantiated formal speci®cations and represent the interaction and instructional issues of the target EVR. The ISW architecture incorporates tools for the analysis of these speci®cations and produces executable speci®cations for prototyping purposes. The paper is organized as follows: in the next subsection, the current speci®cation models and VR software architectures, modeling tools and languages are presented. The second section presents the Speci®cation, Design and Prototyping Framework. The third section presents the design issues of EIKON educational VR software and the validation of
DESIGN OF INTERACTIVITY
293
Virtual-MFG and ISW on EIKON. The paper concludes with the summary conclusions in the fourth section.
Current models for the speci®cation of the interactive dialogue and VR software architectures, modeling tools and languages Formal speci®cation models and cognitive models have been proposed and established for the design of conventional software applications. The Petri Nets (Murata, 1977) are a powerful graphical modeling tool with ®rmly incorporated mathematical foundations that represent a system as a set of interacting active and passive components. In the graphical representation, a PN is a bipartite, weighted, directed graph with two types of nodes (transitions and places), and arcs from a node of one type to a node of the other type. The subject of Cognitive Ergonomics is the modeling of the cognitive behavior of computer users and constitutes an application of Cognitive Science to the ®eld of interactive computer applications. On the subject of modeling cognitive tasks, the GOMS model (Card, 1983) must be mentioned. Regarding VR, no interaction speci®cation models have been presented so far, even recently thought interaction design was recognized as an important issue for the implementation of highly interactive VR systems that exceed a simple 3-D interface. Most architectures and systems that have been presented in the literature are general purpose systems and have not being speci®cally designed for the development of EVRs. This application area can exhibit theoretical frameworks and working prototypes (Bricken and Byrne 1992). Among the most prominent approaches and attempts that have been presented in the area of architectures are the VB2 and the AVIARY architectures (Snowdon and West, 1994). In the area of 3D tools, the Conceptual Design Space (Bowman and Hodges, 1995), the MR (Minimal Reality) Toolkit (Shaw et al., 1993) and the Virtual Reality Interface Toolkit project (Billinghurst and Savage, 1996) which focus on the design of software tools for the production of VR systems, must be mentioned. On the domain of agent technology in VR systems, the Agent-Based Architecture to Developing Intelligent VR-based Training Systems (Lin et al., 1999) forms an approach for a communication architecture of the instructional software agents that exist inside an EVR application. The Virtual Reality Modeling Language (VRML) (The VRML Consortium Incorporated, 1997) is an interchangeable ®le format and an international standard for the description of VR microworlds, and many internet-intranets VRML browsers have already been introduced. Many of the design issues and approaches that presented previously, have been adopted by software manufacturers and a list of VR development systems have already been presented, like Superscape's VRT, Sense8's WorldUP and World2World, VREAM's VR Development System, Autodesk's Cyberspace Developer Kit, StrayLight's PhotoVR, etc. Most systems include the capability of importing and exporting VRML format ®les.
294
DIPLAS AND PINTELAS
The Speci®cation, Design and Prototyping Framework (SpeDeProF) The proposed Speci®cation, Design and Prototyping Framework (SpeDeProF) includes: the ISW generalized software architecture, the Virtual-MFG as the underlying graphical formal model of ISW, a methodology suggesting the following main phases for EVR applications design and development: i) the microworld construction, ii) the tutoring interaction speci®cation and design, iii) the prototyping in-cycles via executable speci®cations and iv) the testing and ®ne tuning of the ®nal EVR application.
The Virtual Multi Flow Graph (Virtual-MFG) graphical formal model Virtual-MFG is a graphical formal model for the interaction speci®cation with capabilities to represent the instructional aspects of the target EVR application. It is based on highlevel, colored Petri Nets (Murata, 1977) and de®ned as an extension of IMFG (Kameas, et al., 1994), which was developed and used for the speci®cation and design of conventional interactive applications. Virtual-MFG incorporates the powerful analysis techniques of Petri Nets and the most prominent features of dialogue description models and cognitive models of user behavior. The graphical notation of Virtual-MFG makes the speci®c model one of the most suitable, among other similar approaches (Harrison and Duke, 1994) for interaction speci®cation in VR applications. The components of Virtual-MFG model are: Actors, that correspond to the actions, tasks or goals of the users of the EVR application's elements. Links, that correspond to the visualization of the different information ¯ows that occur in an interactive EVR application. Tokens that correspond to the data or=and control information that exist into the EVR. Tokens represent abstract data or control structures that are produced or consumed by the Virtual-MFG components. The actors are always preceded and followed by links and produce or consume tokens. The links store tokens. The Virtual-MFG components have a 3D representation symbolism shown in Figure 1. The user and the EVR application's virtual objects are all considered as participants acting inside a common space and may cause events. Actors model the interactive responses that must be performed by these participants, as a consequence of the occurrence of an event. There are four types of actors: Action actors: they represent a single response which is performed by any EVR participant. Actors have an internal structure which contains a rules part. The rules part of each action actor de®nes how this single task is implemented. Virtual-MFG does not
DESIGN OF INTERACTIVITY
295
Figure 1. Virtual-MFG components symbolism.
give a clear description of the task to be done, but speci®es the goal decomposition or the task analysis in order for the goal to be achieved. Context actors: their internal structure represents the task or goal decomposition into sub-tasks or sub-goals, via a number of other context or action actors. Guide actors: are used to represent the way the task or goal decomposition is achieved. AND and OR decompositions are provided, since these two fundamental actions can model any task or goal decomposition. For complex task representation (e.g. task interleaving, task interruption etc.), the Virtual-MFG model provides the actor-ready list, the condition links and the link usage properties (read-only, debit, OK). Virtual actors: used for a graphical grouping of actors, without any other signi®cance and serve as reusable components during design process. Links describe the situation that precedes and results from a user or system action, through the storage of tokens. Links too have an internal structure similar to that of actors. There are six types of links: Event links: Describe the events that are caused by the participants in the EVR application. Event links can be used to describe the communication of events either in the dialogue between the user and the EVR virtual objects (external dialogue) or in the dialogue between the EVR virtual objects (internal dialogue). In VR applications the events may be more composite compared with other interactive applications. For example, in immersive VR applications, the user causes events using hardware interface devices such as data gloves or exoskeletons and uses more complicated interaction methods than a simple key-press or a mouse movement. A number of diverse interaction methods have already been proposed in (Fuchs and Bishop, 1992) and the event generation process in numerous cases (e.g. where gesture interaction methods are
296
DIPLAS AND PINTELAS
used) includes an internal structure. Virtual-MFG provides event links decomposition, so the designer can explicitly specify how events can be caused. Condition links: represent the global or local conditions that precede and result from any participant action that takes place into the EVR. Consequently they represent priority of execution and availability of actors. Moreover condition links describe whether any of the EVR participants is ready to process another participant's action, that will lead to the achievement of a subgoal into the EVR. Data links: represent the data or control ¯ow, inside the EVR application. Moreover, they represent the content of messages that pass between the participants. Context links: represent the context in which a number of interactions take place. Consequently, context links describe the context to which a speci®c goal or task belongs and indicate whether a major goal is decomposed into sub-goals, so that a new session for the accomplishment of each sub-goal starts. Moreover, context links contribute to the representation of the system's memory and knowledge issues which are of major importance for the EVR applications. Every actor that belongs to a speci®c context knows the goal of all the other actors that belong in the same context and the Virtual-MFG can model long-term memory by maintaining the actor-ready list and the content of context links, since the short-term memory is represented directly inside the current context. Communication links: model the effects that the separate micro-worlds existing inside the entire EVR application may have on one another, since there exists a separate VirtualMFG for each virtual micro-world. Learning Links: are created in order to provide the designers-developers of EVR applications with capabilities to represent the instructional and tutoring aspects of the target EVR application. Learning links are classi®ed as follows: ± Learning Context Links: Specify the tutoring and learning context in which the current trainee-system interactions take place. Moreover, specify the current learning scenario which is applied, as this will be shown into the next sections. ± Experience Ampli®cation Learning Links: Are used to represent system responses that are directed from the input, or to the output devices, in order to provide the user of the EVR application with the capability to perceive with natural ways special EVR responses (e.g. haptic feedback, position orientation feedback activities etc.). These links make Virtual-MFG model capable of representing immersion techniques with input-output sensors and advanced hardware interface devices. ± Assessment and Evaluation Learning Links: These links model the activation of the software module which is responsible for the application of a student assessment and evaluation mechanism. ± Simulation Learning Links: Specify the execution of a pre-recorded simulation, demonstration or navigation in an EVR, without user operation.
DESIGN OF INTERACTIVITY
297
The ISW Architecture The Interaction Speci®cation Workspace is a software architecture for the speci®cation and design of EVR applications and includes three major modules: The Interaction3Design Editor, the Prototyping Subsystem and the Target EVR Application which is as an external entity belonging partially to ISW. The virtual objects' geometry, the real-time graphics, the lights of the virtual scene etc, are produced with a third-party VR development tool. Figure 2 presents the overall structure of the ISW architecture. The interaction designer uses the Interaction3Design Editor and constructs the VirtualMFG 3D graphs that represent the interactive and tutoring sessions of the target EVR application. The Interaction3Design Editor is a 3D tool which proposes the `virtual programming' metaphor, as the evolution of the contemporary visual programming. In Figure 3 a snapshot of the Interaction3Design Editor is depicted. The Translator extracts the lexical, syntactical and semantical data and the Analyzer uses these data in order to check the graphs for correctness, evaluate the integrity of the interaction sessions and produce the Virtual-MFG Data Tables. The Presentation and Dialogue Managers read the Virtual-MFG Data Tables and generate the corresponding VRML code. This VRML code is the Interaction Speci®cation Data. The Virtual Microworld VRML ®le is updated with this generated code. These interaction speci®cations, which include both the virtual objects' interactive and instructional properties, are executable because the target application's Virtual Microworld VRML ®le is now updated with them, and the designer is now able to evaluate the results which are related with the interaction and instructional issues. This enables the designer to take the place of the user-trainee and check if the application meets its initial requirements. A short description of ISW can be found in (Diplas et al., 1997). Validation of Virtual-MFG Formal Model and ISW Architecture The EIKON EVR educational software (currently under evaluation in a number of selected high schools in Greece) is used as complementary tutoring material to the subject of `Technology'. Being a prototype for only six class hours, it is concentrating on the domain of `agricultural technology' from ancient times to present. EIKON comprises four VR microworlds and can run over a Local Area Network or the Internet (Pintelas et al., 1999). In the design and development of the EIKON EVR application (Figure 4), the VR technology is combined with hypermedia and database software, in order to enable the tutor to customize the educational software and affect, modify or re®ne the functional and behavioral properties of EIKON's objects. Moreover, EIKON incorporates software components for student monitoring and classroom management, and provisions for the evaluation-assessment of the students during a session. The TractorAssemblageScenario is a part of the overall learning EIKON scenario named `Constructions in the Modern Times' which is described to the student as: `Assemble the tractor and turn it on. Take screenshots of all the parts. Use presentation software to make a presentation about the agricultural tasks where the tractor is used. Discussion with students
298
DIPLAS AND PINTELAS
Figure 2. ISW Structure.
about the agricultural technology in modern times with emphasis in ploughing, follows'. In the TractorAssemblageScenario which will be speci®ed with Virtual-MFG graphs in this section, the students must assemble the wheels of the tractor. The wheels lie next to the tractor and the student must assemble all the wheels, carrying them and placing each of them on the correct tractor's axis. This is the major user-goal. The user plan-goal decomposition approach will be applied in order to specify the way the user-student must assemble all the tractor's wheels. This user goal is directly decomposed into two subgoals: view both, the tractor and the wheels by selecting an appropriate viewpoint and assemble each wheel onto the tractor. Three representative graphs will be presented, one for the representation of the AssembleAllWheels main goal and two for the representation of the View and AssembleEachWheel subgoals. In Figure 5 the main goal is decomposed into the two sub-goals, using the AND guide actor (represented with the cylinder which includes the rest Virtual-MFG components). In order to accomplish this main goal, the student must navigate inside the EVR. This navigation is speci®ed with the Navigate event
DESIGN OF INTERACTIVITY
299
Figure 3. Screenshot of the Interaction3Design Editor.
link. The AssembleAllWheelsGoal context link speci®es the current interaction context, and the AllWheelsOK condition link stands for checking if the wheels are mounted onto the tractor's axis correctly. The EIKON Students Activities Recording Subsystem monitors the student-EVR learning interaction, keeps track of the student's activities and records them into the Students List (Figure 4), for further evaluation and assessment. The TractorAssemblageEvaluation tutoring learning link speci®es the activation of this subsystem, while the current learning scenario is described by the TractorAssemblageScenario learning context link. Figure 6 presents the View context actor which comprises the FkeyViewpoint and the FreeViewpoint action actors. The former ®res when the function key assigned to the correct viewpoint is pressed and the latter ®res when the user selects the correct viewpoint by free navigation using the mouse. Tokens are produced into the ViewOK and ViewGoalOK links when either of the two action actors FkeyViewpoint and FreeViewpoint ®res. In Figure 7 the AssembleEachWheel context actor is depicted. This actor ®res when a token is produced into the ViewOK condition link. This link ensures the correct sequence of the user actions. The AssembleEachWheel context actor includes the ClickAndHoldWheel and MoveWheelAndDropOnAxis action actors with the use of an AND guide actor, so both actions actors must ®re, resulting that every wheel is mounted on its correct
300
DIPLAS AND PINTELAS
Figure 4. EIKON High-level Architecture.
position (speci®ed by a token into the EveryWheelReady condition link) and the assemblage of the wheels onto the tractor is realized (speci®ed by a token into the AssembleEachWheelOK context link). The tutoring interaction session completes successfully when all the wheels are mounted onto the tractor and the speci®c part of the current learning scenario ends.
DESIGN OF INTERACTIVITY
Figure 5. The decomposition of the user's main goal.
Figure 6. The View context actor.
301
302
DIPLAS AND PINTELAS
Figure 7. The AssembleEachWheel context actor.
Summary Conclusions The Speci®cation, Design and Prototyping Framework (SpeDeProF) provides the EVR designer with the Virtual-MFG graphical formal model in order to analyze the tutoring interaction speci®cation formally. The model is supported by the Interaction3Design Editor, a 3D editor used by the designer in order to construct the Virtual-MFG graphs. The ISW architecture is platform independent because it uses the VRML standard. Similarly, ISW is application and domain independent and can be used for the interaction speci®cation of any educational (or otherwise) VR application. This allows the EVRs designers and developers to use any VR authoring tool which imports-exports VRML for the construction of the virtual objects and the educational application's virtual microworld. In addition, SpeDeProF shall provide direct prototyping of the speci®ed EVR via the ISW architecture's components and the executable interaction speci®cations which it produces. This is very useful for the EVR designers who only need to design the pure interaction speci®cations and supervise the target EVR interactivity aspects. They do not get involved directly with the de®nition of the virtual objects physical properties, the images and textures rendering, the photorealistic aspects and other similar tasks, which can be done by the developers using a VR authoring tool.
DESIGN OF INTERACTIVITY
303
Also, SpeDeProF bridges the semantic gap and provides uniformity between the design space and the executable one, proposing the `virtual programming' metaphor, as the evolution of contemporary visual programming. Advantages of viewing 3D (graphs in our case) formation, as opposed to the 2D, are reported in (Ware and Franck, 1994). The usage of a 3D editor for the interaction speci®cation in EVR applications, helps the designer to think in user terms. The feedback from the users-designers will be reported in future publication when the implementation of ISW completes. The ISW architecture is intended to provide workgroup features and capabilities for remote collaboration among several designers using a common repository of Virtual-MFG components in VRML format, over a LAN or the Internet. Acknowledgments The described work is part of the project X-Genitor (PENED 99ED68-2285) funded by the General Secretariat for Research and Technology of the Ministry of Development and the European Social Fund. References Bowman, D.A. and Hodges, L.F. (1995). User interface constraints for immersive virtual environment applications. Graphics, Visualisation and Usability Center, Georgia Institute of Technology, TR GIT-GVU-95±26. Billinghurst, M. and Savage, J. (1996). Adding Intelligence to the Interface. In Proceedings of the IEEE 1996 Virtual Reality Annual International Symposium. pp. 168±176. Bricken, M. and Byrne, C.M. (1992). Summer Students In Virtual Reality: A Pilot Study On Educational Applications Of Virtual Reality Technology, (unpublished paper), Human Interface Technology Laboratory (HITL), Washington Technology Center (WTC), University of Washington (UW). Card S.K., Robertson, G.G. and Mackinley, J.D. (1991). Information Visualizer, An Information Workspace, In Proceedings of SIGCHI, pp. 181±188. Card, S., Moran, T.P. and Newell, A. (1983). The Psychology of Human-Computer Interaction. Lawrence Erlbaum Associates, Hillsdale. Diplas, C.N., Kameas, A.D. and Pintelas, P.E. (1997). The Interaction Speci®cation Workspace: Specifying and Designing the Interaction Issues of Virtual Reality Training Environments From Within. In Design, Speci®cation and Veri®cation of Interactive Systems '97, Harrison-Torres (eds), Springer-Verlag Wien, New York, pp. 241±256. Fuchs, J. and Bishop, G. (1992). Research Directions in Virtual Environments: An Invitational Workshop on the Future of Virtual Environments. The University of North Carolina at Chapel Hill, Department of Computer Science, TR92027. Harrison, M.D. and Duke, D.J. (1994). A Review of Formalisms for Describing Interactive Behavior. Amodeus Project Document: System Modelling=WP28. Kameas, A., Diplas, C., Gerogiannis, V. and Pintelas, P. (1994) Encapsulating multiple perspectives in interaction speci®cation. In Proceedings of 20th EUROMICRO Conference, Liverpool. Lin, F., Su, C-J. and Tseng, M. (1999). An Agent-based Approach to Developing Intelligent Virtual Reality-based Systems. In Proceedings of the 11th IEEE International Conference on Tools with Arti®cial Intelligence, Chicago, In press. Murata, T. (1977). Petri-Nets: properties, analysis and applications. In Proceedings of the IEEE, 77(4), pp 541±580. Pintelas, P., Kameas, A., Triantis, A., Vathis, S., Koutalieris, G., Mikropoulos, A. and Katsikis, A. (1999). Design of Virtual Reality Educational Software: EIKON. In Proceedings of the 4th Panhellenic Conference on Didactics of Mathematics and Informatics in Education, Rethimno, Greece, (In press).
304
DIPLAS AND PINTELAS
Shaw C., Green M., Liang J. and Sun Y. (1993). Decoupled Simulation in Virtual Reality with the MRToolkit. ACM Transactions On Information Systems, 11(3), 287. Snowdon, D. and West, A. (1994). The AVIARY Distibuted Virtual Environment. In Proceedings of the 2nd UK VR-SIG, Reading, pp. 39±54. Sense8 Corporation, WorldUp User & Reference Manual (1996). Superscape Ltd. VRT 5.0, Reference Manual (1998). The VRML Consortium Incorporated (1997). The Virtual Reality Modeling Language±VRML97, ISO=IEC 14772±14771. Ware, C. and Franck, G. (1994). Viewing a graph in a Virtual Reality Display is Three Times as Good as a 2D Diagram. In Proceedings of 1994 IEEE Conference on Visual Languages, S. Louis, pp. 182±183.
