An Interactive Augmented Reality System for Engineering ... - CiteSeerX

 2002 UICEE

3rd Global Congress on Engineering Education Glasgow, Scotland, UK, 30 June - 5 July, 2002

An Interactive Augmented Reality System for Engineering Education F. Liarokapis, N. Mourkoussis, P. Petridis, S. Rumsey, P. F. Lister, M. White University Sussex Falmer, UK ABSTRACT: This paper presents an innovative approach to enhance students’ learning and understanding of digital design using complex commercial design flows. The architecture of our experimental system is based on new technologies such as Augmented Reality (AR), XML metadata, and an XML integrated database system. The database provides the content for multimedia presentation in a virtual environment, visualised through AR, which enables students to engage effectively in the process of learning. These include textual descriptions, animated videos, auditory information, images and 3D computer generated objects of related diagrams and designs. An XML interface will allow the teacher to input multimedia information into the database remotely in a simple way. Using an AR interface the content of the database will be visualized in real time using state-of-the-art virtual reality technologies, i.e. head mounted displays, tracking devices, etc. Through this interface, users will be able to interact collaboratively. Using collaborative AR the students will also develop team skills that play a significant role in the learning process. Finally, we provide an illustrative description of how our experimental system will operate, and we present some initial results.

INTRODUCTION Modern education technology is becoming a more complex task than it used to be some years ago. Although the potential benefits of today’s technology to improve the teaching and learning process is obvious, few results have been achieved compared to its technology capacity. Already, some universities have started to develop distance-learning courses utilizing video conferencing and other online services to provide high standards of teaching, but the cost is extremely high. Currently many teachers and academics make use of multimedia tools to present high-quality content learning material. These include projectors, video clips, auditory devices, transparencies and computer generated tools such as simulations [1].

Our current goals are to implement the VMC and extend the concepts using more current technologies such as XML Web Services [3] applying them to an augmented reality environment for engineering education. Underlying XML technologies are thus: SOAP, WSDL, teaching and learning XML and AR. Our system offers the teacher the ability to use more sophisticated techniques that enable better user interaction with teaching materials. Also it gives students a high degree of flexibility and understanding of the teaching materials by providing them in an interactive and augmented way. Depending on the complexity of the application there

OVERALL ARCHITECTURE The architecture of our prototype system is presented in Figure 1. It consists of three parts; the consumer side, the producer side and the server side. Producer Side Modelling and Authoring Tools

Augmented Reality Hardware

File System Visualization (AR Browser)

Material Editor Interface

XML Parse

XML Query

Web Server

XML Web Services CMS Producer

CMS Consumer

DBMS

Database Database Server

Server Side

Figure 1 – System Architecture

Consumer Side

In October 2000 we proposed a new approach to the teaching of top down design of VHDL using a novel Virtual Interactive Teaching Environment (VITE). This environment enables students to learn more effectively using virtual multimedia content, while exploiting XML, and augmented reality. This environment can be adapted for teaching of other subject areas. VITE is an AR teaching system that can provide teachers or trainers with 3D visualisations of teaching materials in an augmented reality environment, e.g. example designs, tutorials, 3D objects, reference manuals, etc. VITE has to support communication and interaction between the users, i.e. the students and the teacher, within the virtual environment. The virtual multimedia content (VMC), which is stored in a database and visualised on an appropriate medium, is the information interface between the AR technology and the users [2].

will be different levels of abstraction and interaction. Further, we believe that a VE will provide a rewarding learning experience that is otherwise difficult to obtain [4]. The practicality of this approach is that it helps the students to visualise basic and advanced principles and therefore test their learning throughout virtual scenarios. Using VMC and AR the students will be able to visualize real life three-dimensional examples of the principles they are studying in real time with the help of dedicated virtual reality hardware.

This architecture is similar to the system we are developing for another project called ARCO (Augmented Representation of Cultural Objects) [5] whose goal is to implement digital collections of museum artefacts for use in E-commerce based virtual museums. Parallel can be drawn between the underlying technologies in that ARCO also adopts XML Web Services to deliver visualisation of virtual representations of museum artefacts over the Internet.

system. The web services (Middleware layer) are comprised of a web-server and its services [6], provided to the clients. The data services (Back-End layer) consist of the database and the DBMS.

Server Side The server side contains a database server, XML Web Services and a web-server. The database server consists of a Database Management System (DBMS) and the Database. The DBMS is responsible for providing a number of functionalities to the application. These include initially the creation of the database and then querying and updating it where required. Other functionalities are the maintenance of data constraints and integrity and the restriction of unauthorised access. Another issue is the implementation of multiple data interfaces, views, and reports and the provision of backup and recovery. Finally, control of concurrent access is necessary for the operation of classroom scenarios. Moreover, the Database of our system is allocated with three core functionalities; the provision of persistent storage for the material, the reflection of our XML schema or DTD to the database schema, and the specification of the VMC for the teaching material. The XML Web Services are based on SOAP, WSDL, UDDI, and DISCO [3]. The XML Web Services expose the VMC to the user, i.e. our students. Connectivity between the database server and the client programs is provided by SOAP, with the SOAP payload being an XML file containing the VMC that is parsed and consumed by the client visualisation browser.

Producer Side The producer side is responsible for creating the VMC data, e.g. 3D data (e.g. avatars) using state of the art modelling tools (3ds max), text, etc. The VMC is temporarily stored into the File System (FS) for further usage. Through the material editor interface the VMC data is packaged into an XML file based on an XML Schema for transmission to the Database server where the CMS Producer (i.e. database Content Management System) writes it to the database.

Consumer Side The last part is the consumer side whose main operation is to query the database via an XML query file generated from the visualisation browser and parse the results via an XML parser. The visualisation browser consumes the parse tree data from the XML parser. The VMC data from the database is thus visualized in real time using Augmented Reality dedicated software and hardware.

THREE-TIER ARCHITECTURE A flexible way of organizing distributed client-server systems is achieved through three-tier architectures where each client is connected to the server. Furthermore, both the server (BackEnd layer) and the client programs (Front-End layer) interact with the intermediate layer (Middleware layer). Figure 2 illustrates the three-tier architecture applied to our application. Our client (Front-End layer) consists of the tools that are provided both by the consumer and producer sides of our

Figure 2 – Three-tier architecture Furthermore, Figure 2 illustrates a logical mapping of the physical architecture. The latter offers a possible scenario of languages and tools that will be used in order to implement our physical architecture. For instance, the object modeller will take advantage of the 3ds max functionality. The use of the three-tier architecture provides significant advantages to our application. The modules used may re-usable and they can be modified separately without the need to provide new access mechanisms. In explanation the look and feel of front-end clients may be modified without the need of altering or adjusting the other two tiers of the system. Finally, the system may be deployed in multiple physical locations [7].

OBJECT MODELLER Through the use of 3ds max 4 the producer side can make the learning process easier and more effective since the student will be provided with a graphic representation of the teaching material. The teaching material can be modified easy and efficient via our modelling and authoring tools. The Object Modeller with the use of animation, numerous key new feature additions and architectural enhancements compliment these three major initiatives, making 3ds max 4 an ideal tool for the 3D animation industry, can create the presentation of the teaching material in a more professional standard. In addition, the interface can be customized according to the user needs and level of expertise, so as to create a personal workspace. Finally, the producer side can export the 3D model in any format.

MATERIAL EDITOR INTERFACE The material editor interface is written in Java and incorporates a dynamic GUI based upon the underlying XML Schema or DTD that is reflected in the database schema. Offering flexibility, portability and customisability, the material editor interface helps the users to concentrate on content and data structure relevant to the database structure while generate wellformed and valid XML based VMC data. We have developed a DTD parser that us to dynamically generate a new material editor interface when the XML based VMC schema evolves. A java programme generates a new material editor interface as the DTD changes, so allowing for a virtual plug-n-play interface between different servers and file systems with minimal or no further development time or costs. Figure 3 and Figure 4 illustrate the evolving DTD and subsequent dynamic generation of the material editor.

Tracker Transmitter

Tracker Receiver

Bullet Camera

Personal Computer

Real World

Augmented View

AR Software

3D Studio Max

3D Models

Head Mounted Display

Figure 5 – Augmented Reality System

Figure 3 – Example of a GUI based on initial DTD

Figure 4 – Material editor interface adapts to the new DTD

The Augmented Reality System consists of a standard desktop PC, a magnetic tracker, a Head Mounted Display (HMD) and a bullet camera. The bullet camera represents the users eyes and its operation is to capture the real world. The tracking device (inertial or magnetic) is mounted on the camera. The tracker tracks continuously the position of the camera with six degrees of freedom in respect to a reference point. The output of the camera and the tracker are processed to provide the video camera model in the 3D world. Thus, 3D objects (generated by the object modeller) can be registered in the real world viewed in the HMD. We have chosen to use either the freeware, e.g. ARToolKit [10], or Sense8 WorldUP [11] to design the visualisation browser. The users can then visualize the synthetic objects inside the real world (the laboratory environment in this case). In Figure 6 and Figure 7 some of the hardware equipment used is illustrated. In reality, the magnetic tracker is too expensive an option, but the Intersense’s inertial trackers are a very good cheaper alternative

VISUALIZATION THROUGH AUGMENTED REALITY A promising area of Computer Graphics (CG) is AR, which is not supposed to be a single technology but a collection of different technologies that operate in conjunction to enhance user’s perception of the real world with digital information. The main characteristic of AR is that it enhances information from the real world, using computer-generated objects onto the user’s world-view. The real word must be matched with the virtual in position and context in order to provide an understandable view. An ideal AR system will be able to compose computer-generated images or videos with the real world in real time in such a way where the user could not tell the difference. Ideally, we can consider an augmented reality system to be the ultimate immersive system [8]. Our interactive system is focussed on both the technological and educational issues so AR technology has to be planned to operate effectively. This careful planning and use of AR technology will then lead to an improvement in the teaching and learning process [9]. This issue is of particular importance, as students will most certainly need to adapt to the way AR hardware is used. This phase of the learning process needs to be short and well planned in the teaching model. It can also provide additional stimulus for the students, and lead to a better understanding of the principles taught. The overall architecture of the AR system utilized is illustrated in Figure 5.

Figure 6 – Example of a user using an HMD

Figure 7 – The magnetic tracking device used [12]

CONCLUSION AND FUTURE WORK The prototype system presented here that is currently being developed is focused on enhancing the teaching and learning process suing state-of-the-art XML technologies and augmented reality techniques. One disadvantage of our system is the cost of hardware, e.g. the magnetic tracker, and HMD, etc. but our proposed system is much more cost effective than a proprietary fully VR immersive system. The system we are currently implementing could be adapted and applied to other educational and commercial applications. For example, other teaching and learning or training environments may include: surgery operations, civil and military services, visualisation for business, architecture and science, entertainment and others [13]. Future work will involve integrating the system components currently being implemented by several doctoral students.

AKNOWLEDGEMENTS Part of this research was funded by the EU IST Framework V programme, Key Action III-Multimedia Content and Tools, Augmented Representation of Cultural Objects (ARCO) project IST-2000-28366.

REFERENCES

4.

Roussos M., Johnson A., Leigh J., Vasilakis C., Barnes C., Moher T., NICE: Combining Constructionism, Narrative and Collaboration in a Virtual Learning Environment. Computer Graphics, A publication of ACM SIGGRAPH, 31,3, 62-63 (1997).

5.

ARCO Consortium, ‘Augmented Representation Cultural Objects’, http://www. arco-web.org

6.

Chester T.M., Cross-Platform Integration with XML and SOAP. IT Pro, (2001).

7.

O’ Connel P., McCrindle R., Using SOAP to clean up Configuration Management. Proceedings of the 25th Annual International Computer Software and Applications Conference (COMPSAC’01), (2002).

8.

Vallino J., Interactive Augmented Reality. Department of Computer Science, Phd Thesis, University of Rochester, New York, 1-25 (1998).

9.

Jay E., White M., and Lister P. F., The Virtual Design Educator: Realistic or Optimistic? An Approach to Teaching Design Virtually. In Proceedings of the 7th National Conference on Product Design Education, 6-7 (2000).

of

10. Augmented and Mixed Reality, AR Toolkit, http://www.hitl.washington.edu/people/poup/research/ar.ht m#artoolkit.

Cooperstock J.R., The Classroom of the Future: Enhancing Education through Augmented Reality. HCI International 2001, Conference on Human-Computer Interaction, New Orleans, 688-692 (2001).

11. SENSE8 Virtual Reality 3D Software, http://www.sense8.com.

2.

White M, Jay E, Liarokapis F, Kostakis C, Lister P.F., A Virtual Interactive Teaching Environment (VITE) using XML and Augmented Reality. The interactive journal of Electrical Engineering Education, IJEEE, (2001).

13. Barrilleaux J., Experiences and Observations in Applying Augmented Reality to Live Training. Peculiar Technologies, 3800 Lake Shore Ave., Oakland, CA 94610, http://www.augsim.com/vwsim99/vwsim99.html.

3.

Tabor R., Microsoft.Net XML Web Services. Indianapolis, Indiana: SAMS, (2001).

1.

12. Polhemus – FASTRACK, http://www.polhemus.com/ftrakds.htm