scratch. With the use of the ARToolKit library the development of. MR/AR applications became .... can be used to build application specific MR components.
A generic framework for a training application based on Mixed Reality Michael Haller, Jürgen Zauner
Werner Hartmann, Thomas Luckeneder
Fachhochschule Hagenberg, MTD Hagenberg, Austria email [haller|jzauner]@fh-hagenberg.at
Institute for Applied Knowledge Processing (FAW) University of Linz, Austria email [whartmann|tluckeneder]@faw.uni-linz.ac.at
Abstract This paper describes the requirements for a training demonstrator based on Mixed Reality in the context of the AMIRE project. Moreover, a generic framework, that allows an easy communication between the objects (MR components) used in the AMIRE applications, is introduced. The overall objectives and ideas of the AMIRE project are to define and implement a software system that allows content experts to easily design and implement mixed reality applications without detailed knowledge about the underlying base technologies of MR. The AMIRE framework uses a component oriented technology and consists of a visual authoring tool for building MR applications.
the original vase. Dobler et al. [6] implemented the ASR (Augmented Sound Reality) application, in which the user can place virtual 3D sound sources in the real world and experience the real 3D sound environment. Both applications use the same object recognition library, the ARToolKit based on markers. Surely, both applications have more common problems and solutions (e.g. object loader, path animation, input and output interfaces etc.), which could or should be combined and offered by a library, too.
Keywords Mixed Reality, Authoring, Generic Framework, Training Application INTRODUCTION Most of the current MR (Mixed Reality) applications focus on the development of AR (Augmented Reality) base technologies [1]. A lot of Japanese MR projects sponsored by Canon and Sony resulted in useful insights in the area of research and development of AR and MR applications, but none of these projects used a structured authoring approach with reusable components [2]. Even many European projects mainly focus on the development of MR applications for a special domain (e.g. technical maintenance), such as ARVIKA, STAR, etc. [3]. Unfortunately, only MR experts are able to develop MR/AR applications and prototyping of MR based applications becomes a very difficult job, because most research institutes have to develop these applications from scratch. With the use of the ARToolKit library the development of MR/AR applications became easier and more popular. The ARToolKit library is free for non-commercial projects and easy to integrate in an OpenGL environment [4]. In addition, it offers the possibility to recognize objects without the usage of high-end tracking systems. Geiger et al. implemented a lot of different applications using the ARToolKit library [5]. In one of these applications they developed an interactive illustration that explains the shape of a museal artefact in the form of an antique vase. The idea is that visitors select physical fragments of the vase and the applications shows, how those fragments fits into
Figure 1: The Sound Augmented Reality Application allows the user to place 3D sound sources in the real world
Fuhrmann and Schmalstieg implemented the well-known Studierstube, an open source framework for the development of Virtual Environments [7]. Studierstube’s approach is an “operating system” for VR/MR based applications [8]. With the lack of authoring tools it becomes difficult for content experts to develop VR/MR applications. Studierstube offers a great solution for VR/MR experts. With the addition of features, the code will become complex and every time a content expert wants to add a new feature or object, he or she will need the help of a programmer. So programmers can become bottlenecks to creativity. If artists and content experts are empowered with the ability to experiment, the application is sure to look and play better. But current MR applications not only lack authoring tools – they still don’t have a component-based architecture. The idea of using
components for Augmented Reality applications has also been used in the DWARF framework [9], but in comparison to the AMIRE’s approach, the components are settled on a higher level. Even many games suffer from the same problem and do not use a generic framework with reusable components and an inter-object communication concept. Consequently, employing a component-based reuse of (third-party) solutions and providing a structured design method supported by visual authoring tools gives us the opportunity to reuse the results of the mentioned projects in a user-centered authoring framework. APPLICATION The AMIRE (Authoring MIxed REality) approach will be tested within the scope of two demonstrator applications. The first one, which is introduced later, will be implemented for the major Austrian oil company, the OMV. The OMV application should test the AMIRE approach at an early stage, to enable the development team to correct possible design weaknesses before they become critical. The second application, developed for the Guggenheim museum of Bilbao, will be built by using the complete AMIRE toolkit including and using the authoring tools. The museum application should demonstrate the usability and the applicability of the AMIRE framework. Application Goals The goal of the OMV application is to offer a different training tool beside common courses and lessons to transfer knowledge from highly skilled operators to new employees. The difficulties that arise by training a new employee at the refinery are illustrated in Figure 2. After learning basic skills a new employee can easily understand a schematic plan of a plant. But it is hardly possible for the new employee to match the schematic diagram of a plant and its “translation” into real machinery. Without the help of a trainer who is familiar with the refinery, a new employee would barely be able to learn how to operate the plant. The OMV application will provide an easy way to deal with this complexity. By using MR based technologies as described later, the application enables the user to easily find corresponding counterparts in the schematic diagram of the plant and the real plant. Figure 3 shows a typical use case of the OMV application. Firstly, the object recognition component of the application identifies a device of a plant and offers the user some ways to get more information about this specific device. Secondly, the user can browse as much of this information as he or she currently needs. The user can retrieve a textual explanation of the device or a schematic diagram of the identified device.
Figure 2: Schema and reality of an oil refinery
Application context The application will be used outdoors in the whole refinery. The user will move across this area wearing the necessary safety dress (special clothing, helmet, safety shoes and safety glasses) while he/she is using the application. During the walk-through in the refinery the user´s attention should not be diverted by the application. Thus, it is impossible to use HMD´s outside in the refinery. A HMD would cumber the user while he moves through the area. The user sees the environment through clumsy glasses, which could constrict his/her perception. An “Outdoor Solution”, consisting of a PDA or a Tablet PC with a camera, would be the best way to cope with the requirements of the OMV. The PDA doesn’t affect the user’s perception and does not hinder him/her.
Nonfunctional requirements End user device As mentioned earlier, the application should run on mobile devices and that do not draw off the attention of the user. According to this requirement, a PDA like a Compaq iPAQ or a Siemens Pocket Loox would be a good choice. On the other hand, the device should also provide a considerable 3D graphics performance that allows rendering at least 20 frames per second. A compromise would be, to use a small tablet PC, which is handy and offers a good performance. But the question of the end device remains still open. Environment The environment of the refinery is an area where highly explosive gasses can occur. Thus, any device that is used inside the refinery must be active explosion proof. Active explosion proof means, that the device or the operation of the device cannot cause an explosion. A description of explosion proof standards of the European community can be found in [10]. ARCHITECTURE AMIRE’s goal and objective is not to develop new base technologies for Mixed Reality applications, but to adopt existing solutions and provide them in a uniform way. Accordingly, the scientific objectives are focused in the area of component based modeling, Mixed Reality and human computer interaction especially in authoring environments. The AMIRE components: Figure 3: Typical use case of the OMV application
Functional requirements Several functional requirements can be identified within the scope of this application. Firstly, the application should provide MR/VR functionalities like video overlaying and object recognition/tracking. Secondly, it should provide a traditional GUI interface that offers an efficient control of the application. Both, VR/MR functionalities and GUI interface should be seamlessly integrated. The application should be easy to maintain and adaptable to changed refinery environments. Both, maintenance and adaptation should be possible without the knowledge of an MR/VR expert. The application that is developed during the AMIRE project only covers a single production line of the refinery. After the OMV application is accomplished and the AMIRE project is completed, it should be possible to roll out the application to the whole refinery. The creation of new scenarios should be possible for non MR experts and programmers.
architecture
includes
the
following
• MR gems, • MR components, and • MR framework Similar to existing gems collections (game programming gems, graphic gems) AMIRE will produce a MR gem collection containing efficient solutions to individual mixed reality problems. A gem could be an object recognition library, a 3ds object loader, a solution for media generation (2D, 3D, audio, text) or a high level animation. In our case we concentrate on MR based gems. The gem collection will be used in the AMIRE framework but can also be used in other Mixed Reality projects. It will be ensured that useful existing MR technology software units will be available in a well-defined way, so that they can be used several times in MR applications as well as in MR authoring tools. There will be MR gems for using Augmented Reality technologies (e.g. MR gems for identifying markers in a video image) as well as for using rich media (e.g. MR gems for handling and mapping video in a computer generated 3D scene). Typically, AMIRE gems can be reused in many different MR based applications. For example, a “magical lens”-gem that allows seeing through walls of a museum showing the location of a special art piece to the visitor may also be
Application
Authoring framework
The component interface, that includes a component loader unit that allows the integration of different well-defined objects into the system.
•
A message handling architecture, in which the message handler identifies the components that are being addressed, delivers the message and maps it to the corresponding function calls.
FRAMEWORK The framework contains a gem collection and interfaces for the inter-object communication. Three different communication concepts are integrated into the framework.
state
out-slots
MR components represent solutions for particular specific problems and they typically combine and extend MR gems towards advanced high-level features of a MR application. MR components feature a unified interface that easily allows configuring and combining them via the MR authoring tools using a suitable component model. For example, an object recognition gem, a path animation gem and a 3D object loader gem may be combined into a “magical lens”-component that illustrate the inner details of a real machine by providing a suitable animation of virtual machine parts.
•
in-slots
reused to explain the inner parts of a machine that cannot be physically opened by the trainee. The MR gems in turn can be used to build application specific MR components as well as a MR framework that defines how MR components can communicate with each other and can be integrated in a MR application.
Component
Component layer
Gem collection
Component interfaces
Gem interfaces
configuration interface
Inter-object communication layer
Figure 5: The component has in- and out-slots Figure 4: The framework of AMIRE's architecture
Figure 4 describes the different layers of AMIRE’s architecture. The base layer contains the inter-object communication layer which allows the communication of both, the components and the gems. This communication layer provides generic interfaces and mechanisms for component management, communication and authoring. In addition, it includes plug-in interfaces for easy gem extensions. Further a gem communication mechanism based on the listener concept is provided. The component layer is based on the gem collection. A component includes one or more different gems. Finally, the authoring tool and the application tool communicates with all underlying layers. The MR framework is the glue of the AMIRE project that integrates both, the MR gems and the MR components. The main requirements of the framework are: •
The integration of the Graphics Library (actually we tend to support OpenGL or/and OpenSG)
•
The integration of an object recognition unit (e.g. ARToolKit)
•
The integration of different tracking systems (e.g. Polhemus)
The first concept is the low level communication by calling methods (e.g. like a compiler). The second is the listener concept. It will be used for the high level communication between gems. It is a simplified model-view-controller concept, which notifies the view when the model changes [11]. In the listener concept there are only two things, a model and listeners. The model notifies the listeners after it has changed. Listeners have already been used in the Java framework for graphical user interface components [12]. The third concept is the communication concept between components. Each component has a set of named in- and out-slots (as seen in Figure 5).
Component B Component A Component B
Figure 6: Connected components
Out-slots can be connected to in-slots. This concept is illustrated in Figure 6. We use typing for these slots to specify the receiver and the emitter data. By comparing the
type of in- and out-slots we can decide which slots are compatible for connection. Using slots for inter-object communication is not new in C++. The graphical user interface library QT [13] uses a similar concept. It is based on signals and slots. QT signals conform with out-slots and QT slots with in-slots. QT can only connect objects at compile time. Further a meta object compiler has to improve the C++ standard.
Property type
Base property type
String property type
State A
State B Prototype Component Y Component Prototypes
Applicationwide states
Component manager
Figure 7: Component manager
Our communication concept is based on prototypes for components. This means the component developer registers the component at a component manager (as seen in Figure 7). So each component must know his name, which enables the creation of a new component instance by its name like in Java. In Java you can create a new object by a class lookup and the call of one of its constructors. In standard C++ this is not possible. For the implementation of a generic authoring tool such a mechanism is inevitable. So you have two possibilities. The first is to improve the language and the second means you have to use a manager containing prototypes. We have decided to use the manager, because it is more compatible with different compilers and uses only standard C++. Now we are able to create new instances of specific components like a geometry component, a navigation component or a movie component. But we also have to configure it in a generic way. This means we have to find a generic data structure with the ability to describe complex components. Java uses properties to configure generic components called beans [14]. These properties are not structured. So they are not practical to configure structures like components.
Float property type Integer property type
Char property type
Double property type
Prototype Component X
Vector property type
Boolean property type
Structured property type
Figure 8: Property types
We have improved the property concept by defining basic property types including boolean, character, integer, float, double, string and structure (as depicted in Figure 8). In addition, we have defined a vector property type for each basic property type. The structured property type is like structures known from programming languages. You have to define the name and type of all fields. After having defined the names and fields you have to register it under a specific name within a property type manager. This will make type comparison as easy as possible. You only have to lookup the structured property type by its name. Each lookup will result in the same instance. The properties are also used for the transfer of data. Each out-slot will emit a property with a specific type. The in-slots also expect a property of a specific type. In- and out-slots are connectable if the type of an in- and an out-slot match. The last problem on the way to a framework for a generic authoring tool is the persistence of components and connections. The component writer (as seen in Figure 9) and component reader (as seen in Figure 10) will handle this problem. Component manager
Component writer
Database
Binary stream format
XML format
Figure 9: Export of components and connections
Database
Binary stream format
recognition gem, combines the rendered image of the arrow with the video stream.
XML format
PDA
Component reader
with
mounted
camera
Technician equiped with PDA
Component manager
Figure 10: Import of component and connections
IMPLEMENTATION The collaboration of framework, gem collection and components can best be illustrated with a small example. Consider for instance a non-transparent tube that transports liquid and the task to show an operator or trainee the liquid’s direction of flow (see Figure 11). To accomplish this task, several different gems and components have to work together. •
An object recognition gem, e.g. ARToolKit, delivers basic geometric information like angular offset and distance between the camera and the tube, respectively between the camera and the marker that is applied at the tube.
•
A scene node reader gem reads the arrow’s definition from a file or a database and transforms it into a geometric model.
•
A rendering gem, e.g. OpenGL, renders the geometric model of the gem.
•
A video overlay gem that uses the video stream from the camera and the data delivered from the object
Additional technician solution
information to find the
helps right
Figure 11: Operator determines direction of flow
All these gems will be encapsulated into a component to provide a well-defined and structured interface. As the scene node reader gem, the rendering gem and the video overlay gem are tightly coupled, they will be combined into a single component that does the job of rendering the arrow and combining the video stream with the rendered arrow. This is a general-purpose component that can be used to augment any video stream with any rendered AR scene. As it is used to show the direction of flow of a liquid, it will be called “direction of flow component” within the scope of this application. The internal coupling of this component can be seen in Figure 12. Direction
In-Slot for current distance and angular offset
This abstraction allows the support of different storage formats like stream based files, XML-files or databases [15]. For the persistence of components you only have to store the configuration properties. So the export and import of components is reduced to the export and import of properties and connections. Properties are generically described and are written by a property writer and read by a property reader. This reader and writer must abstract hardware varieties like little endian and big endian. The component manager manages all components and connections. So the component manager will be able to assign identification numbers to each component. Using these numbers in the import and export will reduce the connection to an emitter number, a receiver number, an emitter slot name and a receiver slot name. With the functionality of creating component instances by the component name, configuration with generic properties and abstract import and export mechanism it will be easy to create an authoring tool under C++.
of
flow
component
Rendered image of arrow Video overlay
renderer
Current camera position and orientation
Geometric model of arrow
Scene node reader Identifier for arrow geometry Configuration interface
Figure 12: "Direction of flow" component and its internal structure
This component can be initialized easily by providing an identifier for the arrow geometry. This can be interpreted as a filename, if the arrow geometry is stored on a disc, or it may be used to form a query if the geometry is stored in a database. The scene node reader fetches the according data and transforms it into a geometric model. This model will
be passed to the rendering gem. After the geometric model is passed to the renderer, the initialization of the component is finished and the component can be used. After its initialization, the component performs two tasks. While the in-slot is idle, the video overlay gem combines the already rendered picture of the arrow with the incoming video stream. If a message at the in-slot occurs, the rendering gem calculates the new camera position and renders a new image of the arrow. Afterwards, the new image is passed to the video overlay gem to combine it with the video stream. The object recognition gem is encapsulated into another component that delivers messages, containing position and angular offset between the scene and current camera position. This component emits a message on its out slot whenever position or angular offset change. The internal structure of this component is depicted in Figure 13.
Object recognition
Current camera position and orientation
Configuration and calibration data
Out-Slot for current distance and angular offset
Camera position component
Configuration interface
Figure 13: Camera position component
After these two components are defined, the application can be built easily by connecting instances of those components by using their slots, as illustrated in Figure 14.
Figure 14: Coupling of Camera position component with Direction of flow Component
Despite the fact that this example is very simple and it only shows the basic features of the framework, some of the advantages that arise from using the AMIRE approach can clearly be identified. Firstly, gems can be grouped into functional, application specific components. These components hide MR/VR complexity from the application developer. Secondly, those components can be coupled
easily by connecting their slots. Thus, building a complete MR/VR application on top of the AMIRE toolkit would barely be more than selecting a component library that is suitable for the application domain, and connecting those components. CONCLUSION AND FUTURE WORK In this paper we presented the first demonstrator of AMIRE’s project and the generic framework, which will be used for the development of this application. The demonstrator will be for an oil-refinery training scenario. Our MR gems (libraries and software solutions) represent the base of the AMIRE framework. Well-designed MR components use these MR gems and are integrated in the framework of the application and in the framework of the authoring tool. This approach provides efficient means to encapsulate different solutions in a uniform way. In turn, it will dramatically foster the combination of different MR methodologies in a single application and will therefore lead to new MR and imaging techniques. Currently we identify characteristics of MR methodologies, we describe the common interface for the components and we design and implement the base framework technology. For September 2002 we expect to have the first results of the prototype based on the concepts described in this paper. ACKNOWLEDGMENTS The work described in this paper has been undertaken in the context of the AMIRE project, which consists of the following partners: C-Lab, Guggenheim Museum Bilbao, Fachhochschule Hagenberg, FAW, Fraunhofer-AGC, Labein, OMV, Talent Code and University of Helsinki. The authors wish to thank all of the members of the project team and acknowledge their contributions. AMIRE is partly supported by the European Commission under the IST program (project 34024). REFERENCES [1] Feiner S., Höllerer T., Gagas E., Hallaway D., Terauchi T., Güven S., and MacIntyre B., MARS Mobile Augmented Reality Systems, Columbia University Computer Graphics and User Interfaces Lab, 2001. [2] http://www.mr-system.co.jp [3] Friedrich W., ARVIKA Augmented Reality for Development, Production, and Service, Tagungsband des Informationsforum Virtuelle Produktentstehung (IVIP), May, 2000. [4] Kato H., Billinghurst M., Blanding B., May R., ARToolKit, Technical Report, Hiroshima City University, December, 1999. [5] Geiger C., Reimann C., Rosenbach W., Rapid Prototyping of Mixed Reality Applications that Entertain and Inform, IWEC 2002, Special Session on Mixed Reality Entertainment, Kluwer, Japan, 2002.
[6] Dobler D., Haller M., Stampfl P., ASR (Augmented Sound Reality), Sketches and Applications, SIGGRAPH’02, San Antonio (TX), 2002. [7] www.studierstube.org [8] Schmalstieg D., Fuhrmann A., Szalavari Zs., Gervautz M. "Studierstube" - An Environment for Collaboration in Augmented Reality, Extended abstract appeared in: Proceedings of Collaborative Virtual Environments '96, Nottingham, UK, Sep. 19-20, 1996. [9] Bauer M. et al., Design of a Component-Based Augmented Reality Framework. Int. Symposium on Augmented Reality, ISAR 2001, New York, 29-30 October, 2001. [10] http://europa.eu.int/comm/enterprise/atex/direct/text94 -9.htm [11] Gamma E.; Helm R.; Johnson R; Vlissides, J., Design Patterns in Smalltalk MVC, Design Patterns, Addison Wesly Longman Inc, pp. 4-5, 1995
[12] Writing Event Listeners, Sun Microsystems Inc, http://java.sun.com/docs/books/tutorial/uiswing/events /index.html, 2002 [13] Qt 3.0 Whitepaper, Trolltech Inc, http://www.trolltech.com/products/qt/whitepaper/white paper.html, 2002 [14] Hamilton G., JavaBeans, Sun Microsystems Inc, http://java.sun.com/products/javabeans/docs/spec.html, 2002 [15] Haller M., Hartmann W., Zauner J., A generic framework for game development, ACM/SIGGRAPH and Eurographics Campfire “Production Process of 3D Computer Graphics Applications – Structures, Roles and Tools”, Snowbird (Utah), 2002.
