ART 03
Developing and Authoring Mixed Reality with MX Toolkit José Miguel Salles Dias,
[email protected] Luis Monteiro,
[email protected]
Pedro Santos,
[email protected] Rui Silvestre,
[email protected]
Rafael Bastos,
[email protected]
ADETTI, Edifício ISCTE, Av. das Forças Armadas, 1600-082 Lisbon, Portugal, http://www.adetti.iscte.pt •
Abstract
The need to write specific code to handle directly the 2D and 3D Open GL-based graphics output, although some libraries do exist to attenuate the problem [3]. • The need to handle directly the video capture devices or other external devices. • The need to manage separately fiducial marker representations as provided by AR Toolkit and associated 3D shapes, although they are logically bound together in the AR Toolkit “philosophy”, through the registration process. To tackle these issues, we came up with a new Toolkit oriented to the mixed reality applications developer, which is defined at a somewhat higher abstraction level than the AR Toolkit software layer, by hiding from the programmer, low level implementation details and facilitating AR/MR object-oriented programming. MX Toolkit is an SDK written in C++ which uses extensively the AR Toolkit library, for all matters regarding markerbased tracking, but also has a simple dialog with Microsoft Foundation Classes and other standards and third party file formats, such as Open VRML, 3D Studio (currently supported) or Open SceneGraph [4] (development under way). This paper is structured as follows: in section 2, we provide a background and state-of-the-art in the issues of Authoring platforms and SDKs for AR and MR. In section 3, we present the MX Toolkit, with a comparative analysis with AR Toolkit, stressing also its modular structure. Section 4 covers the issues of AR/MR authoring using MX Toolkit, by illustrating the development of a dedicated visual authoring tool, refereed to as Mix It. This section details also the hardware and software of the developed prototype. Finally, in section 5, conclusions and future directions of research are given.
This paper describes a software platform oriented to the Augmented Reality/Mixed Reality application developer, aiming at simplifying his/her programming tasks. This platform comprises a software development kit (SDK) for the Windows environment, consisting of a set of C++ Classes packaged into modules. The platform is known as MX Toolkit and utilises extensively the AR Toolkit, for all matters regarding marker-based tracking, but is defined at a somewhat higher abstraction level than the AR Toolkit software layer, by hiding from the programmer, low level implementation details and facilitating AR/MR object-oriented programming. The MX toolkit is presented by means of a comparison with the AR Toolkit processing pipeline and API and by explaining its eight system modules. We expect to provide this package as an aid for AR Toolkit-based applications development. The paper also describes a visual AR/MR authoring tool, the Mix It application, from the user interface point of view, as an illustration of the development support potential of the MX Toolkit SDK..
1. Introduction Since the pioneering effort of Mark Billinghurst and Hirokazu Kato that has lead to the development of AR Toolkit and its deployment as an Open Source platform [1, 2], a large group of Augmented Reality (AR) and Mixed Reality (MR) application developers has emerged, which was well demonstrated by the presence of a large scientific community in ART02, the First AR Toolkit Workshop, held in Fhg-IGD, in Darmstad, Germany, in the fall of 2002 [3]. Although the C-based AR Toolkit API pipeline is simple and comprehensive, some drawbacks do exist for the AR/MR application programmer developer, namely, the following: • The lack of object oriented programming framework, which brings well established programming advantages: data hiding, polymorphism, operator overloading, re-usability of object classes, enforced modularity, program clarity.
2. Background The deployment of Augmented Reality and Mixed Reality SDK platforms and AR/MR authoring tools has emerged as an interesting trend in this research field, since the appearance of AR Toolkit. While not exhaustive, the following text, enumerates some of the more interesting R&D efforts in this domain. Shared Reality Meeting is a proprietary Collaborative Augmented Reality
1
ART 03
The detailed pipeline steps are depicted in Table 1.
Environment [5], oriented to workgroup design tasks and supporting several types of Tangible Interaction, although no specific general purpose SDK or platform is reported by the authors. AMIRE is an European IST project [7] which has released modules to ease AR/MR development tasks, envisaging similar aims as our own research, since its platform relieves the programmer of the detailed knowledge about the underlying AR/MR base technologies. AMIRE resides in the AR Toolkit tracking technology and, as a framework, uses a component oriented technology, a reusable GEM collection and a visual authoring tool for building AR/MR applications. Our MX Toolkit is oriented specifically to the programmer, providing a simple and minimal set of C++ Classes (although they can grow with the contributions from programmers using the Toolkit), packaged into modules to aid the software development tasks. The MR Toolkit C++ SDK, available for a Linux PC environment [7], is able to manage a parallax-less stereo video seethrough HMD. This SDK is composed of a C++ class library for developing runtime MR applications, is able to handle six degree-of-freedom (DOF) input sensors, and uses an alternative marker-based tracking to the AR Toolkit. Our approach is similar, in the programming framework approach, to the former, but still adheres to the AR Toolkit tracking technology, which, although currently based in a single camera tracking technology, has proven to be a stable development framework and has gained the interest of a large community of users. Our MX toolkit thus profits from the growing installed base of AR Toolkit application developers and from the evolution of this Open Source library, by an enthusiastic research community.
Algorithmic Steps of ARToolkit A. Initiatisation.
AR Toolkit Function
A.1. Initialises application: opens the video path, sets the virtual camera, loads the full set of fiducial markers to be recognised, opens the OpenGL graphics window. A.2. Initializes video frame. B. Captures video frame. C. Detects and recognises fiducial markers in the video frame. D. Displays the image obtained by video capture. E. Computes the virtual camera reference frame relative to the recognised marker. F. Draws objects in OpenGL using the virtual camera reference frame, registered to respective markers. G. Ends video capture and image processing.
Init()
arVideoCaptureStart() arVideoGetImage() arDetectMarker()
arDisplayImage() arGetTransMat()
drawObject()
cleanup()
Table 1 AR Toolkit Algorithmic Process
Our proposed MX Toolkit process is, in synthesis, as follows: A. The AR Toolkit, Video Input/Output and Open GL are initialised and logical associations of 3D shapes and 2D fiducial markers (called XObjects) are established. B. The input video stream is analysed, frame by frame. C. Fiducial markers (previously stored) are searched in the image. D. The input video image is rendered in OpenGL. E. For each found and identified marker in the current video frame, updates the corresponding XObject with the current marker to camera reference frame transformation matrix, as provided by AR Toolkit, using a confidence value. F. Draws every XObject found in the previous step, in OpenGL, by traversing an appropriate scene graph. G. Handles Tangible Interfaces and, specifically, Menu interaction in MR. H. Ends video capture and image processing.
3. MX Toolkit, a MiXed Reality Toolkit As we know, the basic AR Toolkit algorithmic steps, organised as a pipeline, are the following [8]: A. AR Toolkit, Video Capture and Open GL are initialised. B. The input video stream is analysed, frame by frame. C. Fiducial markers (previously stored) are searched in the image. D. The input video image is rendered in OpenGL. E. For each found and identified marker, the camera’s viewpoint relative to the marker is determined and a 3D transformation matrix from marker to camera reference frames is computed. F. An OpenGL display list is rendered, with GL graphics registered relative to the respective markers. G. Ends video capture and image processing.
2
ART 03
Algorithmic Steps of MX Toolkit A. Initialisation. A.1 Initializes AR Toolkit with desired vertical and horizontal window size (scaled by a suitable factor) and with a full screen flag. A.2. Initializes OpenGL rendering modes and lights. A.3. Initializes the relevant Xobjects, which are associations of 3D shapes and 2D fiducial markers. A.4. Queries the available video capture devices, initializes the Video Capture, vertical and horizontal video capture resolution and optionally with vertical and horizontal image flipping. B. Captures the current video frame.
ARToolkit::Init(xsize, ysize, fullscreen, factor)
mxInit_Graphics() new XObject(object_name, shape_name, marker_name) DShowVideoCapture:: Init(deviceId, xsize, ysize, flipV, flipH)
DShowVideoCapture:: GetFrame()
C. Identifies markers detected in the current frame.
ARToolkit:: TrackMarkers( dataPtr , threshold)
D. Displays the image obtained by video capture. E. Updates a Xobject (if the respective marker is found in the current frame) with the current transformation matrix, as provided by AR Toolkit. Uses a confidence ∈ [0..1]. F. Invokes mxDrawObject for every visible Xobject, drawing it in OpenGL, applying its internal rotation, scaling and transformation parameters. Currently supports VRML 97 and 3D Studio. G. Handles Tangible Interfaces, menu pop-up, menu options selection and activation.
ARToolkit:: DisplayFrame(dataPtr) ARToolkit:: LinkMarker(Xobject, confidence_value)
H. Ends video capture and image processing.
process, but with a substantive difference regarding the provision of logical associations between 3D shapes and 2D fiducial markes (absent in AR Toolkit), which are initialized in phase A and handled in phase E and F. The following phases, introduce even higher level concepts. Phase F, maintains the explicit notion of a scene graph and, in phase G, totally new concepts of Tangible Interfaces are provided within the MX Toolkit (Magic Ring, Magic Examiner). The detailed pipeline steps are depicted in Table 2. This algorithmic process was devised taken into account the requirements of the Mixed Reality programmer developer. The initialisation process (Step A) is such that a window pops-up, prior to Video Capture, where the user will be able to select the appropriate Device Driver for Video Input, the Input and Output Video resolution and to toggle, Flipping the Image Vertically and Horizontally ON/OFF, as well as Full screen ON/OFF (Figure 2). These features are desirable, given the limited video input/output CPU resources and video see-through set-up, that the user might have available, as depicted in Figure 1.
MX Toolkit Function
Figure 1 A video see-through set-up
Another initialisation task, regards the issue of defining the logical associations between 3D shapes and respective markers, which is crucial for marker-based mixed reality. In our MX Toolkit we call these associations XObjects. An object of this class, has its own name, defines a pathname to a well known gallery of 3D shapes, which is a folder in a local or remote file system and a filename of a 3D shape stored in a given file format (currently VRML 97 and 3D Studio are the supported types). A XObject also includes the associated filename .patt, of an AR Toolkit marker definition and the 3D transformation matrix from marker to camera local reference frames. This means that once a programmer identifies a marker and a 3D shape, she can define a XObject to create a logical association between both. In run-time, as soon as AR Toolkit recognises that marker, the MX Toolkit will draw the associated 3D shape, conveniently registered relative to the corresponding marker, taking also into account its internal rotation, scaling and transformation parameters, as defined in the scene graph, whose details are hidden from the programmer, that is, this scheme
mxDrawObject(Xobject)
mxFinger->Update() mxFinger->Draw() mxMenuBar>update(fin ger) mxMenuBar> Draw() mxCleanup ()
Table 2 MX Toolkit Algorithmic Process
As we can observe, process phases A, B, C, D and E are in essence similar to the ones found in the AR Toolkit
3
ART 03
applies uniformly to any 3D shape format and/or any scene graph supported by the MX Toolkit.
Upon the identification of the markers that are on sight of the vision system, its time to “register” the logically associated 3D shapes. To accomplish this, the system: • Displays the image obtained by video capture (Step D) • For each identified marker, updates the associated XObject with the transformation matrix, returned by AR Toolkit, that defines the camera reference frame relative to the marker reference frame (Step E). The system uses a confidence value ∈ [0..1], which is a hint for AR Toolkit regarding the degree of certainty it should have when finding markers in the current frame. • Draws every Xobject identified in the previous setp, in OpenGL, applying its internal rotation, scaling and transformation parameters, taking into account the camera transformation matrix (Step F). Currently, the system supports VRML 97 and 3D Studio for the 3D shape definition, including animation. The next step (Step G) is to handle Tangible Interfaces, for such tasks as Menu pop-up, Menu option selection and activation, or 3D shape picking, dragging and dropping, which are described in detail in [9]. Currently the only Tangible Interfaces fully supported are the Magic Ring and the Magic Examiner, for 6DOF examination of 3D shapes in AR/MR. A Magic Ring Tangible Interface has a specific marker attached and, as a visual aid, when it is recognized by the system, a blue circle will be displayed on top of it (see Figure 3). By wearing this type of Rings on one or more fingers, we are able to provide an approximation to finger gesture tracking by computer vision means. If the user wears two rings, for example, in both thumbs, a menu is activated when the user presses the marker in her left thumb using the marker in her right thumb. The menu is displayed next to the left thumb (Figure 4) and is then tracked to this finger marker motion. The user is now able to use her right thumb to scroll the menu items. This is done by simply moving his/her finger along the local ‘y’ axis so that its 2D window ‘y’ coordinates match the same ‘y’ coordinates of the corresponding menu item. The visual result of this interaction is similar to the case of using a mouse cursor to scroll/browse through a menu. Finally, the user can now select a menu item, by means of a timeout and a blinking mechanism for the selected option feedback. Its associated menu function will then be executed, which could be a nested menu. One Magic Ring can be used also for picking, dragging, dropping and object transformation operations, and this behaviour is all embedded in MX Toolkit (Figure 5). The MX Toolkit provides also some utilities to draw ASCII text in the Open GL window were the captured video frame is displayed.
Figure 2 Initial window for Video In/Out settings
The next algorithmic step (Step B), is to capture the current video frame and to identify markers detected in it, subjected to a given threshold value. In run-time, the system provides the user with the means to set the value of this threshold. By pressing the ´D´ key in the keyboard, the system will switch from the normal colour video mode to black and white (binary) mode (Figure 3). The user can use this mode to adjust a threshold level, according to the lighting conditions of her working settings. In this mode, the image is analysed and all pixel values (image luminance, which varies between 0 and 255) are compared with a threshold value, which corresponds to a decision level. If the image luminance is above the threshold, then it is converted to 255 (white), otherwise it is converted to 0 (black), thus producing a binary image. Since the lighting conditions affect the observed image luminance, it is most of the time required to adjust manually the threshold value. This task can be accomplished by pressing keys '8' an '2', from the numpad keyboard, which respectively increases and decrease the threshold value.
Figure 3 Switching from binary (Left) to colour modes (right), by pressing the ´D´ key
4
ART 03
scene is being rendered. In fact, more plugins can be developed, being added and used later at runtime. Peer-to-Peer Communication: The Communication Module can serve applications to stream data or communicate in a higher abstraction level, hiding the application from the TCP/IP layer and communication protocol. Based on Windows Sockets 2 technology, this module combines multi-threading and event dispatching functionalities, to make the system more efficient and flexible for the programmer. For instance, if a programmer wants to send a video data stream to another application instance running on another computer in the network, she can use features like image compression that is offered by the Image Processing Module and then send the stream, through the Communication Module, to another node. This allows the creation of an interactive mixed reality network, where all nodes can share information between each other and use this information to augment its environment. If we add to this model a set of features provided by the Marker Tracking Module, that uses the AR Toolkit subsystem to track fiducial markers in an image, we can create a multi-camera tracking system, solving problems like marker occlusion and object registering precision. Peripheral Communication: This module allows the system to communicate with the Wearable System hardware and with other specific hardware by a serial communication port. It includes a set of routines to deal with accelerometers components like the Inertia Cube2 (which makes part of the Wearable System and that enables orientation tracking). This module can be applied also in an independent way, for example, in peer-to-peer serial communication between computers, or even with other hardware components, like microcontrollers, which support RS232 protocol. Database Communication: This module provides a set of methods to enable database communication. It accesses data through a Data Access Object (DAO) based on the Microsoft Jet database engine. This engine provides the access to data, including through ODBC drivers, via a specific database engine. It also supports Data Definition Language (DDL) operations, such as adding tables and other modifying operations. This module intends to allow the use of distributed data objects, which can be obtained by all the other systems running with MX Toolkit. Video Capture: This module allows the system to acquire images from a capturing device, like cameras, VCRs, DVD players, etc. It works with any WDM (Windows Device Manager) and VFW (Video for Windows) compatible device. The user can select the input video device, the video input video resolution and the output window resolution, which will be delivered to the Rendering Module. There is also the possibility of correcting the image rendering, vertically and horizontally, by the Image Processing Module. This module is built under Microsoft Direct Show technology,
Figure 4 Handling menu pop-up windows, option browsing and selection in Augmented Reality
3.1 MX Tooklit Modular Structure The MX Toolkit can be described, in more detail, as a set of eight functionally independent modules, communicating with each other through the main system core. Amongst these eight different modules, three of them are indispensable elements: Core - the only system part that cannot be replaced, but only upgraded; Video Capturing; and 3D Rendering. One of the main advantages of having modules in the system architecture, is the construction of a flexible system, that can be simply upgraded or it’s modules replaced, without having any concern with the remaining components. Core System: This module enables features, such as application, scenes and sound plugin management, that are controlled by a set of three galleries: Application Gallery (a Mx Toolkit application is developed as a DLL plugin, that can be built and later read at run-time by the Core), Shape Gallery (allows the system to load DLL Shape Plugins – 3DS and VRML 97 3D file formats are currently supported) and Sound Gallery (supports Sound Engine platforms, such as Open Audio Library and Microsoft Direct Sound and also contains loaded Sound File Plugins, such as Windows Wave and MPEG Layer 3). The Core contains also a set of base class descriptors that can be used by external plugin applications built with the Mx Toolkit SDK, such as the Abstract Augmented Reality Object (XObject), Augmented Reality Markers, Abstract Shape and Sound Plugins, Tangible Interfaces, Magic Books and even specific Hardware Sensors and Hardware Interfaces. This module also features a set of utilities to manage timers, registry keys, conversion methods, counters and callbacks, in order to provide an efficient and effective mixed reality programming environment. Any kind of plugin or application built and exported as an Mx Toolkit component, can be loaded, started, updated and terminated by the Core Module. So, this module provides all that is needed for all the other modules to work perfectly in a higher abstraction layer, without the need of concerning which application has been started, which plugin has been loaded or even which
5
ART 03
and needs the installation of Microsoft DirectX runtime library, that comes with recent Windows Operating Systems. The use of this module, simplifies all the initializations and image capture procedures, which are usually required in Mixed Reality applications. This module can be accessed by the Core System. The acquired data can be delivered to the main core and processed in order to enable some functionalities that are included in other modules, like marker tracking, motion detection, camera calibration and registration, image enhancement, contrast enhancement, brightness and saturation calibration, etc. Marker Tracking: A library was developed based on the latest available AR Toolkit source code for MS Windows, and all functions were exported to a Dynamic Link Library. An AR Toolkit wrapping object called MX AR Toolkit, was built, which controls external AR Toolkit use and puts the programming user in a higher (objectoriented) level of abstraction. The module provides a set of methods related to marker tracking and creates associations between 3D shapes and 2D markers. The Core system calls a method from this module, every time a frame is being processed, obtaining information and confidence values about the tracked markers or the current camera matrix. Image Processing: This module provides a set of functionalities to process video streamed images that are delivered by the Video Capture Module. One of its features is the compression of live streamed video. It uses any type of compression compatible with the Windows Video Compression Manager. This functionality may be used in applications that require video streaming in conjunction with the Communication Module, due to the low bit-rates that can be achieved with this module. Another functionality of this module is the ability of applying image filters in a single pass, or in various passes with a given sequence. There are five filters available at the present: Flip Image, Brighten Image, Adaptive Contrast, Threshold Image and Adaptive Image Enhance. 3D Rendering: Objects are rendered by this module independently of their geometry and topology. When creating an object to be rendered, it’s geometrical and topological structures must be defined as well as it’s rendering methods (currently using the OpenGL API). So, the system is not concerned with “what is being rendered” but with “which object is being rendered”, calling update and drawing methods for each object in the mixed reality environment. This module also features a set of utilities like Menus, Menu Bars and Text display methods, allowing the user to create, in a simple way, an Augmented Graphical User Interface (AGUI). This Interface is recognized by the Core, and can be controlled by a Tangible Interface just like a simple mouse cursor.
Figure 5 Finger gesticulation to rotate a 3D shape in AR
4. Authoring Mixed Reality One of the plugin applications developed by our team, using the MX Toolkit SDK, provides the possibility of authoring mixed reality worlds using a visual interface. It is referred as Mix It and comprises the Mix It Editor and the AR Viewer. This application adopts the metaphor of the Magic Book [10]. The Mix It Editor supports multiple Magic Books which can be created, managed, changed, deleted and printed by the user. The AR Viewer is the run-time module that enables the user to view and interact within a mixed reality environment. To author a Mixed Reality environment, the user must evolve through the following steps: Step 1 – Create a Book and Add Objects to that Book (Figure 6): After choosing a Book Name, the user should insert one or more Objects Names in the book. Each object is in fact a Xobject, an association between a marker (now being proposed by the tool) and a 3D shape (to be identified in the next step). The user should repeat this action (to add Object Names to the Book) as many times as needed to complete the book. However, the user can also perform this action at a later stage and in an interactive way.
Figure 6 Creating a Magic Book and attaching objects to it
6
ART 03
Step 2 – Associate 3D Shapes to objects, by doubleclicking on the shape in the “Objects Inside Book” dialog box. Upon popping up a dialog box, the user should browse de local or remote file system to select a VRML file, thus effectively creating a XObject association (Figure 7).
Figure 9 The Mix It 3D shapes Gallery
By pressing the right button of the mouse on a given Object of a Book, a menu will pop-up, with the following options: • Renaming the selected Object (Xobject). • Removing the Object (Xobject) from the book. • Changing the 3D Shape associated with the Xobject, if selected from the shapes available in the Mix It Gallery (Figure 9). • Changing the Marker associated with the XObject, selected from the available Markers list (Figure 10). Each marker is randomly created from a set of 100 available markers (this means that each Book has a maximum of 100 markers).
Figure 7 Associating a 3D shape to a 2D marker
Figure 8 A newly created Book
Step 3 - Manage Books and objects. By pressing the right button of the mouse on the Book Name, a menu will popup, providing the means for: • Renaming the Book. • Removing the Book and all associated objects. • Inserting New Object (XObjects) into the Book. • Printing all the Markers of the selected Book, so that the run time module, AR Viewer can later register 3D shapes onto them. Mix It maintains a Gallery of 3D shapes, which can be managed by the user, namely, by importing a VRML file into the Gallery or by removing a VRML file from the same Gallery.
Figure 10 Changing a fiducial marker
The 3D shape proprieties can also be parameterised by the user, by specifying (Figure 11): • The rotation relative to the local XX´, YY´ or ZZ´ axis; • The translation on the local XX´, YY´ or ZZ´ axis; • The scale, by keeping or not the aspect ratio.
7
ART 03
Figure 11 Changing the Translation, Rotation and Scale proprieties of a given 3D shape
Once one or more books are ready, the AR Viewer tool can be used to enable the mixed reality interaction. 4.1. System Deployment The existing mixed reality authoring system prototype, is currently appropriate for static or mobile usage in indoor settings and it consists of an intelligent garment (a vest) with wireless analogue video input/output link and battery powered video see-through glasses. The input/output video is processed by a base station computer, running Mix It Editor (for scene preparation) and AR Viewer (for Tangible Interaction in mixed reality). The typical hardware and software platforms required by our system, are as follows: • Video See-Through Head Mounted Display: Olympus Eye-trek FMD 700, Multimedia Glasses (800x600 pixel), with a wireless video input link with the Base Station Computer and Micro Cam wireless ultra miniature video camera from Swann Security (Web Cam is also supported although loosing the mobility of the system, since it requires a VGA cable to be connected to the base station computer); • Wearable Unit: Jacket equipped with bidirectional wireless analogue video link and batteries • Base Station Computer: • Hardware - CPU: Intel Pentium IV 2.5GHZ; RAM: 1 Gbyte; Graphics card: NVIDIA GFORCE4 440 GO 64 Mbyte; • Software - MS Visual C++ 6.0 enterprise edition; ArToolkit 2.65; Video Input - Direct X 8.1; Graphics - Open GL and Open VRML; • Indoor Tracking method - Computer vision-based approach (AR Toolkit) with sparsely placed fiducial markers.
Figure 12 An intelligent garment (a vest) with wireless analogue video output link and battery powered video see-through glasses, for mobile usage of the system. The input/output video is processed by a base station computer.
5. Conclusions and Future Work We have presented a Software Development Kit (SDK) for the Windows environment, consisting of a set of C++ Classes packaged into modules, the MX Toolkit, that simplifies the task of developing and authoring mixed reality applications and scenarios. Our MX toolkit is based in the AR Toolkit, regarding the marker-based tracking technology, but is defined at a higher level of programming abstraction than the former. We have described the basic algorithmic process of MX Toolkit, from the perspective of the AR/MR applications developer and its modular structure. If we compare the development APIs of AR Toolkit and MX Toolkit, we conclude that the overall logical process is similar, while MX Toolkit enforces and manages higher level concepts, such as associations between 3D shapes and 2D fiducial
8
ART 03 •
markers, independent of the 3D shape file format, Tangible Interfaces, such as the Magic Ring, Menu manipulation in mixed reality and even Magic Books. Given its object oriented nature, MX Toolkit is extensible and will be enriched in the future, although the core set of classes will remain basically fixed. We have demonstrated the usefulness and applicability of the toolkit by describing a complex application developed in the framework of it, the Mix It MR/AR visual authoring system. This authoring tool, supports the production of Tangible Mixed Reality scenarios that have been applied in real industrial product design tasks in The Nederlands and Germany or interactive museum presentations in Portugal. We are currently engaged in a methodology of structured usability testing and evaluation of the Mix It authoring system, by a panel of users (students), to provide feedback for the improvement of system features, which is reported elsewere [9]. We are also developing new Tangible Interfaces, for specific use cases, such as the Magic Pen, able to provide accuracy in pointing and picking, within design scenarios or the Magic Examiner (already supported), for 6DOF examination of 3D shapes in mixed reality. Hybrid indoor virtual camera tracking and object tracking is also being develop at our HP New Media Lab, using a combination of vision-based approaches (both marker-based, using ArToolkit, and texture-based, using our own developments), ultra-sound sensors positioned along the workspace area and accelerometers providing 3 angular degrees of freedom and located on the Wearable Unit. This approach could be a possible solution for the registration problem, providing more flexibility and accuracy in camera and object tracking, enabling, for example, the user to walk freely within a certain working area, while experiencing and interacting through gestures, with augmented or virtual worlds. Concerning the system architecture, enhancements could be made in the mobility, ubiquitous and the cooperative support of the system, to increase the sense of mixed reality, that is, of seamless transition between real, augmented and virtual environments both in standalone and collaborative working scenarios. In fact, as a future evolution, we would support, multiple collaborators (such as in [11]), over wireless links, where discussion meetings and collaborative working sessions could take place amongst local and remote users, immersed on different collaborative environment modalities over the Internet: real, augmented or virtual. Another promising path to enhance this mobile wearable computing paradigm would be to pursue the multimodality in the User Interface aspect, by integrating other modalities, such as: • Calligraphic Interface modality, a recently introduced paradigm for human-computer interaction [12], specially oriented to industrial design tasks.
•
Speech Interface modality (voice Input and Speech Recognition), paying attention to the task conditions (ex.: noisy industrial environments). Biometrics Interface modality [13], as a way to measure monitors the wearer’s physiological signals, such as skin conductivity, heart rate, circulatory flows, respiration rate and muscle activity and, subsequently, to evaluate mental states, such as arousal, attention, effort, autonomic response to stress (e.g, in the alert reaction to environmental changes), working toward the vision of HCI actions/commands derived from mental states.
References [1] Billinghurst, M. Shared Space Homepage, June 2003 www.hitl.washington.edu/research/shared_space/download/ [2] http://www.c-lab.de/art02/ [3] www.openvrml.org/ [4] http://www.openscenegraph.org/ [5] Wagner, M. T., Regenbrecht, H. T., “Shared Reality Meeting - a Collaborative Augmented Reality Environment”, ART02, The First IEEE International Augmented Reality Toolkit Workshop, Darmstadt Germany, 29 September 2002
[6] Grimm, P., et al., “AMIRE, Authoring Mixed Reality”, ART02, The First IEEE International Augmented Reality Toolkit Workshop, Darmstadt Germany, 29 September 2002 [7] Uchiyama, S., et al., “MR Platform: A Basic Body on Which Mixed Reality Applications Are Built”, ISMAR 2002, IEEE and ACM International Symposium on Mixed and Augmented Reality, Darmstadt Germany, Sept. 30 - Oct. 1 2002 [8] Kato, H., “Developing Applications with ARToolkit”, SIGGRAPH 2001 Course Notes 27, 2001. [9] Dias, J. M. S., Jorge, J., et al, “Usability Evaluation of Tangible User Interfaces for Augmented Reality”, submitted to ART03, The Second IEEE International Augmented Reality Toolkit Workshop, Japan, October 2003 [10] Billinghurst, M., Kato, H., Poupyrev, I., “The MagicBook: A Transitional AR Interface”, in Augmented Reality: the Interface is Everywhere, SIGGRAPH 2001 Course Notes 27, 2001 [11] Grasset, R., et al., “MARE: Multiuser Augmented Reality Environment on table setup”, SIGGRAPH 2002, Conf. Abstract. [12] Contero, M., et al.,“CIGRO: a minimal instruction set calligraphic interface for sketch-based modeling”, 2nd International Workshop on Computer Graphics and Geometric Modeling, CGGM'2003, Montreal, Canada, 18-21 May, 2003. [13] Ebrahimi. T., J.-M. Vesin and G. N. Garcia, “BrainComputer Interface in Multimedia Communication,” IEEE Signal Processing Magazine January 2003.
9
