replacing HMDs by immersive projection technology (IPT) such as PowerWalls, CAVEs, Virtual Tables. Projection-based Augmented Reality enhances such ...
Stork et. al.
PBAR in Engineering Applications
Projection-based Augmented Reality in Engineering Applications
André Stork, Fraunhofer-Institut für Graphische Datenverarbeitung, Darmstadt Oliver Bimber, Fraunhofer Center for Research in Computer Graphics, Providence, RI Raffaele de Amicis, GRIS, Department of Computer Science, Technical University, Darmstadt
Abstract Augmented Reality (AR) superimposes computer-generated graphics onto the user's view of the real world. Today, AR applications can occasionally be found in the late phases of the product development process (PDP), such as in training and in maintenance. Head-mounted displays (HMDs) are typically used for AR applications. HMDs still show deficiencies in terms of ergonomics and image quality. Looking at the development of VR technology, it can be stated that VR began its triumphal procession only after replacing HMDs by immersive projection technology (IPT) such as PowerWalls, CAVEs, Virtual Tables. Projection-based Augmented Reality enhances such environments towards augmented reality applications. We believe that this concept opens new application possibilities for AR – especially in the early phases of the product development process. In this paper we will describe the concept of Projection-based Augmented Reality (PBAR), introduce current hardware prototypes and discuss applications of PBAR configurations within the following areas: augmented design review, hybrid assembling simulation, hybrid modeling/sketching and visual inspection of augmented physical parts.
Introduction Head-attached displays (such as head-mounted displays) are the traditional output devices for Augmented Reality. They have first been developed in the mid-sixties and still today own the display monopole in AR field. However, in contrast to the well established projection technology that is mainly applied for Virtual Reality (VR) applications, head-mounted displays have barely improved over the previous years. HMDs still show deficiencies in terms of ergonomics and image quality, i.e. resolution, viewing angle, etc. (see Fig. 1). Current models are still far away from being “ultimate displays“. High-quality, lightweight, and inexpensive, 3D see-through HMDs are not available today. The compromises which have to be made to the hardware-technology hinder AR from having its break-through. Alternative approaches such as video-mixing are evaluated and further developed by some researchers. Video-mixing superimposes the computer-generated information, e.g. 3D graphics, onto a video stream in the background. This approach supports mutual occlusion between real and virtual objects and allows for a pixelprecise registration of both environments. However, due to the technological limitations of the miniature displays, the virtual and the real environments are be perceived in the low resolution that is supported by the displays.
Fig. 1: Two representative Head Mounted Displays (left: Kaiser Electro-Optics, right: VPI)
1/8
Projection-based Augmented Reality (PBAR) is a new concept that extends the capabilities of projection-based VR (immersive projection technology) towards AR by using semi-transparent mirrors as optical combiners. Commonly known stereo glasses (active or passive ones) are sufficient to see a 3dimensional scene, consisting of a real/physical part and a virtual part. PBAR detaches the display device from the user’s head and consequently solves some of the technological, ergonomic and economic problems that can be attributed to traditional head-attached devices. Optional projection
ceiling
semitransparent mirror stereo rear projection
Real carfront
virtual engine
Fig. 2: Sketch of a PBAR set-up built from a stereo rear projection display, a semi-transparent mirror and a physical model For the product development process this approach is of special relevance, since VR hardware technology in form of PowerWalls - as already established in many companies – can be used by just extending them with inexpensive semi-transparent mirrors and the according software technology. Thus, PBAR shows not only advantages with respect to technological criteria but also economical benefits. In contrast to HMDs that evolved to an all-purpose AR display medium over the previous 35 years, PBAR offers application specific solutions that adapt the display and interaction technology to problem specific tasks. By doing this, some of the technological limitations that are related to HMDs –and which still today remain unsolved– can be improved. , Tracking within a well controlled environment, for instance, becomes a less crucial problem since highly precise tracking technology can be used in a stationary set-up. Since PBAR does not apply head-attached miniature displays, but rather detaches the display technology from the user a high and scalable resolution can be provided. Finally, the focal-length problem caused by HMDs is improved, since the graphics is displayed near their real world position. It should not be left unmentioned that not all AR scenarios can be supported with PBAR. Especially mobile applications, or applications that require a direct or close manipulation can mostly not be supported. However, many applications in the product development process can benefit from PBAR, such as augmented design review, hybrid assembling simulation and modelling/sketching. In the remainder of this paper we will discuss some applications of PBAR, but first we present related work topics. Then, we describe the concept of Projection-based Augmented Reality and introduce current hardware prototypes. Finally we conclude and give an outlook to further work topics.
Related Work In contrast to traditional front or rear-projection systems that apply opaque canvases or ground glass screens, transparent projection screens don't block the observer's view to the real environment behind the display surface. Therefore, they can be used as optical combiners that overlay the projected graphics over the simultaneously visible real environment. Pronova’s HoloPro system (Pronova, 2001) is such a transparent projection screen. It consists of a multi-layered glass plate that has been laminated with a lightdirecting holographic film. The holographic elements on this film route the impinging light rays into specific directions, rather than to diffuse them into all directions (as it is the case for traditional projection screens). This results in a viewing volume of 60° horizontal and 20° vertical range in front of the screen, in which the projected images are visible. Regular projectors can be used to rear-project onto a HoloPro screen. However, they have to beam the images from a specific vertical angle (36.4°) to let them appear within the viewing volume. Originally, the HoloPro technique has been developed to support bright projections at daylight and some researchers already begin to adapt this technology for Augmented Reality purposes (Ogi, Yamada, Yamamoto & Hirose, 2001). However, several drawbacks (mainly due to the applied holographic film) can be related to this technology: • • • •
Limited and restricted viewing area. Static and constrained alignment of projector and projection plane (and therefore a no flexibility or mobility). Low resolution of the holographic film (the pattern of the holographic elements are well visible on the projection plane). Reduced see-through quality due to limited transparency of non-illuminated areas.
Computer augmented whiteboards have been a topic of research over the past years. As traditional whiteboards, a computer augmented or electronic whiteboard presents an environment shared by multiple users, which can be used to discuss, draw or present ideas. We’ll analyze the previous work with the focus on the interaction aspects relevant to our system. The videoWhiteboard (Tang and Minneman, 1991) is one of the early systems that provides tools for shared drawing. A camera captures the drawings and the gestures of each participant on the whiteboard and transmits it to the remote user where it is projected giving the illusion that they are on opposite sides of the same whiteboard. The Clearboard (Ishii and Kobayashi, 1992) is based on the same concept adding gaze awareness by overlapping the image of the user and the drawing on the same display surface. Both, the videoWhiteboard and the Clearboard are suitable for remote conferencing. Another pioneering example of a whiteboard application is the Tivoli (Pedersen, McCall, Mora and Halasz, 1993) system, that has been designed support collaborative meetings. It uses a pen as input device and offers the combination of a traditional graphical user interface that is displayed on Xerox Liveboard and a gesture interface. A different interaction approach is applied for the BrightBoard (Stafford-Fraser and Robinson, 1996). Here, the user controls the computer by drawing simple sketches or characters on the board. Through image processing the computer distinguishes the sketches from the rest of the drawing. Written words, for instance, can be interpreted as computer commands. Thus, a seamless transition between handwriting and computer control is achieved. FlatLand (Mynatt, Igarashi, Edwards and LaMarca, 1999) is a system designed to support typical whiteboard applications. A touch-sensitive whiteboard (the SmartBoardTM) accepts normal whiteboard marker input as well as stylus input. Captured strokes are projected onto the board. Gestures serve for managing the displayed visual layout. HoloWall (Matsushita and Rekimoto, 1997) is a wall-size computer display that allows users to interact with their fingers, hands, bodies, or even with physical objects. Inputs are recognized through infrared lights (IR) and a IR video camera. Since the HoloWall can detect two or more hands (or fingers), multihand interaction is possible. Also, user body posture/position or even physical objects that reflect IR light, such as a document showing a 2D-barcode which identifies it, provide an enriched interaction environment.
Experimental Hardware-Setups Since PBAR is an extension of immersive projection technology it can be combined with different IPT set-ups. Up to now, we built-up two PBAR prototypes for the support of engineering scenarios: • •
A Barco Baron Virtual Table (Barco, 2001) that applies rear-projection and active shuttering using CrystalEyes Shutter glasses (see Fig. 3). In this case, two stereo images are projected sequentially onto the projection surface and shuttering is synchronized via an infrared signal. A portable two-screen front-projection system that applies passive shuttering using polarized light. In this case, four beamers project pre-filtered stereo images simultaneously onto the two projection planes. The images are separated via polarized glasses. This is illustrated in figure 4.
In both cases the semi-transparent mirror consists of a mobile Aluminium rack that holds a tiltable wooden frame. The frames comes with a conventional white board which is replaced by a half-silvered mirror (a 40” x 60” large, and 10mm thick glass pane which has been laminated with a half-silvered mirror foil (3M, 2001)) that simultaneously transmits and reflects light. Thus, we refer to it as “transflective board” (Bimber, Stork and Branco, 2001). Technically, the transflective board is used as an optical combiner that reflects stereoscopic 3D graphics off an arbitrary display surface. Using the optical combiner, this graphics is spatially merged with the surrounding real environment.
semi-transparent mirror
semi- transparent mirror
‚shine through’ model physical model
Virtual Table
Fig. 3: Extended Virtual Table: on the right a table-like back-projection system (Virtual Table – here a Barco Baron) with the semi-transparent mirror; on the left the physical model of a car
Fig. 4: The projection device used for this scenario is a mobile two-screen system. The dashed arrow illustrate the flow of light. The dashed lines outline the reflected projection planes. The set-ups are combined with different tracking technology. In one case an electromagnetic tracking device (Ascension’s Flock of Birds (Ascension, 2001)) is used to support head-tracking and tracking of
spatial input devices. In the other case an optical tracking system (AR-Tracking, 2001) is used which allows highly precise 3D tracking of the hand and head positions. In contrast to the magnetic tracker, the user is not faced with cables which gives the system a new acceptance level. In addition, we apply a tool that is usually used to track real markers and cleaning pads on a whiteboard – the Mimio (Dunkane, 2001). The Mimio is a hybrid (ultrasonic and infrared) 2D tracking system for planar surfaces which is more precise and less susceptible to distortion than an electromagnetic tracking device. The Mimio together with the transflective board can provide the same interaction behaviour as a white-board.
Mixed Reality Techniques and Tools In terms of providing a correct superimposition of the rendered graphics over the real environment, we apply the following techniques (these techniques have been describe in full detail in (Bimber, Encarnação & Branco, 2001): • Rendering: Since the rendered graphics is perceived as reflection by looking at the mirror, several transformations have to be applied to see an unreflected virtual augmentation and to provide correct stereo separation. This is realized by introducing two additional transforms to the transformation pipelines of the rendering framework. A view transformation reflects the viewpoints (the left and right camera positions) that are located on the one mirror side (the side of the stereo display) over the mirror plane. A model transformation reflects the virtual scene that is located on the other side of the mirror (the side of the real environment) vice versa. Since these transformations are affine, they can be simply integrated into a hard- or software-implemented transformation pipeline (such as the one realized by OpenGL (Neider, et al., 1993)). Consequently, only minimal modifications to an existing rendering framework have to be made to support planar mirrors as optical combiners in combination with stereo displays. Furthermore, no additional computational cost is required for these transformations during rendering. • Optical Distortion: Optical distortion is caused by the mirror and the projection planes because of refraction and the curvature of the display or the mirror. We have developed several pre-distortion methods that are applied within the 3D spatial space, as well as within the 2D image space. • Tracking Distortion: When electromagnetic tracking devices are used, we apply smoothening filters to filter high-frequent sub-bands, and positional pre-distortion with the pre-sampled magnetic field of the working volume to minimize non-linear distortion, since electromagnetic tracking devices introduce non-linear tracking-distortion and noise over an extensive working volume. • System Delay: In addition to distortion, end-to-end system delay or lag causes a “swimming effect” (virtual objects appear to float around real objects). To reduce the swimming effect, we apply prediction filters (Kalman filters (Azuma, 1995)) for orientation information and linear prediction for position information.
Interaction Direct interaction in front of the transflective board as well as indirect and remote interaction techniques with virtual objects ‘behind’ the transflective board are supported. A variety of techniques have been implemented. For example, a tracked pen is used for direct manipulation in front of the board, as well as for remote interaction behind the board. In addition, a transparent hand-held tablet (which is also tracked) is applied to feature two-handed interaction (direct or remote). Virtual objects can be exchanged between both sides of the board: they can be picked with the pen – either directly or remotely – and can then be pushed, pulled or beamed through the mirror. A virtual laser beam casts from the pen through the mirror to move and place virtual objects behind the board’s tangent plane. Additionally, the user’s viewpoint can be combined with the pen direction to compute an appropriate selector. The ARToolkit optical tracking system (Kato, Billinghurst, Blanding & May, 1999) is employed to detect different paper markers within the real environment. The markers are used to track real world objects or as placeholder for multi-media information (e.g., images, video or textual information). Since different multi-media contents are attached to specific markers, they can easily be exchanged by replacing the markers, or simply be moved within the real environment. The tracked pen and also the pad feature sketch-based interaction
(Bimber, Encarnação, & Stork, 2000): two or three-dimensional freehand sketches are used to build objects by reconstructing their shapes from the sketches, or to interact with the objects (e.g. by sketching assembling steps, etc.).
Engineering Applications Today, maintenance and training are the major application fields for Augmented Reality from the industry’s point of view (ARVIKA, 2001). Augmented Reality will conquer the earlier phases of the product development process where VR is more heavily used. AR can compensate for the shortcomings of VR, such as fine and precise tactile and haptic feedback. Here, mobility is not a crucial issue. Thus, stationary set-ups that ensure high quality images and a more robust tracking than mobile ones will be preferred for these purposes. We envision four application scenarios for projection-based AR in the engineering process: •
Augmented design review: VR systems are mainly used for design review. But the lack of reality especially concerning haptic and tactile feedback still limits the acceptance of VR as a decision base. This lack can be compensated by full functional rapid prototyping (RP) parts that give the user a real tactile and also aural feedback. To produce many RP parts with, for instance, different colors and materials is cost intensive. The combination of RP parts with virtual overlays that let these parts appear in different colors/materials is a promising ‘best-of-two-worlds’-approach.
•
Hybrid assembling: The feasibility evaluation of the assembling/disassembling processes is a common application of VR whereby the product exists in a digital form, only. But products are frequently altered during their lifecycle. In these situations, the products already exist physically and the need to assemble virtual models into physical mock-ups (PMU) arises.
Fig. 5: Hybrid assembly sequence: upper left: virtual model ‘above’ the Virtual Table; upper right: user moves virtual part ‘through’ the mirror; lower left: distant interaction with virtual part; lower right: virtual part assembled into real printer (images taken from Bimber, Encarnação & Branco, 2001)
•
Hybrid modeling/sketching: Although CAS and VR systems have changed the styling and design process considerably, clay models still have their place in the design process. During a review of a physical model, requirements to change some of its parts/features appear. Today, these modification requirements are expressed verbally. An Augmented Reality setup would allow to virtually sketch the change onto/over the PMU.
Fig. 6: Free-form surface sketched ‘over’ physical model •
Visual inspection of molded parts: The development of a product comprises the design of tools to manufacture the product parts. Molded parts shrink when they cool down. Thus, the tools are not simply the inverse product geometry. When starting a new production line, the produced parts have to be compared with the intended geometry – the virtual parts. Again, Augmented Reality can give us an efficient to use visual inspection environment.
Conclusion and Future Work Traditional Augmented Reality displays, such as head-mounted devices entail a number of technological and ergonomic drawbacks that prevent their usage in a many of application areas. With the objectives to overcome some of these drawbacks and to address an efficient and problem specific application of Augmented Reality technology within the engineering domain, we have presented the early stages of a new projection-based Augmented Reality device – the transflective board. Compared to head-attached AR displays, the application of spatial projection displays for Augmented Reality tasks feature an improved ergonomics, a theoretically unlimited field-of-view, a high and scalable resolution, and an easier eye accommodation (Raskar, 1998). Since the technology, as well as the consequential advantages are derived from the well established projection-based VR concept, a seamless combination of VR and AR is also imaginable. However, several shortcomings can be related to our approach. Self-reflection of the user in some situations, for example, is one of the drawbacks of the transflective board. Our current approach to address this problem is to integrate the transflective board into engineering-related application scenarios in such a way that these reflections are minimized or avoided. Future research will tackle further technological and usability issues, such as mobility, robust tracking, efficient interaction and the evaluation of implemented scenarios with the industry. Another topic for future work is the extension towards multiuser scenarios as sketched in figure 7.
Fig. 7: The Virtual Showcase: Multi-user AR set-up using a transflective cone (Bimber, Fröhlich, Schmalstieg, and Encarnação 2001)
Acknowledgement The work described in this paper has been supported in part by the European Commission Grant #IST2000-28169 (SmartSketches project).
References 3M, Corp. (2001) P-18. Retrieved from the World Wide Web: http://www.mmm.com/market/construction/html/products/product73_p.html. Advanced Realtime Tracking, www.ar-tracking.de, 2001 ARVIKA – Augmented Reality in Industrial Applications (2001), Retrieved from the World Wide Web: http://www.arvika.de Ascension Technologies. Corp. (2001) Flock of Birds. Retrieved from the World Wide Web: http://www.ascensiontech.com/products/flockofbirds/flockofbirds.htm. Azuma, R.(1995) Predictive Tracking for Augmented Reality. Ph.D. dissertation, University of North Carolina at Chapel Hill, TR95-007. Barco, Inc. (2001). Baron. Retrieved from the World Wide Web: http://www.barco.com/projection_systems/products/product.asp?GenNr=324. Bimber, O., Encarnação, L.M. and Stork, A. (2000). A Multi-Layered Architecture for Sketch-based Interaction within Three-dimensional Virtual Environments. Computers and Graphics – The international Journal of Systems and Applications in Computer Graphics, vol. 24, no. 6, pp. 851-867. Bimber, O., Encarnação, L.M. and Stork (2001), A. Seamless integration of virtual reality in habitual workplaces. Journal for Industrial Science, Munich University of Technology, vol.55, no.2., pp. 103-112. Bimber, O., Stork, A., and Branco, P. (2001) Projection-based Augmented Engineering. Human-Computer Interaction International (HCII’01), vol. 1, pp. 787-791. Bimber, O., Fröhlich, B., Schmalstieg, D., and Encarnação, L.M. (2001) The Virtual Showcase. IEEE Computer Graphics & Applications, vol. 21, no.6, pp. 48-55. Bimber, O., Encarnação, L.M. and Branco, P. (2001) The Extended Virtual Table: An Optical Extension for Table-Like Projection Systems. To appear in Presence: Tele-operators and Virtual Environments, vol.10, no. 6. Dunkane, Corp. (2001) Mimio. Retrieved from the World Wide Web: http://www.dukane.com/AudioVisual/whiteboard.htm. Ishii, H. and Kobayashi, M. (1992). ClearBoard: A Seamless Medium for Shared Drawing and Conver-sation with Eye Contact. Proceedings of the Conference on Computer Human Interaction CHI’92, pp. 525-532. Matsushita, N. and Rekimoto, J. (1997). HoloWall: Designing a Finger, Hand, Body, and Object Sensitive Wall. In proceedings of UIST’97, 209-210. Mynatt, E. D., Igarashi, T., Edwards, W. K. and LaMarca, A. (1999). Flatland: New Dimensions in Office Whiteboards. In proceedings of the Conference on Computer Human Interaction CHI’99, pp. 346-353. Neider, J., Davis, T., and Woo, M. (1993) OpenGL programming Guide. Addison-Wesley Publ., ISBN 0-201-63274-8. Ogi, T. Yamada, T. Yamamoto, K. and Hirose, M. (2001). Invisible Interface for Immersive Virtual World. To appear at the Immersive Projection Technology Workshop (IPT’01), Stuttgart, Germany. Pedersen, E. R., McCall, K., Mora, T. P. and Halasz, F. G. (1993). Tivoli: An Electronic Whiteboard for Informal Workgroup Meetings. In proceedings of INTERCHI’93, pp. 391-398. G+B Pronova GmbH. (2001) HoloPro. Retrieved from the World Wide Web: http://www.holopro.html. Raskar, R., Welch, G., and Fuchs, H. (1998). Spatially Augmented Reality. In proceedings of First IEEE Workshop on Augmented Reality (IWAR'98), San Francisco, CA. Stafford-Fraser, Q. and Robinson, P. (1996). BrightBoard: A Video-Augmented Environment. Proceedings of the Conference on Computer Human Interaction CHI’96, pp. 134-141. Tang, J. C. and Minneman, S. L. (1991). VideoWhiteboard: Video Shadows to Support Remote Collaboration. Proceedings of the Conference on Computer Human Interaction CHI’91, pp. 157-168.
