Abstract - This paper describes our design and implementations of projector-based computer augmented environment that allows users to interchange digital.
Extensible Interface using Projector-based Augmentation Tae Soo Yun, Dong Hoon Lee Division of Digital Contents Dongseo University Busan, S. Korea
Sang Heon Han, Jung Hoon Kim Visual Contents Department Graduate School of Design & IT Dongseo University Busan, S. Korea Abstract - This paper describes our design and implementations of projector-based computer augmented environment that allows users to interchange digital information between a computer and physical objects. We propose flexible and practical methods with low-cost projector display system only with a dual monitor personal computer although most previous approaches require high-cost equipments and complex technologies for construction these environments. To demonstrate the notion of the proposed extensible interface, we implement three applications. The results show the effectiveness of the proposed methods. Keywords: Augmented Reality, Interactive, Projector, Dual Display.
1
Extensible
Space,
Introduction
These days many researchers try to overcome limitations of the conventional computer interface with augmented reality, tangible interface and so forth. In parallel with this tendency, our working environments are going to be equipped with many computing facilities such as projectors and digital whiteboard. It is becoming quite common, and in the near future people will take cheap yet powerful display systems. Your goal is to simulate, as closely as possible, the usual appearance of typeset papers. This document provides an example of the desired layout and contains information regarding desktop publishing format, type sizes, and type faces. Our interest in this area began with the observation that projectors might be useful tools extending the conventional computer interface into the physical environments. Until now the main focus on the extended interface was, however, the development of a spatially continuous workspace in plenty research which is smoothly integrates existing portable computers with the environments. Therefore the study for this purpose requires high cost equipments and complex technologies for construction these environments. In addition, such environments are limited within well constructed laboratory environments. For example, to support links between digital information and physical objects, most systems require a wireless network for communicating with other computers and computer vision systems for detecting the user's intention.
For above reason, a new practical and flexible approach to extend a conventional interface is necessary. In this paper we describe our design and implementations of a computer augmented environment that allows a user to smoothly extend digital information between their desktop computer and physical objects. Using the virtual digital space supported by dual monitor PC and computer vision techniques for augmenting the digital information on the irregular display surfaces, users can easily move the specific digital information out of the monitor. The rest of the paper is organized as follows. Section 2 introduces previous work on projected displays. We describe proposed methods for extensible interface in section 3. Case study using the proposed methods and some results are introduced in section 4 and then present conclusions in section 5.
2
Previous Works
The use of projectors to change and augment reality was pioneered by media artists such as Tony Oursler [1] and Michael Naimark [7]. The use of projectors to access computer has been initially proposed by Bolt [2] and Wellner [16]. More radical applications of projectors have been proposed by Morishima et al. [6]. In the past years various and many projector-based systems like Large multi-projector walls [5], Steer-able projected display [8], Immersive environments [3], [9], Intelligent presentation systems [11], [13] and Remote-collaboration tools [14] developed. The previous work in using projectors for augmented reality can be divided into two main groups [4]: (1) projection of useful information on a planar surface, and (2) more complex insertion of 3D shapes and attributes into the real world. 2D interactive images can be projected onto flat surfaces in a room to enhance the user's environment. Examples include Luminous room [15], digital desk [16], and smart white boards. 3D augmentation examples include ShaderLamps [c10] and the related project Being There [12]. Most systems for augmented surface come under the first item. In this case they used conventional projector-based display systems which are typically designed around precise and regular configurations of projectors and display
surfaces. While this results in rendering simplicity and speed, it also means augmented surfaces are restricted within the planar surface such as wall and desk. Projectors can be used to change the surface appearance of neutral colored physical models by illuminating them with rendered images [12]. The ShaderLamps is effectively reproduced or synthesizes various surface attributes statically, dynamically, or interactively. In this system, complete illumination of complex 3D shapes is made possible by rendering geometric models of those shapes from multiple overlapping projectors. The geometric models are pre-authored with detailed color, texture, and material properties to reproduce desired surface appearance.
monitor and the extended space. This cause one acknowledges the fact that the two spaces are same one. We want to prevent the transition between digital and real space without any permission, which is allowed by the user's specific operation. In other words, we want real objects play an original role in general conditions and change their role when user intend to use them as an interface. Transparency is an important feature in this case. This is easily done by separating two monitors in display proper-ties offered by Windows OS (see Fig. 2).
The differences between our approach and the cited work on the use of projection-based display to create interactive display are the simple configuration, low-cost and symmetrical display.
3
Technical Issues
The configuration of our system comprises a 1024x768 LCD projector and a recent regular computer that can display a dual monitor. Fig. 1 depicts the hardware arrangement.
Fig. 2. Display properties in Windows XP™.
3.2
Fig. 1. System Configuration.
3.1
Setting up the Extensible Space
In our design, users can bring digital information in their own computer into the environment and put them on the real objects. Then the objects become an extended desk-top for the computer. Most previous system used other computers for augmented display. Therefore, seamless and continuous transition of digital information requires wireless network between computers. In our case, we want to use only one computer for the practical purpose. For this reason, we utilize the dual monitor capabilities that are supported in general video card. From the extended space in the dual monitor PC, the extensible space can be easily set up. Generally, the extended space in the dual monitor PC is arranged by the side or the upper part. In this case, the movements of the mouse can easily go across between the
Transition into the Extensible Space
To perform spatially continuous transition digital space into the extended space, we assign the specific part of the monitor space (generally upper part of the monitor) as a transition region. When passing through the region, the system monitors the event and then transfers the digital information on the extended space. To detect the user's intention for transition, we can implement the specific operations such as a combination of the key strokes, for example, ctrl or shift key. The transition between two spaces X → X’ can be easily implemented by 2D translation motion (T). In addition, since the extended space is bigger than the monitor space, a scale factor (s) also has to be adjusted for continuous transition (See (1)).
X ' = sTX . 3.3
Augmentation Surfaces
on
Irregular
Display
One central problem to be solved in achieving seamless imagery for the general physical objects is that of geometric calibration. Correct registration, together with correct image intensity blending of the overlapped image
Fig. 3. Prototypes for extensible interface: (a) the wall becomes an extension of the computer, (b) the interactive mailing system which notice received e-mails through extended space and (c) the interactive calendar and infinity sphere which mimic the functionality of real objects. areas, creates a powerful immersive effect for the viewer. We achieve geometric registration by recovering a 3D representation of the augmented objects. For this purpose we utilize the Raskar's projector calibration method [10]. Raskar's calibration method is a computer vision technique to calibrate the cameras and recover display surface and projector parameters and is summarized below. During pre-processing Create 3D graphics model, G, of physical object. Create 3D graphics model, B, of background. Approximately position the projector. Find perspective pose, P, of the projector wrt the physical object. During run-time Get user location, U. Get animation transformation, T. Modify G's surface attributes. Render G using the pose P, and user location U. Transform B using T-1, B'. Render B' using the pose P, and user location U. Modify image intensity to compensate for surface orientation. The projector projection matrix, P, is obtained using an off-line calibration process. From a set of fiducials with known 3D locations on the physical object, we can find the corresponding projector pixels that illuminate them. This allows us to compute a 3x4 perspective projection matrix, which decomposed to find the internal and the external parameters of the projector. The rendering process uses the same internal and external parameters to render the preauthored geometric model, G, so that the projected images are registered with the physical objects. Original Raskar's calibration method considers the user location for view dependent effect such as specular highlights. However, in our case, we assume the user sit in front of the desk so that user location is not required. Fig. 4 shows an example of the spatial augmentation.
Fig. 4. The physical model of the building and the same model enhance with shaderlamps.
4
Case Study for Extensible Interface
To verify the effectiveness of the proposed method, we have developed three kinds of prototype systems. Each system was designed to show the respective different possibilities. First prototype system (Fig. 3(a)) shows a spatially continuous work space. A user can utilize the real world as a big virtual window world. In second prototype system (Fig. 3(b)), the real world represents a part of the transparent digital world. A user can interact with the space
or object as if it is a still real world. Therefore, a computer applies the extended space as a part of computer interface space automatically independent of a user's intention. Third prototype system (Fig. 3(c)) is an example to verify the possibility of original functionality of the real objects as a computer inter-face. For this purpose we have created two different scenarios: interactive calendar and infinity ball. Interactive calendar explores a scheduler where calendar and schedules are projected directly onto the surfaces of the object and infinity ball utilizes a property that the sphere makes revolutions on an axis. The revolution of the ball shows a new interface which represents endless information visualization. The remainder of this section outlines them in more detail.
projector. Fig. 5 shows an example of a digital extended interface. This shows how to solve the problem of spatial limitation of a computer monitor because it provides users with large physical space where people can freely display, move, or attach digital data. Fig. 6 illustrates a mailing notification to the user appearing on a display surface. Here, the interactive mailing system connected mail servers detects that the user received an e-mail and gives an intuitional notification. As a result, this system shows a concept that general real world can be a part of computing spaces.
4.2
Interactive Calendar
The next prototype application can be seen a beam of light that can add information on to real world objects by highlighting them and adding text or images. We develop a prototype that supports the binding of digital data and physical objects that have its specific role in world: especially here we use a calendar, so it is possible to interact between them. Fig. 7 shows a white frame on a computer transformed into an interactive calendar. When simply clicking the date of his/her choice to select it, the user can add/edit/delete the schedule and if there is any schedule at the day, it will be marked with a red-dot. Additionally, the schedule can be shown on the right attached post-it by positioning of the mouse at the date to see it. To build more symmetrical and inter-active interfaces, this system used Raskar's algorithm introduced in section 3.3.
Fig. 5. Digital extended display.
Fig. 7. Interactive Scheduler.
4.3
Fig. 6. Interactive mailing system.
4.1
Interactive mailing system
The first prototype application is a personalized and extended display space where can display digital information and can notice received e-mails through LCD
Infinity Sphere
Fig. 8 shows information continuously displayed on the surface we named 'Infinity Sphere'. The basic idea is to define alternative surfaces where information is displayed continuously. Generally, the scroll bar or hyper link are utilized to show amount of information in the limited display space. In Infinity Sphere, according to control of the angle of the sphere, the information shown on the sphere will be changed and be displayed constantly. Therefore, this can be used to display huge amount information by manipulating the real objects. This is a kind
of tangible interface which employs properties of real objects as it is. To detect the direction and quantity of the motion, a panning motor which sends the information into the system is attached in the motion axis of the sphere. This detection mechanism can be substituted to detection algorithms used in computer vision community or devices such as a magnetic tracker.
Acknowledgement This research was supported by the Program for the Training of Graduate Students in Regional Innovation which was conducted by the Ministry of Commerce Industry and Energy of the Korean Government.
6
References
[1] R. Bellour et al., “Tony Oursler”, Ediciones Poligrafa, 208, 2001. [2] R. Bolt, “Put That There: Voice and Gesture at the Graphics Interface”', ACM Computer Graphics, vol.14, no.3, pp. 262-270, 1980. [3] C. Cruz-Neira, D. Sandlin, and T. DeFanti, “`Surround-screen projection-based virtual reality: The design and implementation of the CAVE”, In Proc. of SIGGRAPH, 1993. [4] P. Dietz, R. Raskar, S. Booth, J. V. Baar, K. Wittenburg and B. Knep, “Multi-Projectors and Implicit Interaction in Persuasive Public Displays”, Advanced Visual Interfaces (AVI), pp. 209-217, May 2004. Fig. 8. An example of the image sequence in Infinity Sphere.
[5] T. Funkhouser and K. Li, “Large format displays”, Computer Graphics and Applications, vol.20, no.4, 2000.
5
[6] S. Morishima et al., “HyperMask: Talking Head Projected Onto Real Objects”, in Proc. of Multimedia Modeling (MMM'00), 2000.
Conclusions
This paper described a practical and flexible approach for creating computer augmented environment that allows a user to smoothly extend digital information between a computer and physical objects. We have shown methods to determine to set up the extended space, transit into extensible space and augment on irregular display surfaces. We have also implemented three applications that realize extended interface concepts: interactive mailing system, interactive calendar and infinity sphere. We believe this work as an important contribution to realize the vision of projector-based display for extensible and interactive interfaces without high-cost equipments and complex technologies. There are a number of features that should be improved. Currently, in the case of the registration using the Raskar's method, we can only overlap the digital information in the static scene. Therefore we cannot shift the objects to other location on the fly, which means the proposed method cannot be applied in dynamic scene. In addition, if several real objects are applied to projected surfaces which are placed at different position, a projector has only one focus, which leads to a blurring problem. In future studies, we plan to solve above problems and seek other appropriate applications using the proposed methods.
[7] M. Naimark, “Spatial Correspondence in Motion Picture Display”, SPIE Optics in Entertainment II, vol.462, pp.78-81, 1984. [8] C. Pinhanez, “The Everywhere display”, Proc. of Ubiquitous Computing, 2001. [9] R. Raskar, G. Welch, M. Cutts, A. Lake, L. Stesin, and H. Fuchs, “The office of the future: A unified approach to image-based modeling and spatially immersive displays”, in Proc. Of SIGGRAPH, 1998. [10] R. Raskar, R. Ziegler, T. Willwacher, “Cartoon Dioramas in Motion”, in Proceedings IEEE Visualization, 1999. [11] R. Raskar, G. Welch, K. Low, D. Bandyopadhyay, “Shader Lamps, Animating Real Objects with Image Based Illumination”, Proceedings of the 12th Eurographics Workshop on Rendering, June 2001. [12] R. Raskar and P. Beardsley, “A self-correcting projector”, Proc. of CVPR, 2001.
[13] R. Sukthankar, R. Stockton, and M. Mullin, “Smarter presentations: Exploiting homography in camera-projector systems”, in Proc. of ICCV, 2001. [14] N. Takao, J. Shi, and S. Baker, “Tele-graffiti”, CMURI-TR-02-10, Carnegie Mellon University, 2002. [15] J. Underkoffler, “A View From the Luminous Room”, Personal Technologies, vol. 1, pp.49-51, 1997. [16] P. Wellner, “Interacting with Paper on the DigitalDesk”, Communications of the ACM, vol. 37, no. 7, pp.87-96, July 1993.
