The AR-CAVE: Distributed Collaborative Augmented Reality and Immersive Virtual Reality System Si-Jung Kim1, Denis Gračanin2, Woodrow W. Winchester1, Tae-Yong Kuc3 Department of Industrial and Systems Engineering1, Department of Computer Science2 Virginia Tech, Blacksburg, U.S.A. School of Information and Communications3 Sungkyunkwan University, Suwon, South Korea {hikim, gracanin, wwwinche}@vt.edu,
[email protected]
Abstract Combining augmented reality (AR) environments with immersive virtual environments (VEs) can reduce interaction gaps and provide embodied interactions in distributed collaborative works. It can also provide many opportunities and degrees of freedom for collaboration in both real and virtual worlds. A gesture based AR environment provides variety of input and output modalities to networked and distributed immersive VEs. In this paper, we present the AR-CAVE (Augmented Reality connected CAVE) platform that enables embodied interaction and tangible manipulations from a set of AR remote site to integrate physical free body interactions in a CAVE system. As a case study of the AR-CAVE, a simple 3D ball hitting task called the Ting Ting Together (T3) is demonstrated under a networked immersive VE and a computer vision based AR system.
Keywords: virtual reality, augmented reality, virtual environments, collaboration, distributed, embodied 1
Introduction
An augmented reality (AR) technology based collaboration system provides physically challenging realistic experiences [2], and it also gives many opportunities for integration of different user interfaces and interaction techniques. Distributed and collaborative virtual reality technologies were used for distributed virtual environments (DVEs) since early ‘90s (e.g. simple networked games and simulations) and it still faces many challenges, such as latency, dead reckoning and consistence [5]. In distributed collaborative interactions, interaction techniques are divided into several sub areas according to types and domains. One such area includes interactions among virtual objects within a local virtual environment (VE), and the other area includes interactions between two or more VEs distributed connected. A few application program interfaces (API) such as DIVERSE and VR-Juggler have been developed to facilitate design and implementation of DVEs [1][6]. Some researchers have investigated on collaborative playful spaces and work place using a VE or an AR interaction technique. For example, a playful space, where VE and real environment are mixed, is introduced, especially for children, based on the concept of embodied interaction and tangible interaction techniques to enhance children’s physical and social activities [7]. A tool for interactively mediated reality is presented using a head-worn video mixing display for painting, grabbing and gluing [4], and a 3D table for texture painting works is showed by using the AR Toolkit with several markers as they address the limitations of
physically changing textures [8]. Some researchers have investigated on user techniques from the points of spectators’ view [9]. From the review, most interaction techniques have been investigated in a single and local environment. Typical AR systems are monolithic and designed for a non-immersive augmented application in a single location. In general, the required program and data in AR are selfcontained to enhance the user's perception lacking in immersive effects. On the contrary, general immersive VEs systems are standalone and do not have collaborative configurations to collaborative with other multi modalities such as gesturers and tangible manipulations. In this paper, we present the AR-CAVE (Augmented Reality connected CAVE) platform, a combination of an AR environment with an immersive VE and show an example of the AR-CAVE called Ting Ting Together (T3), which is a simple 3D ball hitting game under the AR-CAVE platform. The T3 consists of two different working platforms to perform particular tasks. One is AR platform which is mainly controlled by human hands or physical objects using a computer vision system, and the other platform is an immersive VE mainly controlled by a wand in the VT-CAVE. To connect two different working environments and to share data of them, an open source virtual reality toolkit DIVERSE is used with the Internet network protocols. This paper divided into three sections. In the first section, motivation and goal is represented, and system architecture is showed in the second section. In the last of two sections, a case study is introduced and conclusion with future works is described.
2
Motivation and Goal
Two main goals are motivated in this research. The first one addresses the needs to provide tangible manipulation input methods to immersive VEs from the outside of its working range. The second one is to make physical activity based collaboration environment between different interaction environments. To share and access each different VE simultaneously, distributed and collaborative virtual technologies were considered for communication with remote users in VEs. According to above motivations, two specific goals are considered in terms of tangible manipulations and AR coupled immersive VEs.
2.1
Tangible Manipulations to VEs
Embodied Interactions in the CAVE
The other one is utilizing the AR-CAVE platform to enable a user to make physical activity. The specific goal of this part is to show an embodied interaction with the AR-CAVE platform to expand virtual space [3]. Most applications based on the CAVE system are static. Usually it is being used for navigation and visualization tasks. However, some applications running on the CAVE are needed dynamic task needing natural inputs such as body gestures or physical movements from users who are locating outside of its working range. For example, a CAVE system will be used for a playful space for children, and to make collaborative interactions with real agents such as robots.
3.1
Designing the AR-CAVE Interaction Layers
The purpose of the AR-CAVE platform is to enable collaboration between geographically and
CAVE space
Platform Layer
Real Space
User 1
User 2
Figure 1: The relationships between interaction platforms and user layers Regarding inputs and outputs modalities on each manipulation layer, human hands and tangible objects are used as major input methods in the AR space and a 3D wand is used a main manipulation method by sensing its position and orientation in VT-CAVE space. Figure 2 represents the input and output modalities in each space. The data in the AR space is being displayed by one project and can be seen without any particular materials. On the contrary, in the VT-CAVE space, data is being displayed on three walls by three projects and can be seen with a 3D shutter glasses. Platform Layer Manipulation Layer 1
In this study, a few things are not considered such as shadow occlusions, network latency and scalability between immersive VEs and an AR environment. VTCAVE and the Ting Ting platform are used as an immersive VE and an AR space respectably.
3
Virtual Space
Manipulation Layer
2.2
AR space Manipulation Layer
One goal is set to provide a tangible manipulation [10] into immersive VEs from an AR environment and we call this system as the AR-CAVE. Although there are some research results describing how to integrate input and output devices into immersive VEs, mostly have been done in combining sensor dependent components with CAVE locally. There were not remotely coupled with input methods providing gestures or tangible manipulations into the CAVE. In other words, the goal of this part is to expand input and output range of CAVE system, because if users are not in the CAVE, they can not input to the CAVE as well as the results are displayed only within CAVE walls. In addition, most AR and immersive VEs are working separately.
functionally separated different VR environments and to support tangible manipulations. To realize this requirement, we adapted a layer concept in terms of data and contents. The one layer is a platform layer and the second is a manipulation layer. The platform layer plays a role of inter-platform interactions between an AR and an immersive VE, and the manipulation layer acts as a facilitator for interactions between a user and each platform by providing unique modalities. The conceptual model representing the relationships between the layers is shown in figure 1. In the platform layer, motion data such as position and orientation is transacted, and in the manipulation layer, image data captured by a computer vision and sensed hands gesture data by a Wand in the VTCAVE is exchanged.
Hands Physical Objects A projector
Manipulation Layer 2
A 3D Wand A 3D Shutter Glasses
Figure 2: Interaction modalities on each layer
3.2
Connectivity
To connect the two space, between AR and the VTCAVE, there needs a particular connection concept to enhance data transactions between two different platforms. A semi-coupled connection structure is
used where one part has TCP/IP and the other part has UDP. TCP/IP is used to send motion trajectory data of 2D objects on a large screen in AR space to the VTCAVE. However, collision detection data generated for a 3D object and a wand in the VT-CAVE space is transferred by UDP network protocol. To share and deliver the motion trajectory and collision data, an asynchronous shared memory called DIVERSE is used between the two environments. The concept of a connectivity of the AR-CAVE platform is represented in figure 3. AR space
Shared Memory
AR Space
VT-CAVE Space
DIVERSE TCP/IP
Immersive Environment
Figure 3: The relationships of a platform and a content layer
3.3
Coordinate Systems
Since the AR and the VT-CAVE has different coordinate systems as shown in figure 4, we need to match them. z
x
(0, 0)
y
(1, 1, 1)
(0, 0, 0)
x y
(640, 480)
(a) AR coordinate
-y
(b) VT-CAVE coordinate
Figure 4: Coordinate Systems To translate between two different coordinate systems, we projected the AR coordinate to the VTCAVE coordinate using a normalization function represented by equations (1) and (2). The equation (1) represents mapping the x axis of the AR space to the x axis of the VT-CAVE and the equation (2) represents mapping the y axis of the AR to the z axis of the VTCAVE. During this mapping process, the y axis of CAVE is set as a constant variable. According to above matching coordinate systems, the expected object’s motion trajectory created from the AR space will be displayed within +x and +z area in the VTCAVE space. CAVE x = CAVE z =
1 AR x (max) 1 AR y (max)
Implementation
The proposed the AR-CAVE system was implemented by combining the Ting Ting AR environment with the VT-CAVE through the TCP/IP and UDP Internet network protocols. The AR space is built under an IBM compatible Pentium 4 PC with a computer vision system and the VT-CAVE space is used configuring by Linux system with four RGB projectors. An image grabber is used in the AR space in order to capture user’s motions at 30 frames rate per second. In the VT-CAVE, a 3D positioning device, a Wand is used to input 3D data to the VT-CAVE system. In the AR space, four main functions include: 1) generating motion trajectory, 2) sending/ receiving motion data, 3) collision detection, 4) feedback effects. Similarly, five main functions include in the VTCAVE space: 1) sending/receiving motion data, 2) storing data into a shared memory, 3) collision detector, 4) collision detection effecter, 5) generating motion trajectory. The implemented major characteristics of the AR-CAVE platform are represented in table 1.
UDP
Real Environment
4
rx , where 1 ≤ rx ≤ 640
(1)
ry , where 1 ≤ ry ≤ 480
(2)
Table 1: The VR-CAVE characteristic Items
AR Space
CAVE Space
Input Methods
Human hands and physical objects
3D Wand
Input Detection
Computer vision
Rendering
Direct Draw
Output Way
One DLP projector
Shared Memory
None
DIVERSE
Data Sending
TCP/IP
UDP
Collision Detection
Colour based
Position based
User Feedback
Changing background colour
Changing 3D object’s colour
5
DTK (DIVERSE Toolkit) DPF (DIVERSE Performer) Three RGB projectors
Case Study: Interaction
A simple hitting ball game called Ting Ting Together (T3) is demonstrated under the implemented ARCAVE. To play the T3 game, one player is in the VTCAVE room wearing a stereo shutter glasses and holding a Wand and the other player is in the AR space located outside of the VT-CAVE room. Human hands and physical objects are used to change the motion trajectory of the 2D object in the AR space and a 3D object is displayed in the VT-CAVE according to the motion path of the 2D object of the AR space. If the 3D object, the clone of the 2D object of the AR space, is hit by the Wand, the background colour is changed as cyan colour in order to notify to a player in the AR space that their object is captured by the opponent player in the VT-CAVE space. The snapshots are shown in figure 5.
also to make collaborative social interactions ans works between human and agents such as robots.
7
Acknowledgements
We would like to thank Patrick Shinpaugh for helping with the VT-CAVE system. (a) AR space
(b) VT-CAVE Space
8
(c) Playing T3 in the AR
(d) Playing T3 in the VT-CAVE
(e) Display an object of the AR in the VT-CAVE Space (f) Captured object by the Wand in the VT-CAVE
Figure 5: Snapshots of the T3 Game
6
Conclusion and Future work
The AR-CAVE (Augmented Reality connected CAVE) platform allowing tangible manipulations and embodied interactions is implemented by incorporating two different VEs such as an AR and the VT-CAVE. The implemented AR-CAVE platform enables networked group collaborations, and provided more mobility and a larger working space compared to the traditional CAVE applications, because the ARCAVE platform supported users from the outside of working range of CAVE system. An illustrative example called T3 is implemented and tested to show the interactive collaboration ability of the AR-CAVE platform. The player playing in the VT-CAVE space felt user awareness considering that a 3D ball in his place is controlled by another player in the AR space. In the T3 game, a 3D object in the VT-CAVE system is controlled by gesture inputs or tangible objects from the AR space, and embodied interactions are performed in the AR sides. One of things that we found from the test by accidentally is that social interactivity and collaboration performance can be increased in VEs if they are inputting data by using their gestures with physical manipulators instead of not only sensing devices such as a Wand. However, calibration issues had to be resolved to synchronize coordinate systems for the two dimensional AR space and the immersive VT-CAVE space. Feedback effects also had to be considered to give better instructive messages to users. We will improve the AR-CAVE platform not only to support various types of tangible and embodied inputs, but
References
[1] A. Bierbaum, C. Just, P. Hartling, K. Meinert, A. Baker and C. Cruz-Neira, “VR Juggler: A virtual platform for virtual reality application development”, Proceedings of IEEE VR Conference 2001, Yokohama, Japan, pp 89-96 (2001). [2] F. Brooks, “What's real about virtual reality”, IEEE Computer Graphics and Applications 1999, 19(6), pp 16-27 (1999). [3] P. Dourish, Where The Action Is: The Foundations of Embodied Interaction, MIT Press, Massachusetts (2001). [4] R. Grasset, J.D. Grascuel and D. Schmalstieg, “Interactive mediated reality”, Proceedings of International Symposium on Mixed and Augmented Reality 2003, Tokyo, Japan, pp 302303 (2003). [5] C. Joslin, G.D. Thomas and N. MagnenatThalmann, “Collaborative virtual environments: from birth to standardization”, IEEE Communication Magazine2004, 42(4), pp 28-33 (2004). [6] K.Kelso, S.G. Satterfield, L.E. Arsenault, P.M. Ketchan and R.D. Kriz, “DIVERSE: A framework for building extensible and reconfigurable device-independent virtual environments and distributed asynchronous simulations”, Presence 2003, 12(1), pp 19-36 (2003). [7] S.J. Kim, M.S. Jang and T.Y. Kuc, “A development of interactive game Ting Ting using real and virtual objects”, IEEE International Conference on SMC 2004, Hague, Netherlands, pp 1191-1197 (2004). [8] K. Matkovic, T. Psik, I. Wagner and D. Gracanin, “Dynamic texturing of real objects in an augmented reality system”, Proceedings of IEEE VR 2005, Bonn, Germany, pp 245-248 (2005). [9] S. Reeves, S. Benford, C. O’Malley and M. Fraser, “Designing the spectator experience”, ACM Conference on Computer-Human Interaction (CHI) 2005, Portland, USA, pp 741750 (2005). [10] B. Ullmer and H. Ishii, “Tangible Bits: towards seamless interfaces between people, bits and atoms”, ACM Conference on Computer-Human Interaction (CHI) 1997, Atlanta, USA, pp 234241 (1997).
