Abstract. We propose the concepts of Virtual Light and Virtual. Shadow with the aim of achieving a Mixed Reality Environ- ment focused on shadows.
Virtual Shadows -Enhanced Interaction in Mixed Reality Environment Takeshi Naemura Stanford University naemura @graphics.stanford.edu
Takuya Nitta, Atsushi Mimura, Hiroshi Harashima The University of Tokyo { takuya, a-mimu, hiro} @hc.t.u-tokyo.ac.jp
Abstract We propose the concepts of Virtual Light and Virtual Shadow with the aim of achieving a Mixed Reality Environment focused on shadows. In this proposal, we divide the concept of Virtual Shadow into four categories, and among them, implement four types of interactive applications: ( a) real to virtual shadow for rigid objects, (b) real to virtual shadow for non-rigid objects, (c ) image-based virtual to virtual shadow, and (d) virtual to real shadow. In these applications, we can see the shadow of a real object projected onto the virtual world and vice versa. These proposed concepts should contribute to the realization of a Mixed Reality Environment that provides a novel sense ofinteraction.
2. Virtual Light An ordinary flashlight emits light and can therefore be used to illuminate spaceas desired. If, however, we were to know the position and direction of a flashlight that a user is holding and then project the corresponding image on a screen,the user could experiencethe samevisual effect as a flashlight evenif the flashlight is not actually emitting light. Here, the interface that provides the sameor evenbetter visual effect of a flashli~t, regardlessof whether the flashlight is emitting light, IScalled "Virtual Light." We can consider some types of Virtual Light device. T:YPe(a) is a 3D sensorfor measuring3D position and direction that a user can hold like a flashlight. T:YPe(b) embeds a CCD camerain the 3D sensorso as to capture images of the real world. T:YPe(c) usesa projector to project a virtual object's shadowonto the real world. 3. Virtual Shadow A shadowsynthesizedand displayed on the basis of Virtual Light is called a "Virtual Shadow." In this regard, an illuminated object and the world onto which the shadowof that object is projected may be either real or virtual. We therefore can consider four types of Virtual Shadows as summarizedin Table I. Table 1. Four types of Virtual Shadow. Real Object Virtual Object
(c) Figure 1. Introduction ~e s~les~
(d) 1. Virtual
Shadow
Examples.
n;terging of the real and virtual worlds is a
world. These techiiiques enable a user to interact with his or her surroundings by shining a handheld flashlight-like device (called "Virtual light") as desired. As an example of interaction using a flashlight-like device, Inoue [3] hasproposedan application that makesa virtual object transparent.Lutz et al. [4] have applied a tracked flashlight to control a virtuallightsource in the CAVE environment. We can expandupon the concept of Virtual Light as a meansof merging the real world and virtual world.
Proceedings of the IEEE Virtual Reality 2002 (VR’02) 1087-8270/02 $17.00 © 2002 IEEE
Real World RR Shadow
Virtual World RV Shadow
VR Shadow
The RR (Real to Real) Shadow includes the casewhere a real object in the real world is illuminated by an ordinary flashlight. There is nothing special in this case. On the other hand. asdescribedby Chikamori et al. [2], a new form of visual expression can be achieved by synthesizing and displaying an unrealistic shadowof a real object. The RV (Real to Virtual) Shadow is a technique that modifies a real object's shadowin the virtual world according to the way in which the user is manipulating the real object. Here, we divide real objects into rigid and non-rigid ones and implement separatesystems for these two cases (seeFigs. l(a), (b)). The VV (Virtual to Virtual) Shadow correspondsto the projection of a virtual object's shadow onto the virtual world. In this paper, in order to treat the virtual object whose shapeis unkown, we apply the image-basedmethod (seeFig. l(c)). The VR (Virtual to Real) Shadow is a technique for projecting the shadowof a virtual object onto the real world.
State et al. [1] have proposed a method for superimposing virtual objects on "images" of the real world. Here, we go beyond image synthesisand consider the actual projection of a virtual object's shadow(seeFig. l(d)).
(a). In addition to the selectivedisplay, we illuminate larger areasas the user moves away from the screen (Z: small), as shown on the right in Fig. 4. Let me explain how to
y
4. Implementation We introduce four examples of implementing Virtual light interaction. Figure 2 illustrates the mechanism behind RV Shadow for rigid objects shown in Fig. I(a). To begin with. we mea-
x~z
Z=O Figure
4. Mechanism
of Fig.1 (C).
take pictures beforehand. Consider a grid on the xy -plane (Z = 0) as shown in Fig. 4. While moving the position of a point light source from one grid cell to another in order, we take one real-world photograph at each position. Figure 5 shows the mechanism behind VR Shadow shown in Fig. 1(d). In this case, we use the Virtual Light Figure
2. Mechanism
VIrtual Light
of Fig.1 (a).
sure the 3D shape of the real object (a toy cat) by a range finder and create a geometric model. Then, to interact, we use the Virtual Light lYPe (a) and attach a 3D sensor to the cat figure. Finally, we synthesize an appropriate shadow image corresponding to the cat's position and orientation by usin~ its geometric model and project that shadow. FIgure 3 illustrates the mechanism behind RV Shadow for dynamic objects shown in Fig. l(b). When a real ob-
Figure
Light
(Camera+
Capturedimage
of Fig.1 (d).
Type (c) to project the shadow of a virtual object onto the real world while actually illuminating the real world. The silhouette of the virtual object (geometric model) when seen from the position of the Virtual light becomes the shadow image that must be projected. Also, to view a virtual object, the user must wear a See-Through HMD or the like.
~ Virtual
5. Mechanism
real objects
Figure 3. Mechanism of Fig.1 (b). ject can changefonn in various ways like a shadowpuppet, its shape obviously cannot be measuredbeforehand. For this reason,we use the Virtual Light Type (b) and use the camerato capture such changesin the real world and reflect them in the virtual world. In this case,the systempicks up both the real world and the virtual world displayed on the screen. This requires real objects to be extracted from the captured image. We do this by assuming that bright sections within camera input correspond to the virtual world and dark sectionsto real objects. Let's considerthe casewherewe can not utilize the shape dataof objects. In Fig. 1(c), imagestaken beforehandunder a variety of lighting conditions are selectively displayed according to the position and direction of Virtual Light Type
Proceedings of the IEEE Virtual Reality 2002 (VR’02) 1087-8270/02 $17.00 © 2002 IEEE
5. Discussion In addition to the three types of Virtual Light, range finders, infrared cameras, infrared floodlights, and omnidirectional cameras can also be considered for this role. Furthennore, we can consider the implementation of various kinds of unrealistic effects such as Magic Light [3] and Kage [2] and expansionto new methods of interaction and communication. References [I] A. State et al. Superior augmentedreality registration... In SIGGRAPH 96, pages429- 438. ACM. 1996. [2] M. Chikamori and K. Kunoh. Kage. In SIGGRAPH 98: Electronic Art and Animation Catalog, page 14. ACM. 1998. [3] S. Inoue. Internal representation... In SIGGRAPH 98: Conterence Abstracts and Applications, page 288. ACM, 1998. [4] B. Lutz and M. Weintke. Virtual dunhuang art cave: A cave within a CAVE. In Eurographics 99. 1999.
