Interactive Virtual Studio and lmmersive Televiewer Environment Laehyun
Kim
Department of Computer Science, University of Southern California, Los Angeles, CA 90089, USA TEL: +I -213-387-4963
Heedong
Ko
Moonho
*Korea Institute of Science and Technology, 39-1, Hawolkok-dong, Sungbuk-gu, Seoul, KOREA TEL: +82-2-958-5646
[email protected]
Park* & Hyeran Byun
Department of Computer Science, Yonsei University, 134, Shinchondong, Seodaemun-gu, Seoul, KOREA TEL: +82-2-361-27 19 (mhpark, hrbyun}@csai.yonsei.ac.kr
[email protected]
through the telephone is popular in interactive TV program.. This paper introduces a novel interactive TV programming by a combination of the physical surrogate
participation
ABSTRACT In this paper, we propose a novel virtual studio system in which an anchor in the virtual set interacts with televiewers as if they were sharing the same environment. A televiewer participates in the virtual studio environment by sensing and controlling a dummy head equipped with camera, speaker and microphones. The dummy head acts as a surrogate televiewer, providing the viewpoint experienced by the televiewer via a video camera and the sound experienced by the televiewer via microphone in its head. The anchor can not only interact with the virtual set elements but also share the physical studio with the surrogate televiewers. A televiewer with a head-mounted display (HMD) may feel immersed in the virtual studio environment seamlessly combining the virtual set elements with the real studio elements and interact with the anchor and vice versa. The proposed system
called “Interactive Virtual Studio (IVSY) and the later is “Immersive Televiewer Environment (ITE)“(see figure 1). In IVS, all participants including televiewers exist as physical type. Each televiewer has his own physical surrogate in the studio and he can communicate with anchor and even other televiewers. By contrast, televiewers with HMD interact with the anchor and the other televiewers in an augmented virtual environment called ITE. In ITE, the anchor is shown as real video image and the other televiewers is represented as avatar. Also, each participant can hear and speakthrough real audio as if they live in same physical space.Through the pair of IVS and ITE, remote participants have experience to share samephysical space.
consists of Interactive Virtual Studio (IVS) environment and Immersive Televiewer Environment (ITE) in which all the physical elements are collected and managed through IVS and the seamlessly mixed virtual and real elements are experienced via ITE. The essential idea is to have a dual universe where what makes a natural interface only physically should remain as physical and what makes easier to represent virtually should remain virtual and these two parallel universe should be coordinated seamlessly to provide the proper mix of the virtual and real mixed reality experience. In practice, this new interactive virtual studio for the immersive tele-meeting environment may be applied to the production of interactive TV
Figure 1. Interactive Virtual Studio(left) and Immersive Televiewer Environment(right)
televiewer
program, tele-conferencing, tele-education and others.
and mixture
2. RELATED
Keywords
of the real and virtual.
The former is
WORK
Recently, many systems have realized 3D tele-meeting session under the concept of shared virtual space[l]. Internet Foyer[2] is an electronic meeting space that spans the physical and virtual
Interactive Virtual Studio, Mixed Reality, Interactive TV, Avatar
1. INTRODUCTION
worlds. It involves the use of collaborative virtual environment technology to create a navigable and populated 3D visualization of an organization’s WWW home pages. However, in the Internet Foyer, each foyer does not appear as continues extension of the other foyer, because the real and virtual world keep their properties. Next, TELEPORT[3], an immersive tele-presence environment, alldws small group of participants, although geographically separated, to meet one another. The features of the system include the use of wall-size display surfaces, optical tracking of a viewer, and real-time cornpositing of live video with synthetic 3D scene. But, only one person in a real room can have his own viewpoint in TELEPORT. Lastly, Inhabited TV[4] involves the public deployment of collaborative virtual
In general, the televiewers can be involved in a TV program using letters, telephone and Internet in a current TV program. Especially, permission to m&e (ijgipdl or hard copies of all or part of this work for personal or classroom use is granted without Fer:provided that copies GTCll”t made or distributed for profit or commercial advarltagc arId that copies bear this notice and the full citation 011 the first page. l‘0 COPY otherwise, to republish, to post on serversor to redistribute to lists, requires prior specific permission and/or a fee.
VRST 99 London UK Copyright ACM 1999 I-581 13-141-0/99/12...$5.00
172
environments (CVEs) so that on-line audiences can participate in TV shows within shared virtual worlds. The potential size of Inhabited TV audiences also challenges the scalability of CVEs
6. CONCLUSIONS
AND FUTURE
WORK
The proposed system has the following characteristics. First, it suggests a new model for interactive TV program. Next, this system allows remote participants to share the physical environment. Finally, the virtual environment is augmented by real 3D audio as well as video image through the dummy head. The IVS and ITE as a pair affords modality dependent mixture of virtual and real environment. Hence, the Mixed Reality[S] system can be extended with not only images but also the other componentsfrom real world. In this paper, we applied the system to an interactive TV game show, but we may be able to apply our system for the other areas such as tele-conferencing and teleeducation. Our system has a problem of scalability to a participant using a dummy head. It is out of the problem to the participants using computer networks like existing DVR systems, however. We will continue to implement realistic facial animation of a dummy head and virtual avatars. Full body animation of virtual avatars is also one of our future work.
3. IVS Is it possible that remote participants share a physical space and interact physically? In most Distributed Virtual Environments, the answer is no. The reason is that each participant communicates with virtual surrogates of the other participants in virtual environment. In IVS, an anchor is allowed to interact physically with televiewers in remote site via the physical surrogate (dummy head) televiewer in virtual studio(see left picture in figure 1). As a result, IVS provides very natural interface among anchor and teieviewers including gaze awareness.Like a tele-operator in telepresencetechnology, a dummy head follows a televiewer’s head motion and delivers his speech, and feed back physical information from IVS to the televiewer at the same time. The dummy head was sculpted by NC machine using 3D head scan data of the participants. Then it was put on a little makeup for a feeling of intimacy. A dummy head is equipped with a speakerfor talking with other participants in IVS, two microphones to transmit stereo sound heard by the dummy head to ITE participant, a stepping motor for rotating the head to follow a televiewer’s head motion, and a camera to send an anchor’s live video image of IVS to ITE
4. ITE ITE is a conceptual mirror that reflects IVS. All participants and physical elements like the desk and camera are located in the sameposition and orientation in both ITE and IVS environments. Figure I shows this relation between IVS and ITE. Spatial and aural coherency between these two environments is based on this mirror relation. In order to provide spatial coherence, we used background-separated video stream using chroma-keying and then calibrated the image to adapt to virtual camera. Figure 1 and 2 shows that an anchor and the other subject are separatedfrom background. In addition to spatial coherency, aural coherency is very important to communicate among remote participants. So, participants should be located in sameposition and orientation in both IVS and ITE. Also, the other physical ob.jects in virtual studio such as desk should be placed on same position in the virtual environment.
5. IMPLEMENTATION:
Figure 2. The snapshotsof TV game show.
L-1
REFERENCES Macedonia, M. R., and Zyda, M., A Taxonomy for Networked Virtual Environment, IEEE Multimedia, 4, 1,48-56.
PI Benford, S. D., Brown, C. C., Reynard, G. T., and Greenhalgh, C. M., in Proceedings of ACM Conference on Computer Supported Cooperative Work (CSCW’96), (Boston, November 1996), ACM Press, 16-20. c31 Breiteneder, C. J., Gibbs, S. J., Arapis, C., TELEPORT
TV GAME SHOW
We implemented a TV game show in order to experiment the proposed system for new TV broadcasting which involves an anchor and two televiewers. The TV game is a word puzzle that the anchor gives two televiewers some brief explanation to help the televiewers guess given words. If any televiewer get the correct answer to a question, the anchor shows the picture of the word to confirm the answer. Figure 2 shows two snapshots according to the view of two televiewers. A view that a left televiewer sees the anchor is given in left of Figure 2. Right of Figure 2 is a view that a right televiewer seesthe anchor.
173
Augmented An Reality Teleconferencing Environment, in Proceedings of 3rd Eurographics Workshop on Virtual Environments (Monte Carlo, February 1996). [41 Greenhalgh,
C. M., Benford, S. D., Taylor, I. M., Bowers, J. M., Walker, G., and Wyver, J., Creating a Live Broadcast from a Virtual Environment, in Proceedings of SIGGRAPH ‘99 (Los Angeles CA, August 1999).
[51 Milgram,
P., and K&no, F., A Taxonomy of Mixed Reality Visual Displays, IEICE Transactions on Information and Systems, E77-D, 12, 1321-1329.
