QoS mapping for networked Virtual Reality System a
b
c
a
b
Seiwoong Oh , Daisuke Kado , Kazutoshi Fujikawa ,
d
Toshio Matsuura , Shinji Shimo jo , and Masatoshi Arikawa
a
Department of Information and Mathematical Sciences Graduate School of Engineering Science Osaka University, Toyonaka, Osaka 560, Japan
b
Graduate School of Information Science,
Nara Institute of Science and Technology
c d
Osaka City University
Hiroshima City University
ABSTRACT
We developed a networked virtual reality system on which one could have live videoes with high iteractive usage. However, since a client in our system receives both live videoes and computer graphics objects continuously from a server while it is rendering the current frame of the virtual world, a quality of the presentation of the virtual space depends on the current available system resources on the client as well as network resources. In this paper, we discuss the notion of QoS for a networked VR system as an application level QoS which is based on the notion of an importance of presence (IoP) of objects in a virtual space calculated by a distance from a current user's position to the object and an angle from the user's perspective to the object. Then, we show a heuristic mapping mechanism of the application level QoS to a network level QoS which determines the allocation of bandwidth of videos and CG objects to maximize the presentation quality. We adopt bandwidth as an example of network level QoS. We are developing our prototype system on the SGI Indigo utilizing the IRIS Performer with the Virtual Reality Modeling Language (VRML) 2.0. Keywords:
networked virtual reality, QoS control, QoS adaptation, vrml, adaptive application, video, multimedia 1. INTRODUCTION
Soon after the invention of virtual reality where people can navigate through computer generated graphic virtual spaces, people can share virtual spaces through exchanging their descriptions of virtual spaces by using Virtual Reality Modeling Language (VRML) . When one just clicks a mouse button in a virtual space, a server transfers a le which contains the description of a virtual space to the client and he can switch his position from one virtual space to another. However, although one can switch to another virtual space by just one click, he must wait for a while until an entire le is retrieved and a new virtual world is constructed on his local client. As the result, a le transfer interrupts a navigation of a virtual space every time when one changes a virtual space. That is, the quality of the navigation is greatly deteriorated by the interruption for transferring the le and by constructing the incoming new virtual space. Although transferring an entire le at the start avoids uncertainty of best e ort network performance, that is, network Quality of Service (QoS), it also makes a large virtual space model intractable and loses responsiveness to the user in the navigation. In that sense, a current networked VR system is not adaptive to the system and network resources. In addition to this problem, an introduction of a realtime video in a virtual space may cause another problem. By introducing a realtime video into the virtual space, one can bring more realistic elements into the virtual space such as a face image of an attendee in the virtual meeting room or an actual outside scenery shown in a virtual 1 2
Other author information: (Send correspondence to S.S) S.S: Email:
[email protected] Telephone: +81-6-879-8793; Fax: +81-6-879-8794
window. A server sends a realtime video to the client as a stream of video frames through the network and the client renders them as images in the virtual space. In this case, the quality of the video is greatly a ected by the current available network QoS such as bandwidth, delay and jitter, and amounts of processing for rendering video frames on the screen. During the execution of the system, these resources may starve and in such a case, the system can not maintain the quality of the VR space. Therefore, it is necessary to de ne a notion of QoS in VR and introduce an mechanism which maps QoS of a VR system to the current available CPU and network resources, that is, system and network level QoSes. In this paper, we propose a QoS mapping mechanism for networked VR system and enhance a current networked VR system into a adaptive application which can adapt to various system and network resources dynamically as well as statically. This paper is organized as follows. Section 2 discusses general terminology for addaptive applications. Related works are reviewed in section 3. Section 4 discusses the proposed QoS mapping mechanism and then implementation and experimental results are presented in section 5. Finally section 6 provides our conclusions. 3
2. ADAPTIVE APPLICATION
As the Internet grows, many networked applications such as WWW and remote conferencing become popular. However, these applications are expected to run on a rather limited system resources and network resources. And also, these limitations may vary network by network and system by system statically as well as dynamically. For example, a network bandwidth between a server and a client may vary client by client statically as well as in time. Since in networked applications, various clients which have di erent performances access to a server and retrieve a same contents, networked applications should adapt to the di erent system and network resources and introduce a QoS (Quality of Service) control mechanism. A networked application without QoS control mechanism can not work with di erent system and network limitations. For example, when one clicks a link connected to a sound content in a WWW page, one must wait until a whole content is transferred. An interactive networked application such as a remote conferencing application can adapt to the situation by trading o performance for quality. That is to say, a remote conferencing application degrades a quality of a video showing when the system and network resources are going to starve. In this case, a user has two choices for a degradation in terms of video quality: degradation of a temporal resolution, i.e. dropping a frame rate or a degradation of a spatial resolution, i.e. losing neness of a picture. These choises are called application level QoS. To achieve a certain level of application level QoS, a certain level of network and system resources are required: These are called system/network level QoS. The mapping between two levels of QoS is called QoS mapping. Since system and network provides di erent type of assurance for their resources from a best-e ort type to a guaranteed type, there needs some control function in between to manage QoS mapping. However, it is dicult for a user to specify an application level QoS because a user usually could not nd what is the best quality for a given situation. If the application becomes more complicated, it becomes more dicult to specify an application level QoS. In this paper, we consider a network VR system as a complicated example of networked applications. First, we enhance the current networked VR system into an adaptive application. Then, we de ne a notion of application level QoS in the enhanced networked VR system. We introduce a notion of Importance of Presence (IoP) as a de nition of an application level QoS which is based on the distance between a user and an object. Notion of IoP is very helpful for a user because it is automatically calculated by the system rather than speci ed by the user. 3. RELATED WORK
In related studies on QoS adaptation for video objects, proposed dynamic QoS control models for continuous media. However, because their applilcation level QoS is xed, it is dicult to select appropriate QoS paramter. Recently, there have been a lot of works in networked virtual reality systems. VRML provides the capability that users can build their own computer generated 3-dimensional virtual spaces easily and publish them by WWW fashion. In such an environment, one can navigate the virtual space constructed by other users, when he speci es the corresponding URL. Moreover, when one just clicks a mouse button in a virtual space, a le which contains the description of another virtual space is transferred and he can go into another virtual 5,6
9{18
17,18
space. VRML provides a basic mechanism for sharing and exchanging virtual spaces among dispersed users on the network. Although one can change a virtual space by just clicking, he must have a long wait until an entire le is retrieved and a new virtual space is constructed completely on his local client. As a result, le transfers interrupt a navigation of a large virtual space which consists of several VRML les whenever one changes a virtual space. In current VRML3 as well as many VR systems, there is a notion of scalability called level of detail(LoD). A user can de ne di erent level of the quality for each object in a virtual space by using LoD of which each level consumes di erent degree of end system and network performance. However, currently use of LoD is focused on only the system performance not on the network performance and most VRML-based systems can't change LoD of objects dynamically according to the current network performance. Funkhouser proposed object streaming and a prediction mechanism for navigating a virtual space in his system called RING. RING is a building walk-through system which supposes a \densely occluded" space partitioned into cells by walls, ceilings and oors. Therefore, this system take account of only the cells where a user is going to be. However, it can hardly handle a large \non-occluded" space, because, in that case, arbitrary region of a virtual space should be considered as a next user's position. Furthermore, its prediction mechanism is used only for cache management of a local disk and not for uncertainty of the network performance. In order for users to walk through such a large virtual space, we proposed a similar mechanism with RING called the Di erential VRML (DVRML). Several networked virtual reality systems take distribution and multiuser aspects into consideration. Participants and entities of these systems is required to exchange a large amount of information such as their behaviors and the de nitions of virtual spaces including texture images. On the Internet, which is a best-e ort network, messages might be delivered over high latency and low bandwidth paths. In such an environment, it is important that messages should be received by participants/entities within a certain delay. Therefore, these systems is expected to reduce the amount of trac among participants/entities and minimize end-to-end delay. Most of networked virtual reality systems utilize an unreliable multicast protocol or a reliable multicast protocol with negative acknowledgments in order to reduce the amount of trac. Even using a multicast protocol, a network congestion may occur as the number of participants/entities increases largely. In order to decrease the frequency of sending information, some systems employ dead-reckoning techniques. To summarize this section, although most virtual reality systems utilize the notion of LoD in order to provide the scalability, they consider only the distance between objects and user's position as a criterion for selecting a proper LoD. In large scale virtual environments, it is reasonable that a scalability mechanism to choose an appropriate LoD or quality of a virtual space, should be provided in terms of the network performance. Moreover, most of those systems can not handle live audio/video information, which is more sensitive to latency. Even although a few systems can handle audio and video information, none of them provides the dynamic quality control mechanism for such information in case of network resource starvation. 13
9
10,11,14{16
14{16
4. THE QOS MAPPING MECHANISM FOR VR
A network VR system consists of servers which store descriptions of VR spaces and clients which retrieve and render these descriptions. A description represents a collection of graphical objects (CG objects) and their geometrical relations in the virtual space which is called a scene graph. In our case, video objects are also included in the scene graph. After receiving a scene graph, a client renders objects and constructs a virtual space. Our purpose is the seamless integration of VR and a realtime video as well as the incremental acquisition of a large VR space. Starvation, however, may occur in the integration of VR and a realtime video which requires more network and local resources constantly. Therefore, we have to keep the QoS of VR as much as possible (navigation speed, interruption and LoD) when network and local resources are going to starve for the seamless integration. Meanwhile for the incremental acquisition of objects, a client have to interact with a server in realtime not in batch(we call it object streaming). For incremental acquisition of a large VR space, the proposed mechanism provides the object streaming which can handle a large space interactively and integrate realtime media such as streaming video. And also, for the seamless integration of VR and a realtime video, it applies the notion of IoP (Importance of Presence) to the video objects as well as the CG objects. To keep the QoS of VR even though network and local resource are going to starve, it 3 At August 1997, the latest version is VRML2.0.
direction of user’s eyes
CG object
l
&H
user Figure 1.
The calculation of IoP in the CG object
introduces the IoP of objects and adapts to the starvation by reducing the quality of the object which has the least . Besides it introduces the prediction of user's movement for smooth navigation.
IoP
4.1. The importance of presence (IoP)
As reducing the QoS of an object which has the least IoP, the proposed mechanism can lighten the load of CPU and network. The object which has a lower IoP tends to have lower resolution in the presentation while one who has a higher IoP has a higher resolution in the presentation when the network or the system is over loaded. We avoids degrading a QoS of a total presentation by losing a quality of each object randomly. Now, we describe how to calculate the IoP for the CG object and video object respectively. 4.1.1. CG ob ject
If an object is far from the user's position or away from the direction of his eyes, he may have little awareness of it. In our mechanism, the position of CG objects, that is the distance and the direction from the user, is considered into the IoP, while the conventional VR systems take account of only the distance from the user. Fig. 1 shows the relative position between the user and the CG object. Let the distance between the user and the CG object be l. And let the angle between the direction of the user's eyes and the direction of the CG object be . The IoP of the CG object is de ned as follows: IoP
l L
= 100 2 (1 0 ) 2 cos
where L is the maximum visible distance in the virtual space. 4.1.2. Video ob ject
As shown in Fig. 2, it seems that a user is likely to perceive a video object (V 1) which is away from the user but occupies larger area than one (V 2) which is closer but occupies smaller area. Therefore, the IoP of a video object is based on its area as well as the direction with respect to the user's eyes. Fig. 3 represents the relative position between a user and a video object. Let the ratio of the area occupied by the video object and the whole display area, the angle between the normal vector of the video object and the direction of the gravity of the video object with respect to the user's position, and the angle between the direction of the user's eyes and the direction of the gravity of the video object be S , and 2, respectively. The IoP of video objects is calculated as follows: 1
IoP
= 100 2 S 2 cos
1
2 cos
2
V1
V2 user Figure 2.
The perception among video objects
&H1
Figure 3.
The calculation of IoP in the video object
4.2. The object streaming
Users in a virtual space wait until the whole VRML le is transferred from the server at the start. When one move from a virtual space to another, the navigation may eventually stop and wait until the whole VRML le is transferred. To avoid this problem, we adopt the object streaming which transfers a subset of objects incrementally. The basic policy of the object streaming is as follows. CG ob ject
The server starts to send a CG object when a object comes into a user's perception.
The server starts to send a video object when a user approaches within a certain distance to the video object.
Video ob ject
4.3. The QoS mapping
The proposed mechanism controls the LoD of each object based on its IoP when the system is going to starve. The LoD of CG object means its resolution while the LoD of video object is composed of a temporal resolution such as frame per second and a spatial resolution such as the number of colors and the number of pixel/inch. For the CG object, the proposed mechanism adjusts a LoD of an object based on an IoP. Meanwhile, for the video objects, the proposed mechanism assigns units of CPU time and network bandwidth in proportion to their IoPs and then adjusts the temporal and spatial resolution to t into the units. For example, if the two video media, the movie 1 and the movie 2, exist in the same VR space at the same time and the system evaluates the IoP of the movie 1 as 20 points and movie 2 as 10 points, the assignment of CPU resource is by the ratio of 2:1. We implement this mechanism on the SGI's threads mechanism where we can add priority to the threads and control CPU resources. As the result, it adjusts the temporal resolution of the movie 1 and movie 2 to 6fps and 3fps respectively. If they are assigned 100 points of the IoPs, the mechanism would adjust their frame rate up to 30fps, that is, in a full scale.
feedback
SERVER System Monitor
JPEG
CG object
...
monitoring
feedback
object information
QoS adjuster
VRML Analyzer QoS controller
Object Streaming Subsystem
VRML file CG object
JPEG
QoS
status
Network JPEG
...
System
QoS controller
Object Streaming Subsystem
Monitor
RGB monitoring
CG object update view
Scene Graph Builder
status
cull
draw
CLIENT Scene Graph
Figure 4.
User Viewpoint Sender
Rendering System
The overview of the prototype virtual reality system
4.4. The prediction
Since the evaluation of IoPs of objects and the decision of what objects are sent in object streaming may cause some delay at the client, the navigation may not work smoothly. For the smooth navigation through a VR space, Funkhouser proposed a prediction mechanism for reducing the delay of disk-to-memory copy. We apply a similar mechanism to the client/server environment. The prediction is expected to work e ectively in VR system because a movement of a user in the VR space has continuity. Our proposed mechanism predicts a user's position in next t seconds at the server based on the current information of the user's moving vector transferred from the client. The speed of the user is set constant in our system because it becomes dicult to predict next user's position when the user moves with a certain acceleration. For the prediction, the client periodically sends the following informations to the server. 8
the user's current position the direction of his viewpoint the type of his motion(straight, rotation) 5. THE IMPLEMENTATION AND EXPERIMENTAL RESULTS
In this section, we describe the design and implementation of the dynamic QoS mapping mechanism for networked VR system on the SGI indigo2 using Performer as a rendering system. 4
5.1. The implementation
The overview of system equipped with the proposed mechanism is as shown in Fig. 4. In Fig. 4, the server is composed of a VRML analyzer, a QoS adjuster, a QoS controller, an object streaming subsystem and a system monitor. The VRML analyzer extracts information such as the location, the rotation and the size of each object from the VRML le and then saves it as a le. After the QoS adjuster calculates the IoP of
each object, the QoS controller adjusts the LoD of CG objects and QoS parameters of video objects corresponding to their IoPs. The system monitor is watching the CPU resource of both the server and client and the network load. The client is composed of a rendering system, a scene graph builder, a user's viewpoint sender, a QoS controller, an object streaming subsystem and a system monitor as shown in the lower part of Fig. 4. The rendering system displays objects in the scene graph on the screen. It updates the viewpoint of the user due to his/her moving, and renders the scene graph and nally displays rendered results on the screen. The prototype VR system displays objects in its scene graph at every 1/30 seconds. The scene graph builder inserts/deletes an object into/from the scene graph. A motion JPEG is used as a format of a video object in the implemented system and the video object is treated as a texture during displaying cycle. Since only RGB type objects can be put on the scene graph, the motion JPEG object is decompressed to a RGB type object in the scene graph builder. In the prototype system, we use the Cosmo compress of SGI which is a hardware accelerator for compressing/decompressing of the JPEG for fast decompressing. The scene graph builder controls the temporal QoS of the presentation. In that case, it changes the time interval of updating the objects in the scene graph. In short, it controls the frame rate of the presentation. In case of a video object, the scene graph builder can change spatial QoS by reducing the size of the presented area of the video object. The user' viewpoint sender receives the viewpoint information of the user from the rendering system and then extracts the coordinates of the viewpoint and the moving vector. Finally it sends them to the server at given interval as mentioned in the previous section. 5.2. The experimental results
We describe the experimental results of our VR system equipped with the proposed dynamic QoS mapping mechanism. To show e ective results of the proposed system, we prepare two systems, one with the proposed mechanism and one without any QoS control mechanism. In both system, a virtual museum is considered as an example of a virtual space. The left side of Fig. 5. shows an appearance of the virtual museum where the picture in the left represents a JPEG object. The right side of the Fig. 5 represents the whole perspective of the virtual museum. The size of the VRML le which describes the virtual museum is 550,432 bytes. The waiting time until he/she can enter the virtual museum is measured on the system both with and without the QoS adaptation mechanism. The table 1 shows the mean waiting time of 10 experiments. The table 1 designates that the di erence of the waiting times is rather wide even though the virtual model is relatively simple. We believe that di erence becomes bigger in the large virtual space model. 6. CONCLUSION
In this paper, we de ne a notion of QoS in VR system and a heuristic QoS mapping mechanism is presented for the seamless integration of VR and a realtime video and the incremental acquisition of a large VR space. For incremental acquisition of a large VR space, the proposed mechanism provides the object streaming which can treat a large model interactively and realtime media such as streaming video. And also, for the seamless integration of VR and a realtime video, it applies the notion of LoD to the video objects as well as the CG objects. To keep the QoS of VR even though network and local resources are going to starve, it introduces the IoP (Importance of Presence) of objects and adapts to the starvation by reducing the quality of the object which has the least IoP. Besides, it introduces the prediction of user's movement for smooth navigation. Experimental results on the SGI Indigo2 show that the VR system equipped with the proposed mechanism is ecient in respect of smooth navigation and smaller user waiting time. Our future work will include the multiuser mode whose prediction will be performed at clients. Table 1.
The comparison of mean waiting time
status without the mechanism with the mechanism
mean waiting time(second) 8.47 5.32
Figure 5.
The virtual museum of art
ACKNOWLEDGEMENTS
This work was supported in part by Research for the Future Program of Japan Society for the Promotion of Science under the Project \Researches on Advanced Multimedia Contents Processing", and by a Grant Aid for Science and Resarch from the Ministry of Education, Science, and Culture of Japan. We wish to thank members of H2O project for many useful conversations. REFERENCES
1. \VRML2.0," http://vrml.sgi.com/moving-worlds/spec/part1/nodesRef.html. 2. O. Hagsand, \Interactive Multiuser VEs in the DIVE System," IEEE MultiMedia, pp. 30{38, Spring 1996. 3. M. Arikawa, A. Amano, K. Maeda, R. Aibara, S. Shimojo, Y. Nakamura, K. Hiraki, K. Nishimura, and M. Terauchi, \Dynamic LoD for QoS Management in the Next Generation VRML," IEEE,Proc. of the International Conference on Multimedia Computing and Systems,Hiroshima, pp. 24{27, June 1996. 4. J. Rohlf and J. Helman, \Iris Performer:A High Performance Multiprocessing Toolkit for Real-time 3D Graphics, " Proceedings of SIGGRAPH, pp. 381{394, July 1994. 5. K. Kawachiya and H. Tokuda: \Dynamic QOS Control Based on the QOS-Ticket Model," Proc. 3rd IEEE Intl. Conference on Multimedia Computing and Systems (ICMCS'96), pp. 78{85,1996.6. 6. Seiwoong Oh, Kazutoshi Fujikawa, Toshio Matsuura, Shinji Shimojo, Hideo Miyahara, \The design of senario description language supporting QoS speci cations in the distributed multimedia system Symphony,"IEICE Transactions on Communications, B-I, vol.J79, No.5,pp.329-337, 1996.5. 7. H. Nakanishi, C. Yoshida, T. Nishimura, and T. Ishida, \Freewalk:Supporting Casual Meetings in a Network," International Conference on Computer Supported Cooperative Work, pp. 308{314, 1996. 8. Thomas A. Funkhouser, \Database Management for Interactive Display of Large Architectural Models," Graphics Interface'96, Toronto, Ontario , pp. 1{8, May 1996.
9. M. Arikawa, A. Amano, K. Maeda, R. Aibara, S. Shimojo, Y. Nakayama, K. Hiraki, H. Nishimura, and M. Terauchi, \Dynamic LoD for QoS Management in the Next Generation VRML," Proceedings of third IEEE International Conference on Multimedia Computing and Systems(ICMCS'96), pp.24{27, June 1996. Hiroshima, Japan. 10. J.W. Barrus, R.C. Waters, and D.B. Anderson, \Locales: Supporting for Large Multiuser Virtual Environments, " IEEE Computer Graphics and Applications, Vol.16, No.6, pp.50{57, November 1996. 11. S.D. Benford, L.E. Fahlen, and J.M. Bowers, \Managing Mutual Awareness in Collaborative Virtual Environments," Proceedings of ACM SIGCHI Symposium on Virtual Reality Software and Technology (VRST'94), pp.223{236, August 1994. Singapore. 12. T.A. Funkhouser, \RING: A Client-Server System for Multi-user Virtual Environments," Proceedings of ACM SIGGRAPH Special Issue on 1995 Symposium on Intereactive 3D Graphics, pp.85{92, 1995. Monterey, CA, USA. 13. T.A. Funkhouser, \Database Management for Interactive Display of Large Architectural Models," Proceedings of Graphics Interface '96, pp.1-8, May 1996. Toronto, Ontario, Canada. 14. O. Hagsand, \Interactive Multiuser VEs in the DIVE System," IEEE MultiMedia, Vol.3, No.1, pp.30{39, Spring 1996. 15. R. Lea, Y. Honda, K. Matsuda, O. Hagsand, and M. Stenius, \Issues in the design of a scalable shared virtual environment for the internet," Proceedings of HICSS'97, January 1997. Hawaii, USA. 16. M.R. Macedonia, M.J. Zyda, D.R. Pratt, P.T. Barham, and S. Zeswitz, \NPSNET: A Network Software Architecture for Large Scale Virtual Environments," Presence, Vol.3, No.4, pp.265{287, Fall 1994. 17. M. Pesce, VRML - Browsing and Building Cyberspace, New Riders, 1995. 18. \VRML 2.0," http://vrml.sgi.com/moving-worlds/spec/. August 1996.
