Eurographics Italian Chapter
July 11-12- 2002
UNTETHERED INTERACTION FOR IMMERSIVE VIRTUAL ENVIRONMENTS THROUGH HANDHELD DEVICES Davide Brunelli, Elisabetta Farella Dip. Elettronica Informatica Sistemistica - Università di Bologna, V.le Risorgimento 2, 40136, Bologna, ITALY Tel: +39 051 6171573, Fax: +39 051 613.72.73, E-mail: {efarella|dbrunelli}@deis.unibo.it
Maria Elena Bonfigli Vis.I.T. Lab - CINECA Supercomputing Centre,Via Magnanelli 6/3, 40033 Casalecchio di Reno (BO), ITALY Tel: +39 051 6171573, Fax: +39 051 613.72.73, E-mail:
[email protected]
Abstract This paper presents the results and the current status of our research project focusing on the design and implementation of interfaces for interaction with Immersive Virtual Environments through handheld devices. The use of this kind of devices, combined with wireless networking, improves the usability of standard VR. This could be particularly useful in application fields (i.e. Virtual Heritage), where, for the vast majority of users, we cannot expect any significant exposure to advanced VR environments and their specialized interfaces. The current implementation includes a Graphical User Interface (GUI) and support for multi-presence based on peer to peer communication.
Keywords: Virtual Reality, Palmtop Computers, Immersive Virtual Environments, Human Computer Interaction, 3D Interfaces Introduction Virtual Reality (VR) interfacing has traditionally been based on specialized devices that are often produced for a relatively small niche market, extremely expensive - e.g. in the case of high-end military or industrial Virtual Environment (VEs) - and subject to several limitations - e.g. the freedom of movement in the VEs- with the result that the average user of computing and electronic equipment is not familiar with VR interfacing devices[1]. Managing interaction through handheld devices is available solution to these problems while preserving the sense of presence in Immersive and semi-immersive VEs (IVEs), as outlined by previous work. [4,5,6]. The contribution of our research project is the design and implementation of interfaces for remote interaction with IVEs through handheld devices that are flexible and scale with the number of users and with the different kind of terminal on which the interaction interface is downloaded General technical requirements for our system are the following: (i)Wireless connectivity in order to augment sense of presence and immersion inside the Virtual World; (ii) Multi-client enabling multi presence on a networked IVE; (iii) Platform independence in order to support different kinds of handheld device (i.e. a Pocket Pc or a laptop, and in future a cellular phone, of any brands with different hardware capabilities). Using handheld devices for interacting with IVEs Recently [1] we have designed a set of Java based User Interfaces for interacting in existing IVEs visualized in a Virtual Theater (VT) through standard, consumer-market Pocket PC - Compaq iPAQ 3630 running Windows CE 3.0. The VT is a structure based on the Reality Center SGI Technology 1
Eurographics Italian Chapter
July 11-12- 2002
dedicated to the semi-immersive three-dimensional computerized visualization with a surround-sound system and a semicircular screen that allows a user to experience a “physical” immersion in the projected VEs wearing stereoscopic glasses. Usually all the VR applications developed for the VT were managed by a “driver” through the Onyx2 console; now are the users that interact with the VEs and among themselves through their own PDAs. To interact through handheld device with the real time VR applications built as a layer on top of Vega and OpenGL Performer, we have taken into account the following specifications: (i) full support for wireless communication; (ii) the server side rendering on VR engine is Vega, supported in our installation with C, C++ procedural interface; (iii) the client side need to be multiple -platform.
Fig. 1 : Interactions and PDA User interface for the Nu.M.E. project From the client side, we have chosen Java to guarantee platform-independence. Furthermore Java provides a rich set of libraries. The mechanism by which the server and the client communicate and exchange information is RMI. At the server side, we combined RMI with Java Native Interface. The use of JNI enables interoperability with applications and libraries written in other languages (C, C++, assembly) to interface with Vega. Wireless connectivity is provided by a IEEE 802.11b High Rate standard wireless LAN interface working at 11 Mb/s. All the User Interface designed and implemented in the framework of our research project include common interaction features individuated as particularly useful. Among them: - An active “2D map” in which the users visualize all their position and orientation in the virtual world, which regions can be reached from their current position, etc. - “Free and guided navigation” of VEs, in order to move from place to pla ce following personal interests and desires and to run automatic tours to provide the users with a quick overview of the virtual environment and of its objects, helping people to acquire spatial knowledge. - The possibility to select objects in order to connect them with bi-dimensional data and to query databases [3]. - The exchange and collection of multimedia data (images, texts, voice, sounds, video/audio streaming) among different users. It is worth noting that the last feature implied the development of peer to peer interaction among clients and scheduling protocols to organize the priority in controlling the VR exploiting the mix of using Java and RMI. Figures 1 and 2 show:
2
Eurographics Italian Chapter
July 11-12- 2002
1. the User Interface developed for interacting via PDA with the Nu.M.E. [2] VE in which we have developed also a time bar to select the historical period in which the users wish to take their virtual visit (Fig.1) 2. the User Interface developed for interacting via PDA with the “Ancient Appian Way 3D Web Virtual GIS” [3] in which the users can also directly manipulate single small Roman tombs (Fig. 2).
Fig. 2 : Interactions and PDA User interface for “Ancient Appian Way 3D Web Virtual GIS" project Towards a customisable interaction We are currently targeting customisation of remote interaction to build personalized interfaces, selecting the interaction features and the type of handheld device (palmtop, laptop or even cellular phone). The final target of the proposed tool is to build distributed applications which are flexible and scale with the number of users and with the different kind of terminal on which the interaction interface is downloaded. A first direction of customisation will be supporting the choice of the controls to create the most effective interface for interaction with a specific IVE, to have only and all the controls, maps and features really needed. The tool would also support the download of the interface directly on the mobile device to interact with the preferred virtual world. Another direction of customisation will be to support the different kinds of platform on which the interface could be downloaded. The choice of the controls and interaction features for the interface is strongly influenced by the processor, the memory, the multimedia capabilities of the hardware. From the client side the application is in fact characterized by portability on terminal of common use like laptops, palmtops of different brands, with different hardware capabilities or cellula r phones. This even supporting different communication interfaces i.e. GSM, Bluetooth or GPRS. Aknowledgments Authors wish to thank Prof. Francesca Bocchi (Bologna University) scientific director of the NuME Project; and Prof. Marco Gaiani (Politecnico di Milano) scientific director of the “Ancient Appian Way 3D Web Virtual GIS” and their collaborators who collaborate with us to accomplish our researches; Professors Bruno Riccò and Luca Benini (Dept. of Electronics, Computer Science and 3
Eurographics Italian Chapter
July 11-12- 2002
Systems - University of Bologna) who coordinate the project in collaboration with CINECA High Performance Systems Division. References [1] L. BENINI, M.E. BONFIGLI, L. CALORI, E. FARELLA, B. RICCÒ " Palmtop Computers for managing Interaction with Immersive Virtual Heritage", in Proceedings of EUROMEDIA2002, pp. 183-189. [2] F. BOCCHI, M.E. BONFIGLI, M. GHIZZONI, R. SMURRA, F. LUGLI, “The 4D Virtual Museum of the City of Bologna, Italy”, ACM SIGGRAPH99 Conference Abstracts and Applications, Los Angeles (USA), August 1999, pp. 8-11. [3] M. GAIANI VR as a Tool for Architectural & Archaeological Restoration: The “Ancient Appian Way 3D Web Virtual GIS”, In VSMM 2001 Proceedings; Berkley, USA, October 2001. [4] L. HILL, C. CRUZ-NEIRA, “Palmtop interaction methods for immersive projection technology systems”, Fourth International Immersive Projection Technology Workshop (IPT 2000), 2000 [5] K. WATSEN, D. P DARKEN, W.V. CAPPS, “A Handheld Computer as an Interaction Device to a Virtual Environment,” Third International Immersive Projection Technology Workshop (IPT 1999), Stuttgart, Germany, 1999. [6] M. M WLOKA,., E. GREENFIELD, “The Virtual Tricorder: A Uniform Interface to Virtual Reality”, UIST’95 Proceedings, 1995.
4
