Lowering the Development Time of Multimodal Interactive Application: The Real-life Experience of the XVR Project Marcello Carrozzino, Franco Tecchia, Sandro Bacinelli, Carlo Cappelletti, Massimo Bergamasco PERCRO – SSSA Via R.Piaggio 34 56025, Pontedera (PI), Italy +39 050 883075
Marcello, Franco, Bacinelli, @sssup.it;
[email protected];
[email protected] ABSTRACT In this paper we present XVR, an integrated development environment for the rapid development of Virtual Reality applications. Using a modular architecture and a VR-oriented scripting language, XVR contents can be embedded on a variety of container applications. This makes it suitable to write contents ranging from web-oriented presentations to more complex VR installations involving advanced devices, such as real-time trackers, haptic interfaces, sensorized gloves and stereoscopic devices, including HMDs. Some case studies are also presented to illustrate the development processes related to XVR and its features.
Categories and Subject Descriptors H.5.1 [HCI]: Multimedia Information Systems -- Artificial, augmented, and virtual realities
General Terms Algorithms, Design, Standardization, Languages.
Keywords Virtual Reality, 3D Graphics, Developer Framework, 3D web
1. INTRODUCTION Originally generated in the Computer Graphics research field, Virtual Reality (VR) has nowadays conquered a full independence as a research topic. Despite being a very popular topic of general discussion, the use of Virtual Reality outside the labs where it is researched still finds many difficulties. Real-life applications emerge at a slower-than-expected pace, and the so often foreseen impact of VR on the everyday life of an individual is still unclear. There are some application fields that have shown a superior receptivity to VR concepts and techniques, the best example being the Games industry, but under a more general view VR has not yet reached a large market.
One of the difficulties to get a broader acceptance of VR is the lack of sound, stable and effective standards for the development of applications. Many excellent tools exist now able to address specific VR-subtasks with optimal or near-optimal solutions, including scene-graph management, sound management, collision detection, physically-based modeling, haptic rendering and others, but as these tools are often provided as separate, independent and well-confined libraries, they often lack the general vision of a truly far fetched and integrated development environment. As a result, code duplication and data redundancies are common, and the development process is made clumsy and unnecessarily problematic simply by the diversity of languages, principles and platform used. Example of tools that often need to be joined together include tools for: Scene Graph management [3][9], collision detection [4][5], device management [6][7], haptic rendering [8], 3D sound [10][11], real-time physics [12],network management [13][14] and many others. In the present article we try to define and propose a general architecture for VR application, adding some development case-study to exemplify its use. We do this through quoting as a real example the development environment that we created during several years of experience in VR development. We do not claim this environment to be the best possible: there are many aspects of our system where things could be changed, improved or expanded, but we nevertheless found the overall approach used to be reasonable and effective, and we hope that the present work could help the definition of a general architecture for a fully standard VR development platform.
2. XVR: AN OVERVIEW Originally created for the development of web-enabled virtual reality applications, XVR has evolved in the recent years to an allaround technology for interactive applications. Beside Web3D content management, XVR now supports a wide range of VR devices (such as trackers, 3D mice, motion capture devices, stereo projection systems and HMDs) and uses a state-of-the-art graphics engine for the real-time visualization of complex threedimensional models that is perfectly adequate even for advanced off-line VR installations. XVR applications are developed using a dedicated scripting language whose constructs and commands are targeted to VR, and give the possibility to developers to deal with 3D animation, positional sounds effect, audio and video streaming and user interaction. The XVR platform comes equipped with a set of tools for the design and implementation of VR applications. There is an integrated developer framework (XVR Studio, Fig.1) containing a script editor, a workspace manager, a compiler, a FTP publishing
module, and an output window to trace messages for debug and profiling purposes. To speedup the relatively standard job of setting up an interactive application, XVR Studio integrates an application wizard which guides the programmer with a series of step-to-step instructions to setup the application. The wizard allows choosing the basic parameters of the virtual scene (number and position of lights, cameras etc.), the basic interaction metaphors and the initial data files to use. Once the procedure is completed, the wizard automatically generates an s3D script corresponding to these functionalities, with a series of “todo” sections which the programmer can fill in with the appropriate code.
necessary initializations are performed and then the application starts to execute the various loops. An XVR program can be represented as a main loop which integrates several loops, each one running at its own frequency, such as graphics, physics, networking, tracking, and even haptics, at least for the hi-level control loop (Figure 2).
Fig. 2 – XVR main loops This straightforward code-flow and its modularity allow XVR to be easily adapted both to the development of web-oriented or stand-alone applications, according to the specific needs.
3. S3D, THE XVR SCRIPTING LANGUAGE Fig. 1 – The integrated development environment XVR Studio
2.1 Run-time playback of XVR contents XVR content can be reproduced using a dedicated XVR Player, that in its current form is an ActiveX component running on the various Windows platforms, and as such can be embedded in several container applications including the web browser Internet Explorer. The XVR player is actually divided in two main modules: the ActiveX Control module (weighting just 30Kbytes), which hosts the very basic components of the technology, like the versioning check and the plug-in interfaces, and the XVR Virtual Machine (VM) module (currently around 550Kbytes), which contains the core of the technology, such as the 3D Graphics engine, the Multimedia engine and all the software modules managing the other built-in XVR features. Depending on the application environment It is also possible to load additional modules which offer advanced functionalities, like the support to VR devices such as trackers and gloves, as we decided to keep them separated so that web applications, which usually do not need any of these advanced features, are not afflicted by additional downloading times. The XVR-VM, like many other VMs, contains a set of bytecode instructions, a set of registers, a stack and an area for storing methods. The XVR Scripting Language (S3D) allows specifying the behavior of the application, providing the basic language functionalities and the VR-related methods, available as functions or classes. The script is then compiled in a bytecode which is processed and executed by the XVR-VM. The downloading mechanism started by accessing a web page hosting an XVR application is somehow typical: after a version checking and an optional VM updating, the bytecode of the application is downloaded and, subsequently, so do the first batch of data files related to the application, such as 3D models, textures, sounds etc. Immediately after this initial step all the
No matter how complex it is, an XVR program is always based on 6 fundamental functions. These are the only predefined functions, and they constitute the basis of any project: - OnDownload () - OnInit () - OnFrame () - OnTimer () - OnEvent () - OnExit () The OnDownload() function is performed at the very beginning of the execution and triggers the download of the data files needed from the application. The OnInit() function is the place where to put the initialization code for the app. All the commands are executed sequentially. All the other functions are not active until OnInit() completes its execution. The OnFrame() is the place for all the functions and methods that produce graphics output and is the only place where the graphics context is accessible. Placing graphics command outside this function would produce no results. The OnTimer() function runs independently (i.e at a different rate) by OnFrame() and it’s where to put commands that must be independent from the rendering task. As the timer is hi-res, it is possible to setup some parameters so that this function can be called up to 1k times per second. The OnEvent() function is independent from both OnTimer() and OnFrame(). It gets called whenever the application receives an event message. Event messages can be external (i.e. some Windows messages) or internal (i.e. generated anywhere in the XVR program). Events and messages are supported in XVR because they add flexibility to the programming environment for task where fixed timers are not the best option. The OnExit() function is called when the application quits or when the user close the page the application is
in. Beyond the basic functions, XVR offer lots of predefined classes, functions and data structures, the user having the possibility to define new ones. The integrated 3D engine, built on top of OpenGL, allows to manage the visual output not only on a standard graphical window (either web or local hosted), but also on more advanced devices such as Stereo Projection Systems, Head Mounted Displays or Chromium-driven PC clusters. The engine uses state of the art algorithms of culling, simplification, normal mapping and image caching to achieve good real-time performances even with high-complexity models.
XVR technology, available both in ASCII and binary format. AAM can be used to export advanced material properties, multiple texture layers, skeletal and non-skeletal animations and in general all the properties not supported by plain VRML. So far an exporter plug-in is available for the Discreet 3DStudioMax modeling software, which produces AAM files containing data related to geometries, materials, smoothing groups, multi-level textures (supporting reflection, lighting and shadow maps), animation frames, skin deformation, hierarchies.
5. USING XVR IN REAL LIFE
Fig. 3 –Real-time INDICA graphics with XVR A feature that has proved to be extremely useful is that, although the S3D scripting language offers high-level functionalities to manage the 3D contents, it also allows mixing low-level functionalities to realize special effects or personal implementations not directly supported by the language. This is achieved through an almost complete wrapping of OpenGL functions that can so be used in an S3D script. By mixing highlevel function calls to low-level OpenGL programming almost every possible “special effect” is achievable, with no need to modify the source code of the XVR engine. Network communication is handled by means of TCP/IP network functions, which offer connection-oriented or connectionless UDP functionalities, and HTML-related functions, which allow the hosting web page to communicate with the XVR application so that they can reciprocally interact and modify their status. Finally XVR offers a set of multimedia related classes and provides the possibility to dynamically create sound sources and to position them in the 3D space. XVR supports also remotely located sound servers allowing the use of more than one planar 3D audio systems in order to realize a complete positional audio system able to localize the acoustical source not only in the plane surrounding the user, but rather in the whole space adding a better perception of the up-down directionality. Many other functions exist, offering various functionalities related to files, strings and math manipulation.
4. THE AAM FILE FORMAT The data files which can be used by XVR applications are, mainly, geometrical description of the 3D models and related materials, image bitmaps, text and multimedia files. The geometrical data of the 3D models can be expressed in the standard VRML[2] format or in the AAM format with planned support for X3D. The AAM format is the native format of the
Since 2002 XVR is the standard platform of the VR applications realized by PERCRO in the course of several projects. Some examples are the website of Piazza dei Miracoli in Pisa (http://piazza.opapisa.it/3D/index.html), the Museum of Pure Form (http://www.pureform.org/virtualGallery.htm), and an educational project for children (http://www.andrea.baroni.name/ scienzademo) which are available on-line. Other examples are related to more complex VR systems, such as INDICA (Fig.3), a forklift motion-based simulator [15], LINK-2, a virtual prototyping system [16] developed with Piaggio Spa, and Assembly Master, an immersive mechanical CAD model manipulator. XVR is also being used as a test developing platform for the “Virtual Environments” course at the University of Pisa. Students develop several projects focused on interactive, VRoriented applications, and they have to do it in a very short time due to of the brief duration of the course. To give examples of the flexibility of XVR technology, we illustrate here two very different projects: the first is a student project realized in just a couple of months, the other is a complex motion-based VR installation. The technology and the development tools are vastly identical in the both cases.
5.1 Case study 1: XVR Wars XVR Wars (Fig.4) is a multiplayer First Person Shooter in which several network-connected players fight against each other in order to remain the only survivor of the arena. The game levels have been modeled in 3DStudioMax and exported in the AAM format. In the web version of the game, the control of the avatar is realized through keyboard and mouse. For the terrain following, the classic technique of the "perpetual fall” was used. Also, effects like inertia or friction are simulated. In order to quickly implement the basic physics, XVR embeds a number of functions related to vectors and matrices. A recent addition is a built-in physical engine through which many of these calculations are automatically performed. The trade-off is represented by additional computational resources, so a balanced application should consider whether to use the full physical engine or not, depending on the features to implement. Adding positional audio to XVR games is then a simple operation. It is only required to create the needed audio3D objects, with their position and their speed, and to set position and orientation of a listener. Subsequently, the only needed operations are to update these data at each frame. The Doppler effect is also supported. On the networking side, a decentralized solution has been chosen for this game. Each node sends broadcasts UDP packets to all the others node. At the beginning there is a logical distinction between server and clients, but during the game this distinction disappears.
6. REFERENCES [1]http://usinfo.org/USIA/usinfo.state.gov/products/pubs/archive/ telecomm/glossary.htm [2] Gavin Bell, Anthony Parisi, Mark Pesce. "The VRML 1.0 Specification," in Proceedings of the Second International Conference on the World Wide Web, Chicago, October 1994. [3] Rohlg, J., Helman, J. IRIS Performer: A High Performance Multiprocessing Toolkit for Real-Time 3D Graphics. Proceedings of SIGGRAPH 94. In Computer Graphics Proceedings, Annual Conference Series, 1993, ACM SIGGRAPH,
Fig. 4 – XVR Wars screenshot Following the web based version of the game, the support to advanced VR functionalities, such as HMD stereoscopy and head and wand tracking, has been provided by means of a additional DLL library, imported in XVR as external modules. We used a NVIS NVisor SX to provide a 60 degrees (diagonal) FOV and an Intersense IS900-VES for head and wand tracking. With small changes in the code the game has been adapted to support stereoscopy, with a truly remarkable final effect, on a passive stereo projection system.
5.2 Case study 2: The INDICA simulator The INDICA project concerned the design and realization of a motion-based forklift driving simulator (Fig.5), used for training purposes, that can replicate on the driver the physical feelings produced by the movements of the vehicle by means of a Stewart platform which supports a real forklift cockpit, two projection systems for front and rear views and a 4-speakers sound system.
[4] Hudson, T.C., M.C. Lin, J. Cohen, S. Gottschalkand D. Manocha (1997): V-COLLIDE: Accelerated Collision Detection with VRML, In: Proc. of Symposium on the Virtual Reality Modeling Language pp. 381–394. [5] G. Zachmann. Rapid collision detection by dynamically aligned DOP-trees. IEEE VRAIS’98 Atlanta, March 1998. [6] Carolina Cruz-Neira.VRJuggler2001 http://www.vrjuggler.org [7] D. Pape, pfCAVE CAVE/Performer Library (CAVELib 2.6), http://www.evl.uic.edu/pape/CAVE/prog/pfCAVE.manual.html. [8] Gregory, A., Lin, M., Gottschalk, S., and Taylor, R.,1999. HCollide: A framework for fast and accurate collision detection for haptic interaction. Proc. IEEE Virtual Reality Conference. [9] W. Bethel, RM Scene Graph, White Paper, December 1999, http://www.r3vis.com/Meta/SiteMap/index.html. [10] N. Tsingos, “A Versatile Software Architecture For Virtual Audio Simulations,” in Proceedings of the International Conference on Auditory Display 2001, Espoo, 2001. [11] E. M. Wenzel, J. D. Miller, and J. S. Abel, “A softwarebased system for interactive spatial sound synthesis,” in Proceedings of the International Conference on Auditory Display 2000, Atlanta, 2000 [12] S. Cotin, H. Delingette, and N. Ayache. Real Time Volumetric Deformable Models for Surgery Simulation. In K. Hohne and R. Kikinis, editors, Visualisation in Biomedical Computing, pages 535 540, Springer, 1996.
Fig. 5 – The motion-based INDICA simulator Three different scenarios are implemented, each one referring to an existing packaging factory. Forecourts and buildings have been photographed, and textures have been created from them and applied on a geometrical model of the scenario. Each scenario was created in 3DStudioMax. The simulation is viewed with a firstperson perspective from inside of the cockpit, which is completely real with the exception of the forks that are virtual. The physics module is also in charge of the hi-level control of the Stewart platform that communicates with XVR by means of a shared memory data structure, whose management is regulated by an XVR external module.
[13] Blau, Brian, Charles E. Hughes, Michael J. Moshell ,and Curtis Lisle, Networked Virtual Environments. ACMSIGGRAPH Special Issue on 1992 Symposium on Inter-active 3D Graphics, (Cambridge, MA), 1992, 157-164. [14] Carlsson, Christer, and Olof Hafsand, Dive: A MultiUserVirtual Reality System.Proceedings of the IEEE Virtual Reality Annual International Symposium, Septem-ber, 1993 [15] Forklift truck simulator for training in industrial environment – PERCRO-SSSA Pisa (Internal Document) [18] Carrozzino, Raspolli, Tecchia, Bacinelli, Bergamasco, “Virtual Prototyping for 2-Wheels Vehicles”, Workshop Via 2004, Compiegne (France) November 2004
