The AVIARY VR-system. A Prototype Implementation David N. Snowdon∗, Communications Research Group Department of Computer Science The University of Nottingham Nottingham NG7 2RD, UK Tel: +44 602 514226 Email:
[email protected] Adrian J. West Advanced Interfaces Group Department of Computer Science University of Manchester Manchester M13 9PL, UK Tel: +44 61 275 6251 Email:
[email protected] To be presented at the 6th ERCIM workshop, 1-3 June 1994, Stockholm. Abstract The AVIARY project at the University of Manchester has sought to understand the needs of a mature VR-based environment in which diverse applications co-exist, and in which the need for perceptual coherency of the environment is also crucial. A novel feature of AVIARY is that multiple, worlds may concurrently exist, each with its own set of laws which may be tailored to suit particular application domains. The overall structure however is hierarchical so as to permit a coherent relationship between worlds. This is useful both for purposes of code reuse, and to aid users in navigating the system. The project has currently produced a prototype implementation which has addressed underlying issues such as distribution across heterogeneous processor networks, dynamic management of objects and message types within the system, the separation of graphics processing, and the management of spatial extent. Implementations of the prototype have been tested on a Transputer array, a network of Sun workstations, and Silicon Graphics Crimson VGXT and Indy SC workstations. We are currently porting the implementation to a 2.4Gflop KSR-1 supercomputer. In this paper we present an overview of the AVIARY prototype, and describe an application implemented using this system.
∗
Author for correspondence
1
1
Introduction
VR offers some very exciting prospects for human-computer interaction. Principally these revolve around the potential to engage the human perceptual systems directly, as opposed to the primarily cognitive interactions we have with conventional interfaces. The current information bandwidth between human and machine appears rather low compared to the raw information that we process effortlessly as we walk through a shopping centre. Although VR hinges quite crucially on the development of peripheral technology we have assumed that these will be addressed commercially, and have provisionally assumed that increasingly capable peripheral technology will appear as a matter of course. The increasing interest of manufacturers, particularly in Japan, together with the advent of digital television, are good indicators of this trend. Our interest has therefore focused on the use to which VR will actually be put in largescale demanding applications. If we imagine a mature vr technology of several years hence, as opposed to the tentative demonstrators of today, we are led to ask what kind of systems will be required to support it. We have sought to address VR from this perspective, and believe that we have gained some insight into the problems to be resolved. The result of this work is a VR system architecture that we have named AVIARY. Here we present our perspective on the requirements and design issues for such a system, and details of a prototype implementation.
2
Design issues
Although VR has been with us for many years, the serious attempt to apply it on a large scale, such as we envisage, is a comparatively new venture. Researchers, quite rightly, hold different opinions on which features are crucial, and which of merely secondary, or cosmetic, importance. As these views determine the problems that one is attempting to solve, and hence the general approach, it is important to make them explicit if we are to avoid misunderstanding. Our strategy has been to study a number of large-scale applications for which VR seems to offer significant benefits. Thinking through their actual use has led us to some conclusions about the nature of the environment that we wish to provide. A more detailed review of design issues in VR systems is given in [1]. In this paper we present a brief list sufficient to set the AVIARY design in its context; a more detailed exposition is given in [2]. Multiple Applications We assume that multiple applications need to be available concurrently. Applications in this sense range from small scale tools available for use in the environment, to large scale activities encompassing much of the perceived environment. Multiple Worlds A single environment provided by the system would not be ideal for all applications as in some cases the requirements of diverse applications may be mutually exclusive. The notion of gravity may for example be implicit for some applications, and highly undesirable for others. We propose a series of environments, termed worlds, arranged in a hierarchy after the fashion of an object-oriented class hierarchy. Appropriate worlds can be constructed by instancing, or further refining those already provided. The roles of worlds and applications Given this system, a clear distinction is required between worlds and applications. We regard a world as providing a set of laws which logically apply to all things which exist within it. This implies that all things in a given world have a set of attributes to which the laws apply, as in the case of mass for inertia. Any other behavior is regarded as due to the volition of applications or users.
Coherency of the model The structural relationship between the different worlds is significant beyond the concerns of code reuse and applications programming. An important feature of VR is the engagement of human perceptual capabilities leading to skill acquisition. These will be greatly hindered if applications or worlds present arbitrary interfaces to the user. We must therefore strive to provide a structure which has a high degree of overall coherency, where the relationship between different environments can be readily comprehended. Whilst this is difficult to express in concrete terms, we feel that it is a particularly vital aspect of large-scale vr environment. Users Users are a significant feature of vr system design as performance critical parameters such as lag and frame rate must be maintained for them. It is therefore tempting to see users as an integral part of the system. Logically however there is little difference between users and applications. Both exist outside the virtual world system and interact with it via artifacts whose behavior they control. By seeing users as applications in this way, we achieve a useful simplification. There is the additional advantage that users can be added to the system, or their representation modified far more independently than if their code were built into the kernel. Extensibility As there is yet much to learn about virtual environments, we would not wish the system itself to impose a rigid idea of what such an environment should be. This unfortunately conflicts with the desire to provide as much assistance to the application programmers and users as possible. The way out of this dilemma is to provide a a rich framework that may be extended as necessary. Object oriented principles provide a good basis for this as we can provide classes of worlds which may be used or further refined as desired. If fundamentally different models are required on occasion, then we can take worlds closer to the top of the structure as the starting point. If maintained properly, then the structure will also provide a good map of the relationship between the different worlds and applications. Platform architecture As a final point here, we also need to consider the target architecture for the system. Decoupling the architecture from any particular platform allows appropriate machines to be used for particular tasks. One example of this is the use of specialised graphics engines for rendering, and the use of parallel supercomputers for the actual world modeling. Given the computational complexity of the tasks involved, the use of parallel processing is essential. On a larger scale we must also account for users at physically separate locations, and therefore require the system to be distributable. The platform architecture is therefore taken to have a hetrogeneous, distributed and parallel character.
3
The Prototype Implementation
These then are some of the guiding observations that we have made in proposing a design for a large scale VR-system. The conceptual model resulting from this embodies the hierarchy of worlds, the roles of applications and users, and the extensibility and distribution aspects as outlined. In order to explore the detailed issues of implementation, a prototype has been constructed. This contains many of the essential mechanisms that we need to investigate in order to assess performance. The current implementation runs on the SUN 3 and SUN 4 series of workstations, the Silicon Graphics 4D range of workstations and the ParSiFal T-Rack [3]. Work
is also currently in progress to port the prototype to the KSR-1 [4] virtual shared memory supercomputer and to produce a more fully featured implementation. The following sections describe this prototype, and some of our experiences and insights gained with it.
3.1 Basic Concepts At its most fundamental level an AVIARY system can be represented as a collection of communicating autonomous objects executing concurrently. All objects hide their implementation behind an exported interface and share a common communication medium which is used to pass messages between them. The word ‘object’ is used here in the sense of ‘object oriented programming’ – that is an object is a software construct incorporating both data and the code which is capable of modifying that data. Unfortunately the term object is rather useful generally, and it is necessary to introduce additional terminology to avoid the inevitable confusion that would otherwise ensue. In particular we are forced to adopt some other word for things that exist within the virtual world, and we use the term artifact for these. An artifact then is something which can be heard seen or felt, or otherwise experienced by the user, and is distinct from whatever software is behind its existence. The software objects which are directly responsible for artifacts are referred to as demons. A object oriented representation was chosen, instead of the shared database representation used by systems such as dVS [5] and DIVE [6], for several reasons: • Objects provide convenient units of parallelism which may be distributed among a collection of processors. • As the system provides a means for managing the code as well as simply the data (as in systems such as dVS and DIVE) there is the possibility of performing transparent load-balancing by moving objects between processors. • As the objects tend to represent something concrete such as an artifact or a device driver, they are conceptually easier to deal with for the application programmer - an important consideration in a large system with many thousands of objects. • As all updates to an object’s data are the responsibility of that object, access control can be implemented on a per class basis. This allows each object to control access to its data in the fashion best suited to it instead of relying on system services – thus access can be efficient whilst retaining the ability to protect sensitive data if required. Given this design approach, we may now consider the additional mechanisms the system needs to provide in order to support vr. These concern spatial management, support for the multiple-world model, the handling of devices and user interaction, and the mechanisms of distribution.
3.2 Spatial management Modeling spatial relationships is an important feature of a vr system, and manifests itself particularly in the treatment of collision detection. Although not required for all applications, when it is used collision detection places a large burden on the system. We have included this feature directly as it clearly has some impact on the design. In a distributed system one means of doing this would be for every object to compare its position with the other objects in a world and determine whether it had collided with any of
EDB1
EDB1
EDB1
EDB2
Figure 1: An EDB transfers part of its database to another EDB. them, although this would lead to prohibitive communications cost. The other extreme would be a single monolithic collision detection object to which all demons reported changes in their artifact’s position. The intensive use of such a centralised resource would however present a significant bottleneck. AVIARY uses a compromise and provides an object called the Environment Database (EDB), purely for the management of spatial extents. This records information on the bounding extents of every artifact in a world. When an artifact moves or changes its shape, its demon informs the EDB causing it to check for collisions with other artifacts. Initially one EDB is allocated per world. If this becomes overloaded, then another EDB is allocated to the world and the volume representing the world is then divided between them. The demons whose artifacts are in the volume handled by the new EDB are informed of the change and direct their updates towards that EDB. When artifacts cross boundaries between volumes handled by different EDBs then negotiation takes place and both EDBs check for a collision. This is illustrated in figure 1 which shows an EDB for a sparsely populated world. When the number of artifacts increases the database is split, and half is transferred to another EDB. The EDB’s are either assigned from a free pool or dynamically created. All of this activity is hidden in standard classes, and so is not visible to the application programmer. The EDB only detects collisions between artifacts and notifies the appropriate demons when this occurs. It is up to the demons concerned to resolve a collision. This separation of function allows different behavior when a collision occurs ranging from ignoring the collision so all artifacts appear to be insubstantial, preventing the artifacts from overlapping so that they appear to be solid, or more complex behavior such as causing the colliding artifacts to rebound and assume new trajectories. All this can be done without modifying the EDB. A novel capability provided by this would be to have orthogonal sets of EDB’s in a world, each concerned with a different class of spatial interaction. This would readily facilitate implementations such as the CSCW oriented spatial model of interaction [7]. Here each artifact is tested for collision not only on its visual representation, but separately on its field of attention. EDB’s can then monitor this separate phenomenon and inform the appropriate demons when collisions occur.
3.3 Input and Output The AVIARY model does not define the particular forms of data which the user can input, or the output presented to the user - it is possible, although not desirable, to run an AVIARY system with only textual input and output provided to the user, and indeed this was done while early development work was in progress. The AVIARY model does define input objects and output objects which manage device drivers and allow other objects to express interest in them. When the state of an input device changes the input object informs all who have expressed such an interest. Similarly, objects
Input Objects Head Tracker
User Demon Output Objects
Hand Tracker
Actions
Visual
Syntax Glove
User Model
Aural
Speech
Figure 2: A User Demon. may request an output object to monitor a particular location in a world and the output object will present a view (visual, aural, tactile, etc) of the environment at that point via the medium of the controlled device.
3.4 The User AVIARY models a user as an entity separate from the vr-system, logically equivalent to an application. There are no special cases required to represent a user, allowing code developed for other demons to be re-used. The only difference being that a user demon will interact with input and output objects whereas most demons typically, will not. The user demon will combine information from a number of input objects, and in general will need to use dialogue management in order to interpret a context sensitive meaning [8]. Implementations of user objects can range from an object that maps position changes reported by an input device into viewpoint changes for a renderer, to a more complex object which provides for interaction with the world and allows its behavior to be specified by a series of rules which the user may alter to customize its behavior. This notion of having different user representations is similar to the DIVE system in which a number of ‘vehicles’ are provided, each with differing capabilties. Figure 2 shows a user demon receiving input from four input objects (a head tracker, a hand tracker, a glove and speech input) and controlling visual and aural output objects.
3.5 Support for Multiple Virtual Worlds Each world is represented by an object referred to as the World Object. This is used to manage demons, objects providing resources, and the various laws and attributes associated with the world. Objects can send messages to the World Object which are then broadcast to other objects in that world. This allows objects to perform updates that effect the whole world without having to be aware of the IDs of every object associated with a world. As it is desirable to transport artifacts (for example, users) between different worlds, we
must separate the attributes that belong to an artifact from those which are a property of the world. For example in a world with a law of gravity, the property of mass is imposed on all artifacts within it. A users name however is purely a local property of the user artifact. The world object is responsible for administering the world specific attributes as artifacts enter or leave it. This mechanism is examined in more detail in [9].
3.6 Communication To facilitate distribution all AVIARY communication is by message passing. A central issue addressed was whether communication should be synchronous or asynchronous. Asynchronous communication is more efficient as the sender does not need to wait for the receiver, however it then becomes difficult to synchronize different parts of the system. Another approach would be to have two types of message send, but this would needlessly complicate the design. The approach chosen was to make message sends asynchronous, however if there is a reply to a message then the sender will block until the reply is received, thus synchronizing the sender and receiver. As messages may be sent to objects residing on different processors it would be awkward to pass parameters by reference all parameter passing is by value. To make it easier to return results multiple values to be returned in a reply to a message. A message consists of the ID of the receiver, the ID of the sender, a series of typed parameters, and a selector identifying the ‘type’ of the message. A selector is assigned whenever a new message type is created and uniquely identifies a particular combination of message name, parameters and return values. The use of selectors assigned in advance means that type information need not be transmitted with each message send, reducing the bandwidth required yet allowing new message types to be created at run time (for example when an application starts up and creates new objects). When a message send occurs the ID of the receiving object is checked to see if it is local to the sending process, if so the message send continues as a normal procedure call. If the receiver is remote then, then the message header and the parameters are encoded into a byte stream and sent to the remote object using the low level communications mechanism (UDP in the UNIX implementations).
3.7 Distributed Computation The AVIARY prototype allows both heavyweight objects and lightweight objects. A heavyweight object is represented by a process running under the host operating system. A lightweight object is similar to objects implemented by standard object oriented programming languages and is not capable of execution without some form of additional support. Lightweight objects have the advantage of lower overhead as they share resources and can be moved between processors in order to perform load balancing. Some objects must reside on particular processors in order to drive devices resident on that hardware whilst others can execute on any available processor. Therefore device drivers and the core set of standard objects are implemented as heavyweight objects, and other objects tend to be implemented as lightweight objects. The applications programming interface provides for the creation of both heavyweight and lightweight objects. In order to distribute computation between the available processors, AVIARY supports a class of objects known as Object Servers. An Object Server provides an environment for lightweight objects, managing their creation, destruction, communication, memory management and scheduling. By creating as many Object Servers as there are processors available the compute load can be spread evenly. As the Object Servers are aware of the code and data for
each of the objects they host, load balancing by object migration is possible. If a new Object Server is created, then some of the pool of objects can migrate onto it releaving other Object Servers of a portion of their load.
3.8 Name Serving and Service Brokering As it must be possible for objects to be created dynamically and to migrate, it must be possible to communicate between objects regardless of where the sender and receiver are situated. An object referred to as the Virtual Environment Manager (VEM) has the responsibilty of satisfying this name-serving function. The VEM also allocates identifiers for classes and messages as these can also be dynamically created. When an object creates a new class or message type it sends a descriptor to the VEM. If the class/message already exists then its identifier is returned, otherwise a new unique identifier is allocated and returned. In this way objects which understand the same messages can communicate, even if they were implemented completely independently as the messages types will have been assigned the same identifier. At a higher level, it is important to have some means of finding an object that can perform a required task in a given world, such as graphical output or collision detection. As objects are created and destroyed dynamically, new services, may be implemented. Static configuration files, as used in systems such as MR [10, 11] and dVS, are therefore not sufficient here. As these objects tend to be on a per-world basis, We introduce the World Manager object which performs the role of a simple broker of available services. It stores the capabilities of objects which have announced their availability for certain services (eg object servers, EDBs, input and output etc). The above discussion provides an overview of the problems AVIARY addresses, and some of the ideas behind the prototype implementation. In that implementation the central mechanisms necessary to evaluate the principle, and performance criteria in particular, have been constructed. In the remaining sections of the paper we discuss a particular application used to test the system and describe its implementation in AVIARY.
4
Example Application – Air Traffic Control
Air Traffic Control (ATC) [12] has some properties that make it interesting for VR. It is innately 3D, has clearly defined objects with dynamic behavior, and scope for interaction. What is more, the current interface of radar and telephones is generally regarded as sub-optimal [13]. The actual implementation used in testing AVIARY picks up on previous work which used radar data of plane movements over Heathrow airport, and rendered the corresponding flight paths as 3D tubes [14]. That work was an initial investigation into the ATC problem, and produces complex spaghetti like pictures of the skies over Heathrow. A more realistic VR solution to ATC would be to use tubes to represent designated highways in the sky through which planes are routed - thereby simplifying the management of the sky, and the visual representation. Such a model is in fact being seriously proposed as a future ATC discipline. The current version, with one flight tube per aeroplane is however useful for developing AVIARY as it has a far higher computational requirement. In this way the distribution mechanisms can be more thoroughly exercised. Aircraft have a position in 3D space, a direction and speed, flight code and an intended flight path. The landscape below the aircraft is depicted as a flat ‘map’ although some form for relief map to show hazards such as tall buildings near the airport or mountain ranges would also be useful.
Figure 3: Close up of aircraft and flight tube.
Figure 4: Display with a small number of aircraft. The aircraft are depicted as simple models with flight paths shown as envelopes in 3D space. The envelopes are like 3D tubes through which the aircraft fly. The laws of the world ensure that constraints for horizontal and vertical separation of aircraft are not violated - or signals potential violations. Figure3 shows a close up of an ’aircraft’ with a flight tube showing the volume of space the aircraft will move through in the near future. Each tube is modeled as a series of straight segments. A trail is also drawn showing the trajectory the plane followed to get to its present position. Figure 4 shows the display with a small number of aircraft each with a flight tube. Figure 5 shows the full display of approximately 200 planes and this is shown again in plan view in figure 6. In this implementation the user is not allowed to modify the flight tubes and
Figure 5: Full display, approc 200 planes, without flight tubes.
Figure 6: Looking striaght down at the aircraft.
Root World
Root World Object
Root Demon
Visual
Solid World
Root User
ATC Tube
ATC Landscape
Simple User
Figure 7: Classes implemented for the ATC Application thus re-route the aircraft. For a realistic ATC system there are other issues to be explored such as the question of interaction and cooperation between controllers, and additional information to display (eg the aircraft flight code and latitude, longitude and height).
4.1 Implementation In the current implementation the application will generate an animated display of aircraft over Heathrow airport using data captured from radar to generate the position and flight paths of the aircraft. Users have the option of specifying whether ‘flight tubes’ should be displayed representing the path the aircraft will take for a specified interval into the future. Figure 7 shows the classes of lightweight objects used in the ATC application. Of the nine classes, arranged as a ‘forest’ of three trees, only two are specific to the ATC application. The classes surrounded by boxes represent concrete classes for which it is sensible to create instances, the others are abstract and exist simply to provide basic functionality to classes that inherit from them. The landscape class (‘ATC landscape’) implements a demon that generates a static artifact representing the landscape. It is a very simple class that performs some initialization and generates a hard coded visual representation. The landscape class inherits from the class ‘root demon’ which provides the basic properties for the artifact specific part of a demon. It also inherits from the class ‘visual’ which provides the methods for handling the visual representation and code for handling communication with objects, such as visualizers, with an interest in the appearance of an artifact. The class ‘ATC tube’ implements the artifact specific part of the demon which represents an aircraft and its flight path. It generates an artifact with a visual representation of the aircraft, and an optional tube showing the future trajectory. This demon is much more complex than the landscape class and contains code for calculating the current position of the aircraft and handling interaction with other demons (including manipulation by a controller). The class ‘root user’ is an abstract class which provides a very basic functionality for communication with the input and output objects associated with the user. The class ‘simple user’ is a concrete class which adds a basic visual representation to the functionality provided by ‘root user’. The classes ‘root world’ and ‘solid world’ implement the world specific parts of a demon – instances of theses classes are World Attribute Objects. ‘root world’ provides a basic 3D world with no constraints of any kind. ‘solid world’ contains objects which are deemed to be
User demon
Inp. Object Servers Inp. Vis World Object VEM EDB
World Mgr
Message send
Figure 8: Objects in a running ATC Application solid and are prevented from overlapping. Finally, the instances of class ‘root world object’ are used to represent a world and provide functions for manipulating a world such as creating and destroying demons in that world and broadcasting to all objects associated with that world. Figure 8 shows the minimum collection of objects required to run the ATC application. Heavyweight objects are depicted as circles and lightweight objects as hexagons. Demons are drawn with a thicker line style. The input objects send messages to the user demon informing it of changes in the user’s position, the user demon then sends messages to the visualizer when the viewpoint or its visual representation changes. The visualizer informs the EDB that it is interested in the volume of space visible from the current viewpoint and the EDB will then inform it when any artifacts enter or leave that volume. When a demon modifies its artifacts position it informs both the EDB and any other objects – in this example the visualizer. The ATC application has provided a particularly useful vehicle for commissioning our prototype system. The large amount of dynamic behavior has exercised some extreme conditions of loading, although is unrepresentative of the common classes of VR application where a smaller proportion of objects are expected to be animate.
5
Conclusion
In this paper we have presented our own opinions on what the significant requirements are for a mature vr system, and discussed some of the conclusions to which these have led. The particular model that we have developed, AVIARY, attempts to encapsulate these ideas and provide a suitable large scale environment for co-exising diverse applications, which retains both flexibility and performance. The next major phase of the project will be to construct a full implementation. This will run on a number of platforms including a KSR-1 parallel supercomputer and using a Silicon
Graphics Crimson VGXT in our laboratory as a rendering engine to drive the head-mounted display. Such a setup will provide a useful vehicle for further research. As a final comment, VR is a very engaging and enjoyable field to be involved in. It brings together many areas of research, both within computer science, and beyond. We would like to acknowledge the assistance, inspiration and enthusiasm of our colleagues. In particular those at the Advanced Interfaces Group and John Churcher and Ray Macauley in the Department of Psychology at Manchester.
References [1] Roger Hubbold, Alan Murta, Adrian West, and Toby Howard. Design Issues for Virtual Reality Systems. In First Eurographics Workshop on Virtual Environments, September 1993. [2] A.J. West, T.L.J. Howard, R.J. Hubbold, A.D. Murta, D.N. Snowdon, and D.A. Butler. AVIARY - A Generic Virtual Reality Interface for Real Applications. In R.A. Earnshaw, M.A. Gigante, and H. Jones, editors, Virtual Reality Systems, chapter 15, pages 213–236. Academic Press, March 1993. [3] P.C. Capon, J.R. Gurd, and A.E. Knowles. ParSiFal: a parallel simulation facility. IEE Colloquium Digest 1986/91, pages 2/1–2/3, May 1986. [4] Steven Frank, Henry Burkhardt III, and Dr. James Rothnie. The KSR1: Bridging the Gap Between Shared Memory and MPPs. In Proceedings of Compcon’93, pages 285–294. IEEE, February 1993. [5] Division Ltd. dVS Technical Overview, Version 2.0.4, first edition, 1993. [6] Christer Carlsson and Olaf Hagsand. The multig distributed interactive virtual environment. In Lennart E. Fahlen and Kai-Mikael Jaa-Aro, editors, Proceedings of the 5th MultiG Workshop, 1993. [7] Steve Benford, Adrian Bullock, John Bowers, and Lennart Fahlen. A Spatial Model of Interaction in Large Virtual Environments, 1993. [8] Appino P.A., Lewis B.J., Koved L., Ling T.D., Rabenhorst D.A., and Codella C.F. An architecture for virtual worlds. Presence, 1(1):1–17, 1992. [9] David N. Snowdon, Adrian J. West, and Toby L.J. Howard. Towards the Next Generation of Human-Computer Interface. In Informatique’93: Interface to Real & Virtual Worlds, pages 399–408, March 1993. [10] Chris Shaw, Jiandong Liang, Mark Green, and Yunqi Sun. The decoupled simulation model for virtual reality systems. In CHI’92, pages 321–328. University of Alberta, 1992. [11] Mark Green and Dani Beaubien. MR Toolkit: Programmer’s Manual, November 1992. [12] David Graves. A Layman’s Guide to United Kingdom Air Traffic Control. Airlife Publishing Ltd, 1989. [13] John A. Hughes, David Randall, and Dan Shapiro. Faltering from ethnography to design. In CSCW‘92 Proceedings, pages 115–122. ACM, November 1992.
[14] B. K. C. Tam. Three-dimensional display for air traffic control. Master’s thesis, Department of Computer Science, The University of Manchester, March 1993.
