The Software Architecture for Scenario Control in the Iowa ... - CiteSeerX

Proc. 4th Computer Generated Forces and Behavioral Representation Conference, Orlando, FL, May 4-6, 1994

The Software Architecture for Scenario Control in the Iowa Driving Simulator James Cremer Joseph Kearney Computer Science Department

Yiannis Papelis Richard Romano Center for Computer Aided Design

University of Iowa

Abstract

Iowa City, Iowa 52242

The Iowa Driving Simulator (IDS) is a high delity ground-vehicle simulator incorporating realtime multibody dynamics, high-resolution image generation, audio, six-degree-of-freedom motion, and force feedback in a large parallel and distributed software system. The variety of current and planned experiments using the IDS requires a exible system for specifying and executing complex scenarios | subject drivers should encounter realistic trac, pedestrians, stop lights, and variable weather conditions. This paper describes the architecture of the scenario control subsystem of the IDS focusing on the representation of the virtual world and the control of vehicle behavior.

1 Introduction Computer-generated vehicles provide real-time driving simulators a sense of dynamic realism that is essential for eective training and experimentation. On a typical roadway, a driver sees and interacts with large numbers of vehicles in close proximity. Plausible computer-generated trac must be programmatically controlled in real-time to create a believable simulation. The synthetic vehicles must obey the rules of road and model complex behaviors such as passing, merging, and navigating through intersections. In addition to modeling the normal driving behavior of individual vehicles, groups of vehicles must be coordinated to create critical situations and events required for studies of human driving performance. This paper focuses on the organization of the software for representing and manipulating the virtual world. We evaluate the successes and failures of the current system and report on the status of the second generation system currently under development.

2 Overview of the Iowa Driving Simulator

Recent technological advances in computational dynamics, parallel computing, and image generation are incorporated in the IDS to create a high- delity, ground vehicle simulator. The simulator serves as a testbed for the study of simulation capabilities and the advancement of simulation technology. The IDS provides a safe, virtual environment in which to study driving behavior; experiments can be conducted that are impossible to undertake in real environments because of potential risks to subjects. These experiments include the in uence of drugs, disease, and disabilities on driving performance; the eectiveness of computer-assisted driving aids such as collision warning devices and intelligent information systems; and the response of humans to critical events such as sudden, unexpected braking by leading vehicles. Driving simulation is also important for virtual prototyping of new vehicle designs and driver training, especially for heavy machinery. The IDS implements multibody vehicle dynamics in real time using high-speed parallel computers [8, 7]. Figure 1 presents a photograph of the motion platform and dome. An interchangeable cab is mounted inside the dome. IDS presently supports two fully instrumented cabs, a HMMWV and a Ford Taurus. High-resolution, textured graphics are projected onto a panoramic screen in front of the driver and a smaller screen behind the driver visible through rear view mirrors. The system is capable of presenting 4,000 textured polygons at a 60 Hz refresh rate. A photograph taken inside the cab is shown in Figure 2. The simulation software is partitioned into functional subsystems that communicate through shared, typed data structures called cells. Twelve distinct subsystems are spread across ve computers that incorporate thirty-six processors. The dynamics subsystem senses the steering wheel,

Page 1

Figure 1: A picture of the Iowa Driving Simulator facility for real-time operator-in-the-loop driving simulation

Figure 2: Scene from the inside of the Ford Taurus cab of the Iowa Driving Simulator

Page 2

accelerator, brake, and gear shift positions of the driver's vehicle, interrogates a high-resolution spatial database, and calculates the vehicle's motion in real time. The motion information is communicated to the visual, motion, instrument, and audio subsystems that control the driver's perception of the virtual environment. The speedometer and tachometer display simulated speed and engine rpm. An audio system reproduces engine, wind, and road sounds composed of processed digital samples of live audio. The steering wheel is loaded by a torque motor to provide feedback to the driver. This paper focuses on the scenario control subsystem that has responsibility for managing the behaviors of objects and changing conditions in the virtual driving environment. Section 3 describes the general organization of the IDS scenario control subsystem. Section 4 discusses representation and management of the virtual environment. Lastly, in Section 5, we present our approach to modeling the behavior of vehicles and other active entities.

3 The Organization of the Scenario Control Subsystem The scenario control subsystem manipulates simulated vehicles, regulates trac control devices, and sets the lighting and weather conditions. It must meet four, sometimes competing, requirements:  ecient representation of a complex, dynamic virtual environment  realistic modeling of basic driving behavior  controlled, repeatable presentation of critical events and situations  real-time performance In the current system, object behaviors are controlled by state machines that react to each other as well as to the motion of the operator's vehicle. A separate state machine is assigned to each computer-generated vehicle. The state machine is responsible for controlling the position of the vehicle at each step in the simulation. A road database maintains information about the state of the virtual environment that is used by the state machines to re state transitions and to set parameters in control laws that determine vehicle motions. All state machines are executed in sequence on a single CPU at 60Hz. At the end of each computation cycle, the road database is updated and vehicle locations are reported to the visual subsystem for display.

The current scenario control program can manage up to 40 vehicles on city streets, rural highways, and multi-lane urban freeways. By concentrating this trac in a region around the driver, we are able to give the appearance of densely populated roadways. As the driver travels along a road, the region in which trac is simulated moves with her. Plausible trac patterns emerge from the interactions of the autonomously controlled vehicles including trac clusters at intersections and motion ripples through chains of vehicles. We are now implementing the second generation scenario control subsystem to address problems encountered with the current system and to meet the growing demands for greater numbers of vehicles and more complex behaviors. In experiments with the current system, we found that when large numbers of vehicles were simulated, the computational requirements could overwhelm the available processing power and real-time scheduling deadlines would occasionally be missed. For example, as a consequence of a missed deadline, a scenario vehicle traveling at 60 MPH in front of the subject vehicle could remain the same place for two frames in a row. Human viewers are remarkably sensitive to very brief interruptions in the continuous motion of vehicle and to abrupt changes in heading. While it is critical to generate physically consistent motions at video rates, the decision-making processing that governs behavior can run at a much slower rate. Small delays in the response of the scenario vehicles to new circumstances are not noticeable. In fact, reactions that are too quick can appear mechanical. For comparison, the human drivers we are mimicing have reaction times on the order of one-half second. Our goal for the next generation system is to build a system that can accommodate the computational requirements of increasing complex behaviors and still guarantee physically plausible motion. Our solution is to partition vehicle control in two components: a behavior model that simulates the decisionmaking processes of the synthetic drivers and a physical model that simulates the dynamics of the vehicle. The behavior model will communicate control parameters (steering angle and accelerator position in the current design) to the physical model. Given the current state of the vehicle and the values of control parameters, the physical model will compute the location of the vehicle using a simple model of vehicle dynamics. The basic structure is pictured in Figure 3. By separating the vehicle control model into behavioral and physical models, we can run the dierent models on separate processors at rates appropriate to the functions they implement. The physical

Page 3

Behavioral Model

Behavioral Model

Behavioral Model

Dynamic Model

Dynamic Model

Dynamic Model

Scenario Database

Figure 3: Basic structure of scenario control subsystem. model can be eciently executed at a very high rate to ensure that motion is smooth and continuous. The behavior model can be executed at a slower rate to accommodate complex decision making processes. The data paths among the software components are summarized in Figure 3. The lower rate behavior model will pass control values to the higher rate physical model. The physical model will use these values for a series of time steps. In our present plans, the physical models will operate in open-loop mode. Because they will contain no feedback loops, they will not exchange information with the scenario database on every iteration. In contrast, the behavior model must sense the virtual world to determine the locations of nearby vehicles, the path of the road, and the state of trac lights. This information will be acquired through queries to the scenario database. To avoid order dependencies and inconsistent views of the environment, the scenario database will be updated at the same rate as the behavior models. We require that responses returned by the database not be in uenced by the timing and order in which queries are posed during a time step and that dierent state machines receive the same response to the same query during the same time step. At the end of a step of the behavior models, their control values will be passed on to the corresponding physical models. At the same time, the virtual environment will be updated based on the current state of the physical model.

4 Representation of the virtual world Simulator subsystems must have knowledge about the dynamic virtual environment including road geometry, roadway features such as intersections, stop lights and speed limits, the locations and velocities of vehicles, and environmental conditions. Simula-

tor subsystems have individualized needs for information about virtual environment. For example, the dynamics subsystem requires much higher resolution information about the terrain than does the visuals subsystem. The design of the virtual environment database is complicated even further by the requirement that the overall system must execute in realtime. Our solution is to represent the virtual environment in multiple ways and at multiple levels of detail to support the diering demands of various subsystems. A virtual environment such as the one illustrated in Figure 4 is represented in three separate, but correlated databases[4]. The visual database represents roads, terrain, foliage, and cultural features in multiple levels of detail by smooth-shaded textured polygons. The representation is customized for display on dedicated image generation systems. The vehicle dynamics surface database represents the geometry of all drivable surfaces in a variable resolution, piecewise uniform grid. In addition to surface shape, the surface database characterizes the surface type, roughness, friction, and soil properties. The surface database supports overlapping terrain, a feature necessary to model bridges where the roads over and under the bridge are both drivable. The scenario database poses the greatest challenges because of the variety and amount of information it must store, because of the the complexity of the queries in must answer, and because it is dynamic. The scenario database represents the geometric and logical properties of roads, and maintains dynamic information about the state of active entities. Road geometry is represented by oriented three-dimensional space curves. Logical properties of roads | speed limits, passing zones, trac signs, etc. | are associated with road segments. Information about road geometry is needed to guide vehicles through the virtual environment. For example, a simulated vehicle must make frequent, ne-grained adjustments in steering angle to track the path of a lane. In the IDS, the roads are represented by a combination of hermite cubic splines and linear interpolators. This dual representation permits us to eectively and eciently support a variety of controllers. For example, kinematic controllers that require smooth and continuous curves can sample the spline representation and controllers that incorporate vehicle dynamics can use coarser, linear interpolants. The logical properties of roads are needed to intelligently control vehicle behaviors. In the next generation system, the topology of the road network will be represented as a directed graph, where edges are roads and nodes are intersections. The direction of an

Page 4

East Canyon 35 mph

35 mph

Ca

ny

on

35 mph

55 mph 55 mph INTERSTATE 74

55 mph

35 mph

Bartelme

25 mph (in town) loop 35 mph

35 mph 35 mph ROUTE 171

55 mph IINTERSTATE 474

55 mph

Valley Park

55 mph INTERSTATE 74

ROUTE 10 35 mph

ROUTE 14 35 mph

Figure 4: Map of the Orchids2 database of the Iowa Driving Simulator

Page 5

N2 N2

N1 N1 N3 N4

N5 N4

N3

N5 N6

N7

N8

N6

N7

N8

Figure 5: Graph representative of a typical road network. edge will be used as a reference to specify the ow of trac along the lanes of the road segment. Attributes will be attached to edges de ning the trac regulations and the locations and identity of trac control devices. Figure 5 illustrates a graph representative of a typical road network. Queries will support requests for information such as:  the distance to the next intersection  trac signs within an interval of the road  lane markings that restrict passing  the geometry of an intersection Vehicles will use this information to make decisions about lane changes, to select a route when approaching an intersection, and to determine a safe speed at which to take a turn. In addition to the static information associated with each segment, the database is responsible for maintaining information about all vehicles and other dynamic entities in the virtual environment. Vehicle behavior models must be able to determine the location and movements of nearby vehicles, including the operator's vehicle. This information is essential for passing and collision avoidance. To facilitate intelligent interactions among vehicles, the database will support queries such as:  what vehicle is ahead of me in my lane

what is the distance between me and another vehicle  what vehicles are approaching me in the opposing lane The question of lane membership is complicated because vehicles often occupy portions of more than one lane. For example, as a vehicle changes lanes or crosses intersections, it lies, for a time, in multiple lanes. In the current system, vehicles are assigned to the single lane they are tracking. An important advance in the next generation database will be to separate lane tracking from lane occupancy, thereby allowing vehicles to reside in multiple lanes at the same time. 

5 Behavior modeling The ultimate goal of scenario control is present a convincing dynamic environment to the subject driver. In this section, we rst present the techniques used in IDS to model the basic behavior of vehicles and trac control devices. Using these techniques, we can create reasonably convincing trac composed of autonomous reactive vehicles. We introduce a notion of hierarchical, concurrent state machines to model basic behavior. Unfortunately, trac composed of such vehicles does not generally meet the needs of experimenters.

Page 6

Investigations of factors in uencing trac safety and training studies require special trac situations { the driver should encounter trac jams, drunken or sleepy drivers, slow or speeding vehicles, vehicles that run stop lights or stop signs, and so on. We brie y describe tools that allow us to direct and choreograph basic vehicle behavior to produce scenarios meeting experimental needs.

Free Driving

Following

Hazard Avoidance

Constraint Resolution

5.1 Basic behavior modeling

The existing IDS scenario control system models vehicle behavior using a complex, one-level state machine. The system is able to generate trac that has a natural, reactive feel in which trac phenomena such as jams and clustering emerge. However, as discussed in [2] the model is dicult to modify and debug; its size and complexity grows rapidly, it lacks abstract and encapsulation, and it does not handle some common situations well. The next generation of scenario control will incorporate a new framework for modeling object behaviors based on hierarchical, concurrent state machines. The step from single-level at state machines to hierarchical state machines is straightforward and natural. It provides a mechanism for abstraction by allowing states and transitions representing coherent activities to be grouped. For example, turning behavior at intersections is a fairly complex activity modeled using a number of states and transitions. The ability to encapsulate this logic within a single \turn at intersection" state makes programming, modifying, and debugging vehicle behavior much more tractable. The concurrency of our state machines addresses problems of attending to multiple constraints or goals during driving. Vehicles must obey speed limits, stop at red signal lights, avoid collisions with nearby traf c, and be alert to hazards in the roadway. At each instant, a driver must integrate all the relevant information and adjust the accelerator and steering wheel to best accommodate all of the appropriate constraints. Consider the example shown in Figure 6. Here, Follower is trailing Leader as it approaches a railroad crossing. Not much further down the road there is an intersection with stop signs. Follower has a number of things to worry about: 

it needs to attend to its basic driving behavior (Free Driving state machine), so that it stays in its lane, obeys the speed limit, and stops at stop signs and trac signals



it needs to monitor the vehicle ahead (Following

STOP

Follower

Leader

STOP

Figure 6: Concurrent, hierarchical state machines and constraint resolution state machine), in order to follow at an appropriate distance it needs to watch for hazards (hazard avoidance state machine), such as other cars encroaching in its lane, pedestrians, and, in this case, a train Suppose that Leader decides to continue across the tracks just in front of the train. If Follower singlemindedly follows Leader, it will be hit by the train. If we choose a simple single-level, non-concurrent state machine approach with a Follow state that takes care of following behavior, we will quickly get bogged down by special-case handling of such situations. The Follow state (and many others!) must worry about myriad other aspects of driving. In contrast, a vehicle controlled by concurrent state machines can simultaneously attend to all of the factors important for safe driving behavior. The state machine sketched in Figure 6 contains three concurrent state machines. Each machine encodes a behavior and may include states that have concurrent substate machines. As the leader approaches the railroad crossing, all three machines would be active. At every simulation step, the Free Driving, Following, and Hazard Avoidance state machines indepen

Page 7

dently produce recommendations about the vehicle acceleration. If Follower is threatened by a possible collision with the train, then the Hazard Avoidance state machine will recommend breaking to stop before the track. A most conservative constraint resolution mechanism would accept this recommendation over the recommendation to either maintain the speed limit or follow Leader. Basic behavior modeling using hierarchical, concurrent state machines is described in more detail in [2, 3].

5.2 Scenario authoring

Beyond basic behavior modeling, we need to provide a natural framework for designing and building complex scenarios that meet experimental needs while maintaining the diversity and realism of autonomous, reactive behavior. Related IDS papers[3, 6, 1, 5, 4], describe some of the tools that allow experimenters to rapidly create, modify, and validate the layout and appearance of the virtual environment. Here, we outline some of those tools and new ideas that we are currently developing and incorporating into the next generation of scenario control. In order to specify controlled experimental situations and event sequences that blend smoothly with the background of autonomous, reactive behaviors, we need tools to coordinate and direct the behaviors of individual vehicles as well as groups of vehicles. We are developing notions of behavior controllers, higherlevel state machines that may be used to exert control over a scenario without causing the sti, regimented, scripted feel we wish to avoid. Behavior controllers are like daemon processes that monitor situations and react accordingly. Consider an situation in which an experimenter wants to keep a certain section of road clear of vehicles. This can be accomplished by placing a behavior controller called a beacon at an intersection and having the beacon instruct vehicles to turn left (or whatever is necessary to keep them o the relevant stretch of road). Another example of beacon use would be to cause a hazard situation in which a scenario vehicle runs a red light in front the subject vehicle as enters an intersection. For a number of reasons, it is not a good idea to choose the hazard vehicle o-line: Subject drivers will arrive at the intersection at dierent times; if a predetermined hazard vehicle is sitting at the intersection waiting for the subject driver arrival, detection of the \set-up" is reasonably likely. Instead, a beacon may be used to watch for the subject vehicle and conscribe an appropriate scenario vehicle to run

the light. The beacon may help set appropriate conditions for the event by sending appropriate \clear the way" messages to other scenario vehicles. In this way, behavior controllers can orchestrate complex scenario situations that retain signi cant reactive components and avoid the staleness of scripting.

5.3 Authoring tools

Basic behavior modeling and scenario choreography is currently accomplished by programming in C. We are implementing graphical authoring interfaces that allow experimenters to create scenarios at a more appropriate level of abstraction. The interface will allow experimenters to de ne new basic behaviors (e.g. police cars, re engines, accident prone drivers), control object behaviors using beacons and similar mechanisms, and coordinate activities by specifying sequences of critical events that are to take place in an experiment.

6 Summary The wide variety of experiments currently being carried out on the Iowa Driving Simulator places heavy demands on the scenario control subsystem. In the coming years, we expect a signi cant increase in the number of similar ground-vehicle simulators and other simulation-based virtual environments. With this growth, the variety of experiments that will be proposed for such systems will increase dramatically. This paper has outlined the second generation of scenario control subsystem in the Iowa Driving Simulator. We (1) identi ed some of the alternatives and trade-os involved in designing a scenario control system, (2) described our general approach to virtual environment representation and database issues, (3) discussed the in uence of hard real-time constraints, and (4) described our work addressing the diculties of modeling vehicle behavior in a way that yields natural, convincing trac that can, at the same, be choreographed or otherwise controlled to meet experimental requirements.

References [1] Michael Bartelme, Michael Booth, James Cremer, Douglas Evans, Joseph Kearney, and Richard Romano. Experiment authoring for virtual driving environments. In Proceedings of the 1st Eurographics Workshop on Virtual Environments, pages 1{15, Barcelona, September 1993.

Page 8

[2] Michael Booth, James Cremer, and Joseph Kearney. Scenario control for real-time driving simulation. In Proceedings of the 4th Eurographics Workshop on Animation and Simulation, pages 1{15, Barcelona, September 1993. [3] James F. Cremer and Joseph K. Kearney. Scenario authoring for virtual environments. In Proceedings of the IMAGE VI Conference, June 1994. [4] Douglas Evans. Correlated database generation for driving simulators. In Proceedings of the IMAGE IV Conference, pages 353{361, July 1992. [5] Douglas Evans. Ground vehicle database modeling. In Real-time Systems Conference, Paris, France, January 1994. [6] Douglas Evans and Michael Bartelme. Essential methods and emerging concepts in developing virtual driving environments. In Proceedings of IMAGE VI Conference, July 1994. [7] J. G. Kuhl and Y. E. Papelis. A real-time software architecture for an operator-in-the-loop simulator. In Workshop on Parallel and Distributed Real-Time Systems, pages 117{126, Newport Beach, CA, April 1993. [8] James W. Stoner, Edward J. Haug, Kevin S. Berbaum, Douglas F. Evans, Jon G. Kuhl, Julia C. Lenel, John F. McAreavy, and Fuh-Feng Tsai. Introduction to the iowa driving simulator and simulation research program. Center for Simulation and Design Optimization of Mechanical Systems Technical Report R-86, University of Iowa, August 1990.

7 Biographies

James F. Cremer is an Assistant Professor of Com-

puter Science at the University of Iowa. He received his Ph.D. from Cornell University in 1989 as part John Hopcroft's modeling and simulation group. He remained at Cornell as a Research Associate from 1989{1992 and led research and development of the Newton dynamics simulator and, with Rick Palmer, initiated the SimLab project. Since moving to Iowa in 1992, he has been active in research in environments for constructing simulators (SimLab and related projects), scenario control for real-time vehicle simulation (with Iowa's Center for Computer-Aided Design and the Iowa Driving Simulator), and techniques for eciently handling constraint changes related to contact, collision, and con-

trol in simulation-based virtual environments (project Isaac with George Vanecek from Purdue University).

Joseph K. Kearney is an Associate Professor and Chair of Computer Science at the University of Iowa, Iowa City. He rst joined the Iowa faculty in 1983. Kearney spent a year as a Visiting Scientist in the Computer Science Department at Cornell University, Ithaca, NY in 1986 and 1987. Kearney received the B.A. degree in psychology from the University of Minnesota in 1975, the M.A. in psychology from the University of Texas 1979, and M.S. and Ph.D. degrees in computer science from the University of Minnesota in 1981 and 1983. His research interests include animation, control, computer vision, mechanical simulation, robot programming, and virtual environments. Yiannis E. Papelis is a Research and Development Project Leader at the Center for Computer Aided Design at the University of Iowa, and a Visiting Lecturer at the Department of Computer Science. Papelis has been with the Center since 1990 and was one of the original members of the team that designed and built the Iowa Driving Simulator. Papelis received a B.S. in Engineering from Southern Illinois University at Carbondale in 1988, a M.S.E.E from Purdue University in 1989, and a Ph.D. in Electrical and Computer Engineering from the University of Iowa in 1993. His research interests include realtime systems, virtual environments and operator-inthe-loop simulator technology. Richard A. Romano is the Manager of Simulator Development at the University of Iowa's Center for Computer Aided Design. Romano has spent the last six years working the the area of operator in the loop simulation. He received his B.A.Sc. in engineering science from the University of Toronto in 1988, and a M.A.Sc. in aerospace engineering also from the Univerity of Toronto, in 1990. His research interests include motion washout algorithms, vehicle dynamics, man-machine interfaces, and human factors.

Page 9