Simulation in development and testing of autonomous

Simulation in development and testing of autonomous vehicles Dr. Hans-Peter Schöner Daimler AG, Sindelfingen

Simulation in development and testing of autonomous vehicles

1 The Challenge of Testing Autonomous Vehicles On the first glance, autonomous vehicles seem to be just a simple continuation of the development of assistance systems which help the driver keeping the lane, holding the distance to other vehicles and avoiding accidents, with the vision of avoiding 80% of all accidents, because they are mainly caused by human errors. However, there is huge challenge with respect to the requirements on system performance and reliability for this step. As Herrtwich mentioned in [1], human drivers do quite well in driving a vehicle without accident, with statistically 7.5 million km between accidents on the German Autobahn network; if an assistance system helps a driver to avoid such accidents in (just for example) 9 out of 10 times, it does a good job by reducing the number of accidents by a factor of ten. However, autonomous vehicles with SAE level 3 or higher face the challenge to avoid or control any critical situation within a statistical distance of 75 million km between accidents, in order to achieve a similar performance compared to a level 2 (driver assisted) system. That includes many situations, which have traditionally been handled by human drivers easily, but might be difficult for automation. As Winner points out in [2], it would need test driving without accidents for hundreds of millions of kilometers to prove statistically, that the risk of autonomous vehicles is low enough to argue the safe operation; this kind of straight forward system verification would not lead to a practical implementation (because of time and cost issues) and furthermore would still leave gaps in an exhaustive safety argument. As in other industries with low risk requirements (aerospace, power supply systems, automated production systems, etc.), other methods have to be used to prove the safety performance of automated vehicles [3]. System design for such systems relies on redundant subsystems with well understood, logic-based safety arguments. But functional safety (based on robust and uncorrelated subsystems) is only one part of the design; functional completeness, i.e. the proof that any conceivable situation can be handled by the system, needs a very systematical system design and thorough verification and validation procedures. For this part of the development and testing task, simulation plays an important role [6]. This paper shall focus on the goals, the required tools and components, and the approaches of simulation for development and testing of autonomous vehicles.

2 Significant Tasks for Simulation First, it should be considered which type of feature can be tested successfully by means of simulation. In [3], five different categories of reasons for exposure to accidents have been identified:

Simulation in development and testing of autonomous vehicles 1.
Failure of components and hardware deficiencies 2.
Deficiencies in sensing road, traffic and environmental conditions 3.
Deficiencies in control algorithms (complex and difficult situations) 4.
Behavior dependent accidents (adequate behavior and rule compliance) 5.
Faulty driver and vehicle interaction (mode confusion and false commanding) Robustness of components and hardware (cat. 1) needs to be tested on benches and proving grounds; for the main testing platform type HIL (hardware in the loop test benches) some kind of environment simulation needs to be created. These tests however are in principle not different from state of the art testing of automotive components, but now with lower acceptable failure rates and higher safety integrity level requirements. Testing of environment sensors (cat. 2) depends very much on weather, environment status, visibility, and lighting conditions. Those are best tested in field tests, in order to find and verify the sensors’ performances under a wide range of conditions. Specific performance tests can be done in special test areas, with the goal to provide reference conditions, e.g. for certification tests. However, if adequate sensor models are available (describing the phenomenological effects of the sensors under different conditions) based on the results of field tests, sensor fusion algorithms can be tested in simulation. Testing the performance of control algorithms (cat. 3) in complex and difficult driving conditions is one of the key fields for SIL (software in the loop) simulations; in combination with actuators, HIL tests will complete the procedure, before tests on the proving ground verify the complete system.

Figure 1: Testing Platforms and the Importance of Simulation [3]

Simulation in development and testing of autonomous vehicles Compliance to rules and adequate behavior (cat. 4) in interaction with infrastructure and during special situational conditions (e.g. presence of emergency vehicles) can be partially verified by SIL simulation. However behavior verification on proving grounds and in field tests are necessary to complete these tests. Validation of driver and vehicle interaction (especially mode confusion and false commanding) are the core field for driving simulator studies (i.e. driver in the loop studies in a simulation environment). Further proving ground and field tests round off this part of testing. SIL test environments and driving simulators are two completely different hardware implementations, with different focus and challenges. However, the core of the simulation tasks, whether it is SIL or driving simulator studies, needs the simulation and precise control of the traffic environment for creating reproducible testing conditions.

Figure 2: Safety Assessment of Driving Situations

Let us have a further look at the type of traffic conditions needed. The risk from traffic accidents – as for any other risk type – derives from three factors (see fig. 2): severity S, controllability C (or its complement uncontrollability 1-C), and exposure E. Assuming an accident has already happened, the severity S of the accident plays the major role. In the automotive industry the severity risk factor S has been reduced by limiting the consequences of accidents, summarized as “passive safety”: for example introducing seat belts, airbags, crumble zones, sturdy passenger cabins and automatic rescue calls. The wide range of “active safety” collision avoidance systems help the driver to better control risky situations: early detection of dangerous situations as well as fast and appropriate reaction increases the controllability C to closer to 100% and thus reduces the

Simulation in development and testing of autonomous vehicles risk effect 1-C of uncontrollable accident-prone situations. Autonomous vehicles carry over many of the active safety systems to ensure a high level of controllability. Autonomous vehicles finally need to make sure, that the vehicle will not be exposed to dangerous situations which are not controllable – this way reducing the risk factor E for such situations; in conventional vehicles this is a significant task of the driver. It includes looking ahead and anticipating dangerous traffic situations, consequently driving carefully but also driving predictively for other traffic participants. In order to do this, the autonomous vehicle needs to have a correct knowledge of its own performance limits, in order to judge correctly which upcoming situations can be considered controllable or must be avoided. Figure 3 illustrates the pyramid of difficulty in traffic situations. Severe accidents are just the top of the pyramid: They are a subset of all accidents, which again result from critical situations. These again normally derive from complex situations which develop from easy standard driving situations. The difficulty in traffic situations is determined by two categories: ●
Independent parameters, like weather conditions (with influence on road surface and visibility), traffic type and intensity, and status of infrastructure (lane markings, construction zones), etc. ●
Controllable factors, like own vehicle speed, temporal headway to leading vehicles, precision of lateral lane keeping, etc. The exposure E to difficult traffic situations might be inadvertently increased by more challenging independent parameters, but in any case it can be reduced by changing the vehicles own behavior.

Figure 3: Autonomous Vehicle Response upon Changing Conditions

Simulation in development and testing of autonomous vehicles These normal tasks of the driver (control critical situations, judge traffic situations and derive adequate actions) have to be addressed by autonomous vehicle control functions. These functions need to be tested and evaluated during the development and testing process of autonomous vehicles. Simulation of traffic situations can help significantly to perform this task. Traffic situations must include not only critical (emergency) situations, but also situations which require anticipatory actions and rule compliance features.

3 Assessing Human Controllability and Interaction in Driving Simulators The use of driving simulators for the evaluation of human interaction (figure 4) with the vehicle has been thoroughly described in [4]. The significant differences with respect to traffic simulations in comparison to SIL testing is the need for a good visualization, motion rendering and sound reproduction – all these factors are rather irrelevant for SIL testing, but significant for human test persons. However, these points should not be considered here in more detail (see instead [4]). Important for both, driving simulator and SIL, is the precise simulation of behavior of other traffic participants. Small changes in speed, distance and lane keeping behavior are subtle indications for the future behavior of and for human drivers. Traffic situation analysis software uses these subtle indications as well – so for a good simulation this vehicle behavior has to be replicated by a traffic simulation model.

Figure 4: Dynamic Driving Simulator in Sindelfingen

Although level 3 (and higher) automation system should be able to perform any emergency manoeuver completely without driver interaction (since reaction times might be too short to wait for the help of the driver – if there is any), the possible interference of the driver with the vehicle action in critical situations has to be validated. On the other

Simulation in development and testing of autonomous vehicles hand, the controllability of critical situations by a human driver must be a guideline for the performance of an automated vehicle: the vehicle performance should be equivalent or better than the normal human performance in a similar situation. Figure 5 shows a typical result from a driving simulator analysis of driver performance. The rightmost line shows the statistical behavior of drivers without assistance system: if the key parameter TTC “time to collision” (when a possible collision object is visible for the first time) is large enough, all drivers can avoid an accident. Some drivers are less responsive than others, so there is a wide distribution of accident avoidance; below a certain TTC even the attentive and fastest driver is no longer able to avoid the accident. Less attentive drivers can be pushed to action by a warning tone, thus improving the controllability to complete accident avoidance to shorter TTC. Fully autonomous braking or steering action can still improve the controllability. But it should be noted, that even for a perfect autonomous driving system it is impossible to avoid accidents, if the collision object rushes into the sensing range without enough time for adequate reaction. Thus, also autonomous vehicles will face unavoidable accidents in given scenarios. The goal must be to avoid (if possible) the exposition to such scenarios.

Figure 5: Controllability of Traffic Accidents [4]

Traditional driving simulator experiments put the driver into the vehicle with the system to be tested. The interaction of the driver with his own vehicle has been normally the matter of investigation. For autonomous vehicles, it makes sense to investigate also the interaction of the automated vehicle with drivers in other vehicles. For example, the distinct and safe behavior of the automated vehicle in intersections and junctions is under investigation; for this purpose a human driver drives manually in his simulator cabin in a virtual environment, in conjunction with one or more automated vehicles

Simulation in development and testing of autonomous vehicles around him. Interaction patterns, misunderstandings and perceived politeness are under investigation in such driving simulator experiments.

4 Components of the Simulation Environment In order to simulate traffic situations in a development or testing environment, the following main components of a Virtual Driving Simulation Platform have to be provided (see figure 6).

Figure 6: Virtual Driving with Model Based Simulation

4.1 Road Model The base for any traffic simulation is the static road network, in which all simulated traffic participants are driving. State of the art is a road network description in the OpenDRIVE format, which provides all necessary formats to describe the road layout, including lanes, curbs and lane markings along the road, intersections, lane connectivity, traffic signs, traffic lights and other components of the road infrastructure. Other inventory of the virtual 3D world may also be included, in order to improve the perception of the environment around the roads, especially for human use in a driving simulator environment. Obstruction of view (by buildings or other objects) is also an important feature for many simulation tasks, which has to be provided by the road and static environment model. Road models can be purely synthetical virtual worlds; however in some cases it makes sense to reproduce a real road network by scanning the real world and transfer the data into OpenDRIVE format. Maps, which are to be used by automated vehicles to provide the look-ahead information for a trajectory planner and navigation, have a large functional overlap with the road model.

Simulation in development and testing of autonomous vehicles

4.2 Traffic Model Vehicles and other moving objects, especially pedestrians, bicycles, but also wild animals, are provided by the traffic model. This includes the 3D shape of all of these objects, together with their motion profiles and their individual behavior model. All traffic needs to have a source and a goal position as well as a planned trajectory (path and speed) on its way. Interaction of the moving objects with each other and with the vehicle under investigation is especially challenging and still a current topic of research activities. As mentioned above, the subtle behavior of traffic participants indicates their intention, their politeness or dominance, and thus is essential in simulating a scene in a virtual world. A typical traffic situation (a constellation of several traffic participants on a specific road network) can be set up with help of the traffic model. If all parameters are defined, this specific situation is called a traffic scenario.

4.3 Sensor Models Automated vehicles in the real world fulfill their function with the help of environment sensors: a large set of radar, camera, lidar and ultrasonic sensors. In the virtual simulation world, the behavior of these sensors has to be modelled. In the simplest case, a sensor model in a virtual world can provide a perfect picture of the surrounding world by extracting the typical sensor signals (recognized objects and their classification, relative position and speed, shape, etc.) from the complete knowledge of the virtual world; only their viewing sector and detection range as well as sight obstruction by other objects in the virtual world need to be considered. This type of sensor model is well suited to investigate the vehicle performance in a specific traffic situation, the challenge here being the complexity or the dynamics of the situation itself. Testing of vehicle performance under non-optimal sensing conditions needs sensor models which reflect typical sensor phenomena, like range reduction or increased signal noise under bad weather conditions, with incorrect, missing or uncomplete information in some sensed parameters, tracking errors, object loss and object fusion, etc. Such models, describing the behavior of different sensor technologies under similar conditions, are especially suited and needed for verifying the performance of sensor fusion algorithms, which has to provide fault tolerant redundancy of environment sensing. In general it does not make sense to implement physical models of the complex behavior of sensors in interaction with the properties of the features in the environment (reflections, damping, glare, etc.), because of computation time and effort. However, such sensor phenomena should be exemplarily modelled and considered by deliberately turning them on and off in order to test for robustness with respect to such behavior. Typical

Simulation in development and testing of autonomous vehicles sensor behavior patterns might be included by post-processed measurement samples, for example by a procedure dubbed “Replay2Simulation” [5]. For the scope of the simulation, the map provided for the autonomous vehicle can be considered like a sensor: with generally precise and unobstructed, but potentially outdated information about the road and infrastructure ahead. Using a map model equivalent to the sensor model with various levels of deterioration provides testing capabilities for unprecise, uncomplete or even missing and wrong map information.

4.4 Vehicle Model The vehicle model describes how the vehicles in the virtual world behave in detail, based on steering, acceleration and braking commands from the driver in the traffic model, or from input of the autonomous drive control software. For many traffic participants (but dependent on the goal of the simulation), a very simple model is good enough: limit the input signals to physically feasible values according to the vehicle type and just follow the desired value of the controller to shift and turn the center of gravity of the moving object. For exact evaluation of collision avoidance, or for any other dynamic driving manoeuver, and for extreme steering situations (like parking manoeuvers), a more complex vehicle dynamics model has to be implemented, including under- or oversteering, considering the tire-road-contact, and many nonlinearities. If a driving simulator has to provide the exact feeling to a human, a precise and validated vehicle dynamics model is essential.

4.5 Integration of the Autonomous Drive Control Software A simulation platform for testing autonomous drive control software has to include an interface for the integration of such software, as any other SIL or HIL implementation. In the real vehicle, the control software has an interface to a huge amount of input signals, and it produces many output signals. All those signals need to be supplied and evaluated by the vehicle model or the sensor models; in some cases separate virtual components need to be implemented, for example a virtual GPS sensor which provides adequate position information deducted from vehicle motion and references to the updated position of the vehicle with respect to the road network. Complexity and completeness of the interface is a challenge; it can only be managed if the interface is integral part of the software design process for the drive control software, including additional signal interfaces just for testability purposes.

Simulation in development and testing of autonomous vehicles

5 Relevant Traffic Scenarios Simulation of traffic scenarios is just one part of the entire verification and validation process; this is discussed – among many other possible citations – in [5] through [7]. The definition of challenging traffic scenarios, which need consideration in the development and testing of autonomous vehicles, is one of the key features of the PEGASUS project [5]. This project addresses the questions “How good is good enough?” and “How can we prove this?” for autonomous vehicle performance, amongst others by setting up a data base of relevant traffic scenarios. For highway driving, figure 7 describes how the three simple situations “following a preceding car”, “reaction on a cut-in vehicle” and “reaction on a cut-out manoeuver” can develop from trivial easy scenarios to extremely challenging difficult scenarios just by variation of a few parameters.

Figure 7: Function Development for Collision Avoidance Scenarios

The car following task turns from easy to difficult, if the preceding car brakes hard; in the extreme situation this car crashes without own braking action into a standing vehicle at the end of a traffic jam. The cut-in task has the challenge of reacting on a new object and reestablishing the safety distance to the new preceding vehicle; if the cutting-in vehicle presses itself into a too small gap and consecutively brakes hard, the situation turns into a difficult scenario. The cut-out task becomes challenging, if the cutting-out manoeuver suddenly gives sight to an unexpected object on the road or even to an oncoming vehicle in the lane. In all those challenging scenarios human drivers might not be able to handle the situation. Simulation of the traffic situation can elucidate, whether an autonomous driving software can handle the situation within a wide parameter range;

Simulation in development and testing of autonomous vehicles and it can be deducted, under which parameter configuration the situation turns uncontrollable (high risk factor 1-C) even for the fast and reliably acting autonomous vehicle. If this parameter set describes a very unlikely scenario, it might be considered as acceptable; if not, the driving control software has to make sure, that such scenarios will be avoided by early and anticipatory action (reduce exposure factor E close to zero). Autonomous city driving requires a lot of further testing situations and challenging scenarios, because of many more different traffic participants and action options.

6 The PEGASUS approach for Testing of Level 3 Systems Figure 8 summarizes the testing approach as current status of the PEGASUS project [7]. The test specification data base includes all scenarios which have to be considered for a sufficiently safe operation of autonomous vehicles. In a virtual proving ground SIL environment these scenarios are evaluated using the highly automated drive (HAD) software, in order to produce a “heat map” from the evaluation of all test runs. Performance measures provide the numerical values for the criticality assessment of the test cases. Uncritical cases might be considered as passed, above a certain criticality further proving ground tests might be necessary to verify the system performance. Highly critical cases show the need for further changes in the control software, or in some cases, lead to the exclusion of the scenario from the allowed parameter range for autonomous driving (driving with limited speed, or only under good weather conditions).

Figure 8: PEGASUS Approach for Testing of Level 3 Systems [7]

Simulation in development and testing of autonomous vehicles

7 Summary Simulation allows for checking the behavior of autonomous vehicles in a huge number of scenarios, environments, system configurations and driver characteristics. It does not make proving ground tests obsolete, but it can help focusing on the necessary proving ground tests to verify the simulation results and for certification measurements. Field tests will contribute with further validation insights, which derive from unexpected driving situations and retroactive effects under real driving conditions. Simulation plays an essential role in the development and testing of autonomous driving software; without simulation the huge number of tests and verification procedures could not be managed.

Bibliography 1.
Herrtwich, R.G.: The promises and pitfalls of vehicle automation; Automated Vehicles Symposium, San Francisco 2014 2.
Winner, H.; Wachenfeld, W.: Absicherung automatischen Fahrens; 6. FAS-Tagung München, München, 2013 3.
H.P. Schöner: Challenges and Approaches for Testing of Highly Automated Vehicles; 3rd CESA Automotive Electronics Congress, Paris 2014 4.
H.P. Schöner; B. Morys: Dynamic Driving Simulators; in: Handbook of Driver Assistance Systems, Edition: 3, http://www.springer.com/de/book/9783319123516, Chapter: 9, Springer Vieweg, Editors: Hermann Winner, Stephan Hakuli, Felix Lotz, Christina Singer, pp.177 – 198; 2016 5.
J. Mazzega, H.P. Schöner: Wie PEGASUS die Lücke im Bereich Testen und Freigabe von automatisierten Fahrzeugen schließt; Expertendialog Methodenentwicklung für FAS und AS, Essen 2016 6.
U. Steininger, H.P. Schöner, M. Schiementz: Requirements on tools for assessment and validation of assisted and automated driving systems; 7. Tagung Fahrerassistenz, München, Nov. 2015 7.
H. Schittenhelm: Testing of level-3 Systems – stepping through the current PEGASUS approach; PEGASUS Symposium, ika, RWTH Aachen, November 9, 2017 Special thanks to Horst Mock, Axel Blumenstock, Hans Grezlikowski, Fabian Römhild and the entire team“Virtual Driving” at Daimler’s Driving Simulation Center for working together on the development of simulation tools for autonomous vehicle design and testing. Picture sources: by Daimler AG or by the author, if not from cited bibliography.