Model-Based Software In-the-Loop-Test of Autonomous Systems ¨ Andreas Bayha, Franziska Gruneis, Bernhard Sch¨atz fortiss GmbH, Germany bayha,grueneis,
[email protected] Keywords: In-the-loop, simulation-based testing, environment models, UAV
Abstract Software for autonomous systems is hard to test, given their complex reactions as well as safety-critical behavior. Here simulation-based testing of the control software using a simulation of the environment and the platform of the system supports verification without threat to system and environment. A model-based construction of the simulated environment enables an effective verification by systematically assessing reliability issues and robustness aspects while improving its efficiency by simplifying the reuse of virtual environment. We demonstrate such a systematic approach for the verification of the control software of an autonomous unmanned arial vehicle, using Simulink as simulation environment.
1.
INTRODUCTION
In the field of UAVs (unmanned aerial vehicles) embedded processors have lead to low-cost quadrocopter platforms. Increasingly autonomously operating UAVs are explored, e.g., for scientific observations or to explore unaccessible areas. The development of such an autonomous behavior requires the implementation of complex functionally. Correspondingly, testing such a complex functionality poses threats to both the equipment under development as well as the test environment and test personnel. To avoid such kind of damages during the development and test of the software and still support in-the-loop testing, virtual integration of the software can be used, i.e., the process of bringing a system into service without actually building the system by simulating it with all components of interest included. This allows to adequately and safely test the system under development on the software as well as on the hardware level; either by simulating hardware, equipment, and environment (’software-in-the-loop’, SiL), or by simulating only (part of) the equipment and the environment (’hardwarein-the-loop’, HiL).
1.1.
The FALTER Unit
The FALTER project [1] implements an autonomous UAV intended to be used indoors, e.g., to explore a factory site after an accident. Since the FALTER is designed for the use of indoor exploration, GPS-based navigation, commonly used in autopilot approaches for UAVs [2], cannot be used. The unit
operates on its own after it received the mission data from the mission control. It starts at a given point with a rough floor plan of the building under consideration, and tries to find a way to a given target area. After collecting situation data, it returns to the start point. Equipped with different sensors – including gyroscopes, accelerometers for position estimation, and ultrasonic sensors for collision avoidance – the unit autonomously controls the actuators – its propellors – to achieve the mission goal. The autonomous behavior includes collecting the sensor data, controlling the actuators via the execution of flight commands, as well as the complete online (re-)planning of the mission.
1.2.
Overview and Contribution
After presenting related work in Section 2., Section 3. gives an overview of the used approach to a systematic test using virtual integration with a virtual model of the platform as well as the environment of the FALTER unit. Section 4. demonstrates the implementation of a simulation, supporting the virtual Software-in-the-Loop test of the FALTER software. Section 5. closes by discussing resulting further directions of investigation. This contribution demonstrates how a model-based approach can be used to systematically test for reliability against platform defects as well as robustness of autonomous behavior. It furthermore demonstrates how to use model-based simulation frameworks – using the example of Simulink – to build reusable modules for both platform and environment models supporting a virtual in-the-loop test for software for autonomous UAVs. The focus of the presented approach is put on the test of autonomous functionalities rather than physically accurate details. While this focus limits the trustworthiness with respect to some aspects of reliability and robustness (e.g., quartz clock drifts, aeordynamics at floor level), it still allows a proof of concept in early stages of the development process (e.g., concerning a validation of the autonomous behavior or the evaluation concerning the right choice of a sensor system). While the level of detail obviously can be approved, the approach already substantially improves early quality assurance of the developed autonomous software.
2.
RELATED WORK
In the virtual integration and simulation of UAVs, the focus is generally put mainly on the modeling of the aerody-
namics. Virtual test beds like [3] or [4] use simulation engines provided for flight simulators, but provide no facilities to simulate sensors other than gyroscopes, accelerometers, altimeters, GPS, etc. The construction of simulation models of sensors like the ultrasonic sensors is generally only found in the field of sensor and/or filter development. Here, the sensor is modeled in very fine-grained detail to describe the signal propagation w.r.t. multiple reflections, or the Doppler effect, e.g., [5]. However, this level of detail is not suitable to efficiently support in-the-loop simulation of systems with multiple sensors and autonomous behavior in a complex environment like a building with several walls. In contrast to those approaches, the presented approach allows to perform functional testingof autonomic UAVs using range sensors via virtual integration, enabling efficient simulation of the system and complex environments. It extends [6] by providing mechanisms to systematically test functional safety aspects with respect to reliability and robustness, using modular scenarios.
3.
SYSTEMATICS OF BASED TESTING
SIMULATION
Embedded systems – with their high requirements concerning system safety – demand a systematic approach to testing. Testing is specifically challenging concerning the identification, execution, and managing of effective test cases in case of mobile autonomous systems. This is due to the high risk of an faulty autonomously acting system to damage the environment or itself during the testing procedure. Furthermore, the large number of required sensors and actors as well as the complex control software – both being influenced by environmental and technical effects – and the interaction between them adds to the difficulty. In total, while testing autonomous systems is more complex for the first reason, there are even more sources of failures for the second reason. Here, for early validation and verification, in-the-loop testing is a suitable approach. The systematic test of ’classical’, control-oriented embedded application application software is rather well-understood. Especially in the field of automotive application, SiL or HiL tests are a best practice. As these application have a small data space – unlike a UAV with its internal map of its environment – relevant test scenarios are well-understood and generally coverage-driven. Furthermore, in general, the focus is put on functional testing, abstracting from reliability of robustness aspects. Thus, here we demonstrate how to systematically address the issues of reliability against platform defects as well as robustness of autonomous behavior in SiL testing for UAVs. We are specifically interested in a modular testing approach, i.e., an approach that provides ’building blocks’ for test scenarios with respect to those two aspects as well as their combination into complex test cases. Our approach provides a dedicated Software-in-the-Loop (SIL) test environment offering model-
ing support for dynamic environments, sensors/actuators, and classes of test cases, as well as support for module and integration tests. For safety-critical systems, standards like the IEC 61508 demand to ensure fault-tolerance against platform defects. As autonomous systems rely on the use of sensors, the treatment of sensor defects is of special interest. These standards recommend to consider sensor failures like stuckat, oscillations, and drifts. For a systematic approach, we distinguish between sporadic and systematic error, since the later are especially hard to be dealt with. Section 4., presents standard failure models to be added to a base model of a sensor to produce the intended faulty behavior of the platform. Sporadic failures occur randomly with a certain probability. They do not result from, or relate to previous system states. Common examples out of this failure class are measurement errors (’noise’), inaccuracies or even a complete sensor system blackout (’stuck-at’). These sensor failures are generally masked or detected by standard means of sensor data plausibilization (e.g., filters). Systematic failure do not follow a random distribution but – like the sensor transfer function itself – depend on the current input and previous state. Representatives for them are sensor drifts or inaccurate calibration. These problems are generally hard to detect and mask, especially since they may be highly dependent on environmental conditions like temperature, humidity, or similar factors. An especially beneficial advantage of a SiL test is the possibility to directly observe and relate the effective and the measured parameters of the system like the altitude of a UAV over surface and the measured distance over surface. Combined with the convenient possibility to observe the internal data state of an autonomous system during simulation, this allows an easy analysis of the systems state estimation, e.g., an analysis of the quality of the sensor data fusion implementation. Here, the SIL approach allows to directly compare the systems internal state estimate to the real state. Figure 6 gives examples of such diagnostic data comparisons within the FALTER SiL framework. A specific functionality of autonomous systems is their capability to deal with a large range of changing environmental conditions. For sufficient robustness these systems need to cope with environmental conditions not easy to predict. In case of a UAV, the system, e.g., has to deal with completely/partially blocked ways or moving obstacles. Here, systematically testing the system w.r.t this form of robustness is a complex task implying to set up specific environmental situations during mission execution. As this setting up is generally laborious and an exact reproduction of test scenarios is often not possible, here specifically in-the-loop testing has substantial advantages over regular system test. Since testing the robustness of the autonomous behavior requires a large set of different test cases, it is important to flexibly combine all of
Case Specification and Management System.
Figure 1. Visualization of the FALTER SiL-Simulation.
the scenarios (i.e., complete/partial blocks, stationary/moving obstacles, etc) into complex test cases without having to modify the test environment. Furthermore, for a systematic approach the test framework must provide a mechanism to easily reproduce specific test cases, e.g., for regression tests.
4.
IMPLEMENTATION
As the tools for SiL simulations directly provide no solutions for the methods mentioned above, we implemented a framework providing these facilities in Matlab/Simulink. Figure 1 shows the graphical user interface for the simulation, showing information on model of the environment (e.g., upper left part), as well as the internal of the FALTER unit (e.g., lower left). We briefly introduce our implementation of the SiL-test framework, then summarize our experiences and results from testing the UAV software.
4.1.
The FALTER SiL-Test Framework
Figure 2 shows the overall architecture of the FALTER SiL-test framework, including Environment Model, Platform Model and Test Case Management, as implemented in form of a Simulink model. The colored blocks (e.g., Environment, Physical Model, FALTER Software) in the model represent major modules of the simulation model; FROM element s(i.e., right-hand block arrows, e.g., GAS, YAW, NICK) and corresponding GOTO elements (i.e., left-hand block arrows) are used as ‘syntactic’ sources and sinks in the signal routing between the blocks to obtain better readability. The following subsections explain this Simulink Model, first giving details on the Environment and Platform Model, and then on the Test
4.1.1. Environment Model The key concept for the environment model are bounded planes (i.e., rectangle surfaces). Planes p are parametrically defined by means of one vertex v and two edges e1 and e2 : ~p (s,t) = ~v + s · e~1 + t · e~2 with 0 ≤ s ≤ 1 and 0 ≤ t ≤ 1. In the case of modeling a wall, v represents a corner, e1 and e2 are the walls edges starting at that corner, with the width and hight of the wall being specified by the length of those edges. Planes are used to represent walls and objects (e.g., obstacles in the path of the UAV) in a boundary representation manner. While restricted in the possible representation of objects, it enables efficient mathematical operations required e.g. for the implementation of spatial range detection sensors. In practice they proved to be sufficient for the test cases needed in the FALTER setting. Finally such a vector based definition makes it possible to quite easily make obstacles move. Translations can be described e.g. by obtaining the vector ~v from a function of the simulation time or similar. And rotations by rotating one or both edges ei . In the Simulink model of figure 2, the static parts of the environment (i.e. the walls) are specified in the Environment block. As shown, the matrix containing the wall data is concatenated with the dynamic objects from the test case management system before it is passed to the sensor systems of the platform model. The platform model – which includes the modeling of the sensors – thus does not have to distinguish between static and dynamic objects. 4.1.2. Platform Model As the FALTER project focusses more on autonomous control than on smooth flight motions, the simulation uses a model of the platform simplifying real flight dynamics. The motion-mechanical aspects of the quadrocopter model are modeled via the block Physical Model of Figure 2 using a six-degrees-of-freedom (6DoF) model of the Simulink Aerospace Block Set with a point of mass and a transformation of commands send from the flight control software into momentums and forces applied on the 6DoF model. As illustrated in Figure 2, the FALTER unit uses four kinds of sensor systems: The Ultra Sonic Range Detectors, the Laser Scanner and the PMD Camera (a time-of-flight camera system), used for perception of obstacles, and the AHRS (attitude heading reference system) complementing the UAVs internal inertial measurement unit. The replication of inertial measurement units like accelerometers and gyroscopes - in our case AHRSs - is modeled by adding individual sporadic and systematic error models to the actual accelerations, rotation rates and angles from the physical model. However, to modeling complex sensor systems as range detectors, more elaborate models are needed. As mentioned
[Walls] Static Objects
[Obj] [dObj]
Environment
[GAS]
[YAW]
GAS
1
Ve (m/s)
[Ve]
Xe (m)
[Xe] [Angle]
Angle (rad) YAW
[DCM]
DCM
[Vb]
Vb (m/s) [NICK]
[ROLL]
NICK
ROLL
w (rad/s)
[w]
dw/dt
[dw]
[Obj]
Walls
[Xe]
Position
[Xe] [US]
Distances
[DCM]
DCM
[BoUS]
Blackouts
Ultra Sonic Range Detectors Walls
[Xe]
Position DCM
[BoP]
Blackout
angles_real
[GAS]
[BoL]
Blackout
[LSR]
[PMD]
gyro−rate_real
[US]
[Ab]
[Angle] [w]
[dObj]
Dynamic Objects
[BoL]
Laser_blackout
[BoP]
PMD_blackout
[BoUS]
US_blackout
gas acc
[YAW]
gier
[NICK]
nick
[ROLL]
roll
noise_acc
acc_noise
angles_measured
gyro
noise_gyro−rate
gyro_noise
angular rates laser dist
[LSR]
pmd cam
[PMD]
internal data
[DIAG]
Walls
PMD Camera
acc_measured
us dist
UAV DCM Scann Result (Distances)
[Obj]
Distances
[DCM]
acc_real
Physical Model
UAV Rotations
[DCM]
Laser Scanner
[Obj]
[Ab]
Ab (m/s^2)
UAV Position
[Angle]
FALTER Software
drift_acc
acc_drift
drift_gyro
gyro_drift
gyro−rate_measured Blackout
AHRS
AHRS_blackout
Test Case Management
Figure 2. Overall Architecture of Simulation Model: Environmental objects and physical model (orange), environment sensors (purple), IMU (blue), UAV software (green), test managment system (yellow). in Subsection 4.1.1., walls and obstacles are represented as bounded planes. Therefore the suitable method of modeling such sensors is by utilizing vectors to represent distance measurements to the walls/planes. Figure 3(a) gives a visualization of an ultra sonic range detector as an example, showing a sheaf of distance vectors originating in a sensor. The number of distance vectors used defined the accuracy of the simulated sensor; while more vectors obviously lead to more accurate model, they reduce the efficiency of the simulation. In order to yield a reusable implementation, we implemented this class of sensors as shown in Figure 3(b). The core component is the block Compute shortest distances, which implements the actual determination of distances to the environing walls by basically intersecting the sensor measurement vectors with the given walls1 . In order to minimize the computation effort, i.e. to increase the performance, the block ’Ignore too distant walls’ skips walls, that are to far to be detected at all. Especially in large environments, this can substantially speed up the simulation, since the actual computation of distances needs to deal with more than just intersection (e.g., when dealing with concealed walls). Additionally some post processing is performed, to provide the measurement results in a format corresponding to the specific sensor system. This modular implementation, separating the sensors measurement beam pattern, the distance computation, and the post processing, allows to reuse the core components for similar new sensor systems. Thus, e.g., the PMD Camera as well as the Laser Scanner use the same mathematical core, demonstrating their reusability and easy extensibility. 1 A detailed mathematical description of the computation process of distances can be found in [7].
Finally, the virtual integration of the control software is done in the block FALTER Software. The software is integrated ‘as-is’, i.e., directly and without modification as used in the actual implementation, using S-function blocks. Hereby every task is represented in a separate S-function, while the inter-task communication is model by regular intratask mechanisms, e.g., by passing pointers to corresponding data structures between the S-functions. 4.1.3. Test Case Specification and Management As described in Section 3., in order to enable systematical testing, an explicit mechnism to specify and manage different test cases is needed. Our approach is based on independently specifying scenarios for each kind of failure-tolerance or robustness aspect, out of which the tester can select and combine complex test cases. This mechanism provides the possibility to reuse scenarios for a single class of error sources within various test cases. An implementation of this is given in Figure 5. In that model, we distinguished between scenarios for sensor value noise and drift as well as sensor blackouts and dynamic obstacles. The example there for example combines the scenarios 2 for sensor noise and 1 for sensor blackout with sensor drift scenario 2 and obstacle scenario 3. To enable efficient modeling of such scenarios, we provide a dedicated library containing reusable blocks for generation of errors or obstacles. It contains, e.g., a block IMU Noise for generation of accelerator and gyroscope noise. The blocks are parameterized to specify the numerical limit to which the measurement errors – i.e., the deviation from the correct value – can be randomly distorted in every simulation step. There are similar, respectively parameterized blocks
Sensor Parameters (Relative Position and Alignment)
6
a
Sensor Parameters
5
z
Position
N
x
4
2 Position
3
1 Walls
2
Position
3 DCM
N Walls1
Walls Distance Limit
Sensor Prototype
Ignore too distant walls
y 2.5
2
1.5
1
0.5
0 y
−0.5
−1
−1.5
−2
−2.5
(a) Simulated Sensor Beam – Top and Side View
d
R
Compute shortest distances to walls for all sensors
b
[17x3]
Determine Range
Set−Up Result Matrix S−Function
Walls
1
0
Distances
DCM
Sensor prototype
uT
res
ret 1 bo Distances Set defect 4 sensors results Blackouts to zero
Reshape Transpose
(b) Structure of the Sensor Model.
Figure 3. Model of an Ultra Sonic Range Detector.
1
Select Noise Scenario
Scenario ID TimedStaticBox S. ID
2
gyro_noise
(App.: 11s) B−Rep
Select Blackout Scenario
Test Cases: 1 3
TimedStaticBox (App.: 15s) B−Rep
1 Obj.
Test Cases: 2 Cube TimedStaticBox S. ID
IMU Noise Scenario Selection
5 acc_noise 6 gyro_noise
BoAHRS
9 AHRS_blackout
BoLaser
2 Laser_blackout
BoPMD
3 PMD_blackout
Scenario ID
1
Box1
S. ID
acc_noise Scenario ID
(App.: 25s) B−Rep
BoUS
Select Drift Scenario 2 Select Obstacle Scenario 3
Acc
7 acc_drift
Gyro
8 gyro_drift
Scenario ID
Drift Scenario Selection Scneario ID
Obj.
Test Cases: 3 Small Box
(a) Obstacles can simply be con- (b) All parameters of a dynamic catenated. They are selected accord- object can be specified within ing to the specified scenario id. the blocks masks.
Figure 4. Test scenarios are specified via dedicated simulink blocks. This example shows the ’Obstacle Scenario Management’ (see figure 5) and how it manages several obstacle scenarios.
for sensor drifts, sensor system blackouts, or dynamic objects as well. Figure 4 gives as an example for scenario specification and management, the implementation of the Obstacle Scenario Mangement subsystem from the model of Figure 5. Each of the object blocks defines one cuboid-formed obstacle, for which one its dimension as well as spatial and temporal position can be specified. The blocks for dynamic objects basically output a boundary representation of the respective object by generating the required bounded planes (as introduced in Section 4.1.1.).
4.2.
Experiences and Results
The simulation approach was applied to the SiL-test of the FALTER software. The software is implemented in a layered architecture, with the bottom layer providing functionality for sensor fusion or actuator control, and the top layers providing functionalities for (re-)planing and plan execution. Due to this layered architecture, the SiL-test framework was used for two different testing purposes: For reliabilty tests of the complete
4 US_blackout
Sensor Blackout Scenario Selection
1 Dynamic Objects
Obstacle Scenario Management
Figure 5. Test Management subsystem with four categories of error sources selectable in combined test cases.
system; or for functional tests of parts of the system. In the first case, safety critical tests of functionalities like motion re-planning (invoked, e.g., if a chosen path is blocked by obstacles) were addressed. In the second case, module tests of the functionalities of the upper layers were tested by directly connecting the control software to the SiL-test framework. Thus, by effectively removing the bottom layer and providing the upper layers with motion-, position- and environmentdata directly from the physical model, the mission and path (re-)planning could be tested individually. Figure 6 shows results of tests performed to evaluate the fault tolerance of the FALTER unit with respect to platform defects in form of sensor data noise. The figure shows the altitude estimation (blue) and control (effectively achieved altitude of the unit – red) w.r.t. different levels of noise on the accelerator measurements. The estimation of altitude is basically performed by integration of the detected accelerations. The measurement/discretization/integration errors are periodically compensated by additionally measuring distance to ground via an ultra sonic range detector. Figure 6 visualizes the intended altitude (green), the actual altitude (red) and the software internal estimate (blue). The safety-related test cases test the altitude estimation and altitude control for faulttolerance against erroneous acceleration data and the com-
1
1
1
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0
0
0 0
0.5
1
1.5
2
2.5
3
3.5
0
0.5
1
1.5
2
2.5
3
3.5
0
0.5
1
1.5
2
2.5
3
3.5
(a) No noise on acceleration data (b) Normaly distributed noise of at (c) Normaly distributed noise of at (perfectly accurate sensor results). most 0.1m/s2 . most 0.2m/s2 .
Figure 6. Effects on altitude estimation and control by applying different levels of noise to the acceleration data.
munication between the tasks responsible for the ultra sonic range detection and altitude estimation. The first graph was taken from a test without any noise. The second one was performed with normal distributed noise of up to 0.1m/s2 , while the third one is the result of applying normally distributed noise as well, but with a maximum level of 0.2m/s2 to the acceleration data. The experiments with the SiL-test shows the usefulness of a model-based framework for the test of autonomous UAVs, both with respect to effectiveness and efficiency of setting up suitable tests for fault-tolerance as well as robustness aspects not easily possible in physical test scenarios. In the FALTER case, the framework helped to identify several critical design and implementation errors that would have been hard to detect in a physical test scenario (e.g., due to corner cases) and very likely would have caused substantial damage to the unit.
5.
CONCLUSION
We introduced a SiL-test framework for autonomous UAVs, using a modular, model-based description of the platform and the environment of the system under test and supporting a modular definition of test cases for the fault-tolerance and robustness of the control software. As a major asset, the framework allows to test the software ’as-is’, i.e., without any modifications or instrumentation for testing purposes. Overall, the framework proved to be an effective and efficient support for both module and system test scenarios. However, the framework still has some short-comings. While it supports debugging of the software (e.g., by externalizing and visualzing parts of the internal data state), advanced debugging (e.g., by stepping through the execution the software ) is not possible. The current framework also does not support automatic checking of test results. As the test verdict for autonomous systems is rather complex (i.e., successful completion of a mission), a complete automation of the test oracle is not yet provided. Currently, the framework does not provide a scripting language for automatic execution of test cases; however,
basically, here corresponding Matlab features can be used. While the framework successfully supports the early validation as well as the safety-specific test of the implementation of autonomous UAVs, the level of confidence gained substantially depends on the quality of the platform and environment models (e.g, voltage effects on processor speed, draft effects on motion). Therefore, while the model-based framework provides the necessary support for a systematic test, the domain-expertise remains the critical factor in the testing procedure.
REFERENCES [1] “FALTER project website,” 2010, http://www.fortiss.org/de/forschung/projekte/falter.html. [2] H. Chao, Y. Cao, and Y. Chen, “Autopilots for Small Unmanned Aerial Vehicles: A Survey,” International Journal of Control, Automation, and Systems, vol. 8, 2010. [3] E. N. Johnson, D. P. Schrage, J. Prasad, and G. J. Vachtsevanos, “UAV Flight Test Programs at Georgia Tech,” Georgia Institute of Technology, Tech. Rep., 2004. [4] R. D. Garcia, “Designing an Autonomous Helicopter Testbed: From Conception Through Implementation,” Ph.D. dissertation, University of South Florida, 2008. [5] D. Albuquerque, J. Vieira, S. Lopes, C. Bastos, and P. Ferreira, “Indoor ultrasonic simulator for moving objects using fractional delay filters,” 2011. [6] F. Mutter, S. Gareis, B. Sch¨atz, A. Bayha, F. Gr¨uneis, M. Kanis, and D. Koss, “Model-Driven In-the-Loop Validation: Simulation-Based Testing of UAV Software Using Virtual Environments,” in Engineering of ComputerBased Systems, ECBS 2011, 2011. [7] F. Mutter, “FALTER - Flight unit for Autonomous Location and Terrain Exploration: Virtual Integration,” Bachelor’s thesis, Technische Universit¨at M¨unchen, 2010.
