Data-Driven Development

Data-Driven Development, A Complementing Approach for Automotive Systems Engineering Johannes Bach, Jacob Langner, Stefan Otten, Eric Sax

Marc Holz¨apfel

FZI Research Center for Information Technology Email: {bach, langner, otten, sax}@fzi.de

Dr. Ing. hc. F. Porsche AG Email: [email protected]

Abstract—Established methods and processes in the field of Automotive Systems Engineering (ASE) are challenged by the rising complexity of current features. Expanding system boundaries, tighter interconnections of functional elements, increasingly complex algorithms and an ever growing operational domain generate a multitude of different scenarios that require consideration during specification, design, implementation and testing. This paper reflects the current practice on the example of the Automotive SPICE process reference for system and software development in the automotive domain. It then contemplates on opportunities of consistent usage of recorded vehicle data throughout all phases of automotive development. Our concept of data-driven development is not intended to replace the current practice but to complement it. A summary of our previous work demonstrates the practicability of the concept on the basis of the development of a Predictive Cruise Control (PCC) feature. The contribution concludes with a scalable concept for the large scale application of data-driven development in ASE.

I. I NTRODUCTION The current trends automation, electrification and connectivity [1] affect the well established development processes in the automotive industry. Established development principles and methods have to be adopted to these trends in order to meet the new and changing requirements. New features, reaching from advanced sensor systems and digital maps to back-end services, impact the entire development process from system specification to Verification and Validation (V&V). The challenges of growing numbers, size, complexity and interconnections of automotive features [2] are magnified by shorter time to market demands and increasingly complex algorithms, which are facilitated by improved hardware performance. Figure 1 depicts current major challenges concerning processes, methods and tools for ASE. The introduction of digital maps and the advancement in sensor technology expands the vehicle’s environmental perception [3]. Newly emerging backend services further enhance this sphere of perception with real time traffic flow data and other information, magnifying the amount of data and types of information available in a modern vehicle. This leads to complex dependencies between data channels and fuzzy system borders, which challenge current specification techniques. Besides the amount of signals and sensors, also the total number of features in a vehicle and the algorithmic complexity is rising. Moreover, newly build features are often of a highlevel nature and utilize other lower-level function elements for sensing and actuation. This in turn increases the complexity and connectivity between features far beyond the border

10%

70

Environmental Perception

Networking & Dependencies

Algorithmic Complexity

Multitude of Scenarios

Fig. 1: Current challenges concerning processes, methods and tools for ASE of a single Electronic Control Unit (ECU). The resulting dependencies between various features require new ways of coordination and communication between the responsible development teams, divisions and companies, since descriptive and conceptual modeling are no longer sufficient to carry all relevant information. To further add to the pool of current challenges, with the increase of algorithmic complexity, also the complexity and extend of required V&V activities for assisting and automating features is growing. Not only the functionality of each element on its own but the complete functional chain has to be tested. A multitude of scenarios and situations, for example varying weather conditions, have to be considered for sufficient test coverage [4]. Worldwide distribution demands proof for compliance with various country specific rules and regulations. The sheer number of possible test scenarios for worldwide valid V&V of assisting or automating features constitutes an enormous problem for development teams. In this contribution, we propose a data-driven development approach to complement the existing methods and strategies for ASE. The approach describes possible benefits and use cases for application of recorded real world driving data in every phase of the development process. It represents an opportunity to overcome the described challenges. The paper is structured as follows. In chapter II the state of the art in ASE is introduced and open challenges are idetified. In section III an outline of approaches is presented, that are summarized under the term data-driven development. The application of the data-driven development is explained by means of our prototype implementation for the development of a PCC feature in section IV. In the subsequent section V, the

System

necessity for scalability is discussed and an extension concept of the prototype implementation is presented. The contribution concludes in section VI with a conclusion and an outlook on future work.

Requirements Elicitation Requirements Analysis Architectural Design

The field of systems engineering encompasses methods, approaches and tools for ”the engineering of complex systems” [5]. In the automotive domain, a widely accepted approach to structure the necessary development activities is the V-Model [6]. The process reference Automotive SPICE1 tailors this approach according to the specific demands and constraints of ASE and provides a guideline for process implementations [7]. It encourages the partitioning of the vehicle, seen as a system, into smaller and better manageable elements. Figure 2 depicts the Automotive SPICE system and software engineering processes and an exemplary partitioning into system, subsystem, component and unit levels, using the example of a PCC feature. While the V-model quickly implies a strict linear development approach, implemented processes often realize an iterative procedure on system and subsystem levels and incremental approaches on component and unit levels. The development begins at system level with the elicitation of the system requirements. The aim of this process is to establish a clear representation of all relevant needs and requirements of the involved stakeholders. The applied methods and techniques include the analysis based on scenarios, the identification of use and misuse cases and more formal techniques such as FODA (Feature Oriented Domain Analysis) [8]. The gathered stakeholder requirements are subsequently analyzed to derive the more formal and technical system requirements, which facilitate the system design process. The requirements analysis can be supported by consistent usage of modeling, visualization and templating techniques. An exemplary approach is the Systems Modeling Language (SysML) [9] with its activity and sequence diagrams, to cite a few. A full traceability and documentation of the system requirements specified strongly supports all other activities. Methods for V&V of requirements range from manual reviews and throwaway prototypes (e.g. paper mock-ups for user interfaces) to more elaborate simulation based approaches and functional prototypes. During the design, a system architecture is developed, that fulfills the specified requirements. The system, including all features, is broken down into individual subsystems, that implement interrelated functional elements of a specific feature or feature group [10]. Based on the system architectural design, the development of mechanical, hardware and software components is split up. In this contribution, we focus on the software part of the development. The software requirements are derived from the results of the system phases. Software specific details complement the functional description of the system level requirements on the component level. The methods applied for this activity pretty 1 Automotive

Software Process Improvement and Capability Determination

Software

II. S TATE - OF - THE -A RT Requirements Analysis Architectural Design Unit Construction

System

Subsystem

Qualification Integration

Component Longitudinal Control

Unit

Qualification Integration

calcDist() float calcDist(const float &dx, const float &dy) { float dist = 0.0f; dist = sqrt(pow(dx, 2) + pow(dy, 2)); return dist; }

Verification

Fig. 2: Outline of Automotive SPICE development phases and partitioning of a Cruise Control feature on system, subsystem, component and unit levels much correspond to the methods of the system requirements analysis. A frequent industry-wide occurrence are well honed prototypes, which serve as an example for the detailed specification of requirements. The software architectural design splits the software component into individual units, which fulfill specific parts of the requirements. It describes the interactions of all elements and the necessary interfaces. Modeling and simulation techniques support the software design with opportunities to evaluate the overall product in early stages [11]. The unit construction represents the implementation of the specified software units according to the detailed design and the allocated requirements. During unit construction various tools for debugging and automated assessments, such as static code analysis, support the developers. To provide them with a possibility to evaluate the interaction of the implemented software unit with its adjacent units, functional mock-ups and simulation based techniques are utilized. The right leg of the V-model is dedicated to Verification and Validation of the implemented and integrated items [12]. The goals of V&V differ on each level. Unit tests aim at verifying the correct functionality of each software unit, whereas the goal of software component tests is to verify and validate the interaction between and integration of software units. On subsystem and system level the overall integration of the implemented software in the ECU hardware and the correct interaction with the mechanical elements need to be proven. This includes the provision of evidence for the requirement consistent realization of the chain of effects of the distinct features. Varying goals require different testing methods on each abstraction layer. Unit verification is usually performed with small software testing blocks or by manual code reviews. For integration tests on the component level, testers utilize mock-ups and simulation based approaches, such as Softwarein-the-loop (SIL) [13]. Specifically designed integration tests make use of these techniques to verify the correct interaction between software units. Software components, that are integrated into an ECU are tested with Hardware-in-the-loop (HIL) frameworks, which emulate the real system. With proceeding integration, these

III. DATA -D RIVEN D EVELOPMENT The term data-driven development appears in context of product and software development2 as well as economics3 . It pools approaches, which utilize data and data analytics to substantially influence business or design decisions. This section presents usage scenarios of data-driven development for ASE. During real world tests with prototype vehicles and series cars for quality assessment, usually the entire communication of the vehicle is consistently recorded with dedicated hardware for data logging. The captured communication messages include measured states of the mechanical components, such as velocity and momentum, as well as objects detected in the vehicle environment by vehicle sensors. The goal of our approach is to skim the benefits that a consistent usage of recorded real world driving data can offer throughout the development process. As shown in Figure 3, we grouped the Automotive SPICE development process in four groups for which we identified well suited methods that help to deal with the typical challenges in each specific Automotive SPICE development stage. In the requirements elicitation and analysis phases, data analysis supports the definition of requirements and helps to understand the system and its environment. Descriptive preparation of significant data facilitates the design 2 Data Driven Development/Lean Programming, The Practical Dev, https: //dev.to/topriddy/data-driven-developmentlean-programming 3 Data-Driven Development, World Economic Forum Switzerland, https:// www.weforum.org/projects/data-driven-development

 Contextual information of system & environment  statistical analysis  identification of patterns

Recorded Scenarios  detailed understanding  Support of debugging and in-depth analysis of signal on component and subsystem interfaces level during implementation  support of critical design choices  analysis of detailed design and  improved coordination implementation decisions

 improved statistical significance  shortened iterations  increased effectiveness and efficiency of testing

Integration & Qualification

Test Campaigns

Construction & Verification

Sources

Field Trials

Requirements Elict. & Analys.

Prototypes

System & SW Design

frameworks are expanded to full subsystems. These frameworks are often combined with simulation techniques, that reproduce unavailable items of the system or the system environment. Models replace these items with abstract representations during the tests. In addition to the described test techniques, different stages of prototype evaluation models are constructed and integrated. They are used in field trials and test campaigns for real world validation of the developed items. The described V&V methods mainly employ specification based techniques. With the introduced increase of product complexity, the detailed manual specification of a sufficient number of variants becomes more and more difficult to muster. Sophisticated methods for automated test case generation often fail due to the inherent complexity of these approaches. In computer vision, usage of video data recorded in real world is a very common approach [14]. Besides the simplicity of gathering this data during anyway inevitable real world tests, an important reason is the immense variety of optical effects within a single frame and the associated modeling effort for sufficiently accurate generation. Up to now, the data gathered during real world validation is mainly used for specific analyses of the individual feature’s functional behavior. The information included within these data can be further exploited to complement the established specification based techniques and to cover a broader range of variants. In the following, we provide an overview of possible usage scenarios for the individual development activities.

Fig. 3: Applications of data-driven development to support different development phases. process and provides the involved engineers and developers with a deeper understanding of the targeted feature. During the implementation phase, specific scenarios in the recorded data are selected for simulation based verification of the functional behavior on system level. In the later Automotive SPICE stages integration and qualification, the exploitation of large amounts of data in system simulation widens the space of covered variants significantly and complements the currently applied specification based approaches for V&V. A. Requirements Elicitation and Analysis During requirements elicitation and analysis recorded data can be used to better understand the system as well as its environment and contextual purpose. With this deeper understanding of the feature and its surrounding elements and inputs, the specification of requirements and constraints on system and subsystem level is facilitated. In this phase a broad variety of test scenarios with limited depth is suitable to be analyzed by a generic set of statistical methods to answer simple questions like why, when, and where certain things happen to whom or what. Recorded driving data contains data from multiple sensors and input signals relevant to the developed system, which allows parameter value observation in a multidimensional space. Data visualization methods facilitate the analysis of the complex dynamical behavior of input signals and allow exploration of the dynamics and ranges to be expected. An analysis of dependencies between signals might uncover other useful information like correlations between allegedly independent parameters or refutation of expected dependencies.

Fig. 4: Angle of steeringwheel and lateral acceleration compared with curvature of detected lanes.

For example an analysis of steering wheel angle, lateral acceleration and the detected lane marking curvature from a camera was conducted. The expected dependency between steering wheel angle, lateral acceleration and road curvature is visible in the bivariate histograms shown in figure 4, but also strong outliers are visible. These cases depict lane swaps, banked road surfaces and detected road curvature from branching lanes, which otherwise might have been overlooked in the requirements specification. Knowledge about cases like these is beneficial for a comprehensive requirements definition and system specification. B. System and Software Design In the system and software design phase the definition of interfaces and the allocation of the functional elements of the feature under development to specific components is an essential task. Thus, recorded driving data, which can be used to determine the expectable characteristics and quality of input data is highly valuable. The in-depth analysis of recorded signals on a broad data base provides information about the overall availability of certain signals and the types and frequency of invalid value occurrences. Such information is beneficial for critical design decisions concerning redundancy, required filter stages and according computing performance. Especially early on knowledge about invalid values and outliers can prove beneficial to the system design and the ability to predict and avoid errors in the implementation phase. Suitable visualization of specific dependencies, uncertainties and erroneous signals with time lines, maps or statistical charts improves the overall system knowledge on which design choices are made. Also timing aspects in the design phase can be sharpened with the help of the recorded data, allowing for a better coordination with other development teams. C. Unit Construction and Verification During software implementation, programmers can apply different methods for usage of the data to test and verify the functionality of the developed software components on different levels. Section IV presents an example for usage of data records in open-loop simulation as well as an example for usage in closed-loop simulation. Based on the results of these ad-hoc tests, better and faster decisions for software design and implementation can be made. A hand-picked test drive, one well known to the developer, can for example help to identify weaknesses and failures in the behavior of individual software units and support identification and understanding of the failure’s root cause. A fast and neat possibility to simulate the implemented software unit in interaction with the other units and components provides the developer with a possibility to analyze the impact on the overall feature. The use of recorded data ensures realistic stimulation and complements the usually used generic scenarios. Comparing the recorded driving data based simulation output of different software versions gives the developer an overview of the effects of the changes made. Geographic visualizations of the changed

behavior allow for a quick evaluation of the context of wanted and unwanted changes in the function’s behavior. Unit verification with one or just a few inputs can easily be done with generated data. However, with an increasing number of input dimensions test cases skyrocket. While corner cases and boundary values are already covered by regular unit tests, the usage of recorded data provides a broad range of additional test coverage of the intervening spaces. This effect amplifies with correlated parameters and time dependent signals. The tedious specification of adequate test vectors, that cover a reasonable part of combinations and signal curves is reduced by usage of the recorded data. Data-driven simulations on component and subsystem level can be used to extract the initial states of units and signal values of interfaces required for unit test specification. These tests however should not fully replace the existing testing methods but rather add an exhaustive coverage of realistic test cases. D. Software and System Integration and Qualification The aim of the integration and qualification phases for software and system items is to provide evidence for specification compliant implementation and interaction. The simulation of the components, subsystems and the full system is not only viable for testing and debugging during the implementation phase, but also an appropriate tool for wide-spread V&V tests. With a vast amount of automotive full logs and the above mentioned methods to facilitate the usage of the recorded driving data in simulation environments, the system can be executed for thousands of kilometers with realistic inputs. This again does not aim to replace existing test methods like HIL and prototype vehicles, but enhance the existing tool set of the developers. Being able to simulate with recorded data and suitably analyze specific scenarios and simulations of the real world before, during and after prototype and field tests benefits the effectiveness and efficiency of V&V activities. The extend of the possible analysis is increased and real world activity can be better focused based on the analysis. For one, the virtual nature of these tests shortens the iteration time of software development and testing drastically and for another the extended test sets can simulate thousands of kilometers with different scenario settings to increase the statistical significance of the executed tests. Different scenario settings include different road types, weather conditions, traffic scenarios or countries. These extended tests can be implemented on a nightly base and thoroughly evaluate the performance of the current software release with a broad amount of test data. Automated reports and continuous integration of the data-driven simulation gives experts a neat overview and, if necessary, detailed in-depth information to validate the expected behavior or inspect undesired effects. IV. A PPLICATON IN C ONTEXT OF P REDICTIVE C RUISE C ONTROL (PCC) We implemented the data-driven development approach for the development of a Predictive Cruise Control (PCC) feature. The PCC constitutes an advanced Adaptive Cruise Control

Desktop System MATLAB

Data Records

Preprocessing & Execution SW-Items MATLAB/Simulink ANSI-C

Assessment & Visualization

Parallelization

(ACC) feature, which uses a model predictive control algorithm to operate the speed of the vehicle not only to preceding traffic objects, but also to the road topology and speed limits [15]. The feature includes open- and closed-loop functional elements on component and subsystem level. We present two use cases to demonstrate the applicability of the datadriven development approach for both open- and closed-loop functions on component and subsystem level. This includes the evaluation of constraints and requirements for newly introduced enhancements of the functional capability and simulation based evaluation of implemented software items. All dynamic testing described in this contribution is performed in a SIL environment on regular desktop PC systems without utilization of virtual hardware and targeted compiler tool chains. The first use case is the evaluation of an open-loop function to estimate the driver state [16]. The function’s purpose is to anticipate the control inputs of the driver in the foreseeable future depending on the perceived driver actions, vehicle state and environment. Since the actually realized control variables, like the vehicle’s velocity and brake applications, are included in the recorded data, the ground truth to assess the prediction output is already available. Therefore, the data used for testing is already labeled. Time line based visualizations help the developers to tweak the parametrization of the function and find weak spots in the prediction model using their desktop computers. This reduces required real world tests with prototypes and enables the developers to draw a statistical significant sample. Geographical visualizations of the faulty predictions allow visual pattern detection, e.g. repetitive faulty behavior in motorway entry situations. The second use case supports the test and validation of the full PCC feature on subsystem level. To enable the simulation of the closed-loop control feature with recorded driving data, we developed the Reactive-Replay approach [17]. It supports the developers with fast and easy configurable testing capabilities for V&V of the implemented software. The approach opens up a large data pool of recorded data from field trials and test campaigns with the included variants of different situations and scenarios for reuse during development. Specific scenarios can be extracted for conceptual assessments as well as for the continuous tracking of software maturity [18]. Test sets, carefully selected by control system experts, facilitate holistic feature tests in the simulated environment and regression based test strategies. Due to the system’s closed-loop characteristic, the ground truth for regression testing has to be generated for assessment of the functional behavior. This is achieved by comparison of the new software release with the simulation result of the latest release of the software, that was successfully validated in the real world [18]. If a faulty situation is discovered during real world tests, e.g. undesired behavior during roundabout drive-throughs, the Reactive-Replay toolkit enables extraction of similar scenarios from the data pool. These can be used for repeated indepth testing of scenarios resembling the identified faulty situation. The potentially large amount of roundabout drivethrough scenarios in the database facilitates the simulation

Results & Reports

Cloud Computing System of N Clusters Cloud Management Framework Cluster 1

Cluster 2 Cluster N-1 Parallel Workers

Cluster N

Data Storage

Fig. 5: Scalability concept for parallel tests of vehicle software with a large amount of recorded data and assessment of a statistically significant sample by control experts. Furthermore, the data-driven approach in combination with a large test data pool can be used for the preparation of country specific field campaigns. Being able to prepone a field campaign with data from a specific country in a virtual test environment improves the effectiveness of the upcoming real world campaign, as less faults can be expected. Changes in the parametrization of the developed features can be evaluated on a broad data base, regarding the achievement of the desired improvement and the absence of undesired side effects. V. U SABILITY AND S CALABILITY To cover a sufficient amount of variants during V&V activities of ASE, large scale simulations are necessary. The data-driven simulation approaches in combination with a continuously growing data pool enable large scale simulations to a certain extend. However, conventional hardware and single threaded simulations don’t scale to required data volumes and a flexible and scalable approach is required. We propose a parallelization approach utilizing MATLAB’s built-in features, that scales from local PCs and workstations up to cluster and cloud computing networks. Figure 5 outlines our approach of a scalable framework that supports data analysis and evaluation of software items via open- and closed-loop simulations. Integration of proprietary C-code functions via dynamic linking in MATLAB gives access to the visualization and data handling tools well known to most development teams in the automotive industry. Thereby, granting easy customization of existing analyses, access to various toolboxes and the possibility for defining new, custom-made assessment algorithms, which are tailored for specific problems. However, with the growing data pool, new challenges arise. Over time, the database of test data will become more and more heterogeneous, seeing as test files from previous prototypes and older vehicles may not include all of the current functionality and therefore not support all of the signals in the current build process. Signal names may deviate and signal formats might be changed. Without a flexible method to address these changes, old data records will become obsolete. Our framework handles these challenges with the help of generalization in the preprocessing step. The mapping from

concrete to abstract signal representations, the sampling of signals to the function’s cycle as well as the substitution of missing signals are systematically supported, detaching the simulation from implementation details, variants and versions. To achieve the described scalability, our concept integrates cloud computing capabilities to distribute the required computations. The underlying cloud management framework provides each machine with the individually required data and gathers the results for combined analysis. It further deploys independent, portable and encapsulated simulation kernels, that enable the processing of even large scale data sets in a feasible time span. The flexibility of cluster and cloud resources leaves almost no limits to resource expansion. To allow adoption to further features, the detailed design of the framework should be as modular as possible and support the flexible exchange of the function under test, evaluation methods and input data. Our concept aims at establishing a fundamental analysis toolkit with common methods for ASE. Though, in-depth function specific evaluation metrics will have to be implemented on a case by case basis. VI. C ONCLUSION We introduced this contribution with an analysis of current challenges in Automotive Systems Engineering. Rising inner product complexity due to denser cross-linkage between former individual system elements, the expanding perception capabilities of upcoming vehicles and the increase of algorithmic complexity render new approaches for ASE necessary. Especially the large amount of relevant situation and scenario variants that need to be considered, included and validated were identified to exacerbate the development of high level features. We analyzed and described the current procedure of system and software development in the automotive domain based on the Automotive SPICE process reference and identified the demand for new approaches to supplement the specification based engineering process. In Section III we introduced what we call data-driven development. We presented beneficial approaches of extensive data usage for all stages and activities of the product development. These include the analysis of the contextual background of a feature, fact based design decisions and simplified debugging as well as an increase in overall Verification and Validation effectiveness and efficiency. The approaches were explained from a more applied point of view using the example of PCC development for which we developed and implemented them. The benefit of complementing existing and established approaches with data-driven methods were clearly outlined. In order to fully exploit the data-driven development approach we identified a demand for scalability and parallelization. We presented a concept, that is a first step towards meeting this demand. Abstraction of variant and version specific information facilitates the generalization of the available data for use with many types of mainly software based features. We hope,

that our consistent development framework based on a tooling well known in the application domain increases the usability and acceptance of data-driven development approaches. Future work needs to evaluate the feasibility of the described framework concept and prove its effectiveness, efficiency and the transferability to further features. A balanced set of methods and tools to analyze and assess the time series data needs to be evolved, that considers the special constraints in the automotive domain. R EFERENCES [1] K. Bengler, K. Dietmayer, B. F¨arber, M. Maurer, C. Stiller, and H. Winner, “Three decades of driver assistance systems,” IEEE Intelligent Transportation Systems Magazine, vol. 6, no. 4, pp. 6–22, 2014. [2] J. Bach, S. Otten, and E. Sax, “A taxonomy and systematic approach for automotive system architectures - from functional chains to functional networks,” in 3rd International Conference on Vehicle Technology and Intelligent Transport Systems (VEHITS), 2017. [3] M. Aeberhard, S. Rauch, M. Bahram, G. Tanzmeister, J. Thomas, Y. Pilat, F. Homm, W. Huber, and N. Kaempchen, “Experience, results and lessons learned from automated driving on germany’s highways,” IEEE Intelligent Transportation Systems Magazine, vol. 7, no. 1, pp. 42–57, 2015. [4] T. Helmer, L. Wang, K. Kompass, and R. Kates, “Safety performance assessment of assisted and automated driving by virtual experiments: Stochastic microscopic traffic simulation as knowledge synthesis,” in Intelligent Transportation Systems (ITSC), 2015 IEEE 18th International Conference on. IEEE, 2015, pp. 2019–2023. [5] A. Korsiakoff, W. N. Sweet, S. J. Seymour, and S. M. Biemer, Systems Engineering Principles and Practice. John Wiley & Sons, Inc., 2011. [6] J. Weber, Automotive Development Process. Springer-Verlag, 2009. [7] Automotive SPICE Process Assessment / Reference Model, 3rd ed., VDA QMC Working Group 13 and Automotive SIG, Berlin, Germany, 7 2015. [Online]. Available: http://www.automotivespice.com/ [8] K. C. Kang, S. G. Cohen, J. A. Hess, W. E. Novak, and A. S. Peterson, “Feature-oriented domain analysis (foda) feasibility study,” DTIC Document, Tech. Rep., 1990. [9] OMG Systems Modeling Language (OMG SysML), Object Management Group, 2012. [10] J. Sch¨auffele and T. Zurawka, Automotive Software Engineering Grundlagen, Prozesse, Methoden und Werzeuge effizient einsetzen, 5th ed. Springer Fachmedien Wiesbaden GmbH, 2012. [11] P. Bourque, R. E. Fairley, et al., Guide to the software engineering body of knowledge (SWEBOK (R)): Version 3.0. IEEE Computer Society Press, 2014. [12] E. Sax, Automatisiertes Testen Eingebetteter Systeme in der Automobilindustrie. M¨unchen: Hanser, Carl, 2008. [13] H. Shokry and M. Hinchey, “Model-based verification of embedded software,” Computer, vol. 42, no. 4, pp. 53–59, 4 2009. [14] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. [Online]. Available: https://www.cityscapes-dataset.com/ [15] K.-L. Bauer and F. Gauterin, “A two-layer approach for predictive optimal cruise control,” in SAE 2016 World Congress and Exhibition, 2016. [16] S. F¨unfgeld, M. Holz¨apfel, M. Frey, and F. Gauterin, “Driver state estimation for prediction of vehicle states within control systems,” in Intelligent Vehicles Symposium (IV), 2016 IEEE. IEEE, 2016, pp. 1386– 1391. [17] J. Bach, K.-L. Bauer, M. Holz¨apfel, M. Hillenbrand, and E. Sax, “Control based driving assistant functions test using recorded in field data,” in 7. Tagung Fahrerassistenzsysteme, 2015. [Online]. Available: https://mediatum.ub.tum.de/node?id=1285215 [18] J. Bach, S. Otten, M. Holz¨apfel, and E. Sax, “Reactive-replay approach for verification and validation of closed-loop control systems in early development,” in SAE Technical Paper 2017-01-1671, 2017.