A Test-driven Approach for Model-based Development of Powertrain Functions Henrik Peters,
[email protected], Institute for Applied Software Systems Engineering (IPSSE) 05/06/2014
Our Everyday Problems
What are we dealing with?
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 3
What are we dealing with?
Model-based . . . (management view) Testing Development not at all
Henrik Peters Institute for Applied Software Systems Engineering
completely
A Test-driven Approach for Model-based Development 3
What are we dealing with?
Model-based . . . (reality)
Testing not at all
Henrik Peters Institute for Applied Software Systems Engineering
Development completely
A Test-driven Approach for Model-based Development 3
What are we dealing with? Model-based . . . Testing not at all
Development completely
More facts Requirements are specified. . . – in natural language – on a too high level Developer and tester is generally the same person Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 3
2014-06-02
A Test-driven Approach for Model-based Development Our Everyday Problems
What are we dealing with? Model-based . . . Testing not at all
What are we dealing with?
Development completely
More facts Requirements are specified. . . – in natural language – on a too high level Developer and tester is generally the same person
Today model-based development is widely used in the automotive world. From a management perspective this is also valid for model-based testing. However, the reality is different. Between model-based development and model-based testing is a big gap. Again, from a management perspective, this gap has to be closed by means of tool support. But unfortunately, the necessary conditions are partially ignored. It starts with the fact that requirements are mostly specified in natural language which makes it difficult to derive test models directly. In addition, requirements are mostly specified at the level of entire ECU functions. But standards, such as the recently approved ISO 26262 (”road vehicles – functional safety“) [ISO], require testing on the level of software units (which we call modules). That leads to the fact that developers and testers are often the same person, so that there is no independent construction of test models.
Our Pragmatic Approach
What have we tried?
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 5
What have we tried?
Idea Integrating test-driven development (TDD)
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 5
What have we tried?
Idea Integrating test-driven development (TDD)
Hoped-for results 1. Testable requirements specification on the level of software modules 2. Reduction of routine-blindness
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 5
2014-06-02
A Test-driven Approach for Model-based Development Our Pragmatic Approach
What have we tried?
Idea Integrating test-driven development (TDD)
Hoped-for results
What have we tried?
1. Testable requirements specification on the level of software modules 2. Reduction of routine-blindness
Since everyone is claiming we’re particularly agile (no we’re not), we have acutally tried to integrate ideas of agile development. More precisely, to tackle our problems, we integrated ideas of test-driven development to our workflow. By accompanying the concept and especially the development phase with requirements engineering we’re generating a directly testable specification in the form of test models at software module level. Although the test model is still not constructed independently, routine-blindness should be reduced by anticipating test activities.
How does that work?
TDD Test-driven Development
Add a test case Execute the test cases
Eliminate duplications [passed]
[failed] Make a change
[Bec02][Doh08]
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 7
MBTDD Model-based Test-driven Development Adaption of TDD for model-based development and testing
Requirement
Eliminate duplications
Add a test case [passed] Execute the test cases [failed] Make a change
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 7
MBTDD Model-based Test-driven Development Adaption of TDD for model-based development and testing
Requirement
Eliminate duplications
Add a test case [passed] Execute the test cases [failed] Make a change
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 7
MBTDD Model-based Test-driven Development Adaption of TDD for model-based development and testing Eliminate duplications
Extend the test model
Requirement
[passed] Generate and execute the test cases [failed] Make a change
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 7
MBTDD Model-based Test-driven Development Adaption of TDD for model-based development and testing Eliminate duplications
Extend the test model
Requirement
[passed] Generate and execute the test cases [failed] Make a change
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 7
MBTDD Model-based Test-driven Development Adaption of TDD for model-based development and testing Refactor the system model
Extend the test model
Requirement
[passed] Generate and execute the test cases [failed] Change the system model
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 7
2014-06-02
A Test-driven Approach for Model-based Development How does that work?
MBTDD Model-based Test-driven Development Adaption of TDD for model-based development and testing
Requirement
Refactor the system model
Extend the test model [passed]
MBTDD
Generate and execute the test cases [failed] Change the system model
As you can see and as probably everyone knows TDD is based on the idea of writing a test case first for any new code that is written. As an advantage the newly generated source code is prespecified. We’re adapting the development cycle of TDD for model-based development and testing. Each cycle starts with a requirement, which has to be implemented. Instead of writing a test case directly, the test model is extended. In the following test cases are generated and executed with tool support. The following steps are closely related to the original TDD approach, except that we refer to models instead of code.
Limitations
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 8
Limitations
Goal Complete engine control unit (ECU) function
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 8
Limitations
Goal Complete engine control unit (ECU) function
But we started small. . . Single software modules Decomposition with well-established methods Interface specification during decomposition
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 8
Limitations Goal Complete engine control unit (ECU) function
But we started small. . . Single software modules Decomposition with well-established methods Interface specification during decomposition
Result of preparatory work Complete module architecture
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 8
2014-06-02
A Test-driven Approach for Model-based Development How does that work?
Limitations Goal Complete engine control unit (ECU) function
But we started small. . . Single software modules Decomposition with well-established methods
Limitations
Interface specification during decomposition
Result of preparatory work Complete module architecture
To integrate the approach into the existing workflow, we have made some restrictions. Although in the long term, an entire ECU function is to be developed, initially we restrict ourselves to single modules (which represent our ”software units“). Therefore, the function to be developed has to be decomposed into components, which are further divided into modules. The decomposition is performed with established processes. In addition to the full module architecture we get also an interface specification. Within this architecture, each module can be developed separately.
The test model
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 9
The test model Simple example: Continuous behavior of a delay element Input value p should be delayed by calculation duration dT to output value q Relatively simple behaviour of q, but internal state depends on complete calculation duration dT of p [Leh04]
R
1
p q
0
dT
−1 −2 Henrik Peters Institute for Applied Software Systems Engineering
0 t
2
A Test-driven Approach for Model-based Development 9
The test model Time Partition Testing (TPT) Oracles (Assesslets) Domain model (Testlets)
System under test (here: software module)
TPT
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 9
2014-06-02
A Test-driven Approach for Model-based Development How does that work?
The test model Time Partition Testing (TPT) Oracles (Assesslets)
The test model
Domain model (Testlets)
System under test (here: software module)
TPT
When talking about model-based testing approaches, everyone thinks of the generation of test cases with oracles from a behavioral model. In our context, we differ from that in two ways. First of all, continuous behavior of embedded systems in the automotive domain and its testing having some specialties. [Bri07] A simple example for continuous behavior is a delay element where an output value q depends on the complete time dT of an input value p. Therefore, most of the time we’re not interested in direct input-output relationships but in waveforms of signals. The second thing is, since Time Partition Testing is set as the testing tool, we’re confronted with the generation of test input data from a domain model. Nevertheless, TPT extends the idea of domain modeling to the opportunity of assessing test results automatically.
Testable specification
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 10
Testable specification
Specified behavior must be observable: 1. At least one signal per requirement 2. At least one requirement per signal
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 10
Testable specification
Specified behavior must be observable: 1. At least one signal per requirement 2. At least one requirement per signal
Evaluation of the test quality Coverage metrics (Decision coverage, MC/DC, etc.)
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 10
Testable specification Specified behavior must be observable: 1. At least one signal per requirement 2. At least one requirement per signal
Evaluation of the test quality Coverage metrics (Decision coverage, MC/DC, etc.) → Acceptance tests as a directly testable requirements specification in the form of TPT assesslets
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 10
2014-06-02
A Test-driven Approach for Model-based Development How does that work?
Testable specification Specified behavior must be observable: 1. At least one signal per requirement 2. At least one requirement per signal
Evaluation of the test quality
Testable specification
Coverage metrics (Decision coverage, MC/DC, etc.) → Acceptance tests as a directly testable requirements specification in the form of TPT assesslets
We’re extending the test model to cover the requirement to be implemented. That happens in the way that the necessary assesslets for specifying and checking the requirement and the necessary testlets to produce the desired input data are modeled. As already mentioned, we’re focusing on acceptance tests. But to specify behavior on the level of modules, this behavior must be observable, which in our context is only given at signals. To do so, we defined two guidelines that have to be followed: On the one hand, we require at least one signal per requirement to make it observable. On the other hand, we demand at least one formulated requirement per signal to avoid unspecified signals. To evaluate the test quality somehow objectively, coverage metrics like branch coverage or MC/DC were used. What comes out are acceptance tests as a direct testable requirements specification in the form of assesslets in TPT.
Running example
Selective Catalytic Reduction (SCR)
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 12
Selective Catalytic Reduction (SCR)
Purpose Specifically reduce nitrogen oxides (NOx ) of exhaust gas with the help of a chemical reaction A reduction catalyser converts NOx to nitrogen (N2 ) and water (H2 O) by injecting a reductant (AdBlueTM : urea solution) into the exhaust gas stream
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 12
Selective Catalytic Reduction (SCR) Purpose Specifically reduce nitrogen oxides (NOx ) of exhaust gas with the help of a chemical reaction A reduction catalyser converts NOx to nitrogen (N2 ) and water (H2 O) by injecting a reductant (AdBlueTM : urea solution) into the exhaust gas stream
Design Consists of different sensors and actuators Typical medium-sized powertrain function
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 12
2014-06-02
A Test-driven Approach for Model-based Development Running example
Selective Catalytic Reduction (SCR) Purpose Specifically reduce nitrogen oxides (NOx ) of exhaust gas with the help of a chemical reaction A reduction catalyser converts NOx to nitrogen (N2 ) and water (H2 O) by injecting a reductant (AdBlueTM : urea solution) into the exhaust gas stream
Selective Catalytic Reduction (SCR)
Design Consists of different sensors and actuators Typical medium-sized powertrain function
We used SCR as a running example and demonstrate our approach by an SCR case study. To specifically reduce nitrogen oxides of the exhaust gas stream, a reduction catalyser converts NOx to nitrogen and water with the help of a reductant being injected into the stream. The system is a typical medium-sized powertrain function whereas it consists of different sensors like pressure or temperature sensors and actuators like pumps, valves or heatings.
Conclusion
Summary
MBTDD helps to cope with missing requirements specification at module level Systematic anticipation of test activities allows construction of a directly testable requirements specification Reduces misinterpretations of requirements at module level Supports further integration steps (for example, the implementation in fixed-point arithmetic)
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 14
2014-06-02
A Test-driven Approach for Model-based Development Conclusion
Summary
MBTDD helps to cope with missing requirements specification at module level Systematic anticipation of test activities allows construction of a directly testable requirements specification
Summary
Reduces misinterpretations of requirements at module level Supports further integration steps (for example, the implementation in fixed-point arithmetic)
We have learned that the test-driven approach for model-based development helps to cope with the missing requirements specification at module level. The systematic anticipation of test activities allowed us to construct a directly testable requirements specification. Furthermore, our approach reduced misinterpretations of requirements at module level, even though the required effort was considered too high from some developers. In addition, we were able to pass the new test artifacts to later stages of integration in order to support them (e. g. the implementation in fixed-point arithmetics).
Outlook Refactorings – Had to be done manually due to the lack of tool support – Effort has to be measured to decide on tool support Quality of functionality – Due to still missing vehicle measurements difficult to evaluate – Rapid prototyping in the future Control flow-based coverage metrics have only limited suitability for control engineering systems Extend existing limitations to components and then to modules
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 15
2014-06-02
A Test-driven Approach for Model-based Development Conclusion Outlook
Outlook Refactorings – Had to be done manually due to the lack of tool support – Effort has to be measured to decide on tool support Quality of functionality – Due to still missing vehicle measurements difficult to evaluate – Rapid prototyping in the future Control flow-based coverage metrics have only limited suitability for control engineering systems Extend existing limitations to components and then to modules
Because refactorings had to be done manually due to lack of tool support, their effort has to be measured in order to decide on tool support. Due to still missing vehicle measurements the quality of the functionality is difficult to evaluate. Rapid prototyping tests are delayed at the moment indefinitely. Due to the fact that in some modules almost exclusively control engineering behavior is found, control flow-based metrics are meaningful only in parts. Finally, the approach of course should be extended to entire ECU functions, for which at the time the right project is being sought.
Q&A
Bibliography I [Bec02] Beck, K.: Test Driven Development. By Example. Addison-Wesley Longman, 2002. [Bri07] Bringmann, E.: Testing the Continuous Behavior of Embedded Systems. In: Proceedings of the 4th Workshop on System Testing and Validation, 2007. [Doh08] Dohmke, T.: Test-driven development of embedded control systems: application in an automotive collision prevention system. PhD thesis. Faculty of Engineering, Department of Mechanical Engineering, University of Glasgow. 2008. Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 17
Bibliography II
[ISO] International Organization for Standardization: ISO/DIS 26262: Road vehicles – functional safety. 2009. [Leh04] Lehmann, E.: Time Partition Testing – Systematischer Test des kontinuierlichen Verhaltens von eingebetteten Systemen. Dissertation. Fakultät IV – Elektrotechnik und Informatik. Technische Universität Berlin. 2004.
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 18
Pictures
c Dieter Schütz / PIXELIO Slide ”Our Everyday Problems“ c Helene Souza / PIXELIO Slide ”Our Pragmatic Approach“ c Thommy Weiss / PIXELIO Slide ”How does that work?“ Slide ”Conclusion“ https://secure.flickr.com/photos/hades2k/ c Tony Hegewald / PIXELIO Slide ”Q & A“ c Tim Reckmann / PIXELIO Slide ”Backup“
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 19
Excerpt Specification of behavior with assesslets Threshold crossings: Investigation whether/how long/how often a signal exceeds a threshold Sequences of events: Investigation of how much time passed between the occurence of two different results Python scripts: Investigation of pretty much anything that can be formulated with Python
Domain modeling with testlets Decomposition of the test scenario in phases with variation points Generating all test cases using the classification tree method Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 21
Excerpt Domain modeling with testlets initialisation
waiting immediately
time threshold
after 10.0 s
pressure buildup pressure threshold controller time threshold
after 5.0 s immediately
post drive time threshold
after 7.5 s post drive finished time threshold
after 2.5 s vehicle measurements
2nd pressure buildup pressure threshold
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 21
Results
Initial development of several modules Evaluation of test quality with the help of branch coverage – 109 of 111 branches were irrelevant or covered → Coverage of about 98 % without considering the control flow during test design
Henrik Peters Institute for Applied Software Systems Engineering
A Test-driven Approach for Model-based Development 22
