Measurement based software execution tracing in HIL (Hardware In the Loop) tests Balázs Scherer
Gábor Horváth
Department of Measurement and Information Systems Budapest University of Technology and Economics Budapest, Hungary
[email protected]
Department of Measurement and Information Systems Budapest University of Technology and Economics Budapest, Hungary
[email protected]
Abstract— Statistics show that in most of the automotive developments the role of the electronics is getting more and more significant. Some say that today, 90 percent of all innovation in the automotive sector are driven by electronics and software. Many of these developments and innovations affect safety critical functions. Standards like ISO 26262 give guidelines for testing electronic control units. Few of these guidelines give specification to the required test coverage for the software of the embedded systems. These requirements are usually based on Code Coverage metrics. This paper describes a novel integration way of a measurement based Code Coverage calculation method into a professional HIL (Hardware In the Loop) development system the NI VeriStand. The advantages of this integration compared to the existing separate coverage tools based solutions are introduced in details. The paper also describes the integration process with the challenges and our solutions given to them from the lower logical layer of execution trace data measurement to the presentation of Code Coverage calculation results. Keywords— Code coverage; HIL (Hardware-In-the-Loop) testing; NI VeriStand; LabVIEW; ARM Cortex; CoreSight; Trace.
I.
SOFTWARE EXECUTION TRACING
The term software execution tracing can mean many things, ranging from timing verifications to data flow verifications, but in this paper we only focus on the instruction execution tracing part. Instruction execution tracing is the basis of calculating Code Coverage metrics, used and required by many embedded system tests and safety standards. A. Code coverage measurements Code coverage measurements are not tests or verifications; they are metrics describing the effectiveness of a test suite. The literature distinguishes between different types of metrics based on their complexity. Safety critical standards specify metric types to be applied for each safety integrity level. As an example for this, the ISO26262 standard [1] used by the automotive industry specifies the requirements shown on “Table 1” (R means Recommended, HR means Highly Recommended). Other safety standards like DO-178B, or IEC 61508 have very similar requirements. Statement coverage checks, whether each statement (or in case of C language each line) in the program has been executed. This is the simplest coverage metric. Tests ensuring
100% statement coverage are suitable only for the least safety critical applications (ASIL A). TABLE I.
CODE COVERAGE REQUIREMENTS OF ISO26262
Methods
ASIL A
B
C
D
HR
HR
R
R
Branch Coverage
R
HR
HR
HR
MC/DC (Modified Condition/ Decision Coverage)
R
R
R
HR
Statement Coverage
a.
Based on ISO-26262- 6 Table 12 Structural Coverage Metrics at the Software Unit Testing.
Branch Coverage checks; whether each branch of each control structures (like IF and CASE structures) have been executed. For example, both the true and false branches of an IF statement, have been executed? Branch Coverage is a moderately difficult to implement metric, and tests ensuring 100% Branch Coverage are suitable for most of the safety critical systems (from ASIL A to ASIL C). MC/DC (Modified Condition/Decision Coverage) checks; whether each decision tries every possible outcome. Each condition in a decision takes on every possible outcome. Each entry and exit point is invoked, and each condition in a decision is shown to independently affect the outcome of the decision. MC/CD coverage is one of the most complex coverage metric, and not easy to implement it. Tests with 100% of MC/DC coverage is needed only for the most safety critical systems. B. Existing sollutions There are two main types of Code Coverage measurement toolsets on the market. The first types of toolsets are using code instrumentations for measuring the coverage, where the second types of toolsets are using hardware trace interfaces as information source. Among other tools the VectorCAST [2] package and the LDRAcover [3] follow the code instrumentation way. This way has the advantage of architecture independency. These tools also have good performance, when executing the tests in a simulator, but they modify the executed code and require hardware resources when executed at the target.
Typical examples for microcontroller trace interface based tools are the IAR Embedded Workbench [4], and the Lauterbach’s trace solutions [5]. These tools are investigating the executed code without modifying it, by using the special hardware trace interfaces of modern microcontrollers. These tools are very architecture depended and requires very costly hardware components for interfacing to the microcontroller trace port.
The first step of complementing a HIL development environment with Code Coverage measurement capabilities is to integrate the hardware trace port interface into the HIL system.
C. Our suggested sollution and its advantages Although the existing solutions presented in sub section B work fine, but we found that an additional, new type of tool could increase the application area and efficiency of tests using Code Coverage as performance metric.
A Typical NI VeriStand based HIL system consists of a host computer, where the HIL test can be configured: selecting the model, which describes the environment, connecting the input/output channels of the model to hardware I/O channels etc. This host computer also contains the user interface.
ISO-26262 standard requires that the test environment of software unit testing shall correspond as far as possible to the target environment. If the software unit testing is not carried out in the target system, then additional analysis and measurements would be needed. Therefore the preference is to test the code execution on the target in a realistic situation.
After the configuration, typically the model, the stimulus profile execution, and hardware handling functions are deployed to a RT target (Real-Time PC, or PXI controller), while the user interface remains in the host PC “Fig. 1”.
In the case of embedded systems, usually to test the target system or the software running on that a complex environment simulation, so called a HIL (Hardware In the Loop) system is needed. Our idea is to complement an existing widespread HIL development system like the NI VeriStand [6] with Code Coverage measurement abilities based on measured microcontroller trace interface data. Of course existing tools are capable for measuring code coverage during HIL tests, but as a separate toolset, which has many disadvantages.
• Test engineers don’t have to learn new platforms; they only need to learn the HIL test development platform. TRACE PORT INTEGRATION INTO HIL TESTS
II.
User Interface
Custom Workspace tool ETM processing and presenting
• Existing solutions require the source code for the measurements, but many times system tests are done by a separate test group, and providing the source code to this group is sometimes prohibited. Our solution can make limited, but useful measurements without the source code. • Integrating the Code Coverage metrics into a HIL test system makes it easily possible to use the metrics as test exit condition, which can lower the cost of the test process. • In the integrated system, automated test pattern modification is also possible to increase the coverage and make the test more efficient. • In case of an error is detected by the HIL test a good estimation can be done for the location of the error (system can log the input-output patters and the program flow for each HIL simulation steps)
(Host PC)
VeriStand Gateway
TCP/IP based LabVIEW Network Stream
Special TCP/IP connection of VeriStand Veristand Engine
Server Communication Custom Device ETM processing or forwarding
Model execution
The advantages of our suggested solution comparing to the existing ones (mainly comparing to the ones using hardware trace interfaces) are the following: • Integrating the Code Coverage measurement into an existing HIL system can significantly reduce the cost. Existing solutions are very expensive: very costly development environment and trace hardware are required. In our solution the hardware used for tracing is also costly, but it can be used to other functions, not just trace capturing. This reduces the overall cost.
Workspace
Stimulus Generation
(RT Target) Channel forcing, processing, calculation
Other Custom devices
Hardware I/O drivers
DMA channels normal VeriStand operation
DMA channel ETM Trace Trace Signal Capture and packet filtering
ETM Preprocessing
Other FPGA functionality
FPGA hardware custom FPGA personality
Other hardwares DAQ cards CAN, FlexRay cards etc.
Hardware I/O
Fig. 1.
Integrating the trace port interface into NI VeriStand
To integrate the trace signals of microcontrollers, (usually a dual data rate interface, with clock rate up to 100 MHz) high speed FPGA hardware is needed with a custom programmable functionality to filter the trace data. A Custom Device driver is also needed at the RT target level to fetch the instruction tracing related data from the FPGA and forward it to the host PC, where a custom workspace tool can process it and present the coverage metrics. These additions to the VeriStand system are presented as dark boxes on “Fig. 1”, and the following sections will describe the functionality embedded into them. The description will not include de details of FPGA hardware selection and the technical details of integrating such blocks into the NI VeriStand environment, because the challenges of this procedure are described in our previous paper [7].
III.
TRACE BASED CODE COVERAGE MEASUREMENT
This paper describes the trace based Code Coverage measurements of ARM CoreSight standard based Cortex M microcontrollers, which are currently among the most widespread architectures. Other standard (like IEEE-ISTO 5001-2003 (Nexus)) based microcontroller’s instruction tracing could differ somewhat. The trace information in the CoreSight architecture [8] is usually generated from three trace sources: the Embedded Trace Macrocell (ETM), the ITM (Instrumentation Trace Macrocell), and the Data Watchpoint and Trace (DWT) blocks. The Trace Port Interface Unit (TPIU), formats the information from these sources into packets, and sends it to an external trace capture device. The ITM and DWT blocks have little or no connection to the instruction tracing and Code Coverage calculation, therefore we focus on the ETM block. A. ETM packets and their processing The ETM block [9] is used for providing instruction and data traces. In most of the cases normal microcontrollers implement only a restricted version of ETM, which is capable for instruction tracing only. We focus on this restricted version of ETM implementation that can be found in Cortex M microcontroller series. Instruction tracing requires much data. To reduce the amount of this, the ETM does not always output exactly what addresses the processor has reached/executed. It usually outputs information about the program flow, like how many instructions were executed or skipped since the last message, and outputs address changes in so called branch packets, only if it is needed. Typical examples of using branch packets are in the case of executing branch instructions and entering or exiting interrupts. Our measurements have shown that there is a need for preprocessing the ETM messages, because the ETM packets contain the information in a very compressed form (especially the branch packets), and the host PC is not the best device to execute the decoding bit operations on this data stream. Our performance measurements have shown, that even an optimized LabVIEW code (best way to integrate the Code Coverage feature into VeriStand) running on a relatively fast Intel Core i5 processor can have problems during a procession of an ETM stream from a microcontroller with 100 MHz CPU clock rate. Consider that in the case of 100 MHz microcontroller clock rate the PC has about 30-40 clock cycles to identify and administer and log an instruction executed by the microcontroller; therefore it is not too logical to perform branch decoding bit processing at the host PC (that is a typical single process job, the advantages of multiple CPU cores cannot be utilized). Our goal is to trace microcontrollers with 200MHz or higher CPU clock rate, therefore a workaround is needed. The best way to do the ETM preprocessing is to implement it in the FPGA hardware, which is designed for such kind of bit manipulation. Our consideration is to perform the branch packet decoding at the FPGA level and provide full branch addresses to the higher layers. This preprocessing significantly increase the transmitted data bandwidth, because the ETM
block transmits only the lower bits of the address affected by the branch event, but after decoding the full 32bit address is transmitted. In the case of a normal program flow this bandwidth increase is about 30-50%, but the processing time drops to less than the half at the host PC or Real-Time PC side. Branch address decoding also has the advantage that there is no address synchronization needed at the processing side, because the transmitted addresses are always the full addresses, not just a changes calculated from a base address. This also makes the computation process faster. The preprocessed data has about N * 300-500 Kbytes/s bandwidth requirement (the exact value highly depends on the program flow structure), where N is the clock rate of the traced microcontroller in MHz. This means about 100Mbytes/s data bandwidth requirement for a 200 MHz microcontroller clock rate, which exceeds the limit of a gigabit Ethernet connection between the Real-Time PC and the host PC. Therefore some part of the future processing steps, like some coverage calculations, needs to be shifted to the Real-Time PC side in the case of high clock rate microcontrollers. To reduce the communication bandwidth requirement only aggregated information for the presentation should be transmitted through the TCP/IP network between the Real-Time PC and the host PC. Between the Real-Time PC and the FPGA there are no real limitations. Even the simplest PXI configurations have data link bandwidth of 110 Mbytes/sec or above, and newer PXI versions can have data link bandwidth up to 800 Mbytes/sec. B. Statement Coverage calculation The preprocessed ETM data contains the addresses of performed branches, and information about the executed and skipped instructions after the branches. This information is appropriate for calculating the Statement coverage. Usually Code Coverage tools provide coverage information for C functions. However in the early state of ETM information processing it is not logical to make the Statement Coverage calculation for C functions, because this requires the identification of the start and end addresses of functions, and checking whether the actual executed instruction is in these ranges. This procedure would require too much calculation and cannot be executed online. That is much more practical to perform the Statement Coverage calculation for the whole program memory space of the investigated microcontroller software. This method requires a 64bit counter associated with every assembly line in the microcontroller program. A result is an array of 64bit counters describing, how many times each line were executed. The job of ETM processing function becomes very simple in this case: if an assembly line is executed, then the ETM processor needs to increment the counter associated with this line. This incrementation can be done by a simple array indexing based on the executed address. In the real world this processing method is a little bit more complex, because the ARM Cortex M architecture uses the Thumb2 instruction set with mixed 32bit, and 16bit instructions. Therefore it is not trivial to make an association between the program memory address and the assembly line address. Other difficulty is that not every line of the program
memory section is used for assembly instruction storage. For example there are the exception vectors at the beginning of the program memory, and occasionally there are free unused spaces in the program memory. To solve this problem the analysis of the compiled program is needed. For example in the GNU environment the objdump utility is capable of giving the disassembly of the whole program memory section of the software. This disassembly information can be used to determine the location of the 32bit and 16bit instructions, and to identify unused spaces. Based on the disassembly, currently we use a solution where every 16bits of the used program memory code is associated with a 64bit counter. This makes the indexing of the counter array very easy, but also means that a 32bit instruction is associated with two counters, where the second counter is unused “Fig. 2”. This results in a memory overhead, but in the Real-Time PC or in the host PC there is plenty of memory, and the counter array requires only a few Mbytes, even with this method.
environment GDB can provide this information. Therefore it is possible to calculate the C line Coverage from the assembly line execution counters. An example of colored source code by our Coverage presenter is shown on “Fig. 3” (this test was started, when the microcontroller was already running. This is the reason, why the instruction before the WHILE loop was signaled as not executed in this measurement).
64 bit counter array 32bit instruction 16bit instruction
1xxxxxxxxx 0xxxxxxxxx
32bit instruction
1xxxxxxxxx
32bit instruction
1xxxxxxxxx
16bit instruction
Fig. 2.
1xxxxxxxxx
1xxxxxxxxx 1xxxxxxxxx 0xxxxxxxxx
Counter array for Satement Coverage calculation
We use the most significant bit of the counters to identify the 32bit instructions. This is important, because the ETM processor has to determine the next address to be executed in the microcontroller program, from the ETM packets. To perform this, after an executed or skipped instruction, the current address needs to be incremented by 2 after a 16bit instruction or by 4 after a 32bit instruction. Storing the information about the instruction’s size in the counter reduce the number of memory operations executed by the PC, and therefore increase the processing speed. Calculating the Statement coverage information should be very easy based on the counter array: the Coverage presenter block reads the important parts of the counter array associated with the function or functions under investigation, and colors the source code lines appropriately (function start and end points can be mined out from the compiled files. In a GNU environment GDB can provide such information). The main problem or difficulty here is the not one to one relation between the C and assembly instructions. Code coverage tools using the instrumentation method do not face with this problem, but the measurement based versions have to solve the inconsistencies caused by the C and assembly statement differences. After compilation it is possible to discover witch assembly lines associated with which C code lines. In the GNU
Fig. 3.
Example for Satement Coverage
Unfortunately C lines usually consist of more than one assembly instructions. This is not a problem for normal C statements, because in that situation every assembly instructions of the C statements are always executed. In the case of conditions like IF or CASE conditions, or special instructions, this is not so simple: depend on the situation some assembly instructions of the C statement are executed and some instructions are skipped. In our presenter we use the highest value assembly execution counter to determine the execution number of C lines. This is a good estimation, but still not gives a good result for every situation. For example the last } in “Fig. 3” should be executed 1398 times, but that C line consists of two separate assembly branch instructions one for the THEN, one for the ELSE path, and the sum of their execution counters would give the correct result (currently the THEN counter is used, because it has the higher value). A detailed investigation of the source code is needed to present every line counter correctly. Unfortunately currently our system is unable to do that. It is also possible that a C line of the source file do not contains assembly instruction (doe to optimalization, or because they do not have real functions like the ELSE statement in “Fig. 3”), these lines are not taken into account in our calculations.
Our tool can calculate the assembly and C Statement Coverage metrics for functions. Note that, however “Fig. 3” shows the execution numbers for each line, but this is not required for Statement Coverage, therefore most of the problems mentioned above not issue for Statement Coverage calculations just for presentation. Also note that for the presentation shown on “Fig. 3” the source code of the program is also needed, but for calculating the metrics that is not required (compiled files like .elf files can contain the C and assembly line information, but not the C source code associated with that lines). C. Calculating more sophisticated coverage metrics Although assembly and C line coverage inconsistency cause problems in the presentation of executed C lines, but it also can have benefits. A 100% assembly statement coverage is a much stronger metric than a 100% C line statement coverage. Consider the following example: if((a & 0x01 == 1) || (b & 0x10 == 0)) { /* Then branch */ } /* There is no else branch, program continues */
The program flow graph for the example above is shown on “Fig. 4” for C statements and assembly statements. C code level program flow
Assembly level program flow
if statement true
If statement #1 condition false
false
Instructions after the if statement
IV.
ACKNOWLEDGMENT The publication was supported by the TÁMOP-4.2.2.C11/1/KONV-2012-0001 project. The project has been supported by the European Union, co-financed by the European Social Fund. We would like to thank the help of National Instruments Hungary for providing us a PXI-7953R FlexRIO card to perform the measurements. REFERENCES
If statement #2 condition
[1]
true
[2]
then branch instructions
[3]
Instructions after the if statement
[4] [5]
Fig. 4.
C program flow vs. assembly program flow
In this case a 100% C statement coverage does not mean 100% branch coverage, but 100% assembly coverage does (the false branch of #2 condition is also an assembly instruction). We found that except extreme situations 100% assembly coverage means 100% branch coverage. Assembly coverage also contains information about, the executed conditions, which is the first step towards MC/DC coverage. Therefore there is the opportunity to make Branch coverage and MC/DC coverage calculations based on the assembly instruction trace. Unfortunately to implement these calculations
CONCLUSIONS
This paper described the integration way of trace based Code Coverage measurements into a professional HIL development system the NI VeriStand. The paper described the challenges and our solutions for every logical level of the integration procedure staring from the trace signal capturing through the ETM packet filtering to the presentation of coverage results. The system presented currently is able to perform C line and assembly line Statement coverage measurements. There are promising experiments to broaden the features of the system towards the Branch Coverage and MC/DC coverage calculation, which are required by devices with higher safety integration level. We found this development a promising one, as shown by the analyzed advantages of this solution.
true
false
then branch instructions
deep investigation of C code structure is required, and currently our system is not able to perform this, but this is the next development plan for the future.
[6] [7] [8] [9]
International Standardization Organization, ISO 26262:2011 "Road vehicles -- Functional safety" standard, ISO 2011. Vector sotware, VectorCAST Embedded Software Testing products: http://www.vectorcast.com/ LDRA Software Technology, LDRAcover Code Coverage Reporting Tool. http://www.ldra.com/index.php/en/products-a-services/ldra-toolsuite/ldracover IAR Systems, “Using trace”, addition to IAR Embedded Workbench® IDE User Guide for ARM®. IAR Systems 2013. Lauterbach, Lauterbach development tools: http://www.lauterbach.com, 2014. National Instruments, “NI VeriStandTM Fundamentals Coures Manual“ Part Number 325785A-01, 2011. Balázs Scherer, Gábor Horváth, ” Microcontroller tracing in Hardware In the Loop tests”, ICCC2014, 2014. ARM Technical Reference Manual, ”CoreSight™ Components “,ARM DDI 0314H. 2004-2009. ARM, “Embedded Trace Macrocell Architecture Specification”, ARM IHI 0014Q (ID101211), ARM Limited, 2011.
