Automatic Diagnostic Validation is not Rocket Science - Vector Informatik

Automatic Diagnostic Validation is not Rocket Science ... Just a Matter of Consequent Exploitation of Existing Possibilities

Technical articles on diagnostic validation often begin as follows: Along the complexity of the ECUs in vehicles increases, the scope of diagnostics is also increasing, and with it the time and effort for diagnostic validation. However, thanks to current testing tools, the time and effort is no longer increasing linearly – fortunately. In particular, it has actually been possible for several years to generate and execute diagnostic protocol tests fully automatically. Despite a further increase in the functional scope, the time and effort for creating and executing tests remains fairly constant. With the established standards for unified diagnostic communication (UDS) and diagnostic description formats (ODX), it is possible to achieve a comparatively high degree of test automation, and high diagnostic quality as a result. However, testing of the diagnostic application and the update software can also be automated. These days there are few areas in the development of automotive electronics with a similarly high potential for test automation. The release of the Unified Diagnostic Services Standard

cally, the diagnostic services implemented in the ECU must

(ISO14229) [1] in 2006 represented a significant step for

be known. The CANdelaStudio tool from Vector Informatik

the automation of diagnostic tests: with the harmoniza-

GmbH is globally established for specifying diagnostics.

tion of the diagnostic services, it became possible for the

Among other things, it supports the international Open Di-

first time to implement cross-manufacturer diagnostic

agnostic data eXchange standard (ODX, ISO 22901) [3],

protocol tests in greater depth. The previous protocol

that was released in 2008 and is now used worldwide by

KWP2000 (ISO14230) [2] still allowed a great deal of free-

most vehicle manufacturers.

dom and thus must be made more concrete in an OEM-spe-

Thus the technical prerequisites for automatic generation

cific way, that makes generally valid test implementation

of diagnostic protocols have been met for a long time al-

difficult. A revised version of the UDS was published in

ready. Therefore, it is no wonder that there have been solu-

2013, which makes it possible to formulate even more de-

tions for automatic diagnostic test generation and execu-

tailed protocol tests.

tion for over 10 years already, such as the CANoe.DiVa tool

Besides the protocol definition, formally described diag-

from Vector.

nostic data represents the second important prerequisite for test generation. The functional scope of a power win-

Very comprehensive tests are generated automatically on

dow differs essentially from that of an engine ECU – and

the basis of diagnostic information in CANdela or ODX for-

the implemented diagnostic functionality also differs ac-

mat and the diagnostic protocol (e.g. UDS, KWP2000,

cordingly. In order to be able to generate a test automati-

OBD, WWH-OBD, …). These tests cover both valid and in-

01

Technical Article / 03/2017

valid requests that put the error handling in the ECU to the

options during remote maintenance, because the required

test. The tests are loaded into the test environment (CA-

diagnostic functions do not work as desired, so time in the

Noe) and executed automatically. The test results are doc-

workshop is necessary (after all). In the worst case, a pro-

umented in a detailed report [Figure 1]. For larger ECUs

tocol error could permit a safety-critical attack on the vehi-

over 10,000 test cases are generated quickly without re-

cle. Due to continuous availability of the vehicle, new use

dundant testing.

cases that the customer can experience will develop based on diagnostic functionality, which will presumably lead to more frequent and broader use of the diagnostic functions. It is thus foreseeable that the quality requirements on the diagnostic implementation will continue to increase.

Insert figure

Software Update Validation Despite all the diagnostic standardization measures, the processes for updating ECU software (flashing) remain OEM-specific. Although the services used are standardized, functions such as incremental update or mechanisms for protecting the transmitted data (identification data, checksums, signatures, etc.) lead to quite different flash

Figure 1: Function blocks of the CANoe.DiVa automated test environment

procedures in practice. Furthermore, different flash container formats are used in order to meet the process requirements of the respective manufacturers. In practice that means that there is also a specific software update

In practice there is now a formal diagnostic description in

tool for nearly every OEM. Therefore a supplier must not

the CANdela and/or ODX format for nearly every ECU.

only work with various tools but often also develop and

Therefore, it is a small step from the diagnostic specifica-

maintain a specific test environment for each OEM. A

tion to a comprehensive, automatic protocol test. Protocol

functioning flash procedure has always been indispens-

errors can be detected for a long time simply with little ini-

able in production and service. Taking future software up-

tial effort.

dates “over the air” (SOTA) into consideration, a reliable software update function in vehicles is more important

AUTOSAR Increases Diagnostic Protocol Quality

than ever. With the new possibilities, new software will

When AUTOSAR software components are used for diag-

presumably be loaded more frequently – possibly at a fre-

nostics, a series of protocol violations by the tester (e.g.,

quency similar to what we are already familiar with for

format violations) are handled by the basic software al-

mobile devices (e.g., to fix security vulnerabilities). Situa-

ready. Since the basic software is usually parameterized

tions in which an ECU no longer functions after a failed

directly with the available diagnostic data, the diagnostic

update should definitely be avoided, of course. But even

responses of an ECU are usually also protocol-compliant.

for established smartphone manufacturers, software up-

The proportion of simple protocol errors falls noticeably

dates do not always run smoothly – despite sophisticated

when AUTOSAR components are used. The time and effort

protective measures.

for test evaluation and troubleshooting fall accordingly.

Cross-manufacturer software updates have been per-

Nonetheless, the protocol test remains important for the

formed for several years already with the reprogramming

following reasons among others: >>Errors can occur during configuration of the AUTOSAR

tool vFlash from Vector, currently for over 90 different boot

components, integration, and application development >>If the diagnostic implementation in the ECU does not

The test automation tool CANoe.DiVa uses the vFlash au-

match with the diagnostic data used by the diagnostic

software update tests for all boot loaders supported by

tester, correct diagnostics in the ECU is not possible

vFlash. Besides the variables (timing, format, ...) in the val-

loader specifications of all relevant vehicle manufacturers. tomation interface and thus additionally offers automated

despite “correct” implementation >>Gaps in the error handling of the application implemen-

id flash procedure, classical error situations are also tested

tation can potentially be used to install malicious code

various points in the flash procedure. This allows the ro-

Since, in times of DoIP and OTA (“over the air”), no direct

bustness of the ECU software to be tested easily and com-

physical connection with the diagnostic tester is required, a

prehensively.

protocol error may have a negative effect on these services

The tests sketched are essentially black box tests. Tests be-

for an entire fleet of vehicles, for example, due to limited

yond these, e.g. plausibility checks for process-relevant

automatically, such as undervoltage and interruptions at

02

Technical Article / 03/2017

data such as identification data or signatures, require

Furthermore, CANoe.DiVa supports the coupling of propri-

manufacturer-specific detailed knowledge. There are al-

etary parameters and DTC descriptions through an Excel

ready specific extensions for well-known manufacturer for

exchange format. In an integrated test environment like

this type of test.

Vector CANoe, all these sources can be combined and used

The software update tests ensure the essential update

in one test.

features such as reliable data transfer and robust behavior

When a virtualized ECU is used in development (e.g., Vector

in the event of errors. The convenience of automation for

vVIRTUALtarget), the test setup for the application test is

them is similar to that for the diagnostic protocol tests.

comparatively simple, since hardware I/Os can be simulated easily in software.

Validation of the Diagnostic Application

The degree of possible automation during application

The diagnostic functions in the ECU must not only satisfy

tests certainly does not reach that of protocol tests; how-

formal aspects but also have correct content, of course.

ever, numerous options arise for semiautomatic or even

This is the only way to ensure sensible vehicle diagnostics.

fully automatic test generation, which it pays to take ad-

The current diagnostic formats (e.g., ODX) focus especially

vantage of.

on the definition of the protocol aspects. Nonetheless, application-relevant information can be extracted from them

Test Scope, Test Evaluation, and Further Processing

for automatic test generation: for example, an automatic

The high degree of automation for diagnostic protocol

test of the plausibility of the transferred values can be de-

validation achieved through corresponding tools permits a

rived from the description of the specified value ranges.

greater test depth through further automated tests or

Specific error causes can be derived from informal error

additional tests of the user’s own – with constant time

codes or diagnostic parameters with heuristic methods

and effort: more and more OEMs are taking advantage of

and/or manufacturer-specific knowledge. In order to be

the automation options in order to systematically safe-

able to put the insights obtained in specific automatic test

guard the diagnostic application cases, e.g. for production

runs into practice, the options for access to the (test) envi-

and service, that are especially important to them. Corre-

ronment, and thus the options for simulation of the ECU,

spondingly, the diagnostic function can be demonstrably

must be known. However, this information is usually avail-

safeguarded in early development phases already by the

able during ECU development in any case: >>The network architecture is described in AUTOSAR (.

supplier. Many of the largest OEMs around the world now rely on the

arxml) or CANdb (.dbc). Signals can be read from the

test support of CANoe.DiVa. In the process they also bene-

bus easily and manipulated through a remaining bus

fit from the comprehensive support for further processing

simulation. >>The memory layout of an ECU is described in a2l files.

of the test results, which in practice saves a great deal of time [Figure 3]:

Parameters can be read and manipulated easily via XCP. >>The electrical inputs and outputs and the test configuration are usually described in machine-readable formats. These can be measured and triggered, e.g. through a Hardware-in-the-Loop (HiL) test in combination with the Vector VT system [Figure 2].

Insert figure

Insert figure Figure 3: Practical experience: testing time and effort for manual validation compared to automated validation [5]

>> Sorting and filtering, for a targeted and needs-based view of the test resultsThe memory layout of an ECU is Figure 2: CANoe.DiVa: Software and hardware in-the-Loop

described in a2l files. Parameters can be read and manipulated easily via XCP. >> Linking errors with the same cause

03

Technical Article / 03/2017

>>Adding comments to individual test results in order to classify the error and its cause, documenting notes on corrective measures, and controlling troubleshooting. >>Transferring the error analysis results from one test run

Christph Rätz Christoph Rätz is the head of the Diagnostics product line at Vector Informatik GmbH in Stuttgart.

to another >>Connecting the test solution to existing test data management or requirement systems for integration into existing processes Conclusion/outlook

Simon Müller Simon Müller studied software engineering at the University of Stuttgart. He is the product manager responsible for CANoe. DiVa at Vector Informatik GmbH in Stuttgart in the Diagnostics product line.

The diagnostic scope of ECUs and the quality requirements will continue to increase. However, automated solutions already exist today that significantly reduce the time and effort for diagnostic validation and, at the same time, signifi-

Translation of a German publication in Hanser Automotive

cantly increase the test breadth and depth. The formally

issue 3-4/2017

described data required for this is usually available in any case. The reuse of this data for test purposes as well is sim-

Image rights: Vector Informatik GmbH

ply logical. With the increase in networking beyond the boundaries of

Literature References:

the vehicle, the visibility of the topic of automotive cyber

[1] - [3] International Organization for Standardization:

security is also increasing rapidly. “Testing security” and

ISO 14229 (UDS), ISO 14230 (KWP2000), ISO 22901 (ODX)

“testing despite security” [4] have now become topics of

[4] Metzger E.; 2016: Presentation: The Vector Security

focus in many areas of development. Expanded vehicle ac-

Solution, Vector Cyber Security Symposium 2016

cess “over the air” opens up new application domains for

[5] Peti P., Timmerberg A., Pfeffer T., Müller S., Rätz C.;

vehicle diagnostics – which must be safeguarded in turn.

2008: Automatic validation of diagnostic services, ATZ

Many innovations can only be brought into serial produc-

elektronik 06I2008

tion if they are supported in software and the central tools from early on in an user-friendly way. This is not rocket science either – just an exciting challenge for software vendors.

04