Automated functional test case generation from ... - Semantic Scholar

Automated functional test case generation from data flow specifications using structural coverage criteria B. Blanc1, G. Durrieu2, A. Lakehal3, O. Laurent4, B. Marre1, I. Parissis3, C. Seguin2, V. Wiels2 1: CEA List Centre de Saclay DRT/DTSI/SOL 91191 Gif sur Yvette, France 2: ONERA/DTIM 2, avenue E. Belin, BP 4025 31055 Toulouse Cedex, France 3: Laboratoire LSR-IMAG 220 rue de la chimie, BP53 38041 Grenoble Cedex 9, France 4: Airbus France 316 route de Bayonne 31000 Toulouse, France

Abstract: This paper presents a study aiming at improving the cost and the thoroughness of testing process of avionic applications developed at Airbus. The proposed approach aims at generating automatically functional tests from formal detailed specification and a functional test objective. This automatic test data generation is guided by the specification structure and the functional test objective. This approach deals with the SCADE development environment and it is illustrated on an Airbus case study. 1. Introduction Airbus France has used formal methods for several years to specify avionics systems. Thanks to these techniques, development cycles have been shortened significantly. Automatic code generation from formal specification played an essential role in this improvement. A first research project showed that using formal techniques for the specification of avionics systems makes possible the use of formal proof techniques for the validation of these systems. Now Scade Prover is a commercial tool to verify properties of a formal SCADE specification. Static validation techniques can thus be used in an operational way and this constitutes a first breakthrough with respect to the classical verification and validation process which was exclusively based on dynamic techniques such as simulation and test. However, static analysis of the SCADE specifications cannot take into account all operational conditions and some dynamic tests will remain mandatory. This industrial fact leads us to seek for improvements of the testing process. The current testing process consists in producing test sequences manually with respect to test objectives corresponding to the functional requirements of the system to be validated. Stopping criteria for tests (enough tests have been defined to validate the system) depends on the system and are not very simple to specify. The current approach

proposes to base these criteria on the structure of the SCADE specification. So, the formal definition of the criteria makes possible to fully automate the assessment of the specification coverage achieved for a given test set. Moreover, automatic test data generation ensuring the specification coverage can be performed. Indeed, the formal specification can be automatically analyzed to symbolically compute test data meeting a given criterion and a functional test objective. The paper addresses both the specification coverage assessment and the automation of the test data generation. For the latter, we have used GATeL, a test data generator for Lustre/SCADE specifications based on constrained logic programming. The combined use of specification structure-based criteria and of a test data generation tool could improve the test process in many ways. First, assessing the coverage of the specification provides the user with useful information on the thoroughness of a test set. Analysing the parts of the specification that have not been covered can help the user to define additional functional test data. Second, automatic test data generation may produce relevant execution scenarios and, hence, can reduce the cost of the testing process and improve its thoroughness. The paper is structured as follows: Section 2 briefly presents Lustre, a synchronous formal specification language that underlies graphical SCADE specifications used at Airbus. Section 3 defines the user needs and presents the motivations for the definition of structural coverage criteria. Section 4 proposes several coverage criteria adapted to Lustre specifications. In section 5 we present GATeL, the test generation tool we experimented. Section 6 explains our approach for integrating the criteria into GATeL while section 7 gives an example that illustrates all the notions presented in the paper. Section 8 concludes the paper with a discussion. 2. Overview of Lustre 2.1. LUSTRE programs

ERTS 2006 – 25-27 January 2006 – Toulouse

Page 1/10

LUSTRE [1] is a functional language widely used for the specification and programming of reactive synchronous software. The latter must satisfy the synchrony hypothesis which states that the computation of the output values is made instantaneously (in practice, this hypothesis holds if the software reaction time is short enough to take into account any evolution of the input values). Moreover, LUSTRE is a real-time data-flow based language. This means that, instead of describing the control flow of the program, as programs written in imperative languages do, a LUSTRE program describes how its input flows are transformed into output flows. Flows are the infinite sequences of values that are taken by the program variables at each computation cycle. More formally, a flow is a function from discrete time (the positive natural numbers) to the domain of the corresponding variable. A LUSTRE program is structured into nodes. A node defines output flows as functions of inputs flows. This definition is given by an unordered set of equations, possibly involving local flows. Equations are built using expressions. Each expression denotes a flow. An expression e may be an immediate constant, a variable, the point-wise application of a logical or arithmetic operator op(e1,e2,…..,en) to a n-tuple of inputs, a conditional (if c then e1 else e2), the application of the delay (pre) or the initialization (→). The language supports many other facilities such as clock sampling and projection (when, current, with,…) The two last mentioned expression constructs are specific to data flow paradigm. Delay (pre) is introduced to allow breaking data-flow loops and then defining a causally correct specification. If e is an expression denoting the sequence (e1,e2,…..,en,…), the expression pre(e) denotes the sequence (nil,e1,e2,…..,en,…) where nil is an undefined value. In other words, pre(e) returns, at time t, the value of the expression e at time (t-1). The initialization operator (→) makes possible to assign an initial value to an expression at time t=0. If e and f are expressions denoting, respectively, the sequences (e0,e1,….,en) and (f0,f1,…..fn), then (e→f) denotes the sequence (e0,f1,…..fn). An example of LUSTRE program is given in Figure 1.This program has a single Boolean input and a single Boolean output. At a given time, the output is true if and only if the input has never been true since the beginning of the program execution. For instance, the program associates the output sequence (true,true,false,false) with the input sequence (false,false,true,false).

ERTS 2006 – 25-27 January 2006 – Toulouse

Figure 1: An example of LUSTRE program LUSTRE does not provide loop operators (while, for) nor recursive calls. As a result, the execution time of a LUSTRE program can be statically computed and the satisfaction of the synchrony hypothesis can be checked. 2.2. Operator networks and paths The most usual representation for LUSTRE programs is a structure, called operator network. An operator network [2], N, is a multi-entry directed graph that consists of a set of N operators and a set E⊆NxN of directed edges between operators. Each operator represents a logical or a numerical computation. An operator is denoted by a set of ordered pairs where ei (i=1..3) is the ith input edge and s is the output edge. With each pair , will be associated a predicate denoting the condition of data flow transfer from the edge ei into the edge s. There is a one-to-one mapping between the operators of the network and the usual operators of the language. In the remaining of the paper, we will consider only network of boolean flows. We will distinguish more particularly the following subset of operators, noted in capital letters: • single-input operators: NOT, PRE, • double-input operators: AND, OR, -> (initialisation) and • triple-input operators IF-THEN-ELSE. All these operators are single-output. In an operator network, there is as many entry (respectively, exit) edges as there are input (respectively, output) variables in the associated LUSTRE program. The entry (respectively, exit) edges have no inward (respectively, outward) operator. The edges with a single inward operator and one or several outward operators are called internal edges. An edge e1 is called a successor of the edge e2 if there is an operator in the network for which e1 is one input and e2 is the output. Figure 2 shows the operator network of the above “Never” program. The graph contains a single-entry edge and a single-exit edge as well as three internal edges and four distinct operators (NOT, AND, PRE and ->).

Page 2/10

Figure 2: The operator network of “Never” program The operator network defines paths within the program. A path is a finite, possibly empty, sequence of edges, such that for all i∈[0,n-1], ei+1 is a successor of ei. An n-path is a path including n edges. An initial path is a path whose first edge is an input edge (entry). A loop is a strongly connected sub-graph of the operator network (containing delays and initialization operators). A unit path is a pair of two successive edges. A complete path is an initial path whose exit edge is an output edge. An elementary path is a path without delays. In the operator network of the Figure 2, there are two initial elementary paths ( and ). is a path with a single loop iteration. 3. Motivation for structural coverage criteria definition at specification level The current Airbus model based system validation and verification process consists in running functional tests on different simulation platforms and test benches. These tests are defined manually from system requirements and their definitions are based on the following principles: • tests are functional and classified under equivalence classes (a representative value of a class is equivalent to a test of other values of the class), •

tests involving singular points and boundary input values are defined,

•

enough tests have been defined when a predefined stopping criterion (depending on the criticality and the nature of the system) is met.

Due to the growth of avionic systems complexity, this test definition process is more and more costly and new methods and tools are searched in order to assist user in tests specification and consequently to reduce testing processeffort. But, this cost reduction shall guarantee the same level of tests quality. In particular, certification requirements shall be met. As we focus on avionic systems specified with SCADE environment, the idea is to take advantage of this formal language to automate as far as possible the testing process in two directions: • generating automatically from the SCADE specification, test vectors (with respect to

ERTS 2006 – 25-27 January 2006 – Toulouse

industrial practices defined above) that exercise system functional requirements, • defining stopping criteria for tests that can be measured by automated means. In any case, the automation of tests definition will not replace the expertise of system designer that will continue to check the validity of the test cases generated automatically against system requirements . But, the achievement of this automatic test generation and the automated measurement of the testing coverage through stopping criteria will dramatically reduce the testing process. The main difficulty of this new testing approach is on the one hand, to find out means to ease the tests generation and on the other hand, to define relevant testing stopping criteria. When testing a piece of source code, the classical stopping criteria are based on the structure of this source code, that is to say conditions and decisions. The stopping criteria are considered as met when running tests, a predefined rate of conditions and decisions have been exercised: a structural coverage analysis is thus done. The main advantage of such a structural criterion is its capability to be measured automatically and to exhibit code that cannot be executed and is not traceable to a system or software requirement (dead code). Our proposal is to transpose this source code stopping criteria up to specification level. So, the proposed stopping criteria at specification level will be based on the structure of the SCADE specification. Such coverage structural criterion will allow: • to get a measurable value to stop testing, • to generate test cases that cover all classes of behaviour that might participate in a functional objective: this criterion will guide the test generation, • to identify “dead specification” (part of the specification that cannot be reached and is not traceable to a system requirement), • to highlight parts of the specification that are not covered with a set of test cases and lead to define new functional test objectives. Henceforth, the main challenge is to formally define structural criteria for SCADE. This is the objective of the next section. 4. Coverage criteria For classical imperative languages, the top level coverage criterion – path coverage [5] which checks that each path of the control flow graph has been executed at least once – is impossible to achieve since it requires an astronomic (or infinite) number of paths to be considered. Therefore, different attempts have been made to approximate path coverage. For example, LCSAJ

Page 3/10

(Linear Code Sequence and Jump) [6] that are finite length sub-paths of increasing size have been introduced to define intermediate criteria between branch coverage and path coverage. Adequacy criteria based on data flow analysis [7], that aim at covering sub-paths between definitions and uses of the program variables play a similar role. As far as Lustre is concerned, many investigations have been conducted on test data generation procedures [8, 9, 10]. However, the definition of coverage criteria which are dedicated to operator networks (instead of control-flow graphs) has not been deeply studied [11]. Applying existing code coverage criteria [5] to LUSTRE programs is not feasible, since they are designed to deal with control flow graphs. Even if some of the intuitions motivating them are translatable to LUSTRE, new criteria suited to the data flow nature of the language are needed. In the following, we give a hierarchy of coverage criteria tailored for data flow synchronous paradigm. Detailed definitions can be found in [3]. The criteria are based on the coverage of paths of increasing length. The condition for a path to be covered is called activation condition and noted AC in the following. 4.1. Activation conditions

Definition 1. Let N be an operator network and let pn= be an n-path, and let be a unit path of N. Then the activation condition AC(pn) of the path pn= is a Boolean lustre expression inductively defined as follows:

false -> pre(AC(pn-1))

-

If NOT(ea)= eb then OC(ea,eb) = true.

-

If AND(ei,ej)= ek then OC(ei,ek)=not(ei) or ej, and OC(ej,ek)=not(ej) or ei

-

If OR(ei,ej)= ek then OC(ei,ek)=ei or not(ej), and OC(ej,ek)=ej or not(ei)

-

If IF ep THEN eq ELSE er = es then OC(ep, es)= true OC(eq, es)= ep OC(ek, es)= not(ep)

-

If ei, -> ej = ek then OC(ei,ek)= true→false OC(ej,ek)= false→true

Intuitively, the activation condition of a path is the condition under which the output computation depends on the input value. It depends on the nature of the operators it contains and also on the way the operators are composed. More details can be found in [3].Formally, the activation condition is a LUSTRE Boolean expression defined as follows.

AC(pn) = true

} and {, , } in N. Then the operator conditions OC associated to each of these unit paths are Lustre Boolean expressions defined accordingly to the operators that connect the edges as follows:

if n=0 if en=PRE(en-1)

AC(pn-1) and OC(en-1,en) otherwise where OC(en-1,en) is the operator condition of the unit path < en-1,en >. OC(en-1,en) is true when the value of the edge en depends on the value en-1 through the operator that connects these two edges. Formally, the operator condition is a LUSTRE Boolean expression defined as follows. Definition 2. Let N be an operator network and let op1, op2 and op3 be respectively occurrences of single input operator, a double input operator and a triple input operator characterised respectively by the following sets of unit paths {}, {,

ERTS 2006 – 25-27 January 2006 – Toulouse

These definitions involve two kinds of activation conditions: instantaneous and temporal. In the first case, the activation conditions depend only on the values of the input edges at the current time. This is the case for NOT, AND, OR and IF-THEN-ELSE operators. In the second case, the activation conditions depend either on the initial value of the input (->) or on the previous value of the input (PRE). For instance, the expression “OC(ei,ek)=true→false” means that the path is activated only at the initial time (t=0). Based on the definitions of activation conditions, we give, in the following, a set of coverage criteria specific to LUSTRE. These criteria provide a progressive approach to assess the thoroughness of a test data set for a LUSTRE program [1]. 4.2. n-path basic coverage The first criterion is based on a simple and intuitive notion of coverage. Informally, a test data set satisfies this criterion if it ensures the activation of all i-paths for i≤n at least once. This criterion is defined as follows. Definition 3. Let N be an operator network with k entry edges (i1, … ,ik). Let td be a test datum for N i.e. a vector of k flows (c1, …,ck).

Page 4/10

We say that td covers the activation criterion AC(p) at time t iff, when the values (c1, …,ck) of td are assigned to the inputs (i1, … ,ik), then the flow AC(p) is true at time t. We note it AC(p)[td][t]=true. Definition 4. Let N be an operator network with k entry edges (i1, … ,ik). Let TS be a test data set for N. Let Pn be the set of all complete paths whose length is less or equal to n. We say that N is covered by TS according to the npath basic coverage criterion if and only if, for each path p∈Pn, there is a test datum td ∈TS and a time t such that AC(p)[td][t]=true. This basic coverage criterion ensures only the activation of paths without considering the values of the paths entries. Indeed, for a given path whose entry is a Boolean edge, the criterion requires exercising the path with the value “true” (or “false”) but not necessarily both. However, one may need to know the outcome of the path for every value of its entry. This new requirement gives rise to a stronger criterion, for which the variation of the path entry valuation becomes mandatory. This criterion is called elementary conditions coverage [3]. 4.3. n-path elementary conditions coverage Definition 5. Let N be an operator network with k entry edges (i1, … ,ik). Let TS be a test data set for N. Let Pn be the set of all complete paths whose length is less or equal to n. We say that N is covered by TS according to the npath elementary coverage criterion if and only if, for each initial path p∈Pn, whose first edge is the entry edge inp, there are two test data td1 and td2 and two time instants t1 and t2 such that: AC(p)[td1][t1]=true and inp[td1][t1]=true and AC(p)[td2][t2]=true and inp[td2][t2]=false This means that - td1 propagates the value true through the path p whereas -td2 - propagates the value false through p. Elementary conditions criteria are comparable by means of a subsumption relation [4]. Indeed, the satisfaction of elementary conditions criterion for npaths guarantees the satisfaction of the criterion for shorter paths. 4.4. n-path multiple conditions coverage

ERTS 2006 – 25-27 January 2006 – Toulouse

With the n-path elementary criterion, a path is exercised regardless to neither internal edges values nor their various combinations. For instance, the output value of p3= depends only on the variations of e0. However, the value of e1 might depend on other edges. The current criterion does not detect such dependencies. Hence, a more powerful criterion has been proposed. In this new criterion, the path output depends on all the combinations of the edges (input edges and internal edges) values. It is called multiple conditions coverage [3]. Definition 6. Let N be an operator network with k entry edges (i1, … ,ik). Let TS be a test data set for N. Let Pn be the set of all complete paths whose length is less or equal to n. We say that N is covered by TS according to the npath multiple conditions coverage criterion if and only if, for each path p∈Pn, and for each Boolean edge ei of p, there are two test data td1 and td2 and two time instants t1 and t2 such that: AC(p)[td1][t1]=true and ej[td1][t1]=true and AC(p)[td2][t2]=true and ej[td2][t2]=false Similarly, multiple conditions coverage criteria are comparable by means of subsumption relation [4]. Indeed, the satisfaction of multiple conditions n-path criterion implies the satisfaction of the criterion for lower lengths. 5. GATeL Given a reactive program and a (partial) description of its behaviour as a Lustre model, the main role of GATeL [9] is to automatically generate test sequences according to user-oriented test cases. These test sequences can then be submitted to the program under test. In both cases where the program has been automatically generated from the model or not (depending on the development process followed), GATeL also provides a basis for an automatic oracle (expected outputs). On top of the Lustre model, GATeL can use two more user inputs: a characterization of some aspects of the environment in which the program will be run, and a declarative definition of desired test cases. This model of the environment is intended to filter out from all the possible behaviours those corresponding to realistic reactions, decreasing the state space to be explored. Un-realistic behaviours may concern incompatible values for two input flows, or for an input flow depending on the occurrence of a past event on an output flow, etc. Each filtering

Page 5/10

expression is stated by an assert directive that must be true at each cycle of the generated sequences. Requested test cases can then be finely characterized in order to exercise meaningful situations. Test cases selection is a crucial part of the testing process. Several approaches have been proposed to automate it, but none of them is universally recognized. On the contrary, we prefer to propose the user the means to define his/her own selection strategies. The first step on this direction is the definition of a test objective. The test objective states some important expected properties of the program under test to be checked. It can be either invariant properties or reachability properties. Invariant properties are stated with assert directives. The properties that must be satisfied in at least one cycle (in fact, in the last cycle of sequences built by GATeL) are stated by reach directives. To build a sequence reaching the test objective, according to the Lustre model of the program and its environment, these three elements are automatically translated into a constraint system. A resolution procedure then solves this system through an alternation of propagation and labelling phases. Propagation checks the local coherence of the system, while labelling aims at incrementally eliminate the constraints by the choice of a variable and a value within its authorized domain. The random aspect of this resolution procedure implies that the input domains are not fairly covered, thus quite distinct sequences may be generated for the same objective (for instance different ways to raise an alarm). A second step in the definition of a selection strategy is to help GATeL to distinguish these sequences. This can be achieved by splitting the constraint system so that each sub-system characterizes a particular class of behaviour reaching the objective. This splitting can be processed interactively by applying predefined decompositions of Boolean-integer-temporal operators in the Lustre expressions corresponding to the current constraint system. For instance, for a constraint "S = if Cond_i then ExpThen else ExpElse" where Cond_i is a variable at - cycle i, by unfolding of the "if" operator we can derive two subdomains. The first sub-domain includes all test sequences such that Cond_i is true, while the second sub-domain includes all test sequences such that Cond_i is false. At each unfolding step, GATeL only shows the operators that can be unfolded: toplevel operators of an expression whose evaluation is needed. A second way to proceed, more automatic, is to state declaratively the various behaviours one wants to observe through a dedicated unfoldable directive "split Var with [Cond_1...Cond_n]". When selected (depending on the visibility of the attached variable), the constraint system is split into n subsystems with each one corresponding to the assignment of one condition to true. These two

ERTS 2006 – 25-27 January 2006 – Toulouse

techniques allow the user to finely tune the kind of selection strategy needed. Finally, test submission consists in reading input sequences generated with GATeL, computing program outputs, then comparing these values to the expected ones evaluated during the generation procedure. When the program has not been automatically generated from the Lustre model, this gives an automatic oracle. On the other case, the truth value of the test objective can play the role of a partial oracle. GATeL is still under development concerning methodological and efficiency aspects. However, it has been successfully experimented on industrial case studies. 6. Approach In this section, we explain how we tuned GATeL to take into account a coverage criterion and the methodology we defined for its practical use. We could have used the criterion presented in section 4 only to evaluate coverage of a given set of generated test cases. We decided to rather use it from the beginning to guide the test generation process. This approach also allows us to experiment several slightly different versions of the criterion on several concrete examples in order to choose the one that was best adapted to our purpose. 6.1 General approach We would like to take into account, in the generation process, a coverage criterion. We are especially interested in the elementary conditions coverage defined in section 4.3. We proceed in three steps. Operator level. We first apply the basic elementary conditions coverage at the operator level. For each basic operator, we define splitting rules based on this criterion (see section 6.2) Symbol level. We then consider an intermediate level: the symbol level. Symbols are library nodes used as basic operators in the specifications of systems. We want to analyse how elementary conditions over the inputs of the symbol are propagated over the outputs without entering into the details of the internal variables of the symbol. So we define the corresponding splitting rules for two symbols that are often used (see section 6.3). These rules are defined following definition 5 and some additional reasoning when needed (to extend the previous definitions when non Boolean edges occur in the operator network of the symbol). System level. At the system level, we want to use GATeL to generate test cases that cover higher order sub-paths. If we unfold systematically all local split directives using GATeL, we obtain test cases that satisfy the multiple conditions coverage. But an additional issue is time. The question is to know whether we need to cover the multiple conditions at each cycle (and in that case for how many cycles) or

Page 6/10

only at one cycle. In section 6.4, we propose a methodology to guide the unfolding at the system level. 6.2 Splitting rules for GATeL We have implemented the - elementary conditions coverage on each basic operator of Lustre (and, or, not, if then else). In order to integrate this criterion inside GATeL, we use the “split with” directive proposed in GATeL to guide the splitting of - test cases. We have thus defined new versions of the basic operators AND, OR, NOT, IF THEN ELSE where the output is “splitted” following the cases defined in section 4.3. For example, the new AND operator is as follows: node pand (A,B: bool) returns (S: bool); let S = A and B; (*! split S with [not A, not B, A and B]!*) tel

This split directive was deduced from definitions 1, 2 and 4 with the following reasoning. We obtain two cases by applying the rules given in definition 2 with the activation condition of the AND operator on A and B: not A or B not B or A

We obtain four cases by applying the definition 4. Three cases are distinct. (not A or B) and A simplified as B and A (not A or B) and not A simplified as not A (not B or A) and B simplified as A and B (not B or A) and not B simplified as not B

tel;

To define the splitting rules for this symbol, we simply apply the rules given in the definitions of section 4 for the Npath 4 elementary conditions coverage and we obtain the following cases: er, not er es and not er, not es and not er, false -> (pre s and not er and not es), false -> (not pre s and not er and not es)

which finally give the following five cases. The second case “not er” is covered by the third and fourth cases above and can be discarded. The fourth case “not es and not er” is partially covered by the two last cases. In order to ensure its coverage at initial time we change it into “(not es and not er) -> false”. Finally we obtain the following five cases. (*! split s with [ er, es and not er, not es and not er -> false, false -> (not er and not es and pre s), false -> (not er and not es and not pre s) ] !*)

Remark: in this particular case, the obtained cases also satisfy the conditions of the multiple conditions coverage criterion-. Example: conf. The confirmer node “conf” returns as output a Boolean that is true if the Boolean input has been true for the last n cycles, n being the second (integer) input of the node. The corresponding Lustre specification is the following:

6.3 Temporal symbols Airbus specifications use a library of operators called “symbols” that define functions often used as basic blocks for the description of systems behaviour. Among these symbols, we distinguish temporal symbols such as a latch or a confirmer where time plays an important role. We decided to define splitting rules for these symbols that are essential and are often used. It resulted in a more efficient test generation for specifications that used these symbols. We will give in this section two representative examples of such temporal symbols (a set-reset latch and a confirmer) and explain how we define the splitting rules in each case. Example: srl. The Lustre specification of the set-reset latch is the following:

For this symbol, we could not directly apply the rules of section 4.3 because of the counter ct. Our method to define the cases that should be taken into account was to reason about what we thought was interesting cases and to confirm this intuition by using GATeL on the symbol itself (with parameterized versions of the basic operators). The splitting cases we have defined for this symbol are the following:

node srl(es,er: bool) returns (s: bool); let s = if er then false else if es then true else false -> pre s;

(*! split S with false -> E and false -> E and false -> E and not E] !*)

ERTS 2006 – 25-27 January 2006 – Toulouse

node conf(E: bool ; n: int ) returns (S: bool) ; var ct: int ; let ct = 0 -> if not E then 0 else pre ct + 1 ; S = ct > n ; tel

[ pre(ct=n), pre(ct>n), pre(ct