tion of safety properties on LUSTRE programs. 1. Introduction. Over the last several years, a wide variety of software techniques has been defined; their ...
Testing Techniques for Data-Flow Synchronous Programs* Farid Ouabdesselam and Ioannis Parissis Laboratoire de Génie Informatique - Institut IMAG BP 53 38041 Grenoble Cedex 9 France Abstract Three approaches to the problem of testing synchronous data-flow programs written in LUSTRE are presented. LUSTRE is a language well-adapted to both the specification and the development of reactive software. The first approach automatically transforms a set of LUSTRE invariant properties characterizing the environment of the reactive program into a constrained random generator of test data sequences. The second approach consists in analyzing the required safety properties (written in LUSTRE) of the software. This analysis may result, in specific cases, in automatic generators of relevant test data. The third approach considers that LUSTRE is used for the implementation of the software, and defines structure-based testing criteria. These criteria are specific to the model which captures the LUSTRE program structure : the operator net. Moreover, an automatic test data generation process is described for this last approach using LESAR, a tool designed to automatically prove the satisfaction of safety properties on LUSTRE programs.
1. Introduction Over the last several years, a wide variety of software techniques has been defined; their properties have been studied and tools have been built based on some of them. However, almost none of these strategies concern software developed using declarative or applicative languages. In addition, just a few works have been devoted to reactive software which is singled out by a continuous reaction to its environment and by the need to consider many behavioral aspects, including not only functional but also timing and safety requirements. So, designing testing techniques which are adequate to reactive software coded in a declarative and functional language has been rarely tackled. This paper describes an approach to these topics in the unified framework of LUSTRE. For critical software, the requirements engineering phase usually ends in a formal description [BS93] including functional features (what the software must do) but also safety properties (what the software must not do). When such a description is available, a formal proof of the safety properties can be carried-out. The most commonly used approach is model-checking [CES86]. The software is portrayed as a logical model, generally a state graph. Every state of the graph corresponds to a distinct value (or set of values) of the software variables while arcs denote actions leading from a state to another. Functional and safety specifications are represented as logical formulae. If a formula is true in the model it is assumed that the associated specification holds in the software. One reason for the development of this proof technique is that model-checking can be automated for some temporal logics. However, it is well known that model-checking (and more generally, formal proof) often needs prohibitive memory and time amounts. Moreover, the model may be a simplified representation of the original software. As a result, when the formula is false in the model it cannot be assumed that the specification does not hold in the software. Testing is another usual approach to software validation. Clearly, when proving a program is impossible, testing remains the only applicable technique involving software execution. However it could be useful to test a software even after its formal verification. Indeed, the latter is very often concerned with only a small set of properties which must hold on the software. Hence, errors not related to these properties may be undetectable by formal proof. Testing can then be valuable for the detection of such errors. We present here two different specification-based approaches to the test of critical software. First, we focus on testing safety properties satisfaction. Next, we address the problem of testing functional features. Moreover, we present a method to automatically create random generators of test data which are valid w.r.t environment constraints. These approaches should allow to detect implementation errors, when the specification has been errone* This is a revised and extended version of the paper entitled “Testing Synchronous Critical Software” presented at the 5th International Symposium on Software Reliability Engineering.
ously coded, but also design errors which are less frequent but of a higher severity [LE92]. The importance of such specification-based approaches is stressed in [ROT89, Ric92]. Testing reactive software, a challenging problem of practical and theoretical significance, is also related to debugging for several reasons. Both processes require the observation of the program execution over a series of cycles (i.e. traces of computation steps); furthermore, both processes are not interested in complete executions but rather in sequences of input/output pairs. Building automatically a means to compare the actual program behavior with the expected one is an important issue : if every process does not determine whether the program behaves correctly, there is nothing to be gained by performing it. Finally, the functional and declarative style of L USTRE makes the associated testing techniques look like partially to debugging techniques. However, unlike debugging strategies, our testing techniques do not address the problem of error localization. The paper is structured in five main sections. Section 2 is a short presentation of the L USTRE language and its use for specifying critical software. In section 3 we present an example of critical system on which are illustrated the testing techniques presented in sections 4, 5 and 6. Section 7 provides a short discussion on the relation between testing of data-flow synchronous software and debugging strategies.
2. Specifying Synchronous Critical Software We assume that the critical software is specified by means of the synchronous data-flow language L USTRE. It has been shown that this language can be used for both programming synchronous software [CHPP87] (i.e. specifying functional features) and expressing safety properties [HPOG89, OGHP89].
2.1. The LUSTRE language A detailed presentation of the language can be found in [CHPP87]. The presentation of LUSTRE is restricted herein to the elements necessary for understanding the paper. A LUSTRE program is structured into nodes : a node is a subprogram specifying a relation between its input and output parameters. This relation is expressed by un unordered set of equations, possibly involving local variables. An equation X = E, where E is a LUSTRE expression specifies that the variable X is always equal to E. A variable is intended to be a function of time which is assumed to be isomorphic to the set of natural numbers. In other words, a variable denotes the sequence of values that it takes. Expressions are made of variable identifiers, constants (considered as constant functions), usual arithmetic, boolean and conditional operators (considered as pointwisely applying to functions) and only two specific operators: the “previous” operator and the “followed-by” operator: • If E is a LUSTRE expression denoting the sequence of values (e0, e1, ..., en, ....), then pre(E) denotes the sequence (nil, e0, e1, ..., en, ...) where nil is an undefined value. • If E and F are expressions of the same type, denoting the sequences (e0, e1, ..., en, ...) and (f0, f1, ..., fn, ...), then E -> F denotes the sequence (e0, f1, ..., fn, ...). Once declared, a node may be instantiated in any expression, as a basic operator. For instance, the following declaration defines a node, of general usage, which returns true whenever its boolean parameter raises from false to true : node Edge(X: bool) returns(EDGE: bool); let EDGE = X -> (X and not pre(X)); tel Then, the expression Edge(not C) is true whenever the variable C has a falling edge. The full functional specification of a synchronous system can be performed with LUSTRE (see for example [HPOG89]).
2.2. LUSTRE for the expression of safety properties Safety properties of critical software specify that dangerous situations will never occur. Actually, one does not care whether an alarm eventually follows a dangerous situation (liveness properties), but rather whether it occurs within a given delay. Therefore, some temporal constructions expressing safety properties have been identified [RRSV87] such as always A from B to C (resp. once A from B to C) requiring property A to continuously hold (resp. to hold at least once) between the instants where properties B and C occur. It has been shown [Glo89] that LUSTRE, which can be viewed as a temporal logic [PH88], allows the expression of the most commonly used safety properties. To do so, a LUSTRE node NP is associated with every safety property P such as P is true if and only if NP returns a true value during every program execution. A detailed presentation of how such nodes NP are written
C A
B
do_AB
do_BC grant_exit
grant_access
UMS on_A on_C
on_B ack_BC ack_AB
Figure 1: The UMS system and its environment can be found in [HPOG89]. For example, for any L USTRE boolean expressions A, B and C, the safety property always A from B to C is implemented by the node Always_from_to(A, B, C) returning a true value if and only if any occurrence of the event B causes the condition A to be continuously true until the next occurrence of the event C.
2.3. Critical software and environment constraints A main feature of critical software is their reactive behavior : they continuously receive information from their environment and issue orders to it. Software testing consists in simulating the environment in order to observe the system’s reaction and, therefore, only valid environment behaviors must be considered. Let’s consider, for instance, an elevator management system. Clearly, it makes no sense to observe its behavior when the elevator is simultaneously stopped at two different floors or when it jumps from the first to the third floor without reaching in the meantime the second floor. However, specifying all environment valid behaviors may be quite complex, incomplete and even erroneous, since the environment is a possibly very complex system [LE92]. Nevertheless, many incorrect behaviors may be removed if a small set of invariant environment properties are considered and if testing is performed while these properties hold. However, if the software specification includes reactions to some erroneous environment behaviors in order to improve the system’s reliability, these behaviors must not be ignored during testing.
3. An example of critical software We illustrate the use of LUSTRE as a language for expressing functional and safety features and environment invariant properties on a subway device. This device is an automatic “U-turn” section management system (later on called UMS), allowing trains, at each terminus of the subway line, to switch from one track to the other and go back to the opposite direction. This example has been presented for the first time in [HRR91] where it has been used to illustrate how LUSTRE programs can be formally verified. We believe that presenting our approach on the same example should allow a better comprehension of the testing techniques as well as a comparison between them and the formal verification technique. The system specification is divided in three parts, respectively presented in sections 3.1, 3.2 and 3.3. The first part is concerned with the software functional specifications, that is the description of what the software must do. The next part describes the safety properties of the software which define behaviors that must never occur (what the software must not do). The last part comprises the environment constraints upon which the software is built.
3.1. Functional specification of the process-control software The U-turn section (see figure 1) is composed of three tracks A, B, C and a switch S. Assuming the entering track is A and the exiting track is C, trains switching from A to C must first wait for S to connect A with B, then transit on B and wait again for S to connect B with C before going back on C. Since UMS must both drive the switch and manage trains movement in order to avoid accidents into the section, its behavior is typically reactive : upon receipt of informations about the U-section configuration (i.e. switch status and trains position inside the section) it should deliver positioning requests to the switch and access grants to trains. These four kinds of events can be modelled by the following signals : • ack_AB and ack_BC are emitted by the switch to indicate whether it actually connects A with B or B with C. When none of these signals is active (i.e. when the actual connection is changing), trains must not take the switch.
• on_A, on_B and on_C are emitted by three sensors, one on each track of the section. They are active as long as there is a train on their respective track. • do_AB and do_BC, emitted by UMS, are the requests for the switch to connect A with B or B with C. • grant_access and grant_exit, also emitted by UMS, are grants for trains to move along the section (i.e traffic lights). The first will allow trains to access the section only if it is empty and the switch connects A and B. The second will allow trains to exit from B only if the switch connects B with C. The above specifications are formally expressed by the LUSTRE node presented in figure 2 : node UMS (on_A, on_B, on_C, ack_AB, ack_BC: bool) returns (grant_access, grant_exit, do_AB, do_BC: bool); var empty_section, only_on_B: bool; let empty_section = not (on_A or on_B or on_C); only_on_B = on_B and not (on_A or on_C); grant_access = empty_section and ack_AB; grant_exit = only_on_B and ack_BC; do_AB = empty_section and not ack_AB; do_BC = only_on_B and not ack_BC; tel; Figure 2: LUSTRE implementation of the UMS control software
3.2. Safety properties of the process-control software We formally express here some safety properties of the UMS system (variables empty_section and only_on_B are assumed to be defined as in the above node, UMS) : • The access is granted to a train only if the section is empty : (1) Implies(grant_access, empty_section) where the node Implies implements the usual implication. • The switch positioning requests, do_AB and do_BC should never be simultaneously active : (2) not (do_AB and do_BC) • The switch must always connect A to B (resp. B to C) from the instant when a train is allowed to enter (resp. to exit) the section until it has arrived on track B (resp. it has actually left it) : (3) Always_from_to(ack_AB, grant_access, only_on_B) (4) Always_from_to(ack_BC, grant_exit, empty_section)
3.3. Environment constraints The following LUSTRE expressions are invariant properties of the U-turn section : • The switch cannot simultaneously connect the track B with both tracks A and C : not (ack_AB and ack_BC) • Once in a given position, the switch remain stable unless it is requested to move to the opposite position : Always_from_to_(ack_AB, ack_AB, do_BC) and Always_from_to_(ack_BC, ack_BC, do_AB) Other properties can also be considered in order to ensure that trains respect traffic lights, for example. They are omitted here for sake of simplicity.
4. Random test data generation in a constrained environment Let’s turn now to the testing problem of the critical software having a reactive behavior. An input vector is the data associated with the input variables at a given execution cycle (or instant). Since the temporal nature of the specifications involves several successive execution cycles, testing their validity requires the generated test data to be sequences of input vectors. Before presenting each of the two testing approaches (i.e. testing safety properties satisfaction and testing functional specifications) we suggest a method of automatically creating random generators which produce valid test data with respect to environment constraints. In fact, we believe that it is important to provide a means to test the software by a valid simulated environment. As it has been shown in section 2.3, the invariant properties of the environment restrict the sequences of events
((1,1),(0,φ)) ((0,1),(0,0)) ((φ,1),(1,0))
((φ,0),(0,0)) ((φ,1),(ψ,0)) ((1,φ),(0,1))
((φ,0),(1,0))
0
2
((φ,0),(1,0))
((0,φ),(0,1)) ((1,φ),(0,1))
((1,0),(1,0))
((1,1),(φ,0))
((0,1),(0,1))
((1,0),(0,0))
1 ((0,φ),(0,1))
Figure 4: Valid input-output sequences recognizing automaton it can emit (i.e. the program’s input domain) : only few input sequences are associated with valid environment behaviors w.r.t. the constraints. We present here a random generation process of valid test data. A simple method to do so consists in generating random data without taking into account the environment constraints and then filtering them in order to select only those which are valid. However, such a method usually leads to the generation of a big amount of invalid sequences before producing a valid one. For instance, the probability of generating a valid 20 vector long sequence for the UMS system’s environment is lower than 10 -8. This remark has led us to design a random generator of which all the produced data are valid, w.r.t. environment constraints. In order to avoid symbolic evaluation - which is usually complex - a dynamic test data generation must be performed : random program inputs should be generated in order to preserve the validity of the environment constraints w.r.t. the previous software inputs and outputs (see figure 1). Such test data generators can be automatically produced providing that environment constraints are written in LUSTRE. Then, the construction process consists of the three following steps, illustrated on the UMS environment :
Random Generator
Ii
Software
Oi Figure 3: Using a constrained random generator • Writing environment constraints as LUSTRE expressions. For instance, let’s consider the following invariant properties of the UMS system’s environment (we assume, for simplicity, that there is no other environment constraints) : not (ack_AB and ack_BC) Always_from_to_(ack_AB, ack_AB, do_BC) Always_from_to_(ack_BC, ack_BC, do_AB) • Compiling these invariant properties produces a deterministic automaton recognizing all valid system’s input and output sequences (in the same way that a finite acceptor recognizes words from a language). Each transition of this automaton is labelled by a pair (O; I) : a sequence (Oi, Ii)i=0..n has been recognized by the automaton (in other words, a sequence of n+1 consecutive transitions with labels (Oi, Ii), for i=0..n has been executed) when, due to the emission from the environment of the sequence (Ii)i=0..n the system has emitted the sequence (Oi)i=0..n (obviously, Ii and Oi are emitted at the same instant) while the above properties are holding. For simplicity, let’s consider that the only signals emitted from the environment are ack_AB and ack_BC and also that only the signals do_AB and do_BC are emitted from the system. The compilation of the environment constraints results in a 3-states automaton. This automaton is shown in figure 4 where φ and ψ correspond to any boolean value and where boolean values are denoted by 0 (for false) and 1 (for true). The transitions are labeled by the values of the pair ((do_AB, do_BC), (ack_AB, ack_BC)). The environment’s current state is 2
(resp. 1) when the connection AB (resp. BC) has become effective and do_BC (resp. do_AB) has never been active since then. The current state is 0 for any other valid configuration of the environment. • Transforming the above automaton in order to obtain the actual test data generator. Input sequences I will then be generated with respect to the output sequences O. To do so, the semantics of the transition label are modified : a transition (Oi, Ii) is executed when the system has emitted the output value Oi; then, the automaton generates the value Ii which is the new input for the system and so on. The resulting automaton is clearly nondeterministic, since several transition labels may match a same output value. So, we transform it in a probabilistic automaton by associating with each transition (Oi, Ii) an execution probability. For instance, when the current state of the automaton shown in figure 4 is 0, there are three possible transitions for the output value (0, 0) consisting in either staying at state 0 and generating the value (0, 0) either switching to state 1 and generating the value (0, 1) or switching to state 2 and generating the value (1, 0). Hence, an execution probability must be associated with the above transitions. Note that the current features of the LUSTRE compilation process allow the application of the above method to safety properties of which the expression uses only boolean variables. Actually, since safety properties very often involve control features of the software, represented by boolean signals, the method can be applied to several cases. However, in order to make the application field of the method larger, an extension of the compiler is necessary allowing to take into account integer and real values. Such an extension using numeric intervals is currently under study. Of course, the use of numeric values will undoubtedly increase the size of the resulting automata. It may then be necessary to ignore some environment constraints. It must also be kept in mind that, generally, the use of only invariant constraints cannot remove all the impossible environment behaviors. In other words, some of the data sequences produced by a random generator built as it is shown above may correspond to behaviors that the environment could never have. Our goal here - as it has been defined in section 2.3 - is to give a simple method to automatically generate valid test data with respect to a small set of environment invariant constraints, and not to build a complete environment simulator. Such a simulator would require a much more complete specification.
5. Testing safety properties satisfaction 5.1. Test oracle for safety properties The test data generation technique presented in section 4 does not take into account the problem of detecting incorrect results during the actual test operation. This quite hard task is so repeatedly performed by the tester that an automatic means to detect errors while testing the software has always been sought for : this is the well-known problem of the test oracle. When the aim of the test oracle is to detect safety properties violation and if the safety properties are expressed using LUSTRE, then the test oracle can be written in a straightforward manner as a LUSTRE node (see figure 2). Its inputs are the inputs and outputs of the software. Its single boolean output has the value of the conjunction of the safety properties. An error is detected during testing if the output of the above node becomes false. node UMS_ORACLE(on_A, on_B, on_C, ack_AB, ack_BC, grant_access, grant_exit, do_AB, do_BC : bool) returns (ok : bool) var
no_collision, exclusive_req, no_derail_AB, no_derail_BC, empty_section, only_on_B : bool;
let empty_section = not (on_A or on_B or on_C); only_on_B = on_B and not(on_A or on_C); no_collision = Implies(grant_access, empty_section); exclusive_req = not(do_AB and do_BC); no_derail_AB = Always_from_to(ack_AB,grant_access , only_on_B); no_derail_BC = Always_from_to(ack_BC, grant_exit, empty_section); ok = no_collision and exclusive_req and on_derail_BC and no_derail_AB; tel; Figure 5: A test oracle for the UMS control software
5.2. Specification-based testing As for the construction of the test oracle, we assume that a LUSTRE description of the safety properties that the software must satisfy is provided. The aim of the specification-based testing is to analyze these properties and to automatically generate relevant input values (that is, input values which are more appropriate to detect violation of safety properties). In order to illustrate the definition of relevant input values, let’s consider the simple property A ⇒ B, where A is a software input and B is a software output, stating that the output B must be true every time that the input A is true. It can be easily noted that input values setting A to false cannot detect a violation of the property, since in that case A ⇒ B will be true for any B. In other words, only input values for which A is true are relevant with respect to that property. A similar, more complex but automated, analysis can be carried out on every LUSTRE boolean expression E corresponding to a safety property. This is true in particular for temporal properties (e.g always_from_to used in section 3.2). This analysis results in a new boolean expression R(E) defined on the software input variables. This expression is true when the input variables take values which are relevant with respect to the property E. It is then possible to automatically generate relevant input values with respect to a property E. The generation process is similar to the one used for constrained random testing (see section 4). Indeed, the latter consists in building a generator of input values according to a set of environment invariant properties, while the former builds a generator according to the expression R(E) characterizing the relevant input values for the property E. Another possible use of such a definition of relevant input values is the adequacy assessment of a given set of input values used for testing a software. Indeed, it is very easy to write a LUSTRE program counting the number of input values for which the formula characterizing the relevant input values has been true. At the end of the testing process, the ratio between the relevant input values and the total input values can be a useful measure of the quality of the testing process.
6. Structural-like testing of LUSTRE programs or specifications The approaches presented in sections 4 and 5 can be applied to reactive software coded in any programming language. Indeed, although they both use LUSTRE for the expression of the safety properties or for the description of the environment, they do not require the use of a particular programming language for the software implementation. Such black-box testing techniques are generally used for the detection of design errors resulting in a violation of the specifications (which are safety properties in the case of the technique of the section 5). On the contrary, structural testing techniques are useful for detecting implementation errors. We consider that specifying functional features with LUSTRE is a kind of programming activity resulting in nodes which are not exempt from errors. Moreover, L USTRE has been successfully used to implement reactive software. Hence, it is natural to devise structure-based techniques to the particular case of LUSTRE nodes. Control graphs constitute the most common representation to capture the flow of control through programs written in sequential programming languages. Every control graph node consists of a sequence of successive instructions without branching statements; the latter are associated with arcs. Such a graph is a suitable representation of the control flow of the program. On this graph, various testing criteria - like path testing or branch testing to name a few - are defined (see for example [Nta88]). These criteria form a hierarchy based on their effectiveness in covering the various program and program model components. Due to the data-flow nature of the language, LUSTRE nodes are represented by an operator net instead of a control graph. For instance, the operator net associated with the UMS node is shown in figure 4. Actually, a node is an unordered set of equations defining the data flow of the program as invariant relations between inputs and outputs. In these circumstances, the classical control flow graph and data dependency graph are not appropriate models on which to base white-box testing strategies. The operator net is a more natural representation of the program structure. We believe that defining structure-based criteria on such a net, by analogy with those defined on a control graph, could be useful for detecting implementation errors in nodes. Thanks to the formal definition of the structure of an operator net given in section 6.1, structure-based criteria can be defined. In section 6.2 it is shown how test data sequences satisfying such criteria can be automatically generated.
empty_section do_AB
ack_AB a1 on_A
a2 grant_access
on_B a3
on_C
only_on_B a5
grant_exit
a4
do_BC ack_BC a6
Figure 6: Operator net of the UMS node
6.1. Definitions Operator net : A path is defined as a finite sequence of successive arcs from the net. Its first arc is a net input and its last arc is a net output. For example, (ack_AB, a1, do_AB) and (on_B, a2, a3, empty_section, do_AB) are paths defined on the net of the UMS node. A path predicate at instant t associated with a path p, noted PP(p, t), is the condition for the last arc to be computed at instant t using the path p. In other words, it is the condition for the path p to be traversed by the data-flow when its last arc is traversed at instant t. Examples of path predicates defined on the UMS net are : PP((ack_AB, a1, do_AB), t) = empty_section, PP((on_B, a2, a3, empty_section, do_AB), t) = not on_A and not on_C and a1. For instance, the first path predicate requires empty_section to take a true value because in the opposite case the value of do_AB, which is the last arc of the path (ack_AB, a1, do_AB), would be false; in that case, the computation of the value of do_AB does not use the path (ack_AB, a1, do_AB) (i.e. it does not depend on the current values of the arcs of the path). Similarly, an arc predicate AP(ai, ai+1, t) is defined for two successive arcs of a path. It is the condition for the arc ai+1 to be computed at instant t using the arc ai. For instance, AP(a4, a5, t) = true while AP(empty_section, do_AB, t) = a1. The construction of a path predicate PP at an instant t for a path p = (ai)i=1...n is achieved by the conjunction of the arc predicates computed separately for each pair of its arcs (aj, aj+1), j ∈ [1, n-1]. Thus, a path p is traversed by the data-flow at a given instant t if and only if its path predicate PP(p, t) is satisfied at this instant t. We use the LUSTRE language to formally define path predicates. Consider, for instance, the node Edge presented in section 2.1 and the associated operator net shown in figure 4. Three paths are defined on this net :
X
pre
a1 a2
a3
->
EDGE
Figure 7: Operator net of the node Edge p1 = (X, EDGE), p2 = (X, a1, a2, a3, EDGE) and p3 = (X, a3, EDGE) where a1 = pre(X), a2 = not a1 and a3 = a2 and X. Consider the variable init = true -> false. In other words, init always takes a false value except at the first execution cycle. Then, the path predicates of p1, p2 and p3 are : PP(p1, t) = PP((X, EDGE), t) = init,
PP(p2, t) = PP((X, a1, a2, a3, EDGE), t) = X and not init PP(p3, t) = PP((X, a3, EDGE), t) = not pre(X) and not init. According to the first predicate, the path p1 can be traversed only at the first execution cycle. On the contrary the paths p2 and p3 can be traversed at any cycle except the first one. Moreover, for p2 (resp. p3) to be traversed, X (resp. not pre(X)) must hold. Formally, the path predicate PP(p,t) of a path p = (ai)i=1...n is computed as follows : PP(p,t) = andi=1...n-1 AP(ai, ai+1, ti), if n > 1, true, if n = 1, where tn-1 = t, ti = ti+1 - 1 if ai+1 = pre ai and ti = ti+1 otherwise. (I) Let n > 0 and let E be any L USTRE expression. We define the function pre as follows: pre 0 (E) = E and pren(E) = pre(pren-1(E)) if n > 0. In other words, pren(E) denotes the LUSTRE expression obtained by n successive applications of the pre operator to the expression E. Arc predicates are computed according to the following rules : 1. If ai+1 is the output of a unary boolean operator or any arithmetic operator, then AP(ai, ai+1, s) = true, 2. if ai+1= pre ai, then AP(ai, ai+1, s) = true. Indeed, the computation of the output value of an operator among those mentioned in the rules 1 and 2 always depends on the value of all its inputs. On the contrary, the output of an if then else operator is equal to one of its outputs depending on the value of the boolean condition : 3. if ai+1 = if ai1 then ai2 else ai3, then AP(ai1, ai+1, s) = true, AP(ai2, ai+1, s) = pret-s(ai1) and AP(ai3, ai+1, s) = pret-s(not ai1). Similar rules are defined for and and or operators : 4. if ai+1= ai1 and ai2 then AP(ai1 , ai+1, s) = pret-s(ai2) and AP(ai2, ai+1, s) = pret-s(ai1) 5. if ai+1 = ai1 or ai2 then AP(ai1, ai+1, s) = pret-s(not ai2) and AP(ai2, ai+1, s) = pret-s(not ai1). Finally, the output of a “followed-by” operator is always equal to its second input except at the first execution cycle : 6. if ai+1 = ai1 -> ai2 then AP(ai1, ai+1, s) = pret-s(init) and AP(ai2, ai+1, s) = pret-s(not init) Example : Let’s use the above rules to compute the path predicates of the paths p1, p2 and p3. 1. According to (I) and the rule 6 : PP(p1, t) = PP((X, EDGE), t) = AP((X, EDGE), t) = init 2. According to (I) : PP(p2, t) = PP((X, a1, a2, a3, EDGE), t) = AP(X, a1, t-1) and AP(a1, a2, t) and AP(a2, a3, t) and AP(a3, EDGE, t). According to the rule 2, AP(X, a1, t-1) = true; according to the rule 1, AP(a1, a2, t) = true; according to the rule 4, AP(a2, a3, t) = X; according to the rule 6, AP(a3, EDGE, t) = not init. Hence, PP(p2, t) = X and not init. 3. According to (I) : PP(p3, t) = PP((X, a3, EDGE), t) = AP(X, a3, t) and AP(a3, EDGE, t). According to the rule 4, AP(X, a3, t) = a2 = not a1 = not pre(X); according to the rule 6, AP(a3, EDGE, t) = not init. Hence, PP(p3, t) = not pre(X) and not init. Definition of test data selection criteria : We can define now structure-based test data selection criteria. For instance the operator coverage is satisfied by a test data set if at least an output arc of each operator belongs to one of the resulting traversed paths (i.e the operator has been activated and its results has been used for computing the path’s output). Similarly, arc coverage is satisfied if every arc of the operator net belongs at least to one resulting traversed path. Path coverage is generally impossible to satisfy since the number of paths may be infinite. However, the number of paths is finite for a fixed length of the test data sequences. Note that paths are defined in such a manner that errors occurring during program execution will be propagated to the output values, even several execution cycles after their effective occurrence.
in
a3
a2 pre
if
->
out
a1
Figure 8: Operator net of the node on_off Similarly to traditional structural testing criteria, path coverage subsumes arc coverage which itself subsumes operator coverage.
6.2. Structural testing Once structural test data selection criteria has been defined, we suggest a method for computing test data in order to satisfy these criteria. This computation performs a symbolic evaluation of the associated path predicates. The method is based on previous work made on formal verification of L USTRE programs. A verification tool, LESAR [HLR92], has been developed, allowing to automatically prove that a property always holds on a LUSTRE program. An interesting feature of LESAR is that it provides a counter-example for properties which do not hold (i.e. an input sequence leading the program to a state violating the property). The method comprises three steps : 1. Computing a finite set of paths which must be executed in order to satisfy the criterion. Let (pi) i=1...n be these paths. 2. Computing the path predicates (PP(pi, ti))i=1...n associated with the above paths according to the section 6.1 where ti has a fixed value for every path. 3. Then, consider the following LUSTRE equations : X1 = true; and for every n > 1, Xn = false -> Xn-1 and pre(Xn-1). In other words a variable Xn denotes a sequence of boolean values of which the n-1 first terms are equal to false while all the other terms are equal to true. Then, the third step consists in attempting to prove with LESAR that not (PP(pi, ti0) and Xti0) always hold, where ti0 is the length of the required test data sequence. In case of success, the path pi is infeasible at instant ti0. Otherwise the input sequence that LESAR provides is executed and the path pi is covered at instant ti0. The use of the variable Xti0 causes LESAR to generate a sequence of a minimum length (equal to ti0). Indeed, thanks to the definition of those variables, a property not (P and Xn) will always be true during the first n-1 execution cycles. Hence, the counter-example provided by LESAR, if any, will be a sequence of which the length is greater than n-1. If such a variable is not used, the minimum length of the generated sequence will be equal to 1. It must be kept in mind that the verification tool LESAR is used here in a quite different manner than for its original purpose which is formal verification. Indeed, when a formal verification is performed, the entire automaton (i.e. all possible states) must be explored in order to prove that properties hold at every state. On the contrary, the operation of test data generation consists in searching a counter-example of the negation of the path predicate. This is usually a very short operation (unless, of course, the path predicate is infeasible). Thus, there is no contradiction in the use of a formal verification tool for testing purposes.
6.3. Examples 1. The path coverage of the node Edge need the paths p1, p2 and p3 to be executed (step 1). The associated path predicates have been computed (step 2) in section 6.1. The path predicate of p1 is satisfied by any input sequence. Therefore (step 3), we ask LESAR to prove not (PP(p2, 1) and X1) = not (X and not init and X1) where X1 = true. The resulting input sequence is (false, true) and covers the path p2 at instant t = 2 (obviously, the path is infeasible for t = 1). Similarly, we ask LESAR to prove not (PP(p3, 1) and X1) = not (not pre(X) and not init and X1). The resulting input sequence, (false, false), covers p3 at instant t = 2 (note that (false, true) covers both p2 and p3). 2. Let’s consider the following node : node on_off(in: bool) returns (out: bool); let out = in -> if in then not pre(out) else pre(out); tel The boolean output value of this node changes every time that its input is true. The associated net is presented in figure 8, on which, clearly, an infinite number of paths can be defined : p11 = (in, out) and p12 = (in, a3, out)
which can be covered in a single execution cycle, p21 = (in, out, a1, a3, out), p22 = (in, out, a1, a2, a3, out), p23 = (in, a3, out, a1, a3, out), p24 = (in, a3, out, a1, a2, a3, out) requiring two successive execution cycles to be covered and so on. For instance, arc coverage is satisfied if the path p22 = (in, out, a1, a2, a3, out) is traversed. Its path predicate is computed as follows : According to (I) : PP(p22, t) = PP((in, out, a1, a2, a3, out), t) = AP((in, out), t-1) and AP((out, a1), t-1) and AP((a1, a2), t) and AP((a2, a3), t) and AP((a3, out), t). According to the rule 6, AP((in, out), t-1) = pre1(init) = pre(init); according to the rule 2, AP((out, a1), t-1) = true; according to the rule 1, AP((a1, a2), t) = true; according to the rule 3, AP((a2, a3), t) = in; according to the rule 6, AP((a3, out), t) = not init. Hence, PP(p22, t) = in and not init and pre(init). LESAR generates for this predicate the input sequence (false, true) which covers the path p22.
7. Conclusion and debugging issues We have presented in this paper three approaches to testing critical software. Their application not only detect inconsistencies between specification and actual code, but also help check the specification completeness and consistency. First, a constrained random test data generation method has been described which allows for environment constraints to be taken into account. The synchronous declarative language LUSTRE has been used for the formal specification of the environment invariant properties. Compiling these properties results in a constrained random test data generator. Next, a specification-based black-box testing method has been presented. According to this method, the formal expression of safety properties is automatically analyzed in order to identify relevant test data (i.e. test data sequences for which properties can be false). The automatic generation of such data is possible in some specific cases. Furthermore, we have shown how a test oracle can be very easily created. Finally, a structure-based testing approach has been defined on the operator net associated with any description which can be viewed either as a functional specifications of the software to be implemented in an other language, or as a program. Structure coverage criteria, traditionally defined on control graphs, such as instruction, arc or path coverage are easily adapted on this net. Other criteria [Nta88, Wey88, CPRZ89] could also be used with this particular representation of the functional specifications. For reasons of brevity and clarity, a simple example has been selected to show how to test reactive software. Actually, the same illustration has been used in [HRR91] to exemplify formal verification. Therefore, it does not serve the purpose of pointing out the differences between the two approaches. This should not lead the reader to the conclusion that, in general, testing is redundant with verification. Testing is a necessity for real applications developed in LUSTRE since most of them can not be formally verified because of the huge size of the state graph. The implementation of the above methods is in progress; it will allow us to confirm some of the advantages of testing over formal verification. Each of the testing techniques which have been described in the preceding paragraphs share a few characteristics with some automated debugging strategies as classified in [Duc93]. In the latter article, concrete evaluation of assertions and consistency checking with respect to assertions are among the most practical and promising verification-oriented techniques. This ensues from the “trend to rely on user-defined assertions which are more declarative and less sensitive to procedural differences”. In addition, the advantage of these techniques is that the specification can be partial and incomplete; consequently, the strategies are much more practical and can be applied to real programs. It is interesting to note the similarities between these techniques and the ones detailed in this paper. Like the debugging strategies, data-flow synchronous program testing requires that an oracle be available : the oracle is derived from assertions which represent the intended program behavior. The assertions allow a modular testing. Execution information must be collected from the run-time environment by a test monitor. Testing sequential systems usually does not require a sophisticated monitor as all execution information can be gleaned from system outcome. But test monitors for reactive systems, by the very nature of their behavior, must capture additional information such as internal events, stimuli and responses. A similar need appears in the validation activity for distributed systems : trace checkers are used to observe the execution of the system under test and to check whether the resulting sequences of events are possible outcomes according to the given specification. Obviously, finite acceptors as the environment simulator and the oracles are examples of observers. However, synchronous observers only operate on a global state and do not have to cope with the problem of non-deterministic interleaving of processes since communication among processes is totally symmetric.
References [BS93] J.P. Bowen and V. Stavridou. Safety-Critical Systems, Formal Methods and Standards. IEE/BCS Software Engineering Journal, 8(4):189–209, July 1993. [CES86] E. Clarke, E. Emerson, and E. Sistla. Automatic Verification of Finite State Concurrent Systems using Temporal Logic Specifications : A Practical Approach. ACM Transactions on Programming Languages and Systems, 8(2):244–263, 1986. [CHPP87] P. Caspi, N. Halbwachs, D. Pilaud, and J. Plaice. LUSTRE, a declarative language for programming synchronous systems. In 14th Symposium on Principles of Programming Languages (POPL 87), Munich, pages 178–188. ACM, 1987. [CPRZ89] L. Clarke, A. Podgurski, D. Richardson, and S. Zeil. A formal evaluation of data flow path selection criteria. IEEE Transactions on Software Engineering, pages 1318–1331, November 1989. [Duc93] M. Ducassé. A pragmatic survey of automated debugging. In Automated and Algorithmic Debugging. First International Workshop, Linkoping, Sweden, May 1993. Springer Verlag. [Glo89] A-C. Glory. Vérification de propriétés de programmes flots de données synchrones. Thesis, Université Joseph Fourier, Grenoble, France, December 1989. [HLR92] N. Halbwachs, F. Lagnier, and C. Ratel. Programming and Verifying Real-Time Systems by Means of the Synchronous Data-Flow Programming Language LUSTRE. IEEE Transactions on Software Engineering, Special Issue on the Specification and Analysis of Real-Time Systems, pages 785–793, September 1992. [HPOG89] N. Halbwachs, D. Pilaud, F. Ouabdesselam, and A-C. Glory. Specifying, Programming and Verifying Real-Time Systems, using a synchronous declarative language. In Workshop on automatic verification methods for finite state systems, LNCS 407, Grenoble, France, June 1989. Springer Verlag. [HRR91] N. Halbwachs, P. Raymond, and C. Ratel. Programming and Verifying Critical Systems by Means of the Synchronous Data-Flow Programming Language LUSTRE. In Conference on Softwrare for Critical Systems, pages 112–119. ACM SIGSOFT, 1991. [LE92] J.C. Laprie (Ed.). Dependability : Basic Concepts and Terminology. Springer-Verlag L.N.C.S, 1992. [Nta88] S. Ntafos. A comparison of some structural testing strategies. IEEE Transactions on Software Engineering, pages 868–874, June 1988. [OGHP89] F. Ouabdesselam, A-C. Glory, N. Halbwachs, and D. Pilaud. Verification of Data-Flow Synchronous Programs. In 2nd International Symposium on Lucid and Intentional Programming, Tempe, May 1989. [PH88] D. Pilaud and N. Halbwachs. From a synchronous declarative language to a temporal logic dealing with multiform time. In Symposium on Formal Techniques in Real Time and Fault Tolerant Systems, Warwick, September 1988. Springer Verlag. [Ric92] D. Richardson. Specification-based Test Oracles for Reactive Systems. In Proc. of the 14th Int’l Conf.on Software Engineering, pages 105–118, Melbourne, Australia, May 1992. [ROT89] D. Richardson, O. O’Malley, and C. Tittle. Approaches to Specification-Based Testing. In Proc. of the ACM/SIGSOFT’89(TAV3), pages 86–96, December 1989. [RRSV87] J. L. Richier, C. Rodriguez, J. Sifakis, and J. Voiron. Verification in Xesar of the Sliding Window Protocol. In Proc 7th International Symposium on Protocol Specification, Testing, and Verification, Zurich, Switzerland, May 1987. [Wey88] E. Weyuker. The evaluation of program-based software test data adequacy criteria. Communications of the ACM, pages 668–675, June 1988.
