matic proof, to automatically generate test data or to derive ... take into account operational profiles. ...... Telecommunications Systems V. IOS Press, 1998.
Using CLP to Automatically Generate Test Sequences for Synchronous Programs with Numeric Inputs and Outputs∗ Besnik Seljimi, Ioannis Parissis Université Joseph Fourier - Laboratoire LSR-IMAG BP 53 - 38041 - Grenoble Cedex 9 {Besnik.Seljimi, Ioannis.Parissis}@imag.fr
Abstract
application domain is the validation of high level functional specifications of the control part of software. Lutess translates Lustre formal specifications into test data generators, which are input-output finite state machines, randomly simulating the software environment behavior. The test data generators implement several testing strategies, such as random environment simulation, operational-profile guided testing or behavioral-pattern and safety-property guided testing. These strategies aim at improving the ability of the generated test sequences to lead the software to faulty situations or to states where the software safety is challenged. The Lutess test generators feed the software under test with input data sequences. At the same time, a test oracle checks the produced inputs and outputs in order to detect software failures. The test oracle is usually built from the software safety properties and, thus, a software failure occurs when the safety properties are violated. Dealing with software handling numeric inputs and outputs has always been considered as a necessary extension of Lutess, since many synchronous real-world applications have such interfaces. In this paper we propose an approach to this extension based on constraint logic programming (CLP). CLP is a simple and efficient means to achieve this extension, without developing ad hoc constraint handling environments. We are particularly interested in two main issues of such an extension. First, introducing numeric inputs and outputs requires adapting the existing representation techniques to deal with more complex input domains. Second, transforming a Lustre specification into CLP constraints requires adapting the test generation strategies. For instance, using CLP makes possible to enhance the capability of Lutess to take into account operational profiles. Indeed, an occurrence probability can now be associated with any expression returning a boolean value (for instance, a relation between two integer expressions), as opposed to the initial version of Lutess, where only input variables could have an occurrence probability assigned.
Lutess is a testing environment designed for synchronous software specified with Lustre, a synchronous data-flow language. It makes possible to automatically generate test input sequences in conformance with a specification of the software external behavior and of guiding directives such as operational profiles and behavioral patterns. Lutess deals with software and specifications involving only boolean inputs and outputs. In this paper we propose an extension of Lutess, using Constraint Logic Programming (CLP), making possible to deal with numeric inputs and outputs. In particular, we define an appropriate execution model for test input generation and show how test data generation according to the main guiding facilities of Lutess can still be performed. Furthermore, operational profile based generation becomes more powerful thanks to the introduction of CLP solvers capabilities which make possible to associate occurrence probabilities to any boolean expression.
1 Introduction Since synchronous software is usually part of safetycritical applications, thoroughly testing is a very important but time-and-effort-consuming process. Formal specification is usually required to model the behavior of the various system components at several stages of the development process. Such a specification can be used to perform automatic proof, to automatically generate test data or to derive test oracles. Lutess is a specification-based testing tool designed to automate the test process in synchronous reactive software development. Lutess has been initially made to test software with boolean interfaces (boolean signals or events). Its main ∗ This work is supported by DANOCOPS, a project of the “Réseau National des Technologies Logicielles”, a research network funded by the French Government
1
it
i t+1
software environment
Environment description ∆
software ot
ot+1
dynamically produced input data
input data generator
Test harness
system under test
Σ
program output
oracle
verdict
Ω
trace collector
Time Communication link
Object provided by the user
Figure 1. Synchronous software operation Figure 2. The Lutess environment The paper is structured in three sections. Section 2 briefly introduces the essential concepts on testing synchronous software and presents the formal models and implementation principles of Lutess. In section 3 we present the extension of Lutess to numeric inputs and outputs. More precisely, we present the principles of translating the environment specification into an equivalent CLP representation and we discuss the generation strategies that can be applied to automatically generate test data. Section 4 provides a comparison with relevant related work.
2 Background: The Lutess testing environment 2.1
Overview
Lutess [2] is an environment for testing synchronous software. Synchronous programs have a cyclic behavior (see figure 1): at each tick of a global clock, all inputs are read and processed simultaneously, and all outputs are emitted. Lutess is based on Lustre, a synchronous declarative data-flow language [4]. Lustre offers usual arithmetic, boolean and conditional operators and two specific operators : The pre operator which refers to the "previous" value of an expression, and the "followed-by" (->) operator which is used to set the initial value of a flow. Let E and F be two expressions of the same type denoting the sequences of values (e0 , e1 , ..., en , ...) and (f0 , f1 , ..., fn , ...) - fi is the value of F at instant i; pre(E) denotes the sequence (pe0 , pe1 , ..., pen , ...) = (nil, e0 , e1 , ..., en , ...) where nil is an undefined value while E->F denotes the sequence (e0 , f1 , ..., fn , ...). Lustre is an executable specification language, which also provides the main characteristics of a linear temporal logic of the past [5]. Therefore, temporal logic formula can be easily implemented as Lustre programs. The user can define her/his own logical or temporal operators to express invariants or properties. To perform the test operation, Lutess requires three components: the software environment description (∆), the executable code of the system under test (Σ) and a test oracle
(Ω) describing the system requirements (see figure 2). The system under test and the oracle are both synchronous executable programs, whose inputs and outputs are boolean vectors. The environment description is composed of a set of properties, stated as invariants, that the environment of the system is assumed to satisfy. These invariants are Lustre boolean expressions and constrain the possible behaviors of the environment. Lutess builds a random generator from the environment description as well as a test harness which links the generator, the system under test and the oracle. Lutess coordinates their execution and records the sequences of inputs and outputs as well as the associated oracle verdicts, thanks to the trace collector. The test is operated on a single action-reaction cycle, driven by the generator. The generator randomly selects and sends an input vector to the system under test which reacts with an output vector and feeds back the generator with it. The generator proceeds by producing a new input vector and the cycle is repeated. The oracle observes the program inputs and outputs, and determines whether the software specification is violated. The testing process is stopped when the user-defined length of the test sequence is reached. The Lutess generator selection algorithm chooses a valid input vector, i.e. a vector which is in compliance with the environment description. Basically, a fair selection is performed: in each environment state, any valid input vector has the same probability to be selected. This generation mode, called random environment simulation, corresponds to random testing. In addition to it, operational profiles [10], behavioral patterns[2] and safety-property guided testing [11] are supported by Lutess.
2.2
Using Lutess
The software environment specification is included in a special Lustre node called testnode. A testnode has as inputs (resp. outputs) the outputs (resp. inputs) of the software under test. Its body mainly consists of a list of Lustre invariant environment properties (environment operator). In addition to these properties, guiding directives can be in-
testnode Env( returns (); var ; let environment(Ec1 , Ec2 , ...., Ecn ); prob(C1 , E1 , P1 ); ... prob(Cm , Em , Pm ); ; tel;
OnOff
IsOn
Tlow Tok
Cold
AC
Thigh
Idle Hot
Figure 4. The air-conditioner example Figure 3. Testnode syntax cluded in a testnode. For instance, occurrence probabilities (resulting from operational profile specification) can be assigned to input variables thanks to the prob operator. The general form of a testnode is given in figure 3. Each Eci is a Lustre boolean expression which can refer to the present values of the inputs and only to past values of the outpus. A valid input vector must satisfy ∀i = 1..n, Eci = true. The expression prob(C,E,P) means:
node AC(OnOff, Tlow, Tok, Thigh : bool) returns (IsOn, Cold, Idle, Hot : bool) let IsOn = OnOff -> pre IsOn and not(OnOff) or not(pre IsOn) and OnOff; Cold = IsOn and Thigh; Idle = IsOn and Tok; Hot = IsOn and Tlow; tel
Figure 5. Air-conditionner controller • Cold, true when cold air is issued,
• C is a condition relating to the past values of the input/output parameters,
• Idle, when no air is issued,
• E is an input variable,
• Hot, when hot air is issued.
• P is a real constant in the interval [0.0..1.0], • if the condition C holds, then the probability for E to hold is equal to P. One can consider that any environment invariant property Eci (included in the environment operator) have an occurrence probability equal to 1.0. In other words, environment(E)⇔prob(true,E,1.0).
2.3
An instance of a testnode associated with this program is given in figure 6. The environment operator defines that exactly only one among the signals Thigh, Tlow and Tok can be true at a given time and that switching from Tlow to Thigh requires passing from Tok. testnode TestAC(IsOn, Cold, Idle, Hot : bool) returns (OnOff, Tlow, Tok, Thigh : bool); let environment(#(Tlow, Tok, Thigh ), Tlow or Tok orThigh, true -> not(Thigh) or not(pre Tlow)); prob(pre Tok and pre Idle, Tok, 0.8); tel
Example
Let’s consider an air-conditioner controller (figure 4), with boolean inputs and outputs, implemented by the Lustre node of figure 5. The controller has four boolean inputs: • OnOff, true when the user pushes the OnOff button, • Tlow, true when the external temperature is lower than the temperature chosen by the user,
Figure 6. Lutess testnode example An occurrence probability has been specified: Tok has an occurrence probability of 0.8 when the air conditioner is Idle and Tok was true in the previous instant. Table 1 shows an example of generated test data.
• Tok, true when the external temperature is equal to the temperature chosen by the user,
2.4
• Thigh, true when the external temperature is higher than the temperature chosen by the user,
The expressions used in a testnode refer to past values of input, output and local variables by means of the pre operator. Assuming that a state variable sve is associated with every boolean expression pre E, the environment specification can be assimilated to a finite state machine Menv = (I, V, S, trans, env, prob) where :
and four boolean outputs: • IsOn, true when the air-conditioner is on,
Modeling the testnode as a FSM
OnOff
Tlow
Tok
Thigh
IsOn
Cold
Idle
Hot
1 1
0 1
0 0
1 0
1 0
1 0
0 0
0 0
1
1
0
0
1
0
0
1
0 0
0 0
1 1
0 0
1 1
0 0
1 1
0 0
1 1
0 0
1 0
0 1
0 1
0 1
0 0
0 0
1
1
0
0
0
0
0
0
0 1
0 1
1 0
0 0
0 1
0 0
0 0
0 1
1
0
1
0
0
0
0
0
0 1
1 1
0 0
0 0
0 1
0 0
0 0
0 1
0
1
0
0
1
0
0
1
Table 1. Generated test data for the airconditioner
• I and O are respectively the inputs and outputs of the software under test; let Vi and Vo be the associated set of values; • S is the set of states (it corresponds to the set of values of the state variables sve); • The function env : S × Vi → Bool defines the set of valid input values for a given state (values satisfying the environment operator); • The transition function t : S × Vi × Vo → S computes the next state of the environment after an actionreaction cycle. As a result, the above defined testnode (fig. 6) can be abstracted to the following FSM: • i = {OnOf f, T low, T ok, T high} • o = {IsOn, Cold, Idle, Hot} • sve = {sv0 , sv1 }, initial state sve0 = {(sv0 , sv1 )|sv0 = true} • t : sv0 = f alse, sv1 = T low • env: if sv0 then true else not(T high) or not (sv1 )
2.5
Implementation of the test generator
The FSM associated with a testnode is symbolically represented. This means that only the state variables and the t and env functions are represented, in conformance with the above equations. Such a representation avoids state explosion problems. All the input, output and state variables
of the specification are boolean, and therefore the functions involved in the testnode model are implemented as Binary Decision Diagrams (BDD) [1]. At every action-reaction cycle, the env BDD is randomly traversed to determine the next input value that the generator will issue. Moreover, the env function is represented by a labeled BDD making possible the fair generation of inputs [10]. Indeed, the number of distinct input vectors valid for a given state is computed before the test data generation. This information is added as labels in the BDD.
3 Extending Lutess 3.1
Overview
In order to take into account software with numeric inputs and outputs, we extend the syntax of the testnode to allow expressions including, in addition to the boolean operators, relational operators (=,...) and arithmetical operators (+,-,*,div,mod,...) operating on integers. Hence, any relation between integer expressions can be included in the environment operator. Such an extension raises two issues. First, a new representation must be found for the input and output domains which become more complex and BDD are not an appropriate representation. Second, the input data random selection strategies must be adapted to this new representation. Constraint Logic Programming is an attractive solution to these problems. Indeed, the input domain can be represented by a set of adequately defined constraints. On the other hand, the test data generation strategies can be defined as different solution research policies. Translating the environment specification into constraints can be performed in various ways. Our approach consists in translating the abstract model associated with a testnode (that is, a FSM). It is illustrated in sections 3.2 and 3.3. The test generation requires computing the current state and then solving the associated constraints to determine the set of solutions among which an input will be chosen. Different algorithms can be used to determine the set of solutions as well as to choose one of them. For instance, one can first solve a boolean abstraction of the numeric constraints and then compute the numeric values of the inputs (indeed, this is the choice made in the Lurette tool[13], discussed in section 4). Finally, according to the intended test data generation strategy, the way to choose a value among those satisfying the constraints may vary. Section 3.4 is concerned with these issues.
3.2
The air-conditioner revisited
3.2.1 Introducing integer inputs and outputs To illustrate the extension to numeric inputs and outputs, we modify the above presented example of the air-conditioner controller by introducing integer values to represent the temperatures involved in the system operation. The fig. 7 shows the signature of the new version of the air-conditioner controller.
testnode EnvAC(IsOn: bool; Tout: bool) returns (OnOff: bool; Tamb, Tuser : int) let environment( -- the user can choose a -- temperature between 10◦ and 40◦ Tuser >= 10 and Tuser = -20 and Tamb implies(pre IsOn and pre (Tout -Tamb) > 0, not(Tamb < pre Tamb)) , -- the temperature cannot increase -- if cold air is emitted
node AC(OnOff: bool; Tamb, Tuser : int) returns (IsOn: bool; Tout: bool)
Figure 7. The interface of the air-conditioner with integer I/O
The program has one boolean and two integer inputs :
true -> implies(pre IsOn and pre (Tout -Tamb) < 0, not(Tamb > preTamb)) ); tel;
Figure 8. Description of the environment for the air-conditioner
• OnOff is true when the user pushes the On/Off button, • Tamb is the ambient temperature expressed in Celsius degrees,
3.2.3 Translating the environment specification into a FSM
• Tuser is the temperature selected by the user, and two boolean outputs : • IsOn indicates that the air-conditioner is on, • Tout indicates the temperature of the air emitted by the air-conditioner. This program should compute, according to the difference between the ambient and the user selected temperature, the temperature of the air emitted by the air-conditioner.
The formal model associated with such an extended testnode remains a FSM. But numeric variables are now involved in the definition of the states and the transition function as well as in the expession of the invariants. Of course, this means that the potential number of environment states will be much bigger than before. But the FSM is not explicitely represented (only a symbolic representation is used) and, hence, there is no effective state explosion problem. If we introduce a state variable for each expression of the type pre E in the air-conditioner environment, we obtain the FSM defined in figure 9.
3.2.2 Specifying the environment constraints In this new version of the air-conditioner controller, it is possible to express properties on the temperature variation according to the respective values of the ambient temperature and of the issued air temperature. For instance, one may want to specify that the air-conditioner is powerful enough to change the ambient temperature. This can be expressed by means of two properties, stating that if the air emitted by the air-conditioner is hotter (resp. colder) than the ambient temperature, the latter cannot decrease (resp. increase). Moreover, we can specify that the ambient temperature remains in some realistic interval. We can write such properties with usual relational and arithmetical operators of the Lustre language. The example in fig. 8 specifies these properties within the environment operator.
sv0 : bool - distinguishes the initial state sv1 : int - pre(Tout-Tamb) sv2 : int - pre Tamb sv3 : bool - pre IsOn initial state : sv0 = true Transition function: sv0′ = f alse sv1′ = T out − T amb sv2′ = T amb sv3′ = IsOn Environment invariant: Ec1 = T user ≥ 10 and T user ≤ 40 Ec2 = T amb ≥ −20 and T amb ≤ 60 Ec3 = sv0 or (not(sv3 and sv1 > 0) or not(T amb < sv2 )) Ec4 = sv0 or (not(sv3 and sv1 < 0) or not(T amb > sv2 ))
Figure 9. the Finite state machine of the example
3.3
environment(CurrState,Inputs) :CurrState=[Sv0,Sv1,Sv2,Sv3 ], Inputs=[OnOff,Tamb,Tuser], Tuser #>= 10 and Tuser #=< 40, Tamb #>= -20 and Tamb #=< 60,
Translating the environment into constraints
3.3.1 Definitions A constraint satisfaction problem (CSP) is defined by a set of variables, each one taking values in a finite domain, and a set of constraints holding on these variables. Each constraint can be viewed as a mathematical relation defining the allowed combinations of values for a subset of the problem variables. An assignment of all the variables satisfying the set of constraints, is called a solution of the CSP. Standard algorithms for resolution of a constraint system combine filtering (removing from the variable domain the values that obviously cannot be part of the solution) and enumeration (assigning arbitrary values to variables from their domain). 3.3.2 Environment specification as CSP Actually, the environment constraints define the set of allowed values for the input parameters, depending on the current state. In other words, the environment specification is a relation between the software input variables and the state variables. We propose to model the problem of satisfying the environment invariant properties as a constraint satisfaction problem. Modeling the environment consists in defining the constraints between the state variables svet and the input variables. Each variable is strongly typed and will have the initial domain set to its type. The state variables are computed through the transition function and, hence, their value is known at each generation step. In our implementation, the constraint system is generated from the environment description file as ECLiPSe Prolog predicates. We use the ic solver with finite domain constraints. More precisely, the finite state machine is represented by three predicates that are automatically generated: • initial_state(S0) : sets the current state to the initial state, • transition(CurrState, I, O, NextState) : defines the transition function i.e. the constraints between the next and the current state, according to the values of the input/output variables, • environment(CurrState, I) : defines the environment constraints. For instance, if we consider the Finite State Machine of figure 9, we will generate the environment/2 predicate of figure 10. The test sequence generator performs a loop starting at the initial state. At each step of the loop, the environment
or(Sv0, neg(Sv3 and Sv1 #> 0) or neg(Tamb #< Sv2)), or(Sv0, neg(Sv3 and Sv1 #< 0) or neg(Tamb #> Sv2)).
Figure 10. environment/2 predicate constraints are posted. Any solution of the system of constraints is a valid input vector. Hence, generating a test input vector requires finding a solution of the constraint system. At each step, one solution between all the possible solutions is chosen. The selection methods determine the test strategies and are discussed in 3.4. Once a solution has been found, the input vector is submitted to the SUT. The latter produces an output which is used to compute the next state, and so on.
3.4
Test data selection methods
Once all constraints have been posted, the obtained constraint system represents all the properties that inputs should satisfy at a given clock tick. Each solution of the constraint system, represents an input vector which is compliant with the environment description. Usually, there will be more than one solution to the constraint system. The way of determining which of the solutions will be picked up defines a testing strategy. In the following we focus on random test data generation and operational profile guided generation. 3.4.1 Random test data generation Random testing consists in randomly selecting, at any clock tick, an input vector among the solutions of the constraint system. The boolean version of Lutess ensures a fair random selection of inputs: at a given state, all the valid input vector values are considered with the same occurrence probability. Such a fair random generation has been possible since only boolean variables were involved in the generation process. Moreover, the BDD representation of the environment constraints make possible to compute the number of valid input vectors for any state. When numeric constraints are involved, such a fair generation is much more difficult to achieve. Indeed, random selection consists in assigning a random value to variables in their domain. In a system of constraints with potentially more than one solution, the total number of different solutions cannot be known, unless we extensively enumerate them all. Therefore, it is hard to randomly choose between all possible vectors in an fair way.
OnOff
Tamb
Tuser
IsOn
Tout
0 0
43 9
40 36
0 0
39 45
1
30
27
1
26
1 1
56 -19
31 22
0 1
23 35
1 0
-2 7
35 38
0 0
47 48
0
15
22
0
24
1 1
-3 5
40 33
1 0
54 42
1
-14
38
1
55
1 0
23 -12
33 17
0 0
36 26
1
5
32
1
41
Table 2. Generated test data - version 1 Moreover, when the size of the variable domains are very different, fairness is not always meaningful. Let’s consider, for instance, the following constraint: b ∈ (0..1) ∧ x ∈ (0..232 − 1) ∧ (b ⇒ x = 0). If a fair random selection among all the solutions was made, the probability to get b = 1, x = 0 would be negligible ( 2321+1 ), even if this solution is obviously of a certain interest. A solution could be to assign values to variables in their domain, starting from variables the domain of which is the smallest. This would ensure a fair selection for boolean variables, which is important in control-command systems where the logic of the system is mainly implemented by boolean signals. Table 2 shows a test sequence generated using the environment description in figure 8. 3.4.2 Guiding by means of probabilities Operational profile definition [9] is a useful and powerful tool to guide test data selection. In Lutess, this facility is integrated as conditional probability definition. In short, a software input is set to true according to a user-defined conditional probability. The latter is ignored if the associated condition is false. Extending Lutess with CLP make possible to enhance this feature by allowing conditional probability associations with any boolean expression. Let’s consider the figure 11. The previous example of the air-conditioner environment specification has been modified: some of the invariant properties are now specified as expressions that hold with some probability. Also, probabilities have been added in order to have longer sequences with a working airconditioner (IsOn = true). The associated constraints are: 1. Ec1 = T user ≥ 10 and T user ≤ 40 2. Ec2 = T amb ≥ −20 and T amb ≤ 60
testnode EnvAC(IsOn: bool; Tout: bool) returns (OnOff: bool; Tamb, Tuser : int) let environment( -- the user can choose a -- temperature between 10◦ and 40◦ Tuser >= 10 and Tuser = -20 and Tamb pre IsOn and pre (Tout -Tamb)>0, true -> Tamb < pre Tamb, 0.1 ); -- if cold air is emitted, t -- tthe ambient temperature hardly inreases prob( false -> pre IsOn and pre (Tout -Tamb) Tamb > pre Tamb, 0.1 ); -- High probability to press the OnOff button -- when the air-conditioner is not On prob( false -> not(pre IsOn), true -> OnOff, 0.9 ); -- Low probability to press the OnOff button -- when the air-conditioner is On prob( false -> pre IsOn, true -> OnOff, 0.1 ); tel;
Figure 11. Specifying occurrence probabilities for expressions
3. C1 = not(sv0 ) and (sv3 and sv1 > 0), E1 = sv0 or T amb < sv2 , P1 = 0.1 4. C2 = not(sv0 ) and (sv3 and sv1 < 0), E2 = sv0 or T amb > sv2 , P2 = 0.1 5. C3 = not(sv0 ) and not(sv3 ), E3 = sv0 or OnOf f , P3 = 0.9 6. C4 = not(sv0 ) and sv3 , E4 = sv0 or OnOf f > sv2 , P4 = 0.1 In order to satisfy a list of probability associations prob(C1 , E1 , P1 ), ..., prob(Cn , En , Pn ), the test data generator will randomly draw n real values x1 , . . . , xn in the interval (0.0..1.0). Then, for each i it will post the constraint Ci ⇒ (xi ≤ Pi ⇔ Ei ), in addition to the environment constraints. As a result, during the constraint resolution, the Ei will be taken in conideration only if Ci is true and if xi ≤ Pi (that is, with a probability of Pi ). As for the boolean version of Lutess, no consistency check is made on the probability definitions. This means that the user can, in fact, specify an operational profile which is impossible to satisfy. For instance, prob(pre Tlow, Thigh, 1.0) is inconsistent if it is added to the testnode of figure 6, since an invariant property prevents Thigh from being set to true when pre Tlow is true. Thus, it may happen that the whole constraint system cannot be satisfied. In this case, there are no input values for the software under test that satisfy the environment specification and we can consider that the probability specification is inconsistent. Depending on the user’s point of view, we can consider two different cases:
OnOff
Tamb
Tuser
IsOn
Tout
0 1
19 53
32 23
0 1
36 13
0
38
40
1
40
0 1
41 41
25 20
1 0
20 13
1 0
58 30
30 28
1 1
21 28
1
27
38
0
41
1 0
9 12
34 19
1 1
42 21
0
14
31
1
36
0 0
32 41
34 20
1 1
34 13
0
55
26
1
17
Table 3. Generated test data - version 2 • The user requires the test generation to strictly comply with the probability specification. In that case, the test data generator should stop the test operation and the tester should be requested to modify the assigned probabilities. • By specifying probabilities, the user intends to guide the generation and does not care if they are always verified. In this case, at each step, the generator should try to satisfy as much constraints as possible in order to generate input values. One simple way (without overhead) of implementing this kind of guided generation is to use the backtracking mechanisms provided by logic programming languages. The main idea is to create a choice point for each posted probability constraint. When the generator fails to satisfy all the specified probabilities, it backtracks in order to search solutions that satisfy the remaining constraints. The order in which constraints are posted is important, because the lastly posted probability constraints are rejected at first, and, hence, the earlier a probability is posted the more important it is. Determining the order of the probabilities should be a user task. Table 3 shows test data generated with respect to probabilities specified in figure 11.
3.5
Concluding remarks
3.5.1 Other test data selection methods Existing testing strategies in Lutess (such as safetyproperty-guided testing) could also be adapted to the extended version of the tool. But other methods for test data selection can be designed, in particular to track faults related to the use of numeric inputs and outputs (for instance
consisting in selecting values near the boundaries of the input variable domains). 3.5.2 Float variables In our implementation, we use a typical interval solver on finite domain variables. Even if in this paper we consider only integer values, the approach still hold when introducing float variables, provided that an adequate float solver is given. 3.5.3 Scalability of the approach When generating a test sequence, a new independent set of constraints is posted at each step. The number of posted constraints, as well as the number of the problem variables, is the same at each step and is syntactically related to the originating description file. Thus, the memory used to store the constraints does not depend on the generated sequence length (the computing time is linear). The time needed to generate an input vector for one step is less predictable, since it depends on the type of constraints. More significant case studies should be carried out to assess this complexity.
4 Related work Constraint Logic Programming [6] has often been used to test data generation purposes, and for testing reactive systems [12]. In particular, a CLP-based tool has been proposed for Lustre specification testing: Gatel [8] is a test data generation tool translating a Lustre program and the specification of its environment in a equivalent Prolog representation and, then, computing a test input satisfying a userdefined test objective. Gatel has been successfully used to test realistic programs. The main drawback of this tool is that, to reach a test objective, it unfolds the constraints associated with the Lustre program and the test objective: according to the latter, this leads, sometimes, to a memory explosion. The user must interact with the resolution process to avoid such explosions. As opposed to Gatel, in Lutess the Lustre source code of the program under test is not involved in the test data generation process. Furthermore, the representation of the environment specification we use is an abstract model of the underlying behavior, and not a representation of the program flows, as in Gatel. As a result, the number of constraints needed to generate tests does not depend, for most of Lutess strategies, on the test objective nor the sequence length. Another tool (not based on CLP) dedicated to the test of Lustre programs is Lurette[13]. Lurette makes possible to test Lustre programs with numeric input and outputs. A boolean abstraction of the environment constraints is first
built: any constraint consisting of a relation between numeric expressions is assimilated to a single boolean variable in this abstraction. The concrete numeric expressions are handled by an ad hoc environment dedicated to linear arithmetic expressions. The generation process first assigns a value to the variables of the boolean abstraction and, then, tries to solve the corresponding equations to determine the values of the numeric variables. The main difference between Lurette and the new extended version of Lutess is that we use CLP instead of an adhoc resolution environment restricted to linear expressions. It is then possible to use the power of the CLP environments to express occurrence probabilities as well as other guiding directives in a very simple and efficient way. Moreover, the Lurette resolution principle is based on a boolean abstraction of the constraints: this is not the case in Lutess.
5 Conclusion CLP offers an attractive and efficient solution to the test data generation for synchronous programs with numeric inputs and outputs. Even if the scalability of the approach should be confirmed on case studies, CLP-based test tools [12][8] seem to be rather scalable in many cases. The above presented selection methods have been integrated in the Lutess environment and other testing strategies integration is currently under study. Future work also includes studying the applicability and usefulness of the proposed extensions to other kinds of software. Indeed, Lutess is mainly based on a model of the environment in which a software component is running and not on the implementation of the component itself. Thus, the approach can be used to generate test data for non synchronous systems provided that a synchronous communication with the component can be made (it has been shown that Lutess can be used in testing feature interaction in telecommunications systems [3] as well as in testing multimodal interactive systems [7]).
References [1] S.B. Akers. Binary Decision Diagrams. IEEE Transactions on Computers, C-27:509–516, june 1978. [2] L. du Bousquet, F. Ouabdesselam, I. Parissis, J.-L. Richier, and N. Zuanon. Lutess : a testing environment for synchronous software. In Tool Support for System Specification, Development and Verification, pages 48–61. Advances in Computer Science. Springer Verlag, June 1998. [3] L. du Bousquet, F. Ouabdesselam, J.-L. Richier, and N. Zuanon. Incremental feature validation : a syn-
chronous point of view. In Feature Interactions in Telecommunications Systems V. IOS Press, 1998. [4] N. Halbwachs, P. Caspi, P. Raymond, and D. Pilaud. The synchronous data flow programming language lustre. Proceedings of the IEEE, 79(9):1305–1320, 1991. [5] N. Halbwachs, F. Lagnier, and C. Ratel. Programming and verifying real-time systems by means of the synchronous data-flow language lustre. IEEE Trans. Software Eng., 18(9):785–793, 1992. [6] Lassez J.-L. Jaffar J. Constraint logic programming. In 14th. ACM Symposium on Principles of Programming Languages (POPL’87), pages 111–119, 1987. [7] Laya Madani, Catherine Oriat, Ioannis Parissis, Jullien Bouchet, and Laurence Nigay. Synchronous testing of multimodal systems: An operational profilebased approach. In ISSRE, pages 325–334, 2005. [8] Bruno Marre and Agnès Arnould. Test sequences generation from lustre descriptions: Gatel. In ASE, pages 229–, 2000. [9] J. Musa. Operational Profiles in Software-Reliability Engineering. IEEE Software, pages 14–32, 1993. [10] F. Ouabdesselam and I. Parissis. Constructing operational profiles for synchronous critical software. In 6th International Symposium on Software Reliability Engineering, pages 286–293, Toulouse, France, october 1995. [11] I. Parissis and J. Vassy. Strategies for automated specification-based testing of synchronous software. In 16th International Conference on Automated Software Engineering, San Diego, USA, November 2001. IEEE. [12] H. Pretschner A., Lötzbeyer. Model based-testing with constraint logic programming: First results and challenges. In 2nd ICSE Workshop on Automated Program Analysis, Testing and Verification, pages 1–9, 2001. [13] Pascal Raymond, Xavier Nicollin, Nicolas Halbwachs, and Daniel Weber. Automatic testing of reactive systems. In IEEE Real-Time Systems Symposium, pages 200–209, 1998.
