Generating Optimal Distinguishing Sequences with a Model Checker Christopher Robinson-Mallett
Tilo Mücke
University of Potsdam Hasso-Plattner-Institute Software Engineering Germany
[email protected]
University of Brunswick Institute for Programming and Reactive Systems Germany
[email protected]
Peter Liggesmeyer
Ursula Goltz
University of Kaiserslautern Fraunhofer IESE Germany
[email protected]
University of Brunswick Institute for Programming and Reactive Systems Germany
[email protected]
ABSTRACT
1. Introduction
This paper presents an approach for the automatic generation of shortest Distinguishing Sequences (DS) with the Uppaal model checker. The presented method is applicable to a large number of extended finite state machines and it will find an optimal result, if a DS sequence exists for the considered automaton. Our approach is situated in an integrated testing environment that is used to generate checking sequences. The generation method is based on a DS model, which is derived from the same test model that is used for generating test cover sets. The problem of generating DS is reduced to the definition of a DS model and for this reason the complexity of our approach depends mainly on the used model checking algorithm. This means, that the presented method is automatically improved, when the model checking algorithm is improved. This includes the generation of optimal DS depending on the ability of the model checker to produce optimal results.
An automata-based checking sequence is usually constructed from a transition cover set and a characterization sequence. A transition cover set of an automaton M, contains a specific input sequence for each transition t that starts in an initial state of M and finally executes t. A characterization set contains input sequences to allow the identification of each state by input/output-observation. Although, many effort has been spent on the construction and the automatic generation of coverage based test sequences, e.g. a transition cover set, only few publications exist on the construction of characterization sets. The use of model checking for generating test-cases was presented in a number of publications, but to our knowledge none exist on the generation of characterization sequences with a model checker.
Categories and Subject Descriptors F.1.1 automata, F.2.2 computations on discrete structures, D.2.5 testing tools D.2.4 model checking
Statechart
derive
Uppaal Test Cover Model generate
derive
Uppaal State Characterization Model generate
apply
General Terms Algorithms, Verification.
Checking Sequence
Cover Set
Characterization Set
Keywords distinguishing sequence, state identification, model checking, test generation, extended finite state machines, verification PERMISSION TO MAKE DIGITAL OR HARD COPIES OF ALL OR PART OF THIS WORK FOR PERSONAL OR CLASSROOM USE IS GRANTED WITHOUT FEE PROVIDED THAT COPIES ARE NOT MADE OR DISTRIBUTED FOR PROFIT OR COMMERCIAL ADVANTAGE AND THAT COPIES BEAR THIS NOTICE AND THE FULL CITATION ON THE FIRST PAGE. TO COPY OTHERWISE, OR REPUBLISH, TO POST ON SERVERS OR TO REDISTRIBUTE TO LISTS, REQUIRES PRIOR SPECIFIC PERMISSION AND/OR A FEE. A-MOST’05, MAY15–16, 2005, ST. LOUIS, MISSOURI, USA. COPYRIGHT 2005 ACM 1-59593-115-5/00/0004…$5.00.
Figure 1. Checking Sequence Generation Process This paper presents a novel approach for the automatic generation of shortest Distinguishing Sequences (DS) [7,12] with the Uppaal model checker [2]. The presented method is applicable to finite state machines (FSM) and extended finite state machines (EFSM), where an EFSM is extended with data, transition guards and complex data operations. Our approach produces optimal results for each minimal EFSM, for which a DS exists. It is situated in an integrated testing environment that is used to generate checking sequences for statecharts variants with Uppaal. In consequence, a Uppaal timed automata test model exists, which also can be used for DS generation. The construction of those test models is not a
topic of this work, so we will simply assume a test model to be given. The generation method is based on a state characterization model, which is derived from the test cover model. In figure 1 the process of checking sequence generation is schematically presented. A DS is an input sequence of a minimal state machine, which produces a different output for each state, in which it is executed. Therefore, a DS allows the identification of each state of an automaton just by observing its input/output behavior. A DS model is derived from a given test model to map the output behavior of the test model on data. This mapping allows the definition of liveness properties [4] which require the existence of a DS, referred to as DS properties. A DS property holds, if a path exists, which is executed from every state and leads to a state, where each output variable was different to each other output variable at least once during path execution. The mapping of the output alphabet to data values has to be accomplished in conjunction with the derivation of the test model from the statechart under test. During the derivation procedure the statechart alphabets have to be mapped to the, usually restricted, system description formalism of the used model checker. A common restriction is the existence of parameterized events in statecharts, which have to be mapped to atomic symbols in the model checker test model. In this step the output alphabet of the test model is abstracted by means of data abstraction and equivalence partitioning. It is important to note that the generation of cover and characterization sets strongly depends on this mapping. This paper is structured into seven sections, including the introduction in section 1. In section 2 we discuss related work and position our approach into the research area. In section 3 the most important basics of timed automata and Uppaal are presented. In section 4 our approach of generating DS with Uppaal is presented in detail and its complexity is discussed in section 5. In section 6 two examples for FSM and EFSM are given. In section 7 we conclude this paper and present future research topics.
2. Related Work The development of automata-theoretic testing methods was originally motivated by checking problems of sequential circuits. The adoption of these methods to software has been an important research topic for decades. A detailed overview of automatabased testing methods can be found in a number of papers, e.g. [3, 8, 11]. One of the earliest methods is the DS method [7,12]. The major advantage of the DS method is the production of relatively short checking sequences. Each of these automata-based testing methods is demanding a minimal, complete finite state-machine. Just a few publications exist on the topic of generating characterization sequences. Sun et. al. presented in [9] an efficient method for the construction of Unique Input Output (UIO) sequences [13], which are a generalization of DS, though this method is not generating optimal results. Although a UIO is a generalized form of a DS, our approach has to be extended to generate UIOs. The generation of UIOs with an extended approach is not a topic of this paper, but will be presented in the near future. In [5] Lee and Yannakakis presented a detailed study on the complexity of the construction of DS and UIO, with the negative result that both are PSPACE-complete. Of course our approach
suffers from the same structural restrictions as Lee and Yannakakis presented. Though, in our experiments it seemed applicable in a large number of cases, depending on the length of the DS and the quality of the model checking algorithm. Furthermore, the time complexity is less or equal to test case generation on the same test model. Therefore, our approach is applicable whenever test case generation with model checking is applicable, and we are quite optimistic about the practical benefit of our approach. The presented approach of generating DS is based on Computational Tree Logic (CTL) [4] and it can be easily integrated into any CTL model checking approach of test-case generation. The problem of automatically generating DS is reduced to the definition of a model, which can be used by a model checker to generate traces, which fulfill specific requirements. The complexity of our approach is mainly depending on the used model checking algorithm and it is implicitly improved, when the model checking algorithm is improved. This includes the generation of optimal DS depending on the ability of the model checker to produce optimal results. To our knowledge, there exist no other approaches for the generation of DS, and the generation of characterization sequences at all, which combine these benefits in a similar way.
3. Timed Automata A timed automaton consists of locations, transitions, and clocks [1]. The passing of time is mapped to locations, while conditions, synchronizations, and variable updates are mapped to transitions. A clock is initialized with zero and can be reset on a transition. Clocks are used for the definition of invariants and enabling timing constraints on transitions. Every transition can have a label, which is used for synchronization in a set of timed automata. At least one location of each timed automaton has to be an initial location. The semantics of a timed automaton is given in terms of a transformation to a transition system with an infinite number of states, which are tuples of locations and clock values. Beside typical transition semantics exist transitions, which uniformly increment all clocks and which are enabled if no invariant is violated.
Initial Location
Location with Invariant
Urgent Location
Committed Location
Figure 2. Uppaal Timed Automaton In 1995 the model checker UPPAAL was presented [2]. It supports an extended version of timed automata. Some of the extensions are integer variables and constants, send (!) and receive (?) synchronization, urgent and broadcast channels, and urgent and committed locations. An synchronization e is defined between a sending transition (e!) and a receiving transition (e?).
The synchronization over an urgent channel is preferred to any conflicting synchronization over a not urgent channel. A broadcast channel allows to synchronize multiple automata in one step. A transition, which leaves an urgent state, cannot be delayed and must not possess guard conditions. In a committed location time must not be passing and an outgoing transition must be taken immediately. The example in figure 2 presents an UPPAAL timed automaton, which defines locations of the four different types. The transition from the initial state is taken and clock c is set to 0. If c is greater than 3, the reflexive transition is taken, which synchronizes over channel sync1, increments variable i, and sets c to 0. If i is greater than 5 the transition to the urgent location is taken, which synchronizes over channel sync2 and sets i to 0. The transition to the committed location is urgently taken, when it receives a synchronization command over channel sync2. The transition that is leaving the committed location is immediately taken, and synchronizes over channel sync1.
4. DS Generation Method A DS of a minimal finite state machine M consists of an input sequence for the automaton and a unique output sequence for each state of M. An input sequence DSM of M, which is executed from state si, produces an output oi that characterizes si. The output oi is different to any output oj, which is produced when DSM is executed in any state sj, with i≠ j. To generate a DS for a test model M, a DS model is needed, in which for each state si an automaton Mi with the initial state si exists that is derived from M. Each automaton Mi is equipped with an instrumentation variable outi, which is initialized with –1 and contains the actual output code on an execution path of Mi. Each output of M is mapped to a unique natural number that is assigned to outi, where the output is generated. The empty output is mapped to zero. In order to guarantee the uniqueness of the input sequence, each automaton Mi must execute synchronically with every Mj of the same DS model of M. Therefore, each automaton Mi is synchronically triggered via broadcast channels by a driver automaton MD, which also observes and evaluates the output variables outi. For each pair of output variables (outi,outj), i≠ j a boolean variable distinctij is defined, which is initialized with false and set true, if on an execution path the condition outi≠ outj was at least once true. Consequently, the driver automaton evaluates after each input and every corresponding answer of the automata Mi the conditions: n −1
n
i =0
j =i +1
∏ ∏ out
i
A DS demands that each variable distinctij has become true on an execution path. Therefore, a query to generate a DS, referred to as a DS query, can be defined in Uppaal logic as follows: E dist0,1 &&
dist0,2 &&
…
&&
dist0,n
&&
dist1,2 &&
…
&&
dist1,n
…
&&
&&
distn-1,n
The Uppaal model checker is able to produce shortest traces, which fulfill a given liveness property. Therefore a shortest trace, which fulfils a DS query, is an optimal DS. The generation of a DS for an EFSM demands to take the values of the extended state variables and transition guards into account. In contrast to simple finite state-machines, path feasibility is not guaranteed. The presented method is fully applicable to EFSM, when the final extended states of all test paths in the cover set, e.g. transition cover, are considered. Additional automata are introduced to the DS model. For each configuration of the extended state variables cf in a final state sf of a test path in the cover set, an additional automaton is needed, which has the appropriate initial state sf and the initial variable configuration cf. The construction of the EFSM DS model follows the same rules as for FSM, except that the number of automata in the DS model is ruled by the number of configurations instead of the number of states.
A
FSM
b/e
a/f
b/f
B
b/e
a/e
C
a/f
D a/f Uppaal Test Model
≠ out j
After each input and the corresponding outputs, the Uppaal driver automaton evaluates these conditions and updates the distinctij variables, what is expressed in the Uppaal system description language as: n −1
n
i =0
j =i +1
∏ ∏ (distinct
ij
:= outi ≠ out j?true : distinctij )
...
&&
Figure 3. FSM Test Model
b/f
5. Complexity The complexity of CTL model checking has been discussed in a number of publications, e.g. [4]. The complexity of the presented approach of generating DS with the Uppaal model checker, is depending on the input model and the model checking algorithm. A DS model does not imply a higher time complexity than the test model, from which it is derived. Therefore, the presented approach is applicable in most cases, whenever cover set generation with a CTL model checker is applicable. Although, the considered automaton M is multiplied n times, where n is the number of states in M, and additional variables are needed, none of these modifications increase the time consumption of the model checking problem. Only the space consumption is increased, due to the enlarged state representation, which is linear to the number of additional automata in a DS model. In [10], it was demonstrated that breadth first search of a liveness property, which is likely to be fulfilled on a considerably short path, is of complexity O(nl), where n is the system’s degree of nondeterminism and l is the trace-length. The length of a shortest DS is depending only on the automaton structure itself. The nondeterministic degree is at most the size of the input alphabet for a deterministic automaton. The additional variables do not increase the time complexity, because they neither increase n nor l, but they increase space complexity through enlarging the state representation. The multiplication of the number of states by n is not increasing time complexity, because the deterministic automata are executed synchronously with the driver automaton. Therefore, the time complexity of our approach of generating DS is independent of the number of states, polynomial to the size of the input alphabet, and exponential to the length of the minimal DS. Our experiments showed that in most cases an optimal DS is of such a short length, that it can be found in acceptable time with breadth first search. Further complexity reduction can be achieved by alternative instrumentation, which indeed do not guarantee optimal results. A possible alternative instrumentation checks whether all outputs differ once in a single state along a trace. This instrumentation does not identify every DS, but it reduces the state space consumption, because the number of needed distinction variables is reduced from n2 to n.
A≠B, A≠C, A≠ D, B≠C, B≠ D, C≠ D corresponding boolean variables AB, AC, AD, BC, BD, CD are defined. The DS model driver in figure 4 is defined for the example in figure 3 and contains an initial state and an urgent state. From the initial state the transitions to the urgent state with the sending synchronizations a! and b! are enabled, and nondeterministically triggered. The driver stays in the urgent state until each transition in the DS model, which synchronizes over the channels e and f, has triggered. The transition back to the initial state is triggered, if no more enabled transitions in the automata MA, MB, MC, or MD in figure 7 exist. On this transition the values of the output conditions are evaluated and the corresponding distinction variables are redefined. Table 1. Shortest DS Example Figure 3 baa A
eff
B
eef
C
fff
D
efe
A DS exists, if the query E AB&&AC&&AD&&BC&&BD&&CD becomes true. The generation of the shortest trace with Uppaal delivers the input sequence baa, which is a shortest DS of the given example, as shown in table 1.
A
EFSM
b/e,x:=0
a/f b/f
b[x>2]/ ef,x:=0
a/e,x++
B b[x2 holds and b occurs in state B, the outputs e and f are produced, what is represented in the UIO model by an additional event ef that is coded with 3. A transition cover set for this example is given in the first column of table 2. The generation of a DS for the EFSM has to take the extended states of the test sequences into account. Therefore, the DS model contains the automata MA, MB0, MB1, MB2, and MC, which represent the extended states (A,x=0), (B,x=0), (B,x=1), (B,x=2), (C,x=0)
As mentioned before, the existence of a DS for the non-extended finite state machine part does not imply the existence of a DS for the extended states in general. Nevertheless, for the given example, there exists a DS under consideration of the extended states of the cover set, which can be generated with the presented approach. By addition of the distinction variables B0B1, B0B2, B1B2 to the driver and to the liveness property the shortest DS aabaab, presented in table 3, is generated. Table 3. DS for Extended States of Example Figure 4 Input
The corresponding driver in figure 5 evaluates the outputs of these automata and updates the variables AB0, AB1, AB2, AC, B0C, B1C, B2C. The conditions B0≠B1, B0≠B2, and B1≠B2 are not used for DS generation here, because it cannot be guaranteed that there exists a DS for the extended state-space in general. However, the aim of the application of the DS generation to the extended states in the example is not to identify each extended state, but to assure that the DS is identifying each state under the considered extended states.
State
Extended State
DS: aabaab
a
B
x=0
eeeeee
aa, b
C
x=0
eeffef
bb
A
x=0
feffef
ba
B
x=1
eeeeeef
baaab
B
x=2
eeefeee
7. Conclusion In this paper we presented an approach for the generation of optimal DS with a model checker. The presented approach can be easily integrated into any CTL model checking approach for testcase generation. The problem of generating DS is reduced to the definition of a DS model. The DS generation itself is automatically achieved by the model checking tool. For this reason the complexity of our approach is mainly depending on the used model checking algorithm. Therefore, the presented method is automatically improved, when the model checking algorithm is improved. This includes the generation of optimal DS depending on the ability of the model checker to produce optimal results. To this state we haven't taken any complexity reducing techniques, e.g. abstraction, into account. Furthermore, we know little about the practical applicability of this approach. For future research we would recommend a number of industrial and experimental case studies to evaluate the approach. A DS is a special form of a Unique Input Output Sequence, in which each state possesses the same input sequence. Our approach can be adapted to the generation of UIOs for the benefit of explicit state identification and shorter checking sequences.
Figure 6. EFSM DS Model Driver The model checker produces for the query E AB0, AB1, AB2, AC, B0C, B1C, B2C the DS ab, which produces the outputs in the extended states, presented in table 2. Table 2. DS under Extended States of Example Figure 4 Input
State
Extended State
DS: ab
a
B
x=0
ef
aa, b
C
x=0
ee
bb
A
x=0
fe
ba
B
x=1
ef
baaab
B
x=2
ef
The presented approach has shown to be identifying extended states. We are verifying only that a DS holds under the considered extended states, but our approach also can be applied to the identification of extended states. Although, it is not guaranteed that a DS exists for the considered extended states this might be of help in a number of practical applications. A similar problem occurs, when the state machine under test is not minimal or there exists no DS. In these cases, it might be helpful to construct a state machine with the identifiable states and composite states, which contain the equivalent states. The generation of such an abstraction method could be an interesting future research topic.
References [1] R. Alur, L. Dill. A theory of timed automata. Theoretical Computer Science, 126(2):183–235, 1994. [2] J. Bengtsson, K. G. Larsen, F. Larsson, P. Pettersson, W. Yi. UPPAAL - a tool suite for automatic verification of real-time systems. In Proceedings of the 4th DIMACS Workshop on Verification and Control of Hybrid Systems, 1995. [3] G. v. Bochmann, A. Petrenko. Protocol testing: review of methods and relevance for software testing. In Proceedings of the 1994 international symposium on Software testing and analysis, pages 109–124. ACM Press, 1994. [4] M. Clarke, O. Grumberg, D. A. Peled. Model Checking. MIT Press, Boston, 2000. [5] D. Lee, M. Yannakakis. Testing Finite-state Machines: State identification and verification. IEEE Transactions on Computers, 43(3), March 1994. [6] E. F. Moore. Gedanken-Experiments on Sequential Machines. Automata Studies (Annals of Mathematics Studies), (34), 1956.
[7] A.D. Friedman, P.R. Menon, Fault Detection in Digital Circuits, Prentice Hall, Englewood Cliffs, NJ, 1971 [8] Deepinder Sidhu and Ting-Kau Leung. Formal Methods for Protocol Testing: A Detailed Study. IEEE Transactions on Software Engineering, 15(4), April 1989. [9] Dechang Sun, Bapiraju Vinnakota, and Wanli Jiang. Fast State Verification. ACM Press, Proceedings of the 35th Annual Conference on Design Automation Conference, 1998. [10] Michaela Huhn Tilo Mücke. Generation of Optimized Testsuites for UML Statecharts with Time. In Robert M. Hierons Roland Groz, editor, Testing of Communicating Systems, Berlin, March 2004. Springer. [11] H. Ural. Formal methods for test sequence generation. Computer Communications, 15(5):311–325, June 1992. [12] F.C. Hennie, Fault-Detecting Experiments for Sequential Circuits, Proc. 5th Annual Symposium on Switching Circuit Theory and Logical Design, NJ, 1964 [13] K. Sabnani, A. Dahbura. A Protocol Test Generation Procedure, Computer Networks and ISDN Systems, 15, North-Holland, 1988
MB
MA
MC
MD
Figure 7. FSM DS Model
MA
MC
MB0
MB1
x=0
x=1
MB2 x=2
Figure 8. EFSM DS Model
