Automated Boundary Test Generation from JML Specifications

Automated Boundary Test Generation from JML Specifications Fabrice Bouquet, Frédéric Dadeau, and Bruno Legeard Laboratoire d’Informatique (LIFC) Université de Franche-Comté, CNRS - INRIA 16, route de Gray - 25030 Besan¸con cedex, France {bouquet, dadeau, legeard}@lifc.univ-fcomte.fr

Abstract. We present an original approach for the automated computation of model-based test cases from specifications written in Java Modeling Language (JML). We aim at activating all the behaviors from the JML method specifications. Therefore, we extract each behavior and we compute the pertinent test data for the input parameters; we select the boundary values of the ordered domains, and we consider specific features for the objects, involving inheritance and aliasing. Finally, a preamble for each test case is computed by symbolic animation of the JML specification using constraint logic programming techniques. Thus, we are able to automatically generate executable Java test sequences to be run on the system under test. Moreover, this process requires the less possible intervention from a validation engineer. Keywords: Test generation, model-based, Java Modeling Language, automated, boundary values.

1

Introduction

Model-based testing (MBT) [21] has become an efficient way for validating an implementation. While the program is being developed, based on informal requirements, the formal model is written, validated and verified. Tests are then derived from the model and run on the system under test (SUT). Different kinds of testing can be performed. In particular, conformance testing aims at observing the responses of the SUT w.r.t. a specification-compliant use of this latter. If the program is correct, then the test should succeed. On the other hand, robustness testing consists in observing the responses of the SUT w.r.t. an incorrect use of the system. These non-nominal cases also have to be specified in the model. The Java Modeling Language (JML) [12,13] is an assertion language for Java, that can be used either to design a formal model or to strengthen the code with assertions. The main advantage of JML is that it makes it possible to provide both model and code in the same file, sharing the same class attributes and methods. This is a very interesting point since one important problem in MBT

This work has been realized within the GECCOO project of program “ACI Sécurité Informatique” supported by the French Ministry of Research and New Technologies.

J. Misra, T. Nipkow, and E. Sekerinski (Eds.): FM 2006, LNCS 4085, pp. 428–443, 2006. c Springer-Verlag Berlin Heidelberg 2006

Automated Boundary Test Generation from JML Specifications

429

is to “connect” the variables of the specification with the variables of the SUT. In object-oriented testing, a key issue is to compute the oracle, which is the expected result of the test case. Thanks to its assertions JML provides a natural oracle for the Java programs. Indeed, the JML Run-Time Assertion Checker [7] compiles JML-annotated source to a Java byte-code containing on-the-fly checking of the JML assertions. A previous work [3] has presented a way to express an interesting subset of the JML specifications within a set-theoretic framework, so that we were able to perform a symbolic animation of the JML specification. Using underlying constraints solvers, we represent object system states as constraints systems. From there, we simulate the invocation of methods by solving a constraint satisfaction problem involving the state before, the before-after predicates that describes the behavior of the method, and the state after. We present in this paper the direct application of this work, which is the generation of boundary test cases applied both to conformance and robustness testing. This approach has been applied on model-based test generation from B machines within the BZ-Testing-Tools project [1]. The symbolic representation of the system is employed to compute the test data, and the symbolic animation engine is employed to compute the preamble of the test cases. The main contributions of the paper are the following. We introduce an approach to model-based testing for Java based on the JML specifications. We use model coverage for selecting our tests involving structural coverage of the specification and data coverage using a boundary analysis for numerical data. This is, to our knowledge, a novelty for Java/JML. This approach is fully model-based, and aims at generating automatically functional test cases, i.e., black-box test cases, by using the specification both as an oracle, and as a support for computing the test data. The paper is organized as follows. Section 2 introduces the Java Modeling Language, and presents an example to illustrate its principles, it also describes the symbolic representation of a JML-specified system, introducing the notion of underlying constraint stores. Section 3 presents the test selection we apply to JML, decomposed into two parts: the structural coverage of the JML specifications, detailed in Section 4, and the test data computation, explained in Section 5. Section 6 details the generation of the test cases, by computing the preamble. The implementation of our approach and an experimental result on a case study is given in Section 7. Section 8 presents the related work on modelbased test generation for Java and states on the originality of our approach. Finally, Section 9 concludes and announces the future work.

2

Java Modeling Language

This section presents the Java Modeling Language [13] and introduces an example that will be used throughout the remainder of the paper to illustrate our approach. Then, we describe the symbolic representation of an object-oriented system using underlying constraint solvers. This representation will later on be used for the computation of the tests.

430

2.1

F. Bouquet, F. Dadeau, and B. Legeard

Overview of JML

The Java Modeling Language is the specification language dedicated to the Java programming language. The model is expressed by assertions embedded within the Java code. These assertions describe the specification associated to the class, in terms of static or dynamic properties. The static properties are the invariant (invariant clause), and the history constraints (constraints clause), which are applied to the entire class. On the contrary, the dynamic properties are applied to the methods, through method specifications, and they describe the behavior of the method using a pre- and post-condition semantics. The method specification may contain several clauses, such as the preconditions (requires clause), the frame condition (assignable clauses), the normal post-condition (ensures clause) and the exceptional post-conditions (signals clauses). Since JML is based on the Design By Contract principles [17], the pre-condition represents the contract that the system has to fulfill for the method to be executed. In this case, the normal post-condition describes a property that is established when the method terminates normally, i.e., without throwing an exception. The exceptional post-conditions state the properties that are established when the method terminates by throwing a specified exception. Apart from that, the frame condition indicates which fields are modified during by the execution of the method. Method clauses are gathered within method behaviors, separated by also. The syntax of the JML predicates is similar to the Java syntax, enriched with special keywords, beginning with a \, such as \result which represents the return value of a method, or \not_modified(X) whose meaning is obvious. The JML clauses are written using these first-order logic predicates. The history constraints and the postconditions are written using before-after predicates in which it is possible to refer to the before value of an expression expr by surrounding it by \old(expr). Figure 1 illustrates the use of JML through an example. The specification describes an electronic purse (Purse class), that is extended by a limitation (LimitedPurse class) which requires the balance of the purse to be limited by a given value max. It is possible to add money to the purse (credit(short) method) or to remove money from it (debit(short) method). Moreover, it is possible to transfer the content of a purse (transfer(Purse) method) by copying its balance. 2.2

Symbolic Representation of Object-Oriented Systems

Our symbolic representation of object-oriented and especially Java/JML systems, involves the use of underlying constraint solvers. They manage a constraint store, involving symbolic variables that are used to designate different elements of the system, such as the instances, the value of the attributes, etc. Thus, this representation relies on an solver on finite domain integers, CLP(FD), and a set-theoretic solver, named CLPS-BZ [4], and part of the BZ-Testing-Tools project [1]. This latter is able to manage constraints on sets, functions and relations. This section summarizes our previous work presented in [3].


431

Fig. 1. Example of a JML specification

In our approach, an object-oriented system state is defined by: (i) a set of existing instances, whose dynamic types are classes, and (ii) each instance possesses attributes, whose values are either a built-in type value (such as integers, characters, etc.), or an object-typed value (i.e., another instance). Definition 1 (Symbolic States). We represent a symbolic states as a pair composed of an environment and an associated constraint system Cs (V ), that manages constraints on the environment’s variables. An environment is defined by a set of specification variables identified by a module name M and a data name N , mapped to a kind K (input, output, constant, variable, prime), a type T , and a set of variables V that occurs in Cs (V ). Cs (V ), M × N → V × K × T

(1)

We define a special module, named model that describes the heap, and stores the dynamic type of the different instances. The heap is a set of atoms, containing memory addresses represented as atoms null,addr0,addr1,. . .,addrN where N is a user-defined number. The set of addresses that is used is stored in set variable, named instances, constrained to be a subset of the heap. The dynamic type of the instances is known by a function that maps the created instance to

432


a corresponding class name. Each class is considered as a module. Each module has a variable named instances that represents the set of created instances of this class. Each class attribute is a module variable. If this attribute is not static, its value depends on the considered instance and so, it is represented as a total function –a set of pairs– mapping the instance to its value. If the attribute is static, its value is directly given. Example 1 (Symbolic representation of the system states). Consider the classes presented in the example in Fig. 1. The symbolic state representing all the possible configurations of the system is defined by: {A = {null, addr0, . . . , addrN}, B ⊆ A, null ∈ B, C ⊆ B, D ∈ B → {Purse, LimitedPurse}, E ⊆ C, F ∈ E → −32768..32767, G ⊆ E, H ∈ −32768..32767}, {(model, heap) → (A, constant, set(atom)), (model, instances) → (B, variable, set(atom)), (model, accessible) → (C, variable, set(atom)), (model, typeof) → (D, variable, set(atom)), (Purse, instances) → (E, variable, set(atom)), (Purse, balance) → (F, variable, set(pair(atom,int))), (LimitedPurse, instances) → (G, variable, set(atom)), (LimitedPurse, max) → (H, variable, int)}

where A, B, C, D, E, F, G and H are environment variables on which constraints are applied within the store. From this representation, we are able to perform the symbolic animation of the JML specification, by considering the predicates extracted from the method specifications and translated in our internal representation’s syntax. Thus, executing a method for animating the specification is equivalent to solving a constraint satisfaction problem between two symbolic states. More details about it can be found in [3].

3

Test Selection Criteria for JML

The test selection criteria defines what motivates the way we build our test cases. It can be either a test case specification provided by a validation engineer, such as in TOBIAS [14] or STG [8], a stochastic/probabilistic approach as in Jartege [19], or a model coverage criteria, which is our choice for the JML specifications. This criteria focuses on producing the tests by exploiting the informations contained in the specification at two levels. The first level is the structural coverage, composed by the transition coverage, and the data coverage. The transition coverage aims at activating the behaviors of the system, and the decision coverage aims at covering the decisions within the predicates describing the behaviors. The second level is the data coverage, achieved, in our case, by performing the boundary analysis of the data w.r.t. the behaviors. Figure 2 summarizes the process of our approach. A JML model is analyzed and partitioned w.r.t. model coverage criteria, selected by the validation engineer and automatically applied on the specification. This produces test targets, which will be used to generate the executable Java test cases.


433

Fig. 2. Summary of our approach

Definition 2 (Test Target). A test target is defined by the activation of a behavior B within a specific context defined by: Inv ∧ B ∧ Pspe

(2)

where Inv is the system invariant, B designates the behavior expressed as a before-after predicate and Pspe designates a specialization predicate that specifies additional constraints. The specialization predicate can be specified by the validation engineer; by default, it is set to true. Pspe is then enriched during the computation of the test data, in order to require a particular parameter or attribute to be set at a particular value, e.g. an extremum of its domain, as explained in section 5. Before that, the next section explains the structural coverage applied to the JML specification.

4

Structural Coverage of the JML Specification

This section firstly focuses on the extraction of the behaviors from the JML method specifications. Secondly, we present the different rewritings to apply on the decisions nested in the predicates of the considered behavior. These rewritings have to be selected by the validation engineer. 4.1

Extraction of the Behaviors from the Method Specifications

The partitioning of the JML method specifications into behaviors is illustrated by Fig. 3. In this figure, Pk (k ∈ 1..N ) are the precondition predicates, A gives the frame condition, Qk (k ∈ 1..N ) are the normal postconditions, Sp (p ∈ 1..M ) are the exceptional postconditions related to the exceptions Ep . The terminations are distinguished by T , which might be either no exception indicating a normal behavior, or any of the declared exceptions Ep . We call a behavior a branch of this graph. Each behavior represents a particular transition of the system according to the different possible terminations. We require the terminations to be exclusive between the normal termination and the exceptional terminations. From this before-after predicate expressed in the Java/JML logic, we apply user-defined decision coverage criteria as defined hereafter. Notice that the inconsistent behaviors are filtered.

434


Fig. 3. Extraction of the behaviors from a JML method specification

4.2

Application of the Decision Coverage Criteria

In order to exploit the specification, we perform a flow-graph coverage, based on the rewriting of the disjunctions contained in the first-order logic predicates composing the behaviors. We distinguish 4 possible rewritings, presented in Fig. 4, each one of them corresponding to a different decision coverage criteria. They create an additional deployment of the original method specification, adding more granularity to the test targets. In the figure, we denote by [] (read “choice”) the choice-point between predicates. Rewriting 1 consists in leaving the disjunction unmodified. Thus, the first branch that succeeds at being evaluated is enough. This rewriting satisfies the Decision Coverage (DC). As an extension, it also satisfies the Statement Id 1 2 3 4

Rewriting of P1 ∨ P2 P1 ∨ P2 P1 [] P2 (P1 ∧ ¬P2 ) [] (¬P1 ∧ P2 ) (P1 ∧ P2 ) [] (P1 ∧ ¬P2 ) [] (¬P1 ∧ P2 )

Decision Coverage DC and SC D/CC FPC MCC

Fig. 4. Definition of the rewritings of the disjunctive predicates


435

Coverage criteria (SC). Rewriting 2 consists in creating a choice between the two predicates. Thus, the first branch and the second branch independently have to succeed when being evaluated. This rewriting satisfies the decision/condition coverage criteria (D/CC) since it satisfies the DC and the Condition Coverage (CC) criteria. Rewriting 3 consists in creating an exclusive choice between the two predicates. Only one of the sub-predicates of the disjunction is checked at one time. This rewriting satisfies the full predicate coverage (FPC) [18] criteria. Rewriting 4 consists in testing all the possible values for the two sub-predicates to satisfy the disjunction. This rewriting satisfies the multiple condition coverage criteria (MCC). Example 2. Consider the method credit from class LimitedPurse given in the example presented in Fig. 1. The application of rewriting 4 on the behavior from the method specification gives the following deployment. Behavior predicates a >= 0 && balance + a < 0 && balance + a < 0 && balance + a >= 0 && balance +

a a a a

max max max max

&& && && &&

balance balance balance balance

== == == ==

\old(balance) + a (1) \old(balance) (2) \old(balance) (3) \old(balance) (4)

Notice that the behavior (2) is inconsistent w.r.t. the LimitedPurse class invariant balance >= 0 && balance