Supporting the Software Testing Process through ... - Semantic Scholar

Supporting the Software Testing Process through Specification Animation Tim Miller and Paul Strooper Software Verification Research Centre School of Information Technology and Electrical Engineering The University of Queensland Brisbane, Qld 4072, Australia. [email protected] [email protected]

Abstract

part of the behaviour) of an implementation. In this paper we extend the methods from our previous papers by reusing the testgraph from the animation of specifications to generate test cases for the corresponding implementation. We check the implementation against the specification using a passive oracle approach. A passive oracle, as opposed to an active oracle, simply verifies the behaviour of an implementation without replicating its behaviour. This eliminates the problem of non-determinism and under-specification as described by Ho¨ rcher [16]. Our oracles act as self-checking versions of the implementation, and the verification of the implementation is performed using the animator. We use abstraction functions to relate the data types of the implementation to the corresponding data types in the specification. We begin this paper with a summary of related work, and then in Section 3 provide the background on the tools and terminology used in this work. Section 4 presents the generation of oracle classes. Section 5 discusses testgraphs, and Section 6 discusses how to use the testgraphs with the oracle class. Section 7 discusses the application of the method to two case studies. Section 8 concludes the paper and discusses future work in this area.

Achieving consistency between a specification and its implementation is an important part of software development. In this paper, we present a method for generating passive test oracles that act as self-checking implementations. The implementation is verified using an animation tool to check that the behaviour of the implementation matches the behaviour of the specification. We discuss how to integrate this method into a framework developed for systematically animating specifications, which means a tester can significantly reduce testing time and effort by reusing work products from the animation. One such work product is a testgraph: a directed graph that partially models the states and transitions of the specification. Testgraphs are used to generate sequences for animation, and during testing, to execute these same sequences on the implementation.

1

Introduction

Achieving consistency between a specification and its implementation is an important part of software development. Specification animation allows us to view specific examples of a specification that can be constructed quickly and automatically. In previous papers [23, 24], we discuss animating specifications using testgraphs: directed graphs that partially model the states and transitions of the specification. We have also investigated supporting testing with animation. In [22], we present a method with tool support that allows the animation and testing of a component to be combined into one script. One problem with this method is that this uses an active oracle. An oracle is a means of checking the behaviour of an implementation during testing, and an active oracle is an oracle that replicates the behaviour (or

2

Related Work

In this section, we present related work on animation and testing, including animation-based testing.

2.1

Animation

There are several animation tools that automatically execute or interpret specifications such as PiZA [13], Possum [12], Pipedream [18], the B-Model animator [29], and the CLPS-BZ solver [2]. Kazmierczak et al. [18] outline one of the few approaches for systematic specification animation using 1

Pipedream containing three steps: performing an initialisation check; verifying the preconditions of schemas; and performing a simple reachability property.

the specification and implementation of an air-traffic control system using this approach. There does not seem to be any significant tool support for their method.

2.2

3

Testing

Module and class testing can often be automated to save time during testing. There are several research and commercial testing tools available for automated module and class testing, such as the Java testing tools Roast [6] and JUnit [17]. Formal specifications can aid in testing software by providing a starting point for selecting test inputs, executing test cases and determining expected outputs. Research in this area includes work on generating test cases for individual operations [10, 26, 27], generating finite state machines for the purpose of testing [7, 14, 28], generating test oracles [16, 20], and frameworks for specification-based testing that support test case generation, execution, and evaluation [5, 8]. The papers most related to our work are those of McDonald et al.[20] and H¨orcher [16]. Both of these papers present methods for generating passive test oracles. Our approach differs from H¨orcher’s because firstly, we are combining testing with animation, and secondly, we use the animator as the oracle instead of generating code in the target language to do the checking. Callahan et al. [4] and Ammann et al. [3] discuss the application of mutation analysis combined with modelchecking to generate test data sets. They apply slight syntactical changes to a specification to generate mutants, and use model checking to detect counter-examples, which are used to derive sequences for testing.

2.3

Background

In this section, we discuss the example used in this paper and the Roast testing framework [6]. For clarity, specification text appears in italics (IntSet) and implementation text appears in typewriter font (IntSet).

3.1

Example - IntSet

The example used to demonstrate our method is a bounded integer set, called IntSet. The specification is shown in Figure 1. maxsize == 10

Testing using Animation

Legeard et al. [19] present a highly-automated specification-based testing technique for testing Z and B specifications, called the BZ-Testing-Tools (BZTT) method. They use the specification to generate states to test, and then use a constraint solver to search for a sequence of operations that reaches each state. The behaviour of the implementation is compared against the simulation of the specification. This approach differs from ours in several ways. One main difference is that we generate passive test oracles instead of active oracles. The BZTT method uses animation to search out states to test, whereas we use testgraphs. Also, we execute the specification during testing, but the BZTT method derives and refines all tests before execution is done. Aichernig et al. [1] discuss validating the expected outputs of test cases by comparing them with the outputs from the IFAD VDM interpreter. They revealed problems in both

state intset : P Z

init ∆state

#intset ≤ maxsize

intset 0 = ∅

add ∆state x? : Z

remove ∆state x? : Z

x ? 6∈ intset #intset < maxsize intset 0 = intset ∪ {x ?}

x ? ∈ intset intset 0 = intset

size Ξstate size! : N

hasMore Ξstate hasMore! : B

size! = #intset

hasMore! ⇔ #intset > 0

isMember Ξstate x? : Z out! : B

next ∆state el ! : Z

out! ⇔ x ? ∈ intset

\ {x ?}

#intset > 0 el ! ∈ intset intset 0 = intset \ {el !}

Figure 1. Z Specification of IntSet The IntSet specification is written in Z [25]. The IntSet state schema consists of a state variable intset (a set of in2

public class IntSet { final static public int MAXSIZE = 10;

String open(String file); void schemaCheck (String schema, ZVar [] zvars, boolean expval);

//stores the integers in the set protected int [] intset; //the number of elements in the set protected int count;

Figure 3. Methods in the Possum interface

4.1

public IntSet(); public void add(int x) throws FullExc, DuplicateExc; public void remove(int x) throws ItemNotFoundExc; public boolean isMember(int x); public int size(); public boolean hasMore(); public int next() throws NoMoreExc;

In this work, we have used the Possum animator [11, 12], and we have implemented a Java class to communicate with Possum. The interface is generic in order to easily accommodate changes to Possum and possibly other animators. The two methods in this class that we will use are open and schemaCheck, whose interfaces are shown in Figure 3. open opens the file given as a parameter, sends the contents of the file to the animator, and returns the response given by the animator, and schemaCheck, which checks whether the array of ZVar variables, satisfies a schema (i.e. no failure occurs). ZVar objects represent variables used in Z schemas (input, output, and state variables). A ZVar object has two instance variables: a name, which is the name of a variable in a schema, and an object representing the variable’s value. There are two subclasses of ZVar: ZState, which represents state variables; and ZInOut, which represents input and output variables. If the value of the satisfiability check of schemaCheck is different to the boolean parameter expval, then the method asks the animator for more detail. For example, if expval is true, but the simplification failed, the animator is asked to return all possible post-states and outputs for the inputs and prestate specified in the schema object, and this is printed in an error message.

}

Figure 2. Java Interface for IntSet

tegers), and a state invariant, which restricts the intset to a maximum size of 10, defined by the constant maxsize. The seven operations in the IntSet specification are: init, which sets intset to be empty; add , which adds an integer to the set if that integer is not already in the set and the set is not full; remove, which removes an integer from the set provided it is in the set; isMember , which returns a boolean indicating whether an integer is in the set; size, which returns the size of the set; hasMore, which returns true if and only if there are elements in the set; and next, which non-deterministically removes an element from the set and returns it. Figure 2 shows the Java interface for the IntSet implementation of the specification from Figure 1. Note that the init schema from the specification becomes the constructor IntSet(). For the violation of preconditions, we have chosen to raise an exception in the implementation. This is considered a more robust manner to implement a precondition violation than assuming the precondition is true when a method is called. McDonald et al. [20] choose to extend Object-Z by adding exceptions. We prefer to use the original Z specification language, to avoid having to change existing tool support.

4

Communicating with the Animator

4.2

Deriving the Oracle Class

In this section we present a systematic way to derive an oracle for a class. We demonstrate our approach using the next() method from IntSet, which returns an element from the set and removes that element, throwing an exception if no element exists. In the IntSet specification, the operation non-deterministically selects the elements to be removed and returned. We base our approach around McDonald and Strooper’s approach for deriving passive test oracles from Object-Z specifications [20]. They present two different types of oracle: loosely-coupled and closely-coupled. We use the closely-coupled approach. A passive oracle does not replicate the behaviour (or a subset of the behaviour) of the implementation, but instead simply verifies that the behaviour of the implementation is correct. This eliminates the problems of under-specification and non-determinism described by H¨orcher [16].

The Oracle Class

In this section, we discuss using an animator as the backbone to an oracle class. The oracle uses the animator to check the behaviour of the implementation. We try to structure our approach so that the animator is transparent, and no knowledge of the animator is needed by the tester. 3

4.2.1

void abs(ZState [] zst, boolean primed) { ZSet zs = new ZSet(); for (int i = 0; i < count; i++) { zs.add(new Integer(intset[i])); }

The State of the Oracle class

The state of the oracle class contains an instance of the class that communicates to the animator. The rest of the state is obtained by inheriting the class-under-test (CUT). This inheritance allows us to execute the inherited operations, and check their behaviour. The checking is performed inside the oracle to produce a self-checking implementation of the specification, which is implemented as a wrapper around the inherited class.

if (!primed) { zst[0] = new ZState("intset", zs); } else { zst[0] = new ZState("intset’", zs); } }

4.2.2

The Abstraction Function Figure 4. The abs Function for IntSetOracle

Data types in specification languages often differ from those in implementation languages. For example, the specification of IntSet represents the state as a set of integers. However, Java does not have a set data type, and therefore the implementation must use a different data type, such as an array or a linked list. As a result, if we are to compare the behaviour of the specification with the behaviour of the implementation, we have to define a mapping between the two. H¨orcher [16] advocates abstracting the concrete state of the implementation, because this eliminates problems with non-determinism. We also choose this approach for the same reason, but more importantly because we can get the animator to do the checking if we use the abstract state space. Following [20], the tester must implement an abstraction function in the oracle class. This abstraction function takes the concrete state of the CUT and returns the representation of that state in an abstract form. To support this, we have implemented a Z toolkit for Java to give us the abstract types supported by Z. We have implemented classes for the the basic Z types set, sequence, relation, function, and pair. These types all take Java objects, and hence can be nested. Integers and naturals are represented by the Integer class, and booleans by the Boolean class. Any other types, such as free types or given types, must also have corresponding implementations in Java. The abstraction function is called abs and takes two parameters. The first is an array of ZState objects, which is used as an output parameter. The second parameter is a boolean, which indicates whether the current call to abs is a post-state call, in which case we append a prime (’) to the variable names. The abs function in the IntSetOracle class is shown in Figure 4. In this figure, intset is the array of integers that contains the elements in the set and count is the number of integers in the array (the size of the set). Note that because the oracle class is a class that inherits IntSet, the oracle has access to all non-private instance variables declared in IntSet (including intset). This is a restriction on the method since all data members must be defined as public or protected. Alternatively, the tester

public IntSetOracle () { super(); ZState [] state = new ZState [1]; //set up Possum possum = new Possum(); StringBuffer response = possum.open("IntSet.zed"); abs(state, true); ZVar [] vars = {state[0]}; possum.schemaCheck("init", vars, true); }

Figure 5. Constructor for IntSetOracle

can implement the abstraction function using the methods available, change the permissions of the data members for testing, or perform tests based solely on the class interface.

4.2.3

Testing the Constructors

Figure 5 shows the constructor for the IntSetOracle class. For each constructor in the CUT, we want to check that the initial state satisfies the initialisation schema of the specification. To do this we implement, for each constructor, a constructor in the oracle class with the same parameters. Each constructor must firstly make a call to super(params), which calls the superclass’s constructor, passing the same parameters that were passed to the sublass’s constructor (params). The constructor then obtains the state of the CUT using the abstraction function, and checks the specification using the schemaCheck method from the Possum class. 4

public int next() throws NoMoreExc { ZState [] preState = new ZState [1]; ZState [] postState = new ZState [1];

itive to be returned. 4.2.5

abs(preState, false); ZInOut el = new ZInOut("el!", new Integer(super.next())); abs(postState, true);

In the next() example, the type int is converted to an Integer object. However, more complex problems arise with other data types. Input and output types to methods in the implementation can be different to that of the specification. For example, an operation in the specification may return a set, but the implementation returns an array. We must abstract the value of the output in order to compare the two using the animator. We have implemented a class called Abstraction that contains a library of abstraction functions for common data types, such as from an array to a set, an array to a sequence, and from primitive types to their corresponding object types. If there is no abstraction function for a particular data type, the tester can define their own in the oracle class, or update the Abstraction class. The tester can also use this library class to help with implementing the abstraction function for the state of the class.

ZVar [] vars = {preState[0], postState[0], el}; possum.schemaCheck("next", vars, true); return ((Integer)el.value).intValue(); }

Figure 6. IntSetOracle.next() 4.2.4

Abstraction of Data Types

Testing the Methods

For each method, we want to check that the pre-state and input do not violate the precondition, the post-state and output are related to the pre-state and input via the operation in the specification, and the state invariant is maintained after each operation is executed. To do this, each public method in the CUT is overridden in the oracle. Before calling the inherited implementation of the method from the CUT (the superclass) using the super object, the oracle obtains the abstract pre-state using the abstraction function. Any outputs from the inherited method are recorded during the execution of the method. The abstract post-state is then obtained using the abstraction function. Now, we check whether these pre- and postconditions are related by first constructing an array of ZVar objects from the inputs, outputs, pre-, and post-state, and then using the schemaCheck method to check that these variables satisfy the schema. Note the state invariant of a Z specification is appended to the precondition and postcondition of every schema that includes the state schema, therefore this check also tests whether the state invariant is maintained. Figure 6 shows the next() method in the IntSetOracle class. In this figure, the ZState objects for the pre- and poststate are created. A call to super.next() is generated, capturing the output into a ZVar object called element, and the pre- and post-states are retrieved before and after the call respectively. When the output is captured, the primitive type int (the output from next()) is converted into an object type Integer, because ZVar objects hold other objects, not primitives. The ZVar array is created from the pre-state, post-state, and output. The schemaCheck method is then called, and finally, the value of the output is returned. Because the primitive int value was converted to an Integer object, we must obtain the value of the prim-

4.2.6

Checking Exceptions

The animator can also assist with the checking of exceptions. If an exception occurs in the implementation, we want the specification to fail for that case (Possum reports that ‘no solution’ has been found if the schema is not satisfiable by the values bound to its variables). If the specification does not fail, but an exception occurs in the implementation, then our implementation is not fulfilling the requirements of the specification. In order to check exceptions, we make use of the schemaCheck method again. We add a try/catch statement to our operation check. A try/catch statement in Java allows us to catch and handle an exception that has occurred in a piece of code. To check exceptions in the oracle class, we put the method calls in the try part. If an exception does not occur when calling the inherited method, then the checking will behave as normally. However, if an exception does occur, the try statement will stop executing at that point, and jump to the catch part. We expect that the precondition to the operation has been violated, so we check that the operation in the specification fails for the pre-state and inputs. We do this by constructing a ZVar array consisting of only the pre-state and inputs, and sending this to the schemaCheck method. However, the difference between this and the normal case is that we want the binding to fail, so the third argument (expval) to schemaCheck is false. Figure 7 shows an excerpt from the try/catch statement for next(). Note that we still throw the exception because we want the behaviour of the oracle to replicate that of the implementation. 5

try { code from Figure 6 } catch (NoMoreExc e) { ZVar [] vars = preState[0]; possum.schemaCheck("next", vars, false); throw new NoMoreExc(); }

as parameters, which the tester must implement. Alternatively, the tester can add their abstraction function to the Abstraction class, and generate the oracle class again. The tester can choose to not generate calls to the abstraction function for the state of the class, so that if the tester wants to test based only on the interface, they can skip checks on the state, avoiding the use of an abstraction function. To automatically generate the oracle, the implementation must follow certain guidelines, such as: the parameters of the methods must appear in the same order as in the specification; if an operation has only one output declared in the specification, then that must be the return type, otherwise all output must be returned via parameters.

Figure 7. Exception checking for next() Also note that this approach does not check that the correct exception occurred, but rather that the specification allows an exception to occur. Because Z does not support exceptions, and because we did not extend Z to support exceptions, it is not possible to check if the correct exception has occurred using animation. However, Z specifications can be structured using error schemas to identify exceptional cases (see [25]). For example, the next operation can be defined as:

5

We use testgraphs to define and traverse a number of interesting states for both animation and testing. A testgraph is a directed graph that partially models the states and transitions of the CUT. We use testgraphs because they are straightforward to specify, and generating test cases from testgraphs can be done automatically. The similarities between a specification and its implementation mean that we want to test the same behaviour, therefore we use the testgraph for both.

next = b (next0 ∧ Success) ∨ NoNextExc where next0 is the next schema from Figure 1, Success is a schema with an output variable indicating that the operation was successful, and NoNextExc specifies the case in which there is no element to return and has an output variable indicating that the operation failed. We can identify whether the correct exception has occurred by replacing the call to schemaCheck in Figure 7 with:

5.1

possum.schemaCheck("NoMoreExc", vars, true);

Deriving a Testgraph

The state and operations of a specification provide important information about the selection of states to test. For example, the next operation in the IntSet specification will behave differently when the set is empty to when it is not empty. Standard testing practice advocates several heuristics for selecting special states, such as the interval rule. For the IntSet specification, we select our states based on the size of the set. We test an empty set, a set containing one element, a set that is half-full, a set with maxsize −1 elements, and a set that is full. For our testgraph states, we split the set with multiple elements into two different sets: those containing odd elements and those containing even elements. We split these states and we choose the odd and even states because they are easy to generate. For each of these special states, we select one possible instantiation of the state. For example, the state containing one element becomes the set {0}. Each of these instantiations becomes a node in our testgraph. Once we have our testgraph nodes, we derive arcs for the testgraph to be used as transitions during testing. We require each node, except the start node, to have at least one arc leading in to it, otherwise the node will be unreachable.

This will check that the NoMoreExc schema is satisfied with the input variables, which will fail if this exception cannot occur.

4.3

Testgraphs

Tool Support

Tool support for generating the oracle class has been developed. This combines information from the specification with information from the implementation to generate the checks for the constructors and methods. The abstraction function must be implemented by the tester because the relation between the specification state and the implementation state cannot be derived automatically. The oracle-generation tool checks the data types used in the specification and implementation, and determines whether abstraction of the inputs or outputs is needed. If so, the code produced makes use of the Abstraction class to perform the refinement. If the Abstraction class contains a method to abstract from the concrete to abstract types, then it uses that method. Otherwise, it simply generates a call to a local abstraction function with these types 6

ODD {1,3,5,7,9} ADDEVEN

ADDODD

EMPTY

ADD1

ONE

ADDMOST

{0}

{}

ADD9

FULL−1 {0,1,2,3,4,5,6,7,8}

REMOVEODD

FULL {0,1,2,3,4,5,6,7,8,9}

REMOVEEVEN

EVEN {0,2,4,6,8}

differs slightly because we make no calls to the CUT itself, but instead to the oracle, which implements a self-checking version of the implementation. The tester implements a class called Driver that contains an instance of the oracle, and provides three methods. The reset method resets the oracle into its initial state. This is used at the start of a path derived from the testgraph. The arc method generates the transition associated with a testgraph arc. The node method tests the behaviour of the CUT by making calls to the oracle. We traverse the testgraph, calling the appropriate method for each new path, each arc, and each node. The pseudocode for testgraph traversal is:

Figure 8. Testgraph for IntSet for each path P to be traversed call the reset method in Driver for each arc A(S , D) in P where S is the source and D the destination call the arc method in Driver for arc A call the node method in Driver for node D

In addition to making sure each node is reachable, we also add arcs to cover specific transitions that we want to test. Figure 8 shows the testgraph for the IntSet specification. Here, we have six nodes representing the states derived above: EMPTY, ONE, ODD, EVEN, FULL-1, and FULL. EMPTY is the start node and this is indicated by the node being shaded. The seven arcs on the testgraph change the state of IntSet from one state to another. For example, the ADDODD arc represents a transition that adds 1, 3, 5, 7 and 9 to the set. This takes us from EMPTY to ODD.

5.2

To construct the testgraphs, we use the testgraph editor from the Possum Testgraph Framework [23]. If the animation has been performed using this framework, the testgraph should already be constructed. However, the tester may wish to add extra states or transitions for implementation-specific testing. For example, the state of the IntSet specification is a set of integers. However, the implementer may decide to use a binary tree. The tester may then wish to test specific structures of the binary tree by varying the insertion order, which might not have been known at the time of animation, and would not make sense to animate anyway. Therefore, the testgraph is extended to test these special cases. Figure 9 shows an outline of the Driver class for Intset.

Traversing the Testgraph

We generate test sequences by traversing the testgraph to achieve arc coverage. Two other examples of coverage are node and path coverage. Node coverage does not force the traversal of every transition in a graph, and path coverage is infeasible for graphs with a cycle. Arc coverage traverses every arc, visits every node (provided all testgraph nodes and arcs are reachable from the start node), and is straightforward to achieve. To traverse the testgraph, we make the transition specified on an arc, and then perform checking when we reach the destination node of that arc (see Section 6). We continue until all paths calculated for arc coverage have been traversed.

6.1

The reset method resets the oracle into its initial state, re-constructing the instance(s) of the oracle object used for testing. Figure 9 shows the reset method from the driver class for IntSet.

6.2

6

The reset Method

The arc Method

The Driver Class The arc method is called when traversing a testgraph arc. The tester must implement an arc method that takes a string representing a label, and from the label, decide which behaviour to invoke. Figure 9 shows a part of the arc method of the driver class for IntSet. In this figure, if the arc label passed to the method is ADDEVEN , then arc inserts all even numbers from 0 to IntSetOracle.MAXSIZE into the oracle using the add method.

Traversing a testgraph manually, or deriving an automated test script to traverse a testgraph, is tedious and timeconsuming for even the smallest of testgraphs, and any change in the testgraph would result in a change in such a script. For this reason, we automatically traverse the testgraphs. To perform automated traversal, we use an approach similar to Hoffman and Strooper’s [15]. However, our approach 7

public class Driver {

7.1

Mass Transit Railway

IntSetOracle orc;

The MTR specification was devised by us as a case study for an earlier paper [21] and was manually translated from [9]. The specification describes the Hong Kong Mass Transit Railway network. This specification is of particular interest to us because it contains more than one component: three low-level component and one top-level component that uses the three low-level components to perform its services. The MTR specification contains 23 operations, and about 400 lines of Z. The implementation of the MTR was written manually in Java by one of the authors. Each of the four specifications was implemented as a Java class.

public Driver() { orc = new IntSetOracle(); } public void reset() { orc = new IntSetOracle(); } public void arc(String label) { if (label.equals("ADDEVEN")) { //add even elements } } public void node(String label) { if (label.equals("EMPTY")) { //check NoSuchElementExc occurs } orc.size(); }

7.1.1

The MTR consists of a set of passengers and a set of stations between which the passengers travel. To enter the network, a passenger must obtain a ticket, which is supplied to the system upon entering and exiting the network. Tickets can be single-trip, multi-trip, or season tickets. Each ticket has an expiry date and a value. The value of the ticket is decremented by the fare amount when the passenger leaves the network. Fare amounts are stored in a database that supports the addition of new fares and the updating of existing fares. All tickets can be reissued, but only as the same type as they were originally issued. The current date can be incremented.

}

Figure 9. Outline of the Driver class

6.3

The node Method

The node method is used to check the behaviour of the CUT by making calls to the oracle. Any errors uncovered by the oracle will be reported at run-time. Like the arc method, the node method is passed the label of the current node in the testgraph. Figure 9 shows a simple node implementation for IntSet. We do not provide expected outputs for our test cases, such as orc.size(). It is still possible to provide the expected output, but the oracle will check the behaviour using the animator. If expected output was included, this could be used to verify the specification is correct instead of the implementation. Note that we must give expected exceptions when checking exceptions if we do not use the approach of specifying exceptions using schemas as discussed in Section 4.2.6. As discussed in Section 4, this is because with our exception checking in the oracle, we can only use the animator to check that an exception has occurred, not which exception has occurred.

7

Informal Description of Behaviour

7.1.2

Method

We used a bottom-up testing approach, in which the lowlevel classes were tested first. When the top-level class was tested, we used the oracle classes from the supporting classes instead of the supporting classes, allowing us to more easily pinpoint the cause of errors, because the error message will be printed for both the lower-level class where the error occurred, and the top-level class to which the error propagates. Using the oracle classes also meant the abstraction function of the top-level class was far easier to implement, because we could use the abstraction functions from the low-level oracle classes. The MTR specification had been animated using the testgraph approach previously [24], so the testing was straightforward. To test each class, we only have to generate the oracle classes, which is done automatically, define the abstraction functions and the Driver class. The Driver class was also straightforward because the behaviour of the transitions had already been specified for the animation, so we hand-translated them to equivalent calls on the implementation.

Experience

As well as the IntSet example, we have applied our method to other case studies, including the Mass Transit Railway (MTR) case study, and a Dependency Management System (DMS). In this section, we discuss the application of the method to MTR and DMS . 8

7.1.3

Results

due to the effort and expertise required to develop software formally. In this paper, we presented a method that exploits the similarities between animation and testing by integrating the testing into an already existing framework for systematic animation. The testgraph used for the animation can be reused, and the behaviour of the implementation is verified against the specification using an animation tool. We wrap the implementation in an oracle class with the same interface that communicates with the animator to give a selfchecking version of the implementation. We believe that this extended framework both decreases the time and effort needed for testing, and encourages people to use animation, and as a result, formal specification. We also believe that our oracles could be used within other frameworks with little-to-no modification. We have applied our method to several case studies with positive results. Future work in the area includes:

At the start of testing, the implementation had only been compiled, and never run or tested. We found a total of five errors in the implementation. One error was caused by a misunderstanding of an operation in a Java library class, three were a result of not typecasting objects correctly, and one was an oversight when reading the specification.

7.2

Dependency Management System

The DMS specification was first described by Carrington et al. [5] and was hand-translated by us from Object-Z to Z. The resulting specification was almost identical. The DMS case study was of particular interest because of an interesting abstraction function. The state of the specification contains a secondary variable. A secondary variable is a variable whose value is derived from the value of other state variables. The secondary variable is the transitive closure of a relation. There is no variable mapping to the secondary variable in the implementation because it is only used to simplify preconditions in the specification. 7.2.1

• Applying the method to a sizable, industrial case study to evaluate the scalability of the method. • Applying the method to a system with a user interface. The lack of a call-based interface makes it difficult to use batch drivers for testing of user interfaces. Therefore, we believe that the process to do this will be different to the approach presented in this paper.

Informal Description of Behaviour

A DMS is an important component in a theorem-proving tool, because it tracks dependencies between theorems and proofs to prevent circular reasoning. The DMS specification we used maintains a set of nodes, and tracks transitive dependencies between these nodes. There are 15 operations to add and remove nodes and dependencies, to query dependents and supporters of nodes, and to query the existence of nodes and dependencies. 7.2.2

References [1] B. K. Aichernig, A. Gerstinger, and R. Aster. Formal specification techniques as a catalyst in validation. In P. Srimani, editor, Proceedings of the 5th International High Assurance Systems Engineering Symposium, pages 203–206. IEEE Computer Society, 2000.

Results

At the start of testing, the implementation had only been compiled, and never run or tested. We found a total of six errors in the implementation, and one in the abstraction function that we had implemented. The problem with the abstraction function was due to a misunderstanding of an operation in a Java library class. This was easily detected. Four of the other errors were all due to general programming mistakes such as truth values evaluating to true instead of false and vice-versa, and the remaining two were oversights when reading the specification.

8

[2] F. Ambert, F. Bouquet, S. Chemin, S. Guenaud, B. Legeard, F. Peureux, and N. Vacelet. BZ-TT: A tool-set for test generation from Z and B using constraint logic programming. In Proc. of Formal Approaches to Testing of Software, Workshop of CONCUR’02,, pages 105–120, 2002. [3] P. Ammann and P. Black. A specification-based coverage metric to evaluate test suites. International Journal of Reliability, Quality and Safety Engineering, 8(4):275–300, 1999.

Conclusions and Future Work

[4] J. Callahan, Easterbrook. S., and T. Montgomery. Generating test oracles via model checking. Technical Report NASA-IVV-98-015, NASA / West Virginia University Software Research Laboratory, 1998.

Despite the limitations of testing, it remains the most popular form of verification and validation of software in industry. This is largely due to its low cost and potential for automation. Formal methods and, as a result, animation are currently not widely used in industry. This is largely

[5] D. Carrington, I. MacColl, J. McDonald, L. Murray, and P. Strooper. From Object-Z specifications to 9

Classbench test suites. Journal on Software Testing, Verification and Reliability, 10(2):111–137, 2000.

[17] JUnit. JUnit, testing resources for extreme programming. http://ww.junit.org.

[6] N. Daley, D Hoffman, and P. Strooper. Unit operations for automated class testing. Technical Report 00-04, SVRC, 2000.

[18] E. Kazmierczak, M. Winikoff, and P. Dart. Verifying model oriented specifications through animation. In Asia-Pacific Software Engineering Conference, pages 254–261. IEEE Computer Society, 1998.

[7] J. Dick and A. Faivre. Automating the generation and sequencing of test cases from model-based specifications. In Formal Methods Europe, pages 268–284. Springer-Verlag, 1993.

[19] B. Legeard, F. Peureux, and M. Utting. Automated boundary testing from Z and B. In Formal Methods Europe, pages 21–40. Springer, 2002.

[8] R. Doong and P. Frankl. The ASTOOT approach to testing object-oriented programs. ACM Transactions of Software Engineering and Methodology, 3(2):101– 130, 1994.

[20] J. McDonald and P. Strooper. Translating Object-Z specifications to passive test oracles. In S. Liu J. Staples, M.G. Hinchey, editor, Second International Conference on Formal Engineering Methods, pages 165– 174. IEEE Computer Society, 1998.

[9] R. Duke and G. Rose. Formal Object-Oriented Specification Using Object-Z. MacMillan Press Limited, London, 2000.

[21] T. Miller and P. Strooper. Animation can show only the presence of errors, never their absence. In Australian Software Engineering Conference, pages 76– 85, Canberra, Australia, 2001. IEEE Computer Society.

[10] M. Gaudel. Testing can be formal too. In Proceedings of TAPSOFT’95, pages 82–96. Springer-Verlag, 1995.

[22] T. Miller and P. Strooper. Combining the animation and testing of abstract data types. In Asia-Pacific Conference on Quality Software, pages 249–258. IEEE Computer Society, 2001.

[11] D. Hazel, P. Strooper, and O. Traynor. Possum: An animator for the SUM specification language. In Proceedings Asia-Pacific Software Engineering Conference and International Computer Science Conference, pages 42–51. IEEE Computer Society, 1997.

[23] T. Miller and P. Strooper. A framework and tool support for the systematic testing of model-based specifications. TR 02-35, SVRC, 2002.

[12] D. Hazel, P. Strooper, and O. Traynor. Requirements engineering and verification using specification animation. In 13th IEEE International Conference on Automated Software Engineering, pages 302–305. IEEE Computer Society, 1998.

[24] T. Miller and P. Strooper. Model-based animation using testgraphs. In International Conference on Formal Engineering Methods, pages 192–203. Springer Verlag, 2002.

[13] M. Hewitt, C. O’Halloran, and C. Sennett. Experiences with PiZA, an animator for Z. In ZUM’97: The Z Formal Specification Notation, volume 1212 of LNSC, pages 37–51. Springer, 1997.

[25] J. Spivey. The Z Notation: A Reference Manual. Prentice Hall, 2nd edition, 1992. [26] S. Stepney. Testing as abstraction. In Z User Meeting ’95, pages 137–151. Springer-Verlag, 1995.

[14] R. M. Hierons. Testing from a Z specification. Software Testing, Verification and Reliability, 7(1):19–33, 1997.

[27] P. Stocks and D. Carrington. A framework for specification-based testing. IEEE Transactions on Software Engineering, 22(11):777–793, 1996.

[15] D. M. Hoffman and P. A. Strooper. ClassBench: A methodology and framework for automated class testing. In D. C. Kung, P. Hsia, and J. Gao, editors, Testing Object-Oriented Software, pages 152–176. IEEE Computer Society, 1998.

[28] C. Turner and D. Robson. A state-based approach to the testing of class-based programs. Software– Concepts and Tools, 16(3):106–112, 1995. [29] H. Waeselynck and S. Behnia. B-Model animation for external verification. In Proceedings of International Conference for Formal Engineering Methods, pages 36–45. IEEE Computer Society, 1998.

[16] H. M. H¨orcher. Improving software tests using Z specifications. In J. P. Bowen and M. G. Hinchey, editors, Proceedings of the Ninth Annual Z User Meeting, volume 967 of LNCS, pages 152–166, Limerick, Ireland, 1995. Springer-Verlag. 10