Extending the Test Template Framework for speci cation-based testing ...

Extending the Test Template Framework for speci cation-based testing of interactive systems Ian MacColl and David Carrington Software Veri cation Research Centre Department of Computer Science & Electrical Engineering The University of Queensland

fianm,[email protected]

Abstract. In this paper we introduce a framework for speci cation-

based testing of interactive systems. Interactive systems can be analysed and developed in terms of functionality, presentation and behaviour. Testing information for interactive systems can be derived from formal speci cations of these aspects. The paper uses the Multi-modal Airline Travel Information System (MATIS) speci cation of Duke and Harrison which provides a partial speci cation for each aspect. We derive testing information from the speci cations of functionality and behaviour, and we extend the Test Template Framework to encompass behavioural testing information.

1 Introduction An interactive system is one in which \each user entry causes a response from or action by the system" [IEEE, 1990]. Interaction is a signi cant component of contemporary software systems and graphical user interfaces are now almost universal. Interactive systems can be viewed as conceptually and architecturally composed of functionality, presentation and interaction components. Ostrand et al. note that correctness of an interactive system involves both proper behaviour of the user interface (interaction) and proper computations of the underlying application (functionality) [Ostrand et al., 1998]. Testing is one method that is used to increase con dence in software correctness, but testing user interface software is dicult. Although testing of user interface software has elements in common with other software testing, it also presents a number of challenges: size and complexity (compared to functionality); graphical and other

Proceedings of the Twenty Second Australasian Computer Science Conference, Auckland, New Zealand, January 18{21 1999. Copyright SpringerVerlag, Singapore. Permission to copy this work for personal or classroom use is granted without fee provided that: copies are not made or distributed for pro t or personal advantage; and this copyright notice, the title of the publication, and its date appear. Any other use or copying of this document requires speci c prior permission from Springer-Verlag.

presentations; multiple, asynchronous input devices; (apparently) modeless operation; and rapid semantic feedback. Interactive systems have much in common with reactive systems but dier in presenting information to a human user. Solutions proposed to meet these challenges mainly involve automating test execution with tools providing: widget-based veri cation in addition to bitmapbased veri cation; scripting, particularly to encapsulate machine or operating system dependencies; automatic synchronisation; and separate data, control and usability testing [Horowitz and Singhera, 1993; Kepple, 1993; Koning et al., 1996]. Apart from abstracting away from the concrete details of graphical presentations and input events, little guidance is provided on what tests to perform or what results to expect. Formal speci cations can provide this guidance. We are investigating black-box testing of interactive systems based on abstract, multiparadigm speci cations. We derive testing information from such a speci cation to con rm that each view 1 provides useful, but dierent, testing information, and we propose an initial framework for structuring the testing information derived from two of the views. Our examples are deliberately small to simplify presentation. The speci cation we use is the Multi-modal Airline Travel Information System (MATIS) speci cation [Duke and Harrison, 1995]. The speci cation uses Z to de ne functionality and presentation, and CSP to de ne interaction. We are not advocating these notations particularly and we are investigating other model- and behaviour-based notations, such as Object-Z [Duke et al., 1995] and Statecharts [Harel, 1987]. For the functionality and presentation views, we use the Test Template Framework (TTF) [Stocks and Carrington, 1996], a formal, abstract model of testing, used to derive a hierarchy of test information. For the interaction view, here written in CSP and translated to nite state automata, we extend the TTF to accommodate behavioural notations. The TTF permits the human tester to control strategy selection and we provide simple examples. In the next section we present an outline of the Test Template Framework and an extension to support behavioural speci cation notations. We then present the speci cation with minor modi cations and derive testing information for two views2 . Section 3 is concerned with system functionality and Section 4 is concerned with interaction between the system and the user. In Section 5 we discuss related work and conclude. We assume basic familiarity with Z and CSP. The Z content of the paper has been checked with the f uzz typechecker. The contribution of Duke and Harrison's paper was a suggestion that human factors concerns can be incorporated into rigorous development by using models that provide multiple viewpoints. In this paper we demonstrate that dierent 1 2

We use the term view to refer to a partial speci cation. The presentation speci cation is elided in the interests of brevity. As Duke and Harrison note, presentation is an abstraction of functionality and, as the functionality is also speci ed in Z, our initial framework is adequately demonstrated by considering only functionality and interaction.

viewpoints are also useful for speci cation-based testing and we provide an initial framework for structuring the dierent tests.

2 Test Template Framework The Test Template Framework (TTF) is a framework for structuring testing information derived from formal speci cations written in model-based notations such as Z [Stocks and Carrington, 1996]. In Section 2.1 we outline the original TTF and in Section 2.2 we introduce extensions to accommodate behaviourbased notations such as CSP.

2.1 Original Test Template Framework

The Test Template Framework (TTF) is a formal, abstract model of testing, used to derive a hierarchy of test information, including test inputs and outputs, from a formal speci cation [Stocks and Carrington, 1996]. The TTF provides a mechanism for recording a human tester's decisions about testing, including strategies to be applied, coverage levels to achieve, etc. The TTF is not automated but provides a context for the use of testing tools such as test case generators, test execution frameworks and result comparators. The root of the hierarchy for a particular operation is the inputs for which the operation is de ned. The nodes of the hierarchy are constrained subsets of the input values of the operation. Test selection proceeds by selecting a test template (initially the entire valid input space) and applying a testing strategy to yield further test templates. The TTF does not prescribe the use of particular test selection techniques but, rather, advocates the use of multiple techniques by providing a means for structuring testing information. The TTF uses the valid input space (VIS) of an operation as the source of all tests. The valid input space is that part of an operation's input for which the operation is de ned. In Z, the valid input space is the operation's precondition, which is de ned as the operation with after-state and output variables hidden. The basic unit for de ning data in the TTF is a test template (TT) which is a constrained subset of the valid input space. VISOp =b pre Op

TTOp ==  VISOp

Testing information is derived using testing strategies. We take the set of all testing strategies as given. [STRATEGY ] Test templates for an operation are organised into a hierarchy called a test template hierarchy (TTH) which has the valid input space as root. The hierarchy is created by applying testing strategies to existing test templates to derive additional test templates. A test template hierarchy is declared for each operation under test.

TTHOp : TTOp  STRATEGY   TTOp

2.2 Extended Test Template Framework The structures of the original TTF record testing information for a particular operation. Interaction views speci ed in a behavioural notation require a framework that encompasses multiple operations structured, for example, as a sequence or trace of invocations. Actual testing also requires such structuring. Two issues must be addressed in devising such a framework: operation homogeneity and test case structuring. Operation homogeneity refers to assimilating the operations and their testing information to avoid type con icts. We de ne a valid input space and test template type for each operation, for example Op1 and Op2 . We then construct a generic operation by disjoining the individual operations. Note that the signatures of the individual operations are merged to construct the signature of Op 3. In Section 4.2 individual operations are rede ned in terms of the signature of the generic operation. VISOp1 =b pre Op1 TTOp1 ==  VISOp1

VISOp2 =b pre Op2 TTOp2 ==  VISOp2

Op =b Op1 _ Op2

Test case structuring refers to structuring the derived testing information in test template hierarchies and as sequences of operation invocations. These sequences correspond to the testing information derived from a behavioural notation. We de ne a structure that combines the test template hierarchies of the operations with a set of test cases (TC ), each of which is an operation sequence. TEST TTHOp1 : TTOp1  STRATEGY   TTOp1 TTHOp2 : TTOp2  STRATEGY   TTOp2 TC : STRATEGY  (seq( Op ))

In the next two sections we present views of functionality and interaction de ned by Duke and Harrison for MATIS, and we derive testing information from them using the extended TTF.

3 Functionality The Multi-modal Airline Travel Information System (MATIS) allows a user to query a database about ight information via input from modalities such as 3

Type mismatches can arise if disjoined schemas have variables with the same name and dierent types. In our experience this is rare when variable names are meaningful, however, renaming can be applied if required.

speech, keyboard and mouse. In Section 3.1 we abstract from the details of multi-modal input and model queries, results and the underlying database. In Section 3.2 we derive a test suite using the extended TTF.

3.1 Speci cation For the functionality view, the given sets model eld names, eld values and query identi ers. A type for records is also de ned ( denotes a nite, partial function). [Name ; Value ; QID ]

Record == Name  Value

The system state schema MATISF consists of queries, available results and the database. The invariant requires that query identi ers for results also identify queries, that rs contains only records in the database, and that every record in the database contains all elds. Initially the system contains no queries (db is deliberately unconstrained). MATISF qs : QID  Record rs : QID   Record db :  Record dom qs S(ranrsrs) dom  db 8 r : db  dom r = dom(S db )

MATISFI MATISF qs =  rs = 

NIT

Operations are de ned for creating queries, providing name-value pairs for a query, and using a query to search the database. Operations for changing the database are not provided. Such operations would be performed by a dierent user (e.g., a system administrator) and could, therefore, be the subject of additional views. Although the database is unchanged in this speci cation (which could have been speci ed by a customised
MATISF schema), it is used to calculate query results and so is relevant for testing purposes. Operation CreateF models query creation (schema inclusion pre xed by
denotes introduction of the variables of the included schema in both unprimed (before-state) and primed (after-state) form). An unused query is generated internally and is mapped to an empty record. The results and database are unchanged. CreateF
MATISF 9 q : QID  q 2= dom qs ^ qs = qs [ fq 7! g rs = rs ^ db = db 0

0

0

Operation ProvideF models user provision of a value for a named eld on a given, existing query ( denotes relational overriding). All other queries are unchanged ( denotes domain subtraction) as are the results and database. Note that the query to receive input is identi ed by an input (q ?) but that operation CreateF does not output the newly created QID . In the presentation view (not included), the set of queries is de ned to be perceivable by a user. ProvideF
MATISF q ? : QID n ? : Name v ? : Value q ? 2 dom qs qs q ? = qs q ?  fn ? 7! v ?g fq ?g  qs = fq ?g  qs rs = rs ^ db = db 0

0

0

0

Operation SearchF models the generation of results for a given query with all other results unchanged. SearchF
MATISF q ? : QID q ? 2 dom qs rs q ? = fr : db j qs q ?  r g fq ?g  rs = fq ?g  rs qs = qs ^ db = db 0

0

0

0

3.2 Testing We derive testing information for the functionality view using the extended Test Template Framework (TTF). Following simpli cation4, we derive valid input spaces for MATISF operations. We use subscripts to identify the operation from which each valid input space is derived. VISCF =b [MATISF ] VISPF =b [MATISF ; q ? : QID ; n ? : Name ; v ? : Value j q ? 2 dom qs ] VISSF =b [MATISF ; q ? : QID j q ? 2 dom qs ]

We de ne types for all MATIS functionality test templates. 4

We assume, for example, that the cardinality of QID is arbitrarily large.

TTCF ==  VISCF

TTPF ==  VISPF

TTSF ==  VISSF

Testing information is derived using testing strategies. In this paper we use simple strategies to illustrate our approach. A common, intuitive testing strategy is 0-1-many, based on the cardinality of a container type. We refer to this as (one example of) type-based selection which, more generally, involves selecting values for a variable on the basis of its type. We use a degenerate form of typebased selection considering only the 0-many cases. Domain propagation is a partition analysis strategy devised in conjunction with the development of the TTF [Stocks, 1993]. For domain propagation, a partition is formed by considering the subdomains applicable to a particular speci cation operator. We identify type-based selection (TB ) and domain propagation (DP ) as particular strategies. and we construct a generic operation by disjoining the individual operations. TB ; DP : STRATEGY OpMATIS =b CreateF _ ProvideF _ SearchF

We structure MATIS testing information by declaring a schema which includes a test template hierarchy for each operation and a set of operation sequences representing behavioural test cases. We then instantiate the schema. TESTMATIS TTHCF : TTCF  STRATEGY   TTCF TTHPF : TTPF  STRATEGY   TTPF TTHSF : TTSF  STRATEGY   TTSF TCMATIS : STRATEGY  (seq( OpMATIS )) TMATIS : TESTMATIS

For testing CreateF , we recall that only state variable qs is changed. We choose to apply a type-based selection strategy checking empty and non-empty values of qs which is of set type. TTCF 1 =b [VISCF j qs = ] TTCF 2 =b [VISCF j qs 6= ] TMATIS :TTHCF (VISCF ; TB ) = fTTCF 1; TTCF 2g :

:

:

:

For testing ProvideF , we recall that the state variable qs is changed but only at the given query. The rationale for this test is an identi cation that relational override () can be partitioned on whether the overriding has any eect5 . We 5

More precisely, we partition on whether the intersection of the domains of the overridden and overriding relations is empty or non-empty. Given S  T == ((dom T  S ) [ T ) then  is redundant if dom T \ dom S = , i.e., dom T  S = S .

choose to apply a domain propagation strategy over relational override recalling that the overriding relation (right operand) is the single pair (n ?; v ?). TTPF 1 =b [VISPF j n ? 2 dom(qs q ?)] TTPF 2 =b [VISPF j n ? 2= dom(qs q ?)] TMATIS :TTHPF (VISPF ; DP ) = fTTPF 1; TTPF 2g For testing SearchF , we recall that the results are database records matching the given query. We choose to again apply type-based selection and we check empty and non-empty values of the set of results (i.e., whether this is the rst or a subsequent result). TTSF 1 =b [VISSF j rs = ] TTSF 2 =b [VISSF j rs 6= ] TMATIS :TTHSF (VISSF ; TB ) = fTTSF 1; TTSF 2g Test templates are derived until the tester is satis ed that the subdivisions of the valid input space are adequate to achieve satisfactory or required coverage. The Test Template Framework leaves decisions such as adequacy, stopping point, and strategy selection under the control of the (human) tester, which includes the possibility of automated or statistical coverage and strategies. A dierent tester may have chosen, for example, to apply a 0-1-many strategy to VISCF rather than our simple two-way partition. The Test Template Framework permits de nition of concrete test data from the abstract test templates and use of the speci cation as an oracle to determine the success or failure of each test. Examples are available in other TTF papers. :

:

:

:

:

:

:

:

4 Interaction MATIS presents information to users via the visual modality and accepts speech, keyboard and mouse input. Input modalities can be arbitrarily interleaved permitting deictic references6 . The interaction speci cation by Duke and Harrison is at the more abstract level of operation invocation so the issue of modality does not arise. The functionality view in the previous section constrains when a user can invoke abstract operations. For example, searching is not permitted when there are no queries. An interaction (or dialogue) view is explicitly concerned with the interleaving of abstract operations, particularly in conjunction with task and user modelling. We consider interaction views in Section 4.1 and testing under the extended TTF in Section 4.2.

4.1 Speci cation

Duke and Harrison present three models for ordering the MATIS abstract operations. They use CSP and represent queries as processes with the abstract 6

Deictic references use pointing and multiple modalities, for example, pointing to a city on a map with a mouse while saying \That city".

operations of the functionality and presentation views as alphabets (represented here by dropping the trailing F . Query = Create ! ( S  Provide ! S ) 2 Search ! STOP Query = Create ! ( S  Provide ! S 2 Search ! S ) Query = (Create ! ( S  Provide ! Search ! S )) n fSearch g

(1) (2) (3)

The rst model permits a single search from a query. A query is created and then may be repeatedly edited until the user is satis ed and initiates a search. Discarding of the query is represented by STOP and user volition is represented by the external choice operator (2). The second model permits multiple searches from a query. Rather than terminating, searching results in the system providing the user with an ongoing choice to either edit the query or search. The third model moves dialogue initiative for searching from the user to the system. Each query update operation is followed by a search operation, similar to keystroke `clairvoyance' (e.g., emacs incremental search that locates the next partial match as search characters are typed). Note that the search operation has been hidden to indicate that it is no longer available for synchronisation with the environment. Duke and Harrison select the second model for further re nement into an interactor speci cation. We derive testing information from this model in the next section. Figure 1 contains a nite state machine representing this model. I

()*+ /.-, /

C

()*+ /.-, /

e

P ;S

Fig. 1. Finite state machine for multiple searches We are not advocating nite state machines as the most appropriate representation for interactive behaviour but observe that in this case a nite state machine will serve. For example, the three models constrain the behaviour of single queries but provide no guidance on the relationships of multiple queries. The interleaving of multiple queries could be expressed by kni=1 i :Query which would require an extended state machine notation such as Statecharts (or test derivation directly from the CSP).

4.2 Testing There is extensive work on testing derived from behavioural speci cations, particularly in the area of protocol testing. Most of this work uses nite-state machines or notations that can be mapped to labelled transition systems, such as LOTOS and CSP (see [Bochmann and Petrenko, 1994] for a good overview). In this section we consider arc coverage and invalid transitions. Implementations of nite state automata can exhibit errors related to transitions or states.

Transition errors include extra transitions, missing transitions or incorrect transitions. State errors include missing or extra states. Arc coverage involves devising a set of traces that performs every transition as a means to detect missing states, and missing and incorrect transitions. We de ne the operations in terms of the signature of the generic operation. In a full speci cation, a presentation operation would have been included with each functionality operation. Create =b [OpMATIS j CreateF ] Provide =b [OpMATIS j ProvideF ] Search =b [OpMATIS j SearchF ] We distinguish arc coverage (AC ) as a strategy and incorporate the trace of a possible arc cover (derived by inspection) for Figure 1 into the extended TTF7. Traces are assumed to start from the Init state (which is not an operation). AC : STRATEGY TMATIS :TCMATIS (AC ) = fhCreate ; Provide ; Search ig

One approach to detecting extra states and transitions is to attempt to exercise all possible transitions from all possible states. An arc cover exercises all valid transitions so only invalid transitions remain8. We distinguish invalid transitions (IT ) as a test strategy and add a possible set of invalid transitions for Figure 1 to the extended TTF (also derived by inspection). For larger systems the number of invalid transitions will make this strategy impractical and it is used here only for illustration. IT : STRATEGY TMATIS :TCMATIS (IT ) = fhProvide i; hSearch i; hCreate ; Create ig

Tests derived from an interaction view are traces of operation invocations rather than the single invocations derived from the functionality and presentation views. As Dick and Faivre observe, a nite state machine constructed from a model-based speci cation can be used for test sequencing and speci cation coverage [Dick and Faivre, 1993]. We apply this approach to Object-Z speci cations for class testing [Murray et al., 1998], but we suggest that behavioural notations are preferable for de ning interactive behaviour. Each element of TCMATIS is a sequence of sets of Z bindings (accepted by the f uzz typechecker) rather than a sequence of operation schemas. However, Z operation schemas are also sets of bindings, distinguished only by being operands of the schema calculus. Operation schemas can be used in set contexts (as here) but not vice versa. Since we don't apply schema calculus operations to the elements of the sequences, no problems arise. 8 In practice, exercising invalid transitions will require knowledge of implementation decisions, for example, whether invalid transitions are disabled, equivalent to or raise an exception. 7

skip

5 Conclusion In this paper we have proposed an initial framework for structuring testing information derived from partial speci cations of interactive systems. We derived testing information from (slightly modi ed) MATIS views de ning interactive system functionality and interaction. We are not, however, advocating the particular notations or testing strategies used in this paper. There has been considerable related work on formal speci cation of interactive systems, multiparadigm speci cations, speci cation-based testing and user interface testing. However, we are only aware of the work of Yip and Robson on using formal speci cations of interactive systems to derive testing information [Yip and Robson, 1991]. Their approach uses a form of state transition diagram that incorporates presentation information, in conjunction with a notation to express input-output behaviour. Tests are derived for transition coverage (by an algorithm to solve the postman's tour problem) with the input-output speci cation used to derive oracles. Our aim, in contrast, is to use widely accepted, standard notations, with a variety of test derivation mechanisms. We are investigating black-box testing of interactive systems based on abstract, multiparadigmspeci cations. Our initial framework does not include presentation but this is a straightforward extension. There is also scope for testing based on the construction of interactors by view composition [Duke and Harrison, 1993; Hussey and Carrington, 1997]. One open issue in our initial framework is the formation of the nal test set. We can form a nite state machine representation from the test template hierarchies which can then be arc covered. This will yield a set of traces which can be combined with the traces derived from the interaction speci cation to provide the nal test set for the system under test. We are yet to nalise this aspect of our approach. Beyond that, we are developing an abstract model that incorporates a model of interactive system speci cations and a model of speci cation-based testing. We plan to use this to re ne our framework. We are also investigating testing based on labelled transition systems. The contribution of this paper is twofold: a demonstration that dierent viewpoints are useful for speci cation-based testing and the introduction of an initial framework for structuring the testing information derived from the dierent views.

Acknowledgements We thank the anonymous reviewers for their detailed and helpful comments. Ian MacColl is supported by an Australian Postgraduate Award and a Telstra Research Laboratories Postgraduate Fellowship.

References Bochmann, G. v. and Petrenko, A. (1994). Protocol testing: Review of methods and relevance for software testing. In International Symposium on Software Testing and Analysis (ISSTA94), pages 109{124. ACM. Dick, J. and Faivre, A. (1993). Automating the generation and sequencing of test cases from model-based speci cation. In Formal Methods Europe (FME'93), volume 670 of LNCS, pages 268{284. Springer-Verlag. Duke, D. and Harrison, M. (1993). Abstract interaction objects. Computer Graphics Forum, 12:25{36. Duke, D. J. and Harrison, M. D. (1995). Mapping user requirements to implementations. Software Engineering Journal, 10(1):13{20. Duke, R., Rose, G., and Smith, G. (1995). Object-Z: A speci cation language advocated for the description of standards. Computer Standards & Interfaces, 17:511{533. Harel, D. (1987). Statecharts: A visual formalism for complex systems. Science of Computer Programming, 8(3):231{274. Horowitz, E. and Singhera, Z. (1993). Graphical user interface testing. Technical Report USC-CS-93-548, Department of Computer Science, University of Southern California. Hussey, A. and Carrington, D. (1997). Specifying a Web browser interface using Object-Z. In Formal Methods in Human-Computer Interaction, chapter 8, pages 157{174. Springer-Verlag. IEEE (1990). Standard Glossary of Software Engineering Terminology. In IEEE Software Engineering Standards Collection. IEEE. Std 610.12-190. Kepple, L. R. (1993). Testing GUI applications: Some pictures really are worth a thousand words. The X Journal, 2(6):59{66. Koning, P., Strooper, P., and Udink, R. (1996). Testing multi-media systems|A case study. In Asia-Paci c Software Engineering Conference (APSEC '96), pages 413{419. IEEE. Murray, L., Carrington, D., MacColl, I., McDonald, J., and Strooper, P. (1998). Formal derivation of nite state machines for class testing. In Z User Meeting (ZUM98), volume 1493 of LNCS, pages 42{59. Springer-Verlag. Ostrand, T., Anodide, A., Foste, H., and Goradia, T. (1998). A visual test development environment for GUI systems. In International Symposium on Software Testing and Analysis (ISSTA98), pages 82{92. ACM. Stocks, P. (1993). Applying formal methods to software testing. PhD thesis, Department of Computer Science, The University of Queensland. Stocks, P. and Carrington, D. (1996). A framework for speci cation-based testing. IEEE Transactions on Software Engineering, 22(11):777{793. Yip, S.W.L. and Robson, D.J. (1991). Applying formal speci cation and functional testing to graphical user interfaces. In CompEuro 1991, pages 557{ 561. IEEE.