Tools for Test Case Generation - Semantic Scholar

Tools for Test Case Generation Dagstuhl Research Seminar – 01/14/04 Axel Belinfante, Lars Frantzen, Christian Schallhart [email protected], [email protected], [email protected]

University of Twente, Radboud University Nijmegen, Technical University Munich

Tools for Test Case Generation – p.1/39

Introduction A selection of tools will be briefly introduced Up to date information uneasy to obtain Selection cannot be complete or fair Speaker is changing frequently


Tools Tool

Languages

CAR

Access

Lutess

Lustre

A

n.p.

Lurette

Lustre

A

n.p.

GATeL

Lustre

A

n.p.

Autofocus

Autofocus

A

n.p.

Conformance Kit

EFSM

R

no

Phact

EFSM

R

no

TVEDA

SDL, Estelle

R

no

AsmL

AsmL

R

www

Cooper

LTS (Basic LOTOS)

A

www

TorX

LTS (LOTOS, Promela, FSP)

A

www

TGV

LTS-API (LOTOS, SDL, UML)

A

www

STG

NTIF

A

winter ’04?

AGEDIS

UML/AML

?

?

TestComposer

SDL

C

$

Autolink

SDL

C

$ Tools for Test Case Generation – p.3/39

Lutess testing time-synchronous programs based on Lustre SUT, oracle: binary time-sync. programs environment desc. : observer in Lustre testing method parameterization fully automated test generation & execution Tools for Test Case Generation – p.4/39

Lutess – Lustre dataflow oriented programming language time-synchronous declarative


Lutess – Lustre dataflow oriented programming language time-synchronous declarative node COUNTER(val_init, val_incr : int; reset : bool) returns (n : int); let n = val_init -> if reset then val_init else pre(n)+val_incr; tel; even=COUNTER(0,2,false); mod5=COUNTER(0,1,pre(mod5)=4);


Lutess – Testing Method uniform random testing

if condition

is true with probability

operational profile-based testing holds

pattern-based testing

property-guided testing

combination thereof Tools for Test Case Generation – p.7/39

Lutess – Test Sequence Generation sequence is generated during execution BDD for environment transition used to invert the transition relation labelled with weights for sampling pass/fail verdict computed by the oracle SUT/environment limited to Boolean IO Tools for Test Case Generation – p.8/39

Lurette comparable to Lutess SUT, oracle: C-file with a predefined interface environment desc. : observer in Lustre IUT and environment can have numerical IO testing interest part of environment desc. fully automated test generation & execution Tools for Test Case Generation – p.9/39

GATeL SUT, env. desc., test objective: as Lustre source test objective contains assert-statements: safety property reach-statements: liveness property based on Prolog with finite domains (no reals) interactive & automated test generation


GATeL – Test Sequence Generation sequence is generated backwards initial constraint set: conditions to be reached and assertions variables are instantiated uniformly randomly constraint set is expanded backwards iterated until a valid sequence is found, or aborted in case of maximum length or contradiction Tools for Test Case Generation – p.11/39

Autofocus graphical tool to model distributed systems hierarchically time-synchronous communicating EFSMs model of the SUT SUT itself as binary test case specification fully automated test generation & execution (symbolic execution based on CLP) Tools for Test Case Generation – p.12/39

Conformance Kit Developed at KPN Research in the early 90s Not publicly available Aimed at supporting automatic testing of protocol implementations (E)FSMs serve as specifications Gate concept allows communicating EFSMs


Conformance Kit - Fundamental Tools EFSM

FSM converter

FSM minimization EFSM simulator EFSM composer (for communicating EFSMs)


Conformance Kit - Test Generation Transition Tour (if FSM is strongly connected) No end-state correctness is checked Partition Tour – single sequences for each transition: 1. A synchronising sequence to transfer the FSM to its initial state 2. A transferring sequence to move to the source state of the transition to be tested 3. A UIO sequence which verifies the correct end-state of the transition Tools for Test Case Generation – p.15/39

Conformance Kit - Interfaces A textual EFSM-representation is needed (including guard definitions) The generated test suite is expressed in TTCN-MP (TTCN 2) TTCN-GR is possible via an export to LATEX


Conformance Kit - PHACT Philips developed in 1995 a set of tools called PHACT (PHilips Automated Conformance Tester) Extends the Conformance Kit with the ability to execute the TTCN test suite To compile a tester, modules in C, VHDL and Java are supported


TVEDA Developed by the former R&D centre CNet of the France Telecom between 1989 and 1995 The final version TVEDA V3 was released 1995, based on Aimed at supporting automatic conformance testing of protocols Goal: achieving a complete EFSM-transition coverage TVEDA and TGV have been partially incorporated into the tool TestComposer Tools for Test Case Generation – p.18/39

AsmL - Background Yuri Gurevich developed in the early 90s the concept of Evolving Algebras His motivation was to develop a computation model which is capable of describing any algorithm at its appropriate abstraction level Fundament: simple notions from universal algebra Later on Evolving Algebra was renamed to Abstract State Machine, short ASM Tools for Test Case Generation – p.19/39

AsmL At Microsoft Research a group called Foundations of Software Engineering is developing the Abstract State Machine Language, short AsmL It is a .NET language and therefore embedded into Microsoft’s .NET framework AsmL is aimed at specifying systems in an object-oriented manner based on ASMs AsmL and the .NET framework can be freely downloaded Tools for Test Case Generation – p.20/39

AsmL - Test Generation AsmL embeds a conformance test facility: 1. the specification ASM is transformed into a FSM 2. well known FSM-based algorithms (rural Chinese postman tour) are applied to generate a test suite Guidance by the user is necessary to construct test cases


AsmL - Interfaces The SUT must be given as any managed .NET assembly, written in a .NET language The binding of the specification methods with the implementation methods is supported by a wizard A test manager is responsible to carry out the generated test cases


AsmL - Summary Generating a FSM out of a ASM is a difficult task which requires a certain expertise from the tester The resulting FSM may become nondeterministic (even if the specification ASM is not) Quite a new research topic


Cooper Developed in academic/industrial research project LOTOSPHERE (1989-1992) Implementation relation CONF Based on CO-OP method for deriving canonical testers Educational (academic) value: illustrate CO-OP, under control of GUI Available for download as part of LITE


TorX - Background Developed in dutch academic/industrial research project Côte de Resyste: Conformance Testing of Reactive Systems which run from 1998 until 2002, and aimed at: develop methods, techniques, and tools for automated specification based testing based on formal theory and validate developed methods, techniques and tools in practice Tools for Test Case Generation – p.25/39

TorX - Tool Implementation relation IOCO Test generation and test execution on the fly : As soon as a test step is generated, it is immediately executed, after which the next step is generated Experimental campaigns to specify multiple torx runs Open architecture, easy to extend, and to run with TorX components on multiple machines Available for downloading Tools for Test Case Generation – p.26/39

TorX - Test Guidance By default: choices made randomly random walk in spec

verdict is tuple: (instead of traditional

Optional Combinator combines (intersects) behaviour allowed by spec and test purpose )

test purpose tool jararaca uses regular expression-style input language any other formalism that maps on LTS can be used (LOTOS spec in early experiments) Tools for Test Case Generation – p.27/39

TorX - Interfaces Open interfaces to spec (explorer, primer) and SUT (Adapter, endecoding glue) Specification languages LOTOS (BSG, AUT) via CADP, Promela, FSP via LTSA; Test purpose language jararaca any other language that maps on LTS connection to SUT: to be written by user (reuseable, well defined interface)


TorX - Future Symbolic testing (data parametrization) (LOTOS, via Smile), Promela Initial experiments with timed testing Studied in project Stress (Twente/Nijmegen)


TGV Developed by Vérimag and IRISA Rennes Implements implementation relation IOCO Integrated in TestComposer of ObjectGéode Also available as part of CADP tool kit Specification languages: LOTOS, BCG, AUT, SDL, UML, IF Test suite can be generated as TTCN, or In a graph format of CADP (AUT, BCG) Expandable wrt. input/output formats Tools for Test Case Generation – p.30/39

TGV - Test Generation Takes spec and test purpose Test purpose is automaton with special states Accept and Reject and wildcard transitions Formulating test purpose may need some trial and error Computes cross product between spec and test purpose Result can be test suite or single test case


STG Developed by IRISA Rennes (France) Enhances ideas underlying TorX and TGV with symbolic treatment of data (satisfyability of guards, data value generation) Implements (symbolic) IOCONF Test generation (of symbolic test cases) and execution Public downloadable maybe after summer?


AGEDIS AGEDIS: Automated Generation and Execution of test suites for DIstributed component-based Software Project running from October 2000 until the end of 2003 Basically the TGV algorithms are used for the test generation Another tool which partly found its way into AGEDIS is GOTCHA from IBM UML 1.4 based Tools for Test Case Generation – p.33/39

TestComposer Commercially released in 1999 as a component of ObjectGéode by Verilog In December 1999 Telelogic acquired Verilog Together with Autolink (also Telelogic) they form the two major SDL toolsets TVEDA and TGV constitute its fundament


TestComposer - Interfaces SDL specifications serve as inputs An API allows the user to construct interfaces with arbitrary test specification languages A module for TTCN is already included


TestComposer - Summary TestComposer is very similar to Autolink A pragmatic approach was guiding the development As a consequence the tool is embedded into an industrial-strength environment The price paid for this embedding was the sacrifice of the sound theoretical foundation Rumours...


Autolink Developed by Institute for Telematics in Lübeck Based on former work of SaMsTaG project Integrated in Telelogic’s Tau tool kit in 1997 Attention on producing readable TTCN output Implementation relation reverse engineered in [GOGA01]


Autolink - Test Generation Complete SDL system is taken as SUT PCO’s: SDL channels linking to environment Test generation guided by test purposes as MSCs (hand-written, obtained by simulation, or generated fully automatically) Observer process can guide test generation Goal: achieve maximum coverage Basic coverage unit: symbol of specification Support for distributed testing Tools for Test Case Generation – p.38/39

Tool Comparison Approaches: Theoretical analysis Common denominator (implementation relation?) Comparison not always feasible Benchmarking Compare error detecting power (for real!) e.g. feature interaction contest e.g. conference protocol benchmark
