Learning-based Software Testing using Symbolic Constraint ... - KTH

Motivation and Background Learning-based Testing Frameworks Results and Future Work

Learning-based Software Testing using Symbolic Constraint Solving Methods Fei Niu School of Computer Science and Communication KTH Royal Institute of Technology

Licentiate Thesis Presentation, 2011

Fei Niu

LBT using Symbolic Constraint Solving Methods


Outline 1

Motivation and Background Software Testing Black-box Testing

2

Learning-based Testing Frameworks Overview Paper 1: LBT for Numerical Procedures Paper 2: LBT for Reactive Systems

3

Results and Future Work Empirical Performance Evaluation of LBT Future Work

Fei Niu



Software Testing Black-box Testing

Outline 1


2


3


Fei Niu




Background Software systems Ubiquitous Increasingly complex and critical

Software testing One of the most important approaches to high-quality software Remains time-consuming and ad-hoc

High economic impact of software testing: Cost up to 30-50% of a project budget Achievable savings up to $22.2 billion in U.S (2002)

Fei Niu




What is Software Testing Software Testing In One Sentence Software testing consists of the dynamic verication of the behaviour of a system under test (SUT) on a nite set of test cases, suitably selected from the usually innite execution domain, against the expected behaviour. Test case generation (TCG) Test objectives, selection strategies and adequacy criteria

Test case execution Test case evaluation (the oracle) Automate them all! Fei Niu














Classes of Testing Methods

Glass-box testing Internal structure (source code)

Black-box testing Various specications or models More scalable and suitable in many cases Functional testing (Input&Output)

Non-functional testing Quality of software such as scalability, reliability, security...

Fei Niu




Classes of Testing Methods

Glass-box testing Internal structure (source code)

Black-box testing Various specications or models More scalable and suitable in many cases Functional testing (Input&Output)

Non-functional testing Quality of software such as scalability, reliability, security...

Fei Niu




Outline 1


2


3


Fei Niu




Black-box Testing Strategies Random testing (without a strategy) Boundary value analysis Equivalence partitioning Inductive testing ...

Specication-based testing Model-based testing e.g. all-verdict/path/state/transition coverage criteria

Learning-based testing (LBT)

Fei Niu




Characteristics of Learning-based Testing Functional specication-based black-box testing Test objective: generates test cases that falsify given specication

Make use of machine learning techniques Model mining

Make use of the great power of formal methods Automated test case generation and evaluation

Fully automated testing method

Fei Niu



Overview Paper 1: LBT for Numerical Procedures Paper 2: LBT for Reactive Systems

Outline 1


2


3


Fei Niu




Generic Framework of Learning-based Testing Requirements Specification TCG

Input + output

input

(Constraint Solver)

SUT

Oracle (Satisfiability Checker)

Model

Input + output

Learner

Iterative testing Inductive learning Constraint solving Fei Niu





Input + output

input

(Constraint Solver)

SUT


Model

Input + output

Learner





Inductive Learning Inductive learning Infer functions or languages from examples e.g. automata learning (L*, RPNI, CGE, etc)

Classes of (inductive) learning algorithms

Active Passive Incremental (Online) v.s.

v.s.

Oine

In LBT, we use incremental learning Iterative The sequence of hypothesis converges after n iterations Mn =∼ Mn+1 =∼ Mn+2 =∼ . . . (Learning in the limit)

Fei Niu





Input + output

input

(Constraint Solver)

SUT


Model

Input + output

Learner





Constraint Solving Techniques Satisability problem: is a formula φ true over a model M for some assignment α , denoted by M , α |=? φ

In LBT, we use constraint solving algorithms to compute αt that violates the specication Req , denoted by ML , αt |= ¬Req

where ML is the learned model and test cases can be derived from αt . Alternative test case generation method (random) The rst test iteration where the model is empty No solution can be found by the constraint solver Fei Niu




Generic Framework of Learning-based Testing Requirements Specification TCG (Constraint Solver)

Input + output

input SUT


Model

Input + output

Learner

Iterative testing Inductive learning Constraint solving Instantiated with dierent type of SUTs Fei Niu




Procedural and Reactive Systems Procedural systems: Compute from an input to an output (if terminate) f :I →O i

f

o

Reactive systems: Continuously interact with the environment F : (T → I ) → (T → O ) … i2 i1 i0

F

Fei Niu

… o2 o1 o0




Publications

1

2

Karl Meinke and Fei Niu, A Learning-based Approach to Unit Testing of Numerical Software, A. Petrenko et al. (eds). In pp 221-235, Proc. 22nd IFIP International Conference on Testing Software and Systems (ICTSS 2010), LNCS 6435, Springer Verlag, Berlin, 2010. Karl Meinke and Fei Niu, Learning-Based Testing for Reactive Systems using Term Rewriting Technology, B. Wol and F. Zaidi (Eds.). In pp. 97-114, Proc. 23rd IFIP International Conference on Testing Software and Systems (ICTSS 2011), LNCS 7019, Springer Verlag, Berlin, 2011.

Fei Niu




Outline 1


2


3


Fei Niu




Contributions

Proposed the learning based testing framework An ecient modelling method for numerical procedures A test case generation algorithm using an existing quantier elimination algorithm Experiments showing the eciency of LBT compared with random testing

Fei Niu




Specifying Numerical Procedures

Requirements specication language

Hoare triple: {Pre } S {Post } Pre and Post are (quantied) rst-order logic formulas over real closed elds

Example Computing square root by Newton's method: {in ≥ 0} S {|out ∗ out − in | ≤ ε}

Fei Niu




Modelling and Inferring Numerical Procedures (1) Piecewise polynomial interpolation for S (x ) with m tests

S (x ) ≈ f1 (x ) ] f2 (x ) ] · · · ] fm (x ) (d = 3, n = 1, pts = (d + 1)n = 4) y

10

Overlapping Area

r1

5

f1 HxL f2 HxL

-5

5

10

15

x

r2 -5

Centre Points

Fei Niu




Modelling and Inferring Numerical Procedures (2) A numerical procedure S with n-dimension input and m executed test cases S (¯x ) ≈ ym ≡ f1 (¯x ) ] f2 (¯x ) ] · · · ] fm (¯x )

On the next testing iteration with m + 1 executed test cases S (¯x ) ≈ ym+1 ≡ f1 (¯x ) ] f20 (¯ x ) ] · · · ] fn (¯ x ) ] fm +1 (¯ x) S is eciently and incrementally learned Local updating Learning in the limit (by Weierstrass Approximation Theorem)

Convergence by Monte Carlo method Fei Niu




Test Case Generation for Numerical Procedures

Choose the best converged polynomial of center point c¯ = (c0 , c1 , . . . , cn−1 ) and radius r R |= |¯ x − c¯|2 ≤ r 2 ∧ Pre (¯x ) ∧ ¬Post (¯x ) ∧ fc¯ (¯x ) = 0

Constraint solving algorithm for test case generation Quantier elimination method: Hoon-Collins cylindric algebraic decomposition (CAD) Implemented in Mathematica

Fei Niu




Outline 1


2


3


Fei Niu




Contributions

A modelling method for reactive systems: extended Mealy automata over abstract data types (ADTs) A bounded model checking algorithm utilising narrowing techniques Experiments that show the eciency of LBT compared with random testing.

Fei Niu




Extended Mealy Automata over ADTs Extended Mealy automata over abstract data types (EMA) f(x, a) / g(y, b) S

S'

Signature ΣA = (S , Σ, input , output ) of EMA A

initial state: qA0 ∈ Astate State transition function: δA : Astate × Ainput → Astate Output function: λA : Astate × Ainput → Aoutput

Functions to be inferred:

General state transition function: δA∗ : Astate × A∗input → Astate General output function: λA∗ : Astate × A+input → Aoutput Fei Niu




Abstract Data types - STACK Abstract data type STACK specied as (Σstack , Estack ) Sorts: {Stack , Element } Constant and functional symbols

e0 , e1 , . . . , en , error :→ Element empty :→ Stack Push : Element × Stack → Stack Pop : Stack → Stack Top : Stack → Element

Axioms EStack (x ∈ XElement , s ∈ XStack ) Pop (Push(x , s )) = s Top (Push(x , s )) = x Pop (empty ) = empty Top (empty ) = error

Fei Niu




Term Rewriting A symbolic computational model with ADTs A rewriting rule is a directed equation: l → r Term rewriting system (TRS) R A set of rewriting rules Rewriting relations (One step rewriting):

t →R t 0 if ∃l → r ∈ R , σ , p ∈ O¯ (t ) such that t |p = σ l and t 0 ≡ t [σ r ]|p ∗ Reexive transitive closure (many step rewriting) t →R t 0

Example

∗ top (pop (push(e1, push(e0 , s )))) →R e0 , R ≡ EStack

Fei Niu




Term Rewriting A symbolic computational model with ADTs A rewriting rule is a directed equation: l → r Term rewriting system (TRS) R A set of rewriting rules Rewriting relations (One step rewriting):

t →R t 0 if ∃l → r ∈ R , σ , p ∈ O¯ (t ) such that t |p = σ l and t 0 ≡ t [σ r ]|p ∗ Reexive transitive closure (many step rewriting) t →R t 0

Example

∗ top (pop (push(e1, push(e0 , s )))) →R e0 , R ≡ EStack

Fei Niu




Complete TRS Strongly normalising there is no innite sequence of rewrite steps

Conuent (Church-Rosser property) t

t0

*

*

*

Fei Niu

t1

* t'




Inferring and Specifying EMAs CGE - an incremental learning algorithm for Mealy automata Automata learning: infer equivalence classes of states Builds a complete TRS R of states (input strings) More compact than tables

Requirements specication language LTL(Σ, X ) Quantier-free rst-order linear temporal logic (FO-LTL) with equality and free variables in ∈ Xinput and out ∈ Xoutput Includes past time temporal operators i.e. F −1 , G −1

Example

G( (in = x ) ∧ X( (in = y ) Fei Niu

=⇒

X(out = x + y ) ) )




Inferring and Specifying EMAs CGE - an incremental learning algorithm for Mealy automata Automata learning: infer equivalence classes of states Builds a complete TRS R of states (input strings) More compact than tables

Requirements specication language LTL(Σ, X ) Quantier-free rst-order linear temporal logic (FO-LTL) with equality and free variables in ∈ Xinput and out ∈ Xoutput Includes past time temporal operators i.e. F −1 , G −1

Example

G( (in = x ) ∧ X( (in = y ) Fei Niu

=⇒

X(out = x + y ) ) )




Model Checking EMA for TCG (1) Given an EMA A, a satisfaction relation for this LTL language is dened A, n, ¯i , α |= φ φ ≡ true |t = t 0 |¬φ |φ ∧ ϕ| · · · |F φ | · · ·

To generated test cases for A that is specied in Req A, 0, ¯i , α |= ¬Req

This is a model checking problem.

Fei Niu




Model Checking EMA for TCG (2) ¬Req −→ SatSet0 (¬Req ) that is inductively dened by

SatSetn (in = t ) −→ {{¯xn = t }} SatSetn (out = t ) −→ {{λ (δ ∗ (q0 , x¯o x¯1 . . . x¯n−1 ), x¯n ) = t }} SatSetn (in 6= t ) −→ {{¯xn 6= t }} SatSetn (out 6= t ) −→ {{λ (δ ∗ (q0 , x¯0 x¯1 . . . x¯n−1 ), x¯n ) 6= t }} ... SatSetn (φ ∧ ψ) → {Sφ ∪ Sψ |Sφ ∈ SatSetn (φ ), Sψ ∈ SatSetn (ψ)} ... SatSetn (X φ ) −→ SSatSetn+1 (φ ) SatSetn (F φ ) −→ loopbound SatSetn+k (φ ) k =0 ...

¬Req −→ S = {S1 , · · · , Sm } and Si is a disunication problem {s1 =? t1 , s2 6=? t2 , . . . , sn =? tn }

modulo A = E R , which can be solved by narrowing S

Fei Niu




Model Checking EMA for TCG (2) ¬Req −→ SatSet0 (¬Req ) that is inductively dened by

SatSetn (in = t ) −→ {{¯xn = t }} SatSetn (out = t ) −→ {{λ (δ ∗ (q0 , x¯o x¯1 . . . x¯n−1 ), x¯n ) = t }} SatSetn (in 6= t ) −→ {{¯xn 6= t }} SatSetn (out 6= t ) −→ {{λ (δ ∗ (q0 , x¯0 x¯1 . . . x¯n−1 ), x¯n ) 6= t }} ... SatSetn (φ ∧ ψ) → {Sφ ∪ Sψ |Sφ ∈ SatSetn (φ ), Sψ ∈ SatSetn (ψ)} ... SatSetn (X φ ) −→ SSatSetn+1 (φ ) SatSetn (F φ ) −→ loopbound SatSetn+k (φ ) k =0 ...

¬Req −→ S = {S1 , · · · , Sm } and Si is a disunication problem {s1 =? t1 , s2 6=? t2 , . . . , sn =? tn }

modulo A = E R , which can be solved by narrowing S

Fei Niu




Disunication Problems Formalise an E -disunication problem S = {s1 =? t1 , s2 6=? t2 , . . . , , sn =? tn }

R ≡E =R reexive symmetric and transitive closure of →R A R -unier σ such that ∀(s =? t ) ∈ S , σ s =R σ t ∀(s 6=? t ) ∈ S , σ s 6=R σ t

Consider the case R = 0/

Find a unier σ such that σ s1 ≡ σ t1 ∧ σ s1 6≡ σ t1 ∧ · · · Syntactic unication by pattern matching Decidable Fei Niu




Disunication Problems Formalise an E -disunication problem S = {s1 =? t1 , s2 6=? t2 , . . . , , sn =? tn }

R ≡E =R reexive symmetric and transitive closure of →R A R -unier σ such that ∀(s =? t ) ∈ S , σ s =R σ t ∀(s 6=? t ) ∈ S , σ s 6=R σ t

Consider the case R = 0/

Find a unier σ such that σ s1 ≡ σ t1 ∧ σ s1 6≡ σ t1 ∧ · · · Syntactic unication by pattern matching Decidable Fei Niu




Narrowing and Disunication Consider the case R 6= 0/ : semidecidable Narrowing Apply a (minimum) unier on a term and the left hand side of a rule of R to make it reducible by R

Narrowing relation t

[p ,l →r ,σ ]

t0

if ∃p ∈ O¯ (t ), l → r ∈ R , σ such that σ t |p = σ l , t 0 = σ (t [r ]p ) Narrowing all terms in S until a syntactic unier can be found

Fei Niu




Narrowing and Disunication Consider the case R 6= 0/ : semidecidable Narrowing Apply a (minimum) unier on a term and the left hand side of a rule of R to make it reducible by R

Narrowing relation t

[p ,l →r ,σ ]

t0

if ∃p ∈ O¯ (t ), l → r ∈ R , σ such that σ t |p = σ l , t 0 = σ (t [r ]p ) Narrowing all terms in S until a syntactic unier can be found

Fei Niu




An example of Narrowing Search Consider the TRS

0+x

(

R=

S (x ) + y

Step a: z + z [r ,σ ] S (x + S (x )) r : S (x ) + y → S (x + y ) σ = {y , z → S (x )}

→x → S (x + y ) z + z =? S(S(0))

a

0 =? S(S(0))

S(x+S(x)) =? S(S(0))

No solution S(S(0)) =? S(S(0))

S(S(x' + S(S(x')))) =? S(S(0))

True R-unifier: { z→S(0) }

No solutions

Fei Niu





0+x

(

R=

S (x ) + y

Step a: z + z [r ,σ ] S (x + S (x )) r : S (x ) + y → S (x + y ) σ = {y , z → S (x )}

→x → S (x + y ) z + z =? S(S(0))

a

0 =? S(S(0))

S(x+S(x)) =? S(S(0))


S(S(x' + S(S(x')))) =? S(S(0))


No solutions

Fei Niu





0+x

(

R=

S (x ) + y

{z → S (0)} S (0)+S (0)→R S (0+S (0))→R S (S (0))

→x → S (x + y ) z + z =? S(S(0))

a

0 =? S(S(0))

S(x+S(x)) =? S(S(0))


S(S(x' + S(S(x')))) =? S(S(0))


No solutions

Fei Niu




Model Checking EMA for TCG (3) Narrowing Solution complete Infeasibly large search space and frequently fails to terminate

Basic narrowing - a more ecient narrowing by Hullot Restrictions on allowable reducing positions Termination can be guaranteed

Model checking EMA by basic narrowing Bounded model checking Decision procedure for some complete TRSs Sound, complete and terminating

Fei Niu



Empirical Performance Evaluation of LBT Future Work

Outline 1


2


3


Fei Niu




Methodology of Performance Evaluation

Iterative random testing (IRT) as the benchmark Benchmark what? Fault detection power is LBT faster than IRT at nding the rst error?

NOT coverage measurement

All faults are injected by mutating SUTs All results are averaged over a large number of SUT runs

Fei Niu




Case Studies for Numerical Procedures (1) Bubble sort algorithm Random generated SUTs Each randomly generated SUT A set of polynomials of random coecients and degrees Randomly generated specications: equational & inequational Faults of dierent sizes are injected in a controlled way 3000 2500 2000 1500 1000 500

mutation 50

Fei Niu

mutation 100

150

200




Benchmark Results Bubble sort algorithm

LBT is 10 times faster than IRT

Random SUTs

æ

Equational spec.

à

Inequational spec.

Performance ratio IRTLBT 14

à

12

à

10 8

æ æ

6 à

4

æ

2 0.01

0.1 Fei Niu

1

à æ

æ à

10

100

Error size H%L




TCP/IP Stack Case Study for Reactive Systems

'#%&' !

(& #$%&' !

(&& $%-&.$

! ! "

!) &$* %

!

+,

(& !

!

(& !

(& !

!) &$*% +,

! !

!) &$*% +,

!

11 states, 12 input symbols, 6 output symbols 5 requirements Fei Niu




Benchmark Results for TCP/IP Model

At the level of logical performance LBT is always (5.5 to 400 times) faster than IRT

At the level of real-time performance LBT is often but not always signicantly faster than IRT (107 times faster in the best case) Much better performance can be easily obtained by better implementation

Fei Niu




Outline 1


2


3


Fei Niu




Future Work

More sophisticated learning algorithms More ecient automata learning algorithms based on CGE Learning algorithms for innite-state EMAs ...

More narrowing strategies Combined with theorem proving techniques Narrowing strategies to deal with more abstract date types

More case studies

Fei Niu


Learning-based Software Testing using Symbolic Constraint ... - KTH

Learning-based Software Testing using Symbolic Constraint ... - KTH

Suggest Documents

Use of Constraint Solving for Testing Software

Constraint-Based Software Testing - Laboratoire I3S

A Symbolic Execution Algorithm for Constraint-Based Testing of ...

Automated Testing using Symbolic Execution and Temporal ...

Automated Testing using Symbolic Execution and Temporal

SOFTWARE TESTING USING GENETIC ALGORITHMS

Parallel Symbolic Execution for Automated Real-World Software Testing

10 Shadow Symbolic Execution for Testing Software Patches

Parallel Symbolic Execution for Automated Real-World Software Testing

Constraint-Based Techniques for Software Testing - Nikolai Kosmatov

Automated Black-Box Testing of Functional Correctness using ... - KTH

A SYMBOLIC APPROACH TOWARDS CONSTRAINT ... - CS UTEP

Using JPF1 format - KTH

Using RSI format - KTH

Using JPF1 format - KTH

Symbolic Search-Based Testing - CiteSeerX

Automated Testing using Symbolic Model Checking and Temporal

Regression Testing of Virtual Prototypes Using Symbolic ... - arXiv

Testing for Software Vulnerability Using Environment Perturbation

Testing Embedded Software using Simulated ... - Semantic Scholar

Using Software Architecture for Code Testing - CiteSeerX

Automatic Software Structural Testing by Using Evolutionary ...

Hypothesis Testing Using Statistical Software

Automated Software Testing Using Metahurestic Technique ... - arXiv