Search-based Software Verification and Validation

Search-based Software Verification and Validation – A motivation Wasif Afzal SWELL PhD Student Blekinge Institute of Technology, Ronneby, Sweden [email protected]

Agenda • • • • • •

Search-based software engineering Search-based software testing Industrial success story Search-based software fault prediction Industrial success story SWELL – national research school in software verification and validation

Search-Based Software Engineering (SBSE) • Application of optimization techniques (OT) in solving software engineering problems. – Optimization is the process of attempting to find the best possible solution amongst all those available. – Examples of OT: Hill-climbing, Simulated annealing, Tabu search, evolutionary methods (Genetic Algorithms & Genetic Programming)

Why SBSE? • Competing constraints • Requirement of near-optimal solutions • Examples: – Allocation of teams to work products – Set of requirements to be part of next release balancing customer requests, resource constraints and requirement interdependencies. – Software testing

Search-Based Software Testing • SW testing is a suitable candidate: – Generation of software tests is an undecidable problem – A program’s input space is very large, exhaustive enumeration is infeasible

• General concept: – Set of possible inputs to a program forms a search space. – The test adequacy criterion is coded as a fitness function. • Fitness function compare and contrast solutions of the search with respect to the overall search goal.

Motivation: Future impacts

J. Wegener’s Slide

Correct and reliable products … but how? • Software testing is an important analytical quality assurance activity. • But it is expensive ! – Verification, Validation and Testing constitutes 40% of the total lifecycle cost (Boehm, 2001) – Even then complete testing is not possible due to cost and time-to-market constraints (Sörqvist, 1998)

• A possible solution: – Test Automation

Search Based Software Testing in automated test data generation • Application areas: – Structural or white-box testing (e.g. providing coverage of specific program structures) – Black-box testing (e.g. exercising specific program features) – To automatically disprove certain grey-box properties (e.g. stimulating error conditions) – To verify non-functional properties

Few stats

SBSE repository, Y. Zhang

Evolutionary testing – an example of search-based software testing

J. Wegener’s Slide

Search-based testing for nonfunctional system properties • Execution time (e.g. finding inputs causing shortest and longest execution times)

• Quality of Service (e.g. violation of service level agreement between the integrator and the end user)

• Security (e.g. detecting buffer over-flows) • Usability (e.g. construction of covering arrays to cover t-way interactions)

• Safety (e.g. violation of safety properties)

Distribution of non-functional search-based software testing Execution time= 42.86% Quality of service= 5.71% Security= 20% Usability= 20% Safety= 11.43%

Industrial success story Evolutionary execution time testing at Daimler Chrysler AG • Aim: Find the input situations causing shortest and the longest execution times for real-time embedded systems. – Find test data with longest and shortest execution time to check whether they cause temporal error. – Fitness values based on the execution times of corresponding test data.

Industrial success story Evolutionary execution time testing at Daimler Chrysler AG continued … • •

Work done by Wegener and colleagues Numerous experiments with diverse applications showed GA outperforming random testing by finding more extreme times.

J. Wegener’s Slide

And its not just about search based software testing … • There are several studies for search-based software fault prediction (empirical data modelling). • The bug repositories can be mined to find patterns in faults. • These patterns can be modelled using genetic programming to predict the faults ahead in time. • Advantage: market.

better testing resource allocation considering time to

Industrial success story Evolutionary data modelling at Dow Chemical

Kordon’s Slide

Integrated methodology for empirical models development at Dow Chemical

Kordon’s Slide

SWELL Overview

Goals: Sweden leads in SW V&V Drive V&V knowledge innovation: – Develop: Industry-relevant & -close research – Promote: Spread and help implement – Commercialize: Services and tools

Visit Swell and the blog at: www.swell.se http://swell.weebly.com/blog.html

References • • • • •

P. McMinn, Search-based Software Test Data Generation: A Survey. Journal on Software Testing, Verification, and Reliability, vol. 14, no. 2, pp. 105-156, June 2004. J. Wegener’s slides on SBST available at http://www.systematictesting.com/documents/testworkshopTUM2.pdf M. Kotanchek and G. Smits and A. Kordon, Industrial Strength Genetic Programming, Genetic Programming Theory and Practise, Kluwer, 2003. W. Afzal , R. Torkar and R. Feldt. A systematic review of search-based testing for non-functional system properties. Information and Software Technology, 2009. In Press, Corrected Proof. Y. Zhang. Repository of publications on search-based software engineering, http://www.sebase.org/sbse/publications/