Measuring Tests Using COSMIC

2°International Conferen IT Data collection, Analysis and Benchma Tokyo (Japan) - October 22,

Measuring Tests Using COSMIC

Testing ICT Services in the Cloud

Thomas M. Fehlmann, Zü

Eberhard Kranich, Duisb

Measuring Tests Using COSMIC

Goals of the presentation

G1. Understand COSMIC measurements in testing G2. Free software testing from lines of code (LoC) G3. Measure and benchmark software testing

IT Confidence 2014 – October 22, 2014

http://itconfidence2014.wordpress.com

2

Dr. Thomas Fehlmann l

1981:

Dr. Math. ETHZ

l ll

1991: 1999: 1999:

Six Sigma for Software Black Belt Euro Project Project Office Office AG, AG, Zürich Zürich Euro

2011: 2013: 2013:

Net Promoter® ISBSG Certified Associate Vice-President Vice-President ISBSG

l l l l l l l ll ll l

2001: 2001: 2003: 2003: 2004: 2004: 2007: 2007: 2011:

Akao Price 2001 for original contributions to QFD Akao Price 2001 for original contributions to QFD SwissICT Expert for Software Metrics, ICTscope.ch SwissICT Expert for Software Metrics, ICTscope.ch Member of the Board QFD Institute Deutschland – QFD Architect Member of the Board QFD Institute Deutschland – QFD Architect CMMI for Software – Level 4 & 5 CMMI for Software – Level 4&5 Net Promoter® Certified Associate



3

Eberhard Kranich l l Mathematics Mathematicsand andComputer ComputerScience Science Emphasison onMathematical MathematicalStatistics Statistics l l Emphasis MathematicalOptimization Optimization l l Mathematical TheoryofofPolynomial PolynomialComplexity ComplexityofofAlgorithms Algorithms l l Theory l Worked at T-Systems International GmbH in Bonn, Germany l Working at T-Systems International GmbH in Bonn, Germany l Six Sigma Black Belt for Software Development l Six Sigma Black Belt for Software Development l Software Quality Assurance Manager l Software Quality Assurance Manager



4

l

What is a Defect? Defect = Behavior impacting expected or required functionality of software è

How many bugs?

è

By counting the size of defect repositories?

è

By number of entries???



5

l

l

Software Testing as a Game Tester sees selected sequences in the UML sequence diagram Tester can “walk” the data movements when planning or executing tests è

Functionality becomes visible to the agile team

è

Defects impacting functionality become visible to testers


Other Other Application Application

Some Device

Other Device

8.// Move some data 9.// Move some data 10.// Move some data 11.// Move some data


6

l

l

l

Functionality, Defect Size, and Defect Density What happens if data movements have defects? Testers mark the data movement where a defect has been detected


Some Device

Other Device

8.// Move some data Move some data 10.// Move some data 11.// Move some data

Same Metric: è ISO/IEC 19761 COSMIC l

Functional Size è

l

Test Size è

l

Number of Data Movements executed in Tests

Test Story è

l

Number of Data Movements needed to implement all FUR

Collection of Test Cases aiming at certain FURs

Defect Count è

Number of Data Movements affected by some defect detected in a test story



7

Defects Density Prediction? Now he counts the defects!


And counts and adjusts test size

Some Device

Other Device

8.// Move some data 9.// Move some data

By ISO/IEC 19761 COSMIC

10.// Move some data 11.// Move some data


How does he know that he found all the defects? http://itconfidence2014.wordpress.com

8

ISO/IEC Standard 29119 on Software Testing Published as ISO/IEC 29119 (2013-07) International Standard

Defines the Test Process

Calls for suitable Test Measures



9

ISO/IEC Standard 29119 on Software Testing Organisational Test Process Organisational Test Documentation

Feedback on Organisational Test Documentation

Test Management Processes Test Plan Updates

Test Planning Test Plan, Control Directives

Static Test Processes

Test Measures

Test Plan

Test Monitoring & Control Test Plan, Control Directives

Test Completion

Test Plan, Test Completion Report, Test Measures

Test Management Processes

Test Completion Report

Test Plan, Control Directives

Test Measures

Dynamic Test Processes

© Tafline Murnane and Stuart Reid from ISO/IEC JTC1/SC7 WG26 Software Testing IT Confidence 2014 – October 22, 2014


10

l

The SW Testing Qualifications Board ISTQB è

295’000 Certificates

è

Iqnite Conferences (Sydney 2013)

l

Third after ITIL and PMI

l

Importance of Testing grows



11

l

l

But it’s Even Worse… What is Defect Density? è

Defects per KDLOC?

What is Test Coverage? è

Code lines executed by some test case?



12

l

l

l

SW Testing and SW Metrics Counting practices for defect counting are undocumented è

“Number of Defects Found” per Stages / with Tests / etc.

è

How do you count “Number of Defects”?

Is it simply the number of entries in a defect repository? è

How can you avoid double reporting?

è

Or make sure two defects are reported twice and not in a single report?

A successor to the “Defect Measurement Manual” published by UKSMA in October 2000 is under review: “Defect Measurement and Analysis Handbook” è

By European cooperation

è

Important enhancement for ISBSG’s Data Collection!



13

l

l

Counting practices for defect counting are undocumented è

“Number of Defects Found” per Stages / with Tests / etc.

è

How do you count “Number of Defects”?

Is it simply the number of entries in a defect repository?

ts Commen Review

l

SW Testing and SW Metrics

è

How can you avoid double reporting?

è

Or make sure two defects are reported twice and not in a single report?

A successor to the “Defect Measurement Manual” published by UKSMA in October 2000 is under review: “Defect Measurement and Analysis Handbook” è

By European cooperation

è

Important enhancement for ISBSG’s Data Collection!



14

A Simple Data Search Example 1 Entry (E) + 2 eXit (X) + 2 Read (R) + 1 Write (W) = 6 CFP Trigger

User

Data Data

l

1.// Search Criteria

Functional User Requirements (FUR) describe a very simple data search

2.// Write Search 3.// Get Result 4.// Show Result

l

They meet Customer’s Needs

l

And have a

Priority Profile

5.// Nothing Found 6.// Show Error Message

Customer's Needs Topics Y.a Data Access y1 Access Data y2 Repeatable Responses Y.b Data Integrity y3 Cannot impact data

Attributes Always Responses identical No Write allowed

Reliable Frequently Always

Goal Profile derived from Voice of the Customer

Goal Profile 1) R001 Search Data 0.62 2) R002 Answer Questions http://itconfidence2014.wordpress.com 0.69 0.37 3) R003 Keep Data Safe

Functional User Requirements IT Confidence 2014 – October 22, 2014

15

è

Many Test Cases

è

Each Test Case has •

Test Data

•

Known Expected Response

Test Size and Test Profiles can be measured è

by Functionality Covered FALSE Show Defects

Test Size

l

Test Stories (scenarios) have

Priority

l

The Following SW Tests look Appropriate:

Test Story CT-A Prepare CT-A.1 Retrieve Previous Responses CT-A.2 Detect Missing Data IT Confidence 2014 – October 22, 2014

CT-A.3 Data Stays Untouched

Measured Defect Profile

0.43 11 0.74 18 http://itconfidence2014.wordpress.com 0.51 12 41

16

Execute Test Case CT-A.1.1 Test Case Measurements for Test Story CT-A.1

Test Story No. 1 Functional User Requirements

CT-A.1 Retrieve Previous Responses

R001: Search Data

R002: Answer Questions

R003: Keep Data Safe

Expected Response

CT-A.1.1 Enter valid Search String

X001,R001,W001,E001

X001,E001

R001,W001

Return (known) Answer

8

CT-A.1.2 Enter invalid Search String

E001

R002,W001

Invalid Search String

3

Test Size

11

Test Story Contribution (CFP):

5

2

1 Entry (E) + 2 eXit (X) + 2 Read (R) + 1 Write (W) = 6 CFP

Entering valid search string è

Returns expected response

è

Test Size is 4 CFP

4

CFP

Trigger

User

Data Data

1.// Search Criteria 2.// Write Search 3.// Get Result 4.// Show Result 5.// Nothing Found 6.// Show Error Message IT Confidence 2014 – October 22, 2014


17

Total Test Size Test Case Measurements for Test Story CT-A.1



R001: Search Data



Expected Response


X001,R001,W001,E001

X001,E001

R001,W001

Return (known) Answer

8


E001

R002,W001


3

Test Size

11


5

2

4

CFP

CT-A.2 Detect Missing Data

R001: Search Data



Expected Response

CT-A.2.1 Enter valid Search String for No Data

X002,R002,W001,E001

X001,R001,W001,E001

R002,W001

No Data Available

10


R001,W001,X002,E001

X002,E001

R002,W001


8

Test Size

18


8

6

4

CFP


R001: Search Data



Expected Response


W001,E001

X001,R001

R001,W001

Return identical Answer

6

CFP


X002,E001


2

CT-A.3.3 Enter Same String Again

R001,W001,X001,E001

Return identical Answer

4

Test Size

12


2

8

2

Total Test Size is 11 + 18 + 12 = 41 CFP è

Compares to Functional Size of 6 CFP

è

Yields a Test Intensity of 41/6 = 6.8

è

On average, < 7 tests per data movement IT Confidence 2014 – October 22, 2014

Test Size in CFP: 41 Test Intensity in CFP: 6.8 Test Coverage: 100% http://itconfidence2014.wordpress.com

18

l

Recording a Defect Test Case Measurements for Test Story CT-A.2



R001: Search Data



Expected Response


X002,R002,W001,E001

X001,R001,W001,E001

R002,W001

No Data Available

10


R001,W001,X002,E001

X002,E001

R002,W001


8

Test Size

18


8

6

4

CFP

1 Entry (E) + 2 eXit (X) + 2 Read (R) + 1 Write (W) = 6 CFP

Trigger

User

Data Data

1.// Search Criteria

“Bug” in 6.// Show Error Message è

2.// Write Search

Detected with data base containing no data

è

Test Size is 4 CFP

è

1 Defect found!

3.// Get Result 4.// Show Result 5.// Nothing Found 6.// Show Error Message



19

Defect Observed Test Case Measurements for Test Story CT-A.2



R001: Search Data



Expected Response


X002,R002,W001,E001

X001,R001,W001,E001

R002,W001

No Data Available

10


R001,W001,X002,E001

X002,E001

R002,W001


8

Test Size

18


8

6

Defects Observed

4

Data Movements Affected

Name Label

Description

Name Label

#001

Some characters such as 'ä' are wrongly interpreted as escape characetrs in strings

R001

1

Escape Chars

CFP

Get Result

Defect Count

One Defect Found è

Possibly observable in all tests touching the data movement 6.// Show Error Message – named X002

è

Counted once IT Confidence 2014 – October 22, 2014


20

Test Status Reporting Test Status Summary Total CFP: 6 Defects Pending for Removal: 2 Defects Found in Total: 2 Defect Density: 33% l

l

Test Size in CFP: 41 Test Intensity in CFP: 6.8 Test Coverage: 100%

Test Status Characterization è

Test Size is the total number of data movements executed in all Test Cases of all test stories

è

One Data Movement can have only one defect identified per Test Story

However, one misbehavior found might affect more than one data movement and thus count for more than one defect IT Confidence 2014 – October 22, 2014


21

l

Caveat Test size increases with meaningless test cases added to the test stories – little variations of test data with almost identical expected responses might spoil measurements! è

We need a metric indicating that our test strategy is appropriate



22

What is Six Sigma Testing?



23

What is Lean Six Sigma Testing? … go for the Eigenvector!

Replace many Test Stories…


… by those needed for the Eigensolution


24

Calculating the Eigenvector with Jacobi Iterative

x

AT



25

Calculating the Eigenvector with Jacobi Iterative 9 0 1

0 9 0

3 0 5

0 8 3

x


90

0

0

145 24

24

24

0 9 0 8

AT

9 0 3 0 24 35


26

Measuring Test Coverage with Eigensolution Test Stories

Functional User Requirements R001 Search Data

0.62

R002 Answer Questions

0.69

R003 Keep Data Safe

0.37

5 8 2 2 6 8 4 4 2

Achieved Coverage


Goal Test Coverage

Number of data movements executed


Deployment Combinator


Test Stories

0.64 0.66 0.40

Ideal Profile for Test Stories 0.43 0.74 0.51 Convergence Gap 0.42

0.7

0.5

0.04

0.10 Convergence Range 0.20 Convergence Limit



27

Lean & Six Sigma for Software Testing Six Sigma

l

Lean

è

Design of Experiments

è

è

Multi-linear Regression for Root Cause Analysis and process control

Detect Waste (muda, 無駄 )

è

Test-Driven Development

è

Defect-free delivery

l

Lean Six Sigma è

Predict Waste (muda, 無駄)

è

Eigenvectors solutions for Root Cause Analysis and process control

è

Predict Defect Density

è

Q Control Charts for SW Testing



28

Questions?



29