Deriving Metric Thresholds from Benchmark Data

Deriving Metric Thresholds from Benchmark Data Tiago L. Alves, Christiaan Ypma, Joost Visser

September 16th, 2010

T +31 20 314 0950 [email protected] www.sig.eu

Software Improvement Group Services

2 I 18

DocGen • Automated generation of technical documentation • Reduce learning time, assist impact analysis, support migration, …

Software Risk Assessment • In-depth investigation of software quality and associated business risks • Answers to specific research questions

Software Monitoring • Continuous measurement, feedback, and development consultancy • Guard quality from start to finish

Software Product Certification • Five levels of technical quality (maintainability) • Evaluation by SIG, certification by TÜV Informationstechnik Tiago L. Alves - Software Improvement Group / University of Minho “Deriving Metric Thresholds from Benchmark Data”, ICSM’10

Who is using our services?

3 I 18 Financial and Insurance companies

Government

Tiago L. Alves - Software Improvement Group / University of Minho “Deriving Metric Thresholds from Benchmark Data”, ICSM’10

Logistical

IT

Other

The metrics crisis

4 I 18

We need to measure!!! ArgoUML size: 200K LOC

Methods A, B have McCabe 5 and 10

Average McCabe: 20

What is the #1 problem with software metrics? not lack of tools

not lack of metrics

Metrics (by themselves) do not mean anything, we need thresholds!!! Tiago L. Alves - Software Improvement Group / University of Minho “Deriving Metric Thresholds from Benchmark Data”, ICSM’10

What about thresholds? Life sciences vs. software sciences

Cholesterol levels

Complexity

5 I 18

Typical values

µ

Not so good

µ±

15%

!

µ=2 µ+ =6

! Malnutrition - anxiety, depression, suicide

?

Heart attack

Tiago L. Alves - Software Improvement Group / University of Minho “Deriving Metric Thresholds from Benchmark Data”, ICSM’10

Validation?

! !

??? %

Methodology

6 I 18

Requirements 1. Respect the statistical properties of the metric (scale and distribution) 2. Based on data analysis from a representative set of systems (benchmark) 3. Repeatable, transparent, and of straightforward execution.

Benchmark:

Tiago L. Alves - Software Improvement Group / University of Minho “Deriving Metric Thresholds from Benchmark Data”, ICSM’10

Methodology steps Quantile plots

7 I 18

Tiago L. Alves - Software Improvement Group / University of Minho “Deriving Metric Thresholds from Benchmark Data”, ICSM’10

Methodology steps Weighting by size (Vuze)

non-weighted

Tiago L. Alves - Software Improvement Group / University of Minho “Deriving Metric Thresholds from Benchmark Data”, ICSM’10

8 I 18

weighted

Methodology steps Weighting by size (all systems)

9 I 18

non-weighted

Tiago L. Alves - Software Improvement Group / University of Minho “Deriving Metric Thresholds from Benchmark Data”, ICSM’10

weighted

Methodology steps Aggregating with relative size

10 I 18

6

Tiago L. Alves - Software Improvement Group / University of Minho “Deriving Metric Thresholds from Benchmark Data”, ICSM’10

8

14

Thresholds

11 I 18

Metric / Quantiles Unit complexity Unit size Module inward coupling Module interface size

70% 6 30 10 29

80% 8 44 22 42

90% 14 74 56 73

Metric / Quantiles Unit interfacing

80% 2

90% 3

95% 4

Tiago L. Alves - Software Improvement Group / University of Minho “Deriving Metric Thresholds from Benchmark Data”, ICSM’10

From Thresholds to Quality Profiles

12 I 18

Metric / Quantiles Unit complexity

70% 6

80% 90% 8 14

70%

10%

70% low risk

moderate risk

Tiago L. Alves - Software Improvement Group / University of Minho “Deriving Metric Thresholds from Benchmark Data”, ICSM’10

high risk

10%

80%

10%

90%

100%

very-high risk

Analysis

13 I 18

70%

10%

Tiago L. Alves - Software Improvement Group / University of Minho “Deriving Metric Thresholds from Benchmark Data”, ICSM’10

Considerations 1/3 Outliers

14 I 18

All systems

Tiago L. Alves - Software Improvement Group / University of Minho “Deriving Metric Thresholds from Benchmark Data”, ICSM’10

Metric characterization

Considerations 2/3 Influence of size

15 I 18

Tiago L. Alves - Software Improvement Group / University of Minho “Deriving Metric Thresholds from Benchmark Data”, ICSM’10

Considerations 3/3 Scoping / tool configuration

16 I 18

Tiago L. Alves - Software Improvement Group / University of Minho “Deriving Metric Thresholds from Benchmark Data”, ICSM’10

Conclusion

17 I 18

Contribution • Novel methodology to derive metric thresholds • Based on benchmark, statistically sound - depart from expert opinion • Operational & agrees with expert opinion (sensible thresholds)

Challenges to meet • Benchmarks of real data • Repeatability with different tools

Plans for the future • More validation with external characteristics (see Luijten and Visser, “Faster Defect Resolution with Higher Technical Quality of Software”, SQM’10 • Use thresholds to derive system ratings (under progress)

“The” Lesson - Meaningful thresholds can be derived from benchmarks Tiago L. Alves - Software Improvement Group / University of Minho “Deriving Metric Thresholds from Benchmark Data”, ICSM’10

18 I 18

Thank you! [email protected]

Tiago L. Alves - Software Improvement Group / University of Minho “Deriving Metric Thresholds from Benchmark Data”, ICSM’10