Return On Investment @ Model Based Testing

Return On Investment @ Model Based Testing An Industrial MBT Use Case & its Return On Invest

BITKOM – Forum Qualitätsmanagement

Helmut Goetz Siemens AG - Corporate Technology © Siemens AG 2010. All rights reserved.

Siemens Corporate Technology – Who we are?

Page 2

Sept-2010

Goetz, SAG

All rights reserved. © Siemens AG, Corporate Technology

Siemens AG Active in three Sectors Sectors Industry Divisions

Divisions

Drive Technologies Industry Automation Building Technologies Mobility Lighting (OSRAM) Industry Solutions

Page 3

Energy

Sept-2010

Fossil Power Generation Renewable Energy Oil & Gas Energy Service Power Transmission Power Distribution

Goetz, SAG

Healthcare Divisions Imaging & IT Workflow & Solutions Diagnostics


Corporate Technology Networking the integrated technology company Customers Sectors / Divisions

Chief Technology Officer

Chief Technology Office (CT O) Direct support of CTO

Energy

Industry

Healthcare Regions

Review innovation strategies Drive technology based synergies Secure innovation power Technology assessments Governance and guidance

Siemens IT Solutions and Services (SIS) Siemens Financial Services (SFS)

Corporate Research and Technologies (CT T) Global technology fields with multiple impact Pictures of the Future Accelerators for new business opportunities

Corporate Intellectual Property and Functions (CT I) Intellectual Property services & strategy Standardization, environmental affairs Global information research

Corporate Technology (CT) Page 4

Sept-2010

Goetz, SAG


Global Technology Field Software Development Technologies Our role Contribution to the Siemens business units by enhancing efficiency and quality through engineering excellence as

Model Driven Development Requirements Engineering

Code Quality Management

„Evangelists“ -> bring new technologies into the company

Technology hub within in Siemens for software quality and efficiencyrelated issues Partner of academia for research in requirement, design, development, integration, and validation Know-how transfer to business units through assessments, consulting, project support, coaching/ mentoring, workshops, etc.

where you need us … worldwide! Sept-2010

Software Testing

Our approach

Our goal: efficiency and quality of your software development

Page 5

Performance

Tracing and Diagnosis

Software Production

Technology Consultants -> understand Siemens business

Driver for technologies

Dependability

Goetz, SAG


Software Test – Situation of Today

Page 6

Sept-2010

Goetz, SAG


What is Quality? „Quality is Conformance with Requirements“ [Hruschka 1999]

Quality „Wie verständlich, konsistent, vollständig und korrekt die Anforderungen sind, bestimmt wesentlich die erreichbare Qualität der Testfälle“ Require [Jungmayr] ments

Testing

Testen = ein Prozess, bei dem durch Stichproben geprüft wird, ob Anforderungen korrekt implementiert worden sind mit dem Ziel, Fehler zu finden.

Page 7

Sept-2010

Goetz, SAG


Testing Challenge No.1 – Testing Time Release Date is hammered in stone!!

Development “overtime”

RELEASE

Finished by Milestone reached!

In the most projects the release date has the highest priority! The RE-phase is the most challenging phase in the project. Sometimes development needs an “overtime”!

Remaining testing time!

This means that the planned testing time will be often reduced due to fixed release date. Page 8

Sept-2010

Goetz, SAG


Testing Challenge No. 2 – Test Costs Die Aufwände im “klassischen” Testen verteilen sich etwa wie folgt: 10% entfallen auf vorbereitende Tätigkeiten (i.W. die Testplanung); 40% werden für die Analyse der Testobjekte und das Design der Testfälle verbraucht; 45% fallen bei der Durchführung und Protokollierung der Tests an;

Generischer Testprozess nach ISTQB®

5% benötigen die nachgelagerten Arbeiten (Auswertung, Berichterstattung, Archivierung etc.). (vgl. z.B. [Spillner 2008])

Page 9

Sept-2010

Goetz, SAG


Testing Challenge No. 3 – Defect Source

[Source: Plan 2008]

Top Priority - This needs to be optimized! Page 10

Sept-2010

Goetz, SAG


Software Errors – some Numbers… Was kosten Fehler? Geschätzte Verluste durch Softwarefehler in Mittelstandsund Großunternehmen in Deutschland: ca. 84,4 Mrd. € p.a. 35,9% der IT-Budgets für Beseitigung von Programmfehlern entsprechen 14,4 Mrd. p.a. Produktivitätsverluste durch Computerausfälle aufgrund fehlerhafter Software: ca. 2,6% des Umsatzes – 70 Mrd. € p.a. Source: [LOT 2001] Page 11

Sept-2010

Goetz, SAG


What does the Repair of a Software Error cost?

[Source: SQS 2009] Page 12

Sept-2010

Goetz, SAG


Summary

55 %

of all errors will be introduced during the requirements engineering phase.

40 %

of the testing time needs to be spend for analyzing of the requirements and to derive test cases out of it.

1.000 €

90.000 € Page 13

does one software error cost if he will be found during requirement engineering phase, but…

Model Based Testing (MBT)

will cost the same error if he will be detected first after delivery of the software application. Sept-2010

Goetz, SAG


Our MBT- Way “Testing the Requirements” Use Cases / Workflows Product Requirements “Do we build the product right?”

derive

derive transform

System Model generate/ develop

“Do we build the right product?”

Test Model reuse

generate

validate

Application

Test Cases

[source: Schieferdecker 2007]

Model-based testing (MBT) aims to automatically generate test cases using test models with the possibility of reusing of artifacts created during development process. Page 14

Sept-2010

Goetz, SAG


MBT – My Book Publications Basiswissen Modellbasierter Test

W NE

!

Available since Sept. 2010 http://www.dpunkt.de/buecher/3064.html

iX- Studie “Modelbasiertes Testen Modellierung und Generierung von Tests Available since March 2009 http://www.heise.de/kiosk/special/#studien

Page 15

Sept-2010

Goetz, SAG


Status of MBT today Typical customer voices… “MBT can only be introduced if a project will start from scratch…” “MBT looks great, but what shall be done with our existing thousands of tests we had today?” “Modelling means a huge invest in tooling as well as in trainings to be spent for the people, because MBT forces to have experts!” “What is my Return of Investment?” “Do you have any reference customers or projects where MBT is already introduced and running?” “Concern: MBT will radically change my existing Test Process”

Page 16

Sept-2010

Goetz, SAG


Short Introduction of our MBT-Way with TDE/UML

Page 17

Sept-2010

Goetz, SAG


Our MBT-Approach and Tool Our developed MBT Tool TDE/UML which: is a methodology for UML based model based testing is a Plug-In to Eclipse, a popular IDE; but runs as well in a standalone- mode provides an UML modeling environment provides an user interface for rule validation implements a test generation process combining: Graph Coverage (i.e. UML Diagram Coverage Criterion) and Data Coverage (i.e. Category-Partition Methodology) Our main goal - Support an industrial and “affordable“ approach!

Page 18

Sept-2010

Goetz, SAG


UML 2.0 Diagrams Structure & Behavior Diagrams

Class Class Diagram Diagram

Use Case Case Use Diagram Diagram

Object Object Diagram Diagram

Activity Activity Diagram Diagram State Machine Machine State Diagram Diagram

Package Package Diagram Diagram

Structure Diagrams

Sequence Sequence Diagram Diagram

Behavior Diagrams

Component Component Diagram Diagram

Communication Communication Diagram Diagram

Compositestructure structure Composite Diagram Diagram

Page 19

Sept-2010

Deployment Deployment Diagram Diagram Goetz, SAG

Timing Timing Diagram Diagram

Interaction Interaction Overview Overview Diagram Diagram


UML 2.0 Diagrams …and which are currently used

Class Class Diagram Diagram

Use Case Case Use Diagram Diagram

Object Object Diagram Diagram

Activity Activity Diagram Diagram State Machine Machine State Diagram Diagram

Package Package Diagram Diagram

Structural Model

Sequence Sequence Diagram Diagram

Behavioral Model

Component Component Diagram Diagram

Communication Communication Diagram Diagram

Compositestructure structure Composite Diagram Diagram

Page 20

Sept-2010

Deployment Deployment Diagram Diagram Goetz, SAG

Timing Timing Diagram Diagram

Interaction Interaction Overview Overview Diagram Diagram


MBT- Test Workflow with TDE/UML Part 1 – Modeling Focus on UML Use Case & Activity Diagrams including checking of the models & reverse modeling from exist. test cases Part 2 - Automatic Test Case Generation Combines Graph Coverage and Data Coverage to form a Test Suite Part 3 - Code Generation Transforms a Test Suite into any other formats, such as - export to test management systems - generating test specs & test reports - generate executable code (test scripts) - XML, … Part 4 – Quality Assessment After test case execution the results will be transferred back to the model: Model Entry Defect Search & Defect Entry Model Search Page 21

Sept-2010

Goetz, SAG


Modeling From Use Case to Activity Diagramm

Category/Partition variables are used to show data variations. Constraints on edges are used to constrain data variations and constrain tests to consistent paths.

Page 22

Sept-2010

Goetz, SAG


Test Case Generation Graph & Data Coverage Criteria Test Generation Options Graph Coverage Criterion All activities All transitions Round trip criterion All paths criterion Happy Paths Happy Path Grouping

Page 23

Sept-2010

Data Coverage Criterion Sampling Choice-per-suite Choice-pervariable Choice-per-path Exhaustive

Graph Coverage

Data Coverage

All-paths

Exhaustive

4320

All-paths

Choice-per-path

346

All-paths

Choice-per-var.

90

All-activities

Choice-per-path

26

Happy-path

Happy-data

1

Goetz, SAG

# of Test Cases


MBT Project at Trapeze ITS (Switzerland)

Page 24

Sept-2010

Goetz, SAG


Trapeze ITS ist der weltweit führende Hersteller von umfassenden, durchgängigen Lösungen für Betriebsführung, Fahrgastinformationen und Unternehmenssteuerung im öffentlichen Personennahverkehr. Damit erreichen wir für Verkehrsunternehmen einen optimalen Fahrzeugeinsatz, höhere Pünktlichkeit und ein effizientes Flottenmanagement. Reisende profitieren von präzisen Anschlüssen sowie von Echtzeitinformationen an Haltestellen, unterwegs in Bus und Bahn wie auch zuhause und am Arbeitsplatz. Unsere Angebotspalette umfasst Leit- und Managementsysteme, Fahrzeugausrüstungen, Kommunikationslösungen, Anzeiger für Fahrgastinformationen, Integration von Fahrgeldmanagementsystemen sowie kundenspezifische Serviceleistungen. Seit mehr als 40 Jahren bewähren sich unsere Lösungen im täglichen Einsatz – heute bereits in mehr als 200 Städten weltweit. Trapeze ITS ist ein eigenständiges Unternehmen innerhalb der Trapeze Gruppe, die wiederum eine von sechs Geschäftseinheiten der kanadische Constellation Software Inc. ist. Als Trapeze ITS beschreiten wir auch weiterhin zielstrebig unseren Weg als erfolgreiches Unternehmen im Bereich öffentlicher Personennahverkehr.

Page 25

Sept-2010

Goetz, SAG


AVLC System Landscape

Depot Data Management Integrated Control Center Technology

On-board systems

Communication

Traffic Signal Priority

Page 26

Dynamic Passenger Information Sept-2010

Goetz, SAG

All rights reserved. 26 © Siemens AG, Corporate Technology

Integrated Control Center Technology

VICOS-LIO control center: Dispatcher workstation screen

VICOS-LIO control center: Main window of the blockplan

VICOS-LIO control center: Transfer visualization

VICOS-LIO control center: Route diagramm

VICOS-LIO Business Intelligence for comprehensive reporting

VICOS-LIO ActiveForms+ for easy dispatch and documentation

Page 27

Sept-2010

Goetz, SAG

VICOS-LIO control center: Network overview

All rights reserved. 27 © Siemens AG, Corporate Technology

Status Quo Test Approach before introducing MBT

Page 28

Sept-2010

Goetz, SAG


Status Quo Test Approach before MBT Analysis

Design & Realisation Access DB

System Use Case

Test case

Manual Test Exec. 320 Test Categories, each has max. 15 test cases, in total ~ 4800 test cases

Subversion

Busi ness Rules

Test Script Development

~ 215 Business Rules incl. ~ 645 scenarios, i.e. ~ 1.505 DIN A4 pages with pure plain text. Page 29

Test case

Test case

Test Category

~ 800 System UC, whereby 320 only relevant for test) ~ 4000 UC scenarios, i.e.. ~ 5.600 DIN A4 pages with pure plain text.

Execution

Automatic Test Exec.

Test Automation (HP Quick Test Pro) Sept-2010

Goetz, SAG


Challenges of Today Success Criteria for MBT System Use Case & Business Rules Extent the traceability between System Use Case/ Business Rules vs. Test Category to a maximum extent. How to handle ~7.100 DIN A4 pages?

Test Categories & Test Cases Reduce complexity and size of Test Categories (up to 40 pages as PDF) Improve review efficiency of test categories – They are very time consuming, retrieved results after review were poor. Test Tooling Reduce maintenance costs of proprietary MS Access DB Mandatory reuse of existing test cases Test Costs Reduce the cost of test case maintenance Page 30

Sept-2010

Goetz, SAG


Introducing MBT – The new Tool Landscape

Page 31

Sept-2010

Goetz, SAG


New Test Tool Landscape Overview

TDE/UML

Busi ness Rules

Test case

Testmodel

System Use Case

Execution

Realisation

Analysis & Design Test Spec

1..p

Manual Test Exec.

(automatic generated)

HP Quality Center

Test case 1..n

Test case DB

direct import

Test case

Test Protocol

Test Reports & Metrics

1..m

Subversion

Reverse engineering to create models out of existing test scripts

Page 32

Sept-2010

Goetz, SAG

Test Script Development

Automatic Test Exec.

Test Automation (HP Quick Test Pro) All rights reserved. © Siemens AG, Corporate Technology

TDE Step - Code Generation Two different documents - A mouse click away… Test Specification for manual execution

Selected Path through the model Test Case Steps Description

Chosen Test Data for this Test Case

Page 33

Sept-2010

Goetz, SAG


TDE Step - Code Generation Example: Quality Center Import Selection of QC-project and directory

Mapping of model properties to QC user defined fields

Selection which Tests will be imported Page 34

Sept-2010

Goetz, SAG


Introducing MBT – Project Phases

Page 35

Sept-2010

Goetz, SAG


MBT Project Phases The MBT Evolution Steps Pure Evaluation Phase (30-days) Pilot - Get started with model based testing in customer's domain and infrastructure (“out of the box approach”) Rollout of model based testing as customized solution in the organization (“Power User”)

Keep customer happy with their MBT-solution

TODAY!

Sunset within customer development projects

Page 36

Sept-2010

Goetz, SAG


MBT Project Phases Concrete Numbers MBTProjectphase

N/A

Performed by SAG Basic & Advanced Training

Efforts by SAG 48,0 PH

Pilot

Hands-on Support

16,0 PH

Performed by Trapeze Check selected tools

Rollout

1. Customization / Maintenance & Delivery of new TDE-version 2. Hands-on Support 268,0 PH 3. Project Management Tasks

efforts in total:

Page 37

Efforts by SAG

Sept-2010

24,0

PH

Total 24,0 PH

Efforts by Trapeze

Total

Basic & Advanced Training

130,0 PH

178,0 PH

1. Create first models 2. Perform lessons learned / describe best practises 3. Generate Model Pattern

416,0 PH

416,0 PH

Hands-on Support

16,0

PH

32,0 PH

562,0 PH

626,0 PH

Efforts by Trapeze

Total

Performed by Trapeze

64,0 PH Performed by SAG

Efforts by Trapeze

Performed by Trapeze 1. Refinement on Models - major features/ areas 2. Reverse modelling - Create models out of existing test cases 3. Create new models for new introduced features/functionality

2040,0 PH 2308,0 PH

268,0 PH

2040,0 PH

2308,0 PH

332,0 PH

2626,0 PH

2958,0 PH

Goetz, SAG

EUR 240.000

N/A

Efforts by SAG

EUR 50.796

Performed by SAG

EUR 189.204

ToolEvaluation

Major Tasks, Efforts & Costs


Introducing MBT – Return On Invest

Page 38

Sept-2010

Goetz, SAG


ROI@MBT – 1st Result Reduce Complexity (Model Statistics) 1. Test Model size as of today: The complete test model statistic is as following:

# of modelled use cases # of activities in these UC’s: # of single TDE-model files:

2.289 16.789 232

Expectation: Instead of maintaining ~7.100 pages of plain text there is a good chance to have at the end “only” 5000 modeled use cases. This would mean a reduction of ~30 % (assuming 1 UC fits to one page) and a huge reduction of maintenance effort in the future.

Voice of the customer: „Generell ist es aber immer so, dass die Wartung der Modelle machbar bleibt. Prosa-Testfälle dieser Komplexität sind faktisch nicht mehr wartbar.“ Page 39

Sept-2010

Goetz, SAG


ROI@MBT – 2nd Result Early Error Detection Analysis & Design TDE/UML

Busi ness Rules

Test case

Testmodel

System Use Case

Improved Review Technique

Major Result New Feature 1

1..p

NEW! New Feature 2

Test case 1..n

Test case 1..m

System Engineers & Architects

Changed System Use Case

This early error detection would already save 36.000 € assuming that all these three major review findings were the first time found during system integration test! Page 40

Sept-2010

Goetz, SAG


ROI@MBT – 3rd Result Tremendous Savings in the upcoming Versions Classic – Way Effort to generate one test case manually: 2,67 PersonHours

Assuming that for the next version there are 500 new test cases needed!

500 TC x 2,67 PH/TC = 1.335 ManHours

Effort per TC ~74% reduced!

500 TC x 0,68 PH/TC = 340 ManHours

Hourly Rate: 75 € / PH

100.125 €

Savings: ~ 75.000 €

Hourly Rate: 75 € / PH

25.500 €

MBT – Way Effort to generate one test case automatically: 0,68 PersonHours

Page 41

Sept-2010

Goetz, SAG


ROI@MBT – 4th Result Voice of the Customer Generelles Feedback: MBT ist eine sehr professionelle Art und Weise Tests zu spezifizieren. Die Anwendung, bzw. das Ausreizen der Möglichkeiten des MBT erfordert eine gewisse Spezialisierung. Für komplexe Themengebiete unerlässlich um überhaupt pflegbare Testfälle entwickeln zu können. Bei trivialen Funktionalitäten generell auch geeignet, die Vorteile kommen dann aber nicht so stark zur Geltung.

Lesbarkeit: Die Lesbarkeit der Modelle ist für triviale Modelle generell sehr gut. Sobald es komplexer wird (d.h. mit Testdaten, Parameter, etc.) wird es natürlicher Weise schwieriger lesbar. Generell ist es aber immer so, dass die Wartung der Modelle machbar bleibt. Prosa-Testfälle dieser Komplexität sind faktisch nicht mehr wartbar. Improvement Potential: MBT ist sehr artverwandt mit der Arbeit von System Engineers. MBT bietet theoretisch die Möglichkeiten Requirement Modelle und Testmodelle zu verschmelzen oder zumindest sehr aneinander anzunähern. Der prinzipielle Unterschied ist im Prinzip nur das Hinzufügen von Daten im Testmodell Akzeptanz von MBT: Die Akzeptanz ist im Moment noch zweigeteilt. MBT erfordert in einem gewissen Maß abstraktes Denkvermögen. Damit verbunden ist eine generelle Umstellung der Arbeitsweise. Die Erfahrung zeigt, dass wenn ein Mitarbeiter die erste Hürde genommen hat, dass dann die Akzeptanz und Motivation sehr hoch ist. Ich denke das ist eine ähnliche Entwicklung die jeder mitmacht, wenn er z.B. eine neue Programmiersprache lernt. In so fern erfordert MBT teilweise eine Spezialisierung von Test Engineers in Sachen Test Design. Generell ist das allerdings positiv zu bewerten, da dadurch auch die Qualität der Testfälle steigt.

Page 42

Sept-2010

Goetz, SAG


Summary - MBT is THE Fast Lane… …also from the financial Point of View System Use Case & Business Rules Traceability? – Guaranteed by using “check elements” in the test model ~7.100 pages? – Use 5000 models instead of text!

Test Categories & Test Cases Review method? – Improved: Models, not text! Reuse of test specifications? – 100% guaranteed! Used Test data? – 100% guaranteed due to using test model Test Tooling Integrated MBT & test management tool chain Reverse engineering? -- Fulfilled Test Costs Budget increase? – No, budget will be reduced! Test case maintenance? – Guaranteed by having only one source: Test models!

Page 43

Sept-2010

Goetz, SAG


Thank you for your attention!

Helmut Goetz Siemens AG Corporate Technology CT T DE TC 4 Guenther- Scharowsky- Str. 1 91058 Erlangen Germany Phone: E-mail:

Page 44

+49 (0) 9131 7 35563 [email protected]

Sept-2010

Copyright © Siemens AG 2010. All rights reserved. Goetz, SAG © Siemens AG, Corporate Technology

Referenzen [Winter 2008] Winter, M.: Modellbasiertes Testen – Alter Wein in neuen Schläuchen?, TAV 27 / AK TOOP, 06. Juni 2008 [Hruschka 1999] Hruschka, P.: Quality Starts With Requirements, CONQUEST 1999 [SQS 2009] SQS Empirische Daten aus 3000 IT-Projekten entnommen aus: Prof. Spillner, Universität Bremen, 1st Testing Day Franken, Nürnberg, 10-March-2009. [Spillner 2008] Spillner, A.; Roßner, T.; Winter, M. ; Linz, T.: Praxiswissen Softwaretest – Testmanagement. 2., überarbeitete und aktualisierte Auflage, dpunkt.verlag, Heidelberg, 2008. [LOT 2001] Studie der LOT Consulting Karlsruhe, IT-Services 3/2001, S. 31 (s. auch http://www.computerwoche.de/nachrichten/519152/). [Plan 2008] Dr. Plan, O.;Schwabe, S.: Verbesserung von Integrationstests mittels modellbasierter Verfahren, imbus QS-Tag, 2008 [Schieferdecker 2007] Schieferdecker, I.: Modellbasiertes Testen, Journal OBJEKTspektrum, SIGSDATACOM, vol. 03, 2007, p. 39-45. [Jungmayr] Jungmayr, S.,www.testbarkeit.de, 2006

Page 45

Sept-2010

Goetz, SAG


Return On Investment @ Model Based Testing

Return On Investment @ Model Based Testing

Suggest Documents

return on investment

'Return on Investment' Calculations

return on investment

Maximizing Website Return on Investment:

A Social Return on Investment evaluative analysis based on a ...

Social RetuRn on inveStment - Khana

True Return On Investment - WordPress.com

Potential Return on Investment (RoI) on web-based ...

A Social Return on Investment evaluative analysis based on a ...

return on investment models - CiteSeerX

True Return On Investment - WordPress.com

An approach to model the Return on Investment of ...

Return on Investment Analysis for E-business Projects Return on ...

The Return on Investment on Migration

Software Economics: Quality-Based Return-on-Investment ... - wseas.us

Evidence-based management assessment of return on investment ...

Return on Investment of Behavior-Based Safety - Behavioral Safety

Model-Based Testing

On Model-Based Testing Advanced GUIs

On Model-Based Testing Advanced GUIs

Training Return on Investment (ROI)

Maximizing Website Return on Investment - Google Services

Return on Investment and Grants - Eric

ENERGY RETURN ON INVESTMENT - Energy-Reality.org