... different people. â« Change Management Solution for Process Models ... Development process must be iterative and incremental ... [1] M. Fowler: Refactoring.
IBM Research
Incremental Development of Model Transformation Chains Using Automated Testing
Jochen Küster, Thomas Gschwind, Olaf Zimmermann IBM Research - Zurich
ACM/IEEE 12th International Conference on Model Driven Engineering Languages and Systems (MODELS), Denver, October 2009 © 2009 IBM Corporation
IBM Research
Outline Motivation and Case Study Overview Requirements for Development of Transformation Chains Techniques and Approach – Test Design Techniques – Test Architecture – Development Process
Results Conclusion & Outlook
2
© 2009 IBM Corporation
IBM Research
Motivation and Context Process Models are key artifact in modern SOAoriented software development
BPMN
Business-Driven Development comprises – modeling of processes at different levels of abstraction – in different modeling languages (BPMN,BPEL) – in a multi-user environment by different people Change Management Solution for Process Models – see “Language-Independent Change Management of Process Models” by C. Gerth et al – framework for change management of process models
3
© 2009 IBM Corporation
IBM Research
Overview of Change Management Solution BPMN to Workflow Graph Transformation
PST Computation
Correspondence Computation/ Process Matching
Difference Computation
Dependency Computation
Filtering of Non-BPMN Differences
BPMN Workflow Graph 1
BPMN
Workflow Graph 2
Process Structure Tree 1
Process Structure Tree 2
Difference Model Joint Process Structure Tree
Difference Model
BPMN - InsertFragment(V1,”Record Claim”, Difference Difference Decision, fA) Model Model With - InsertActivity(V1,”Retrieve add. Data”, With Dependencies Merge, Decision) Dependencies -…
Transformation for constructing Difference Model – obtained by instantiating the framework for change management
Transformation chain involving 6 individual transformations Each transformation has to be implemented in Java (tool integration issue)
How to ensure quality of the transformation chain? 4
© 2009 IBM Corporation
IBM Research
Challenges in Transformation Chain Design and Implementation BPMN to Workflow Graph Transformation
PST Computation
Correspondence Computation/ Process Matching
Difference Computation
Dependency Computation
Filtering of Non-BPMN Differences
BPMN Workflow Graph 1
BPMN
Workflow Graph 2
Process Structure Tree 1
Process Structure Tree 2
Joint Process Structure Tree
Difference Model
BPMN Difference Model With Dependencies
Difference Model With Dependencies
Dependencies between transformations – changes in correspondence computation affect Difference Model – changes in WFG and PST computation affect Difference Model – …
Dependencies between different input model pairs – functionality written for one input model pair may affect other input model pairs – infinitely many input model pairs make this difficult
Diversity of input models – transformation chain has to produce a Difference Model for all possible combinations
5
© 2009 IBM Corporation
IBM Research
Outline Motivation and Case Study Overview Requirements for Development of Transformation Chains Techniques and Approach – Test Design Techniques – Architecture of Test Framework – Development Process
Results Conclusion & Outlook
6
© 2009 IBM Corporation
IBM Research
Requirements for Development Approach (1) How to ensure quality of a transformation chain? Development process must be iterative and incremental – development will take place in several iterations – transformation chain can be developed in teams – How to organize the development process for a transformation chain?
Testing of transformation chain is required – important for quality assurance – How to design test cases for a transformation chain?
A fully automated test environment is needed – manual testing on large number of input models is impossible in practice – How to establish such an automated test environment?
7
© 2009 IBM Corporation
IBM Research
Requirements for Development Approach (2) How to ensure quality of a transformation chain? Specific transformation chain requirements – software engineer must be able to change/add functionality of a transformation without breaking transformation chain – fixing defects requires a mechanism to ensure that changes of a transformation are side-effect free – refactoring for ensuring code quality of the transformations is required, typical refactorings (e.g. [1]) need to be applied – How to fulfill these requirements in the development environment and in designing test cases?
[1] M. Fowler: Refactoring. Improving the Design of Existing Code. Addison Wesley,1999 8
© 2009 IBM Corporation
IBM Research
Outline Motivation and Case Study Overview Requirements for Development of Transformation Chains Techniques and Approach – Test Design Techniques – Architecture of Test Framework – Development Process
Results Conclusion & Outlook
9
© 2009 IBM Corporation
IBM Research
Test Design Techniques for Model Transformation Chains Test design techniques for deriving test cases for a transformation chain – test cases for transformation chain and each individual transformation Three different test design techniques – Inspected reference outcomes – Invariant validation – Deviation test data
10
© 2009 IBM Corporation
IBM Research
Test Design Techniques for Automated Testing (1) Inspected reference outcomes
textually specified result of a transformation
creation: manually inspect the outcome of the transformation once and save it BPMN Difference Model - InsertFragment(V1,”Record Claim”, Decision, fA) BPMN
Inspected Reference Outcomes
- InsertActivity(V1,”Retrieve add. Data”, Merge, Decision) -…
usage: automatic testing verifies outcome when applying a transformation
advantage: – very valuable during defect removal and refactoring – detection of unwanted side effects when changing transformations
drawback: only limited numbers of test cases possible (manual overhead)
11
© 2009 IBM Corporation
IBM Research
Test Design Techniques for Automated Testing (2) Invariant validation:
invariants about the outcome of a transformation
creation: encode invariants in Java or OCL BPMN to Workflow Graph Transformation
Dependency Computation
PST Computation
BPMN Workflow Graph 1
Process Structure Tree 1
Difference Model
Difference Model With Dependencies
Invariants: WFG and PST properties
Invariants:
• every node is reachable
•no cyclic dependencies
• StructuredNode has exactly one entry/exit edge
12
usage: check invariants automatically when applying transformation
advantage: large number of test cases possible, no manual overhead © 2009 IBM Corporation
IBM Research
Test Design Techniques for Automated Testing (3) Deviation test data
test data about created model structures
creation: determine test data to be computed, compute it automatically for each test case Difference Model - InsertFragment(V1,”Record Claim”, Decision, fA) - InsertActivity(V1,”Retrieve add. Data”, Merge, Decision)
#InsertFragment #InsertActivity #...
Repository for Deviation Test Data
-…
usage: in subsequent test runs, compare data automatically and report deviations
advantage: – valuable while refactoring and defect removal for detecting side effects – identification of input models where deviation occurs
13
© 2009 IBM Corporation
IBM Research
Tests and Transformation Overview BPMN to Workflow Graph Transformation
PST Computation
Correspondence Computation/ Process Matching
Difference Computation
Dependency Computation
Filtering of Non-BPMN Differences
BPMN Workflow Graph 1
BPMN
Workflow Graph 2
Process Structure Tree 1
Process Structure Tree 2
Inspected Reference Outcomes Invariant test
Deviation Test
14
Joint Process Structure Tree
Difference Model
Difference Model With Dependencies
BPMN Difference Model With Dependencies
Inspected Reference Outcomes Invariant test Of Models 1
Invariant test Of Models 2
Deviation Test 1
Invariant test Of Models 3
Deviation Test 2
© 2009 IBM Corporation
IBM Research
Architecture of Test Framework Test case input models are created manually Analysis Component is added for evaluating test results – based on test design techniques presented – produces detailed test result report
Analysis Component Model Transformation Execution Component
Test Case Input Models
15
Model Transformation Chain
Models
Reference Outcome Analysis/ Invariant Validation/ Deviation Analysis/
Inspected Reference Outcomes
Repository for Deviation Test Data
Test Result Reports
© 2009 IBM Corporation
IBM Research
Architecture of Test Framework
Manual Inspection Component simplifies creation of test case reference outcomes
Execution component reuses JUnit framework for test case execution
Test Case Execution Component (JUnit)
Manual Inspection Component
Analysis Component Model Transformation Execution Component
Test Case Input Models
16
Model Transformation Chain
Models
Reference Outcome Analysis/ Invariant Validation/ Deviation Analysis/
Inspected Reference Outcomes
Repository for Deviation Test Data
Test Result Reports
© 2009 IBM Corporation
IBM Research
Iterative and Incremental Development Process Implement model transformation
Create first set of test cases
Run test cases
Defects? Yes No
Improve model transformation implementation by removing defects
Increase set of test cases
17
No focus on requirements analysis Needs the automated testing environment Inspired by agile development
Implement functionality
New test cases required? Yes No New functionality? Yes
No Refactoring?
Refactor Yes
No
© 2009 IBM Corporation
IBM Research
Outline Motivation and Case Study Overview Requirements for Development of Transformation Chains Techniques and Approach – Test Design Techniques – Architecture of Test Framework – Development Process
Results Conclusion & Outlook
18
© 2009 IBM Corporation
IBM Research
Defects versus Test Case Input Models ~300 test case input model pairs with inspected reference outcomes fully automatic test environment based on the test framework architecture (completion in ~ 20 sec) refactoring needs automated testing 140
120
100
80 Test Case Input Models 60
Defects
40
20
0 12:00 10:53 11:23 12:46 16:23 10:17 10:22 11:24 12:19 10:25 11:59 13:47 16:44 15:49 11:10 12:00 17:00 15:09 13:33 11:45 14:35 13:15 13:58 16:29 13:36 13:53 15:19 15:23 10:52 10.10. 28.10. 28.10. 28.10. 28.10. 29.10. 29.10. 29.10. 29.10. 30.10. 30.10. 10.11. 11.11. 13.11. 25.11. 26.11. 26.11. 01.12. 02.12. 04.12. 04.12. 05.12. 05.12. 05.12. 23.12. 23.12. 23.12. 23.12. 08.01.
19
Date and Time
© 2009 IBM Corporation
IBM Research
Lessons Learnt Incremental development for model transformation chains is required in practice Iterative improvement of the transformations is the reality Systematic quality improvement is not possible without automated testing Code quality is an important issue when implementing model transformations in Java One has to choose the encoding of the inspected reference outcomes carefully
20
© 2009 IBM Corporation
IBM Research
Conclusion & Outlook Incremental and iterative development process for developing high-quality transformation chains Test design techniques and test framework for automated testing of transformation chains Further research challenges include: – Model coverage of test case input models – Automated generation of test case input models
21
© 2009 IBM Corporation