other sources, I have given due credit to them by citing them in the text of the thesis ... 3.2 An activity diagram for ATM money withdrawal functionality with ...... Debit amount. Dispense cash. Print receipt. Show error. Eject card. Authorize user ...... and it's reply are represented by an open arrowhead in sequence diagrams.
CONCURRENCY TEST SCENARIO GENERATION USING UML TRANSITION SEQUENCES
Mahesh Raghunath Shirole
CONCURRENCY TEST SCENARIO GENERATION USING UML TRANSITION SEQUENCES
Thesis submitted to the Indian Institute of Technology, Kharagpur For award of the degree of
Doctor of Philosophy by
Mahesh Raghunath Shirole Under the guidance of Professor Rajeev Kumar
DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING INDIAN INSTITUTE OF TECHNOLOGY, KHARAGPUR December 2016 c
2016, Mahesh Raghunath Shirole. All rights reserved.
Revision History Date (MonYear) Jul-2014 Mar-2015
Dec-2016
Version
Description
Text Color Code
1.0 1.1
Initial Draft of the thesis Revised draft after first revision comments
1.2
Revised Draft after second revision comments
Black Modifications as per first revision comments are shown in Royal Blue color Modifications as per second revision comments are shown in Green color
List of Modification – Appendix A
Appendix B
Dedicated To, My parents Shri. Raghunath Shirole and Sau. Indira Shirole
CERTIFICATE This is to certify that the thesis entitled Concurrency Test Scenario Generation using UML Transition Sequences, submitted by Mahesh Raghunath Shirole to Indian Institute of Technology, Kharagpur, is a record of bona fide research work under my supervision and I consider it worthy of consideration for the award of the degree of Doctor of Philosophy of the Institute.
Rajeev Kumar Professor (on lien) Department of Computer Science and Engineering, Indian Institute of Technology, Kharagpur -721 302, India. Currently, Professor, School Computer Systems and Science, JNU, New Delhi 110 067, India. Date:
Acknowledgments It gives me immense pleasure to convey my deep sense of gratitude to my supervisor Professor Rajeev Kumar for his expert guidance and support throughout my research work. He has taught me to work as an independent researcher. His ideas provided the platform of my research. He has been a steady source of wise encouragement. This work would not have been completed without his active involvement. My sincere thanks to my care-taker supervisors, Professor Sudebkumar Prasant Pal and Professor Arobindo Gupta, who have supported me to proceed in this hard journey. Their kind nature helped me in many occasions. I take this opportunity to offer my deep sense of gratitude to my doctoral scrutiny committee (DSC) members Professor Sudeshna Sarkar, Professor Rajib Mall, Professor Sudebkumar Prasant Pal and Professor Pawan Kumar for their valuable suggestions during my research tenure. I sincerely remember and thanks to Late. Anita Rajeev Kumar for her kind nature and support. I sincerely remember the support of office staff members. I am thankful to our work-group members Soma, Amit, and Mounika, for their valuable technical support. Thanks to my all colleagues. I am also thankful to Sharad, Shriniwas, Prasenjit, Dhiraj, Pramod, Bhagwan, Rudragouda, Santosh, Chinmaya, and Jasaswi for their support and suggestions. I am thankful to my friends at Mumbai Madhav, Hari, Madhu, Sandeep and Girish Bhole for their help. I take this opportunity to express my sincere thanks to Veermata Jijabai Technological Institute (VJTI) and former Director Prof. K. G. Narayankhedkar for sponsoring me to this research program. I would like to acknowledge the financial support of All India Council of Technical Education of Government of India for my study. It would have been impossible to achieve anything without the support of my family. I will never forget my parents sacrifice and support during tenure of four years of PhD. I have no words to acknowledge my wife Ulka, who took all the pain to manage all the problems and gave me all sorts of support during the period of my research work. My kids, Pranav and Shrawani whom I missed for the most of period in their childhood. My beloved family supported, listened and made it possible to embark on such an extraordinary journey. Mahesh Raghunath Shirole
Declaration I certify that a.
The work contained in the Thesis is original and has been done by myself under the supervision of my Supervisor. b. The work has not been submitted to any other Institute for any degree or diploma. c. I have followed the guidelines provided by the Institute in writing the thesis. d. I have conformed to the norms and guidelines given in the Ethical Code of Conduct of the Institute. e. Whenever I have used materials (data, theoretical analysis, and text) from other sources, I have given due credit to them by citing them in the text of the thesis and giving their details in the references. f. Whenever I have quoted written materials from other sources, I have put them under quotation marks and given due credit to the sources by citing them and giving required details in the references.
(Mahesh Raghunath Shirole)
List of Abbreviations
List of Abbreviations ADG BPC CCG CFG CS CQS DFS LevelPermute DP-DFS EA ECC ESMD GA IAPC IMPC MARTE MCFG NOCS RMCFG SC SCG SPC SUT TAPC TCS TEIS TS UML XML
Activity Diagram Graph 104 Basic Path Coverage 58 Concurrent Composite Graph 33 Control Flow Graph 45 Context Switch/Thread Switch 54 Concurrent Queue Search 11 Dept-first Search Level Permutation 11 Dependency Preserving Depth-first Search 12 Evolutionary Algorithm 11 Essential Concurrent Message Path Coverage 10 Enhanced State Machine Diagram 80 Genetic Algorithm 28 Interleaving Activity Path Coverage 10 Interleaving Message Path Coverage 10 Modeling and Analysis of Real-Time and Embedded Sys- 172 tems Message Control Flow Graph 12 No Thread/Context Switch 82 Reduced Message Control Flow Graph 12 Synchronization Coverage 10 Structured Composite Graph 33 Simple Path Coverage 138 Software Under Test 15 Total Activity Path Coverage 10 Test Case Suite 45 Total Execution Interleaving Sequences 101 Test Scenario 48 Unified Modeling Language 1 Extensible Markup Language 22
xiii
List of Figures 1.1
An activity diagram for an instance of geometric algorithm . . . . . .
5
1.2
A sequence diagram for a money transfer bank transaction representing concurrent behavior modeled using parallel and critical section fragments
6
2.1
A state transition model for read/write behavior of a shared data for data race analysis (Lei et al., 2008) . . . . . . . . . . . . . . . . . . .
32
Communication of activities inside concurrent flow using Send signal and Receive signal constructs . . . . . . . . . . . . . . . . . . . . . .
43
An activity diagram for ATM money withdrawal functionality with fork-join construct . . . . . . . . . . . . . . . . . . . . . . . . . . . .
44
A non-concurrent sequence diagram for ATM money withdrawal process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
63
A message control flow graph representation of the bank transaction sequence diagram shown in the Figure 1.2 . . . . . . . . . . . . . . .
66
An RMCFG representation of the bank transaction sequence diagram shown in the Figure 1.2 . . . . . . . . . . . . . . . . . . . . . . . . .
70
ATM money withdrawal banking functionality modeled in sequence diagram using parallel fragment to illustrate data access-tags . . . . .
78
An enhanced state machine diagram to analyze and classify test scenario’s concurrency behavior . . . . . . . . . . . . . . . . . . . . . . .
81
5.1
A translation method for synchronous messages of sequence diagrams
94
5.2
A translation method for asynchronous messages of sequence diagrams
94
5.3
A translation method for alternative fragments of sequence diagrams
95
5.4
A translation method for option fragments of sequence diagrams
. .
95
5.5
A translation method for loop fragments of sequence diagrams . . . .
96
3.1 3.2 3.3 3.4 3.5
4.1 4.2
xv
List of Figures 5.6
A translation method for parallel fragments of sequence diagrams . .
96
5.7
A translation method for general ordering inside the parallel fragment of sequence diagrams . . . . . . . . . . . . . . . . . . . . . . . . . . .
97
5.8
SDD: Non-nested (serial) decision-decision activity diagram . . . . . .
98
5.9
SDI: Non-nested (serial) decision-iteration activity diagram . . . . . .
98
5.10 NDD: Nested decision-decision activity diagram . . . . . . . . . . . .
99
5.11 NDI: Nested decision-iteration activity diagram . . . . . . . . . . . .
99
5.12 SDF: Non-nested (serial) decision-fork concurrent activity diagram . . 100 5.13 SFF: Non-nested (serial) fork-fork concurrent activity diagram . . . . 100 5.14 NDFD: Nested decision-fork-decision concurrent activity diagram . . 101 5.15 NDFF: Nested decision-fork-fork concurrent activity diagram . . . . . 101 6.1
An illustration example of non-nested decision iteration and fork-fork activity diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
6.2
A queue structure representing process of queue generation of main queue and sub queues for the Figure 6.1 . . . . . . . . . . . . . . . . 111
6.3
A level permutation algorithm illustration for example 1, which is having equal number of activities per level . . . . . . . . . . . . . . . . . 114
6.4
A level permutation algorithm illustration for example 2, which is having different number of activities per level . . . . . . . . . . . . . . . 115
6.5
An illustrative example of DFS level permutation algorithm . . . . . 116
6.6
A bank transaction example showing shared object access by two threads for read access and write access . . . . . . . . . . . . . . . . . . . . . 120
7.1
An illustration of a crossover operation for paths represented by chromosomes in evolutionary algorithm . . . . . . . . . . . . . . . . . . . 125
7.2
An illustration of a mutation operation for a path represented by the chromosome in evolutionary algorithm . . . . . . . . . . . . . . . . . 125
7.3
A chromosome encoding of a test scenario for the NDD activity diagram shown in Figure 5.10 . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
7.4
A hybrid evolutionary algorithm framework for generating interleaving test scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
7.5
A graph representing a number of test scenarios generated by DFS, Random DFS, and Evolutionary algorithms for non-concurrent activity diagrams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 xvi
List of Figures 7.6
A graph representing time required to generate test scenarios by DFS, Random DFS, and Evolutionary algorithms for non-concurrent activity diagrams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
7.7
A graph representing coverage analysis for SDF: Sequential decisionfork activity diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
7.8
A graph representing coverage analysis for SFF: Sequential fork-fork activity diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
7.9
A graph representing coverage analysis for NDFD: Nested decisionfork-decision activity diagram . . . . . . . . . . . . . . . . . . . . . . 139
7.10 A graph representing coverage analysis for NDFF: Nested decisionfork-fork activity diagram . . . . . . . . . . . . . . . . . . . . . . . . 139 7.11 A graph representing a number of test scenarios generated for concurrent activity diagrams by different algorithms (log scale) . . . . . . . 140 7.12 A graph representing time required to generate test scenarios for concurrent activity diagrams by different algorithms . . . . . . . . . . . 140 8.1
A CFG-like representation for the money transfer bank transaction example shown in the Figure 1.2 . . . . . . . . . . . . . . . . . . . . . 148
8.2
A structured composite graph / concurrent composite graph representation for the money transfer bank transaction example shown in the Figure 1.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
8.3
A sequence diagram illustrating general order construct inside the parallel fragment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
8.4
An RMCFG representation of the Figure 8.3, which has general order construct inside the parallel fragment . . . . . . . . . . . . . . . . . . 161
C.1 An architecture of path generator for CQS . . . . . . . . . . . . . . . 187 C.2 A path generator class diagram . . . . . . . . . . . . . . . . . . . . . 188 C.3 An activity diagram graph class diagram . . . . . . . . . . . . . . . . 189 C.4 An activity diagram graph text file format . . . . . . . . . . . . . . . 190 D.1 An architecture of path generator for DP-DFS path generator . . . . 191 D.2 A class diagram of path generator . . . . . . . . . . . . . . . . . . . . 192 D.3 A class diagram of MCFG graph
. . . . . . . . . . . . . . . . . . . . 193
D.4 A class diagram of message node of MCFG graph . . . . . . . . . . . 194 D.5 A text file format of the MCFG and RMCFG graphs . . . . . . . . . 194 xvii
List of Figures E.1 E.2 E.3 E.4 E.5
A class diagram of the State Machine Executor . . . . . . . . . . . . A text file format of the state machine . . . . . . . . . . . . . . . . . A class diagram of MCFG graph for access trace generation . . . . . A class diagram of message node for access trace generation . . . . . A text file format of the MCFG and RMCFG graphs for access trace generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.1 An MCFG graph representation of a synchronous message for sequence diagrams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F.2 An MCFG graph representation of an asynchronous message for sequence diagrams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F.3 An MCFG graph representation of an alternative combined fragment for sequence diagrams . . . . . . . . . . . . . . . . . . . . . . . . . . F.4 An MCFG graph representation of an option combined fragment for sequence diagrams . . . . . . . . . . . . . . . . . . . . . . . . . . . . F.5 An MCFG graph representation of a loop combined fragment for sequence diagrams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F.6 An MCFG graph representation of a parallel combined fragment for sequence diagrams . . . . . . . . . . . . . . . . . . . . . . . . . . . . F.7 An MCFG graph representation of a general ordering inside the parallel fragment for sequence diagrams . . . . . . . . . . . . . . . . . . . . .
198 199 200 200 202
204 205 205 206 207 207 208
G.1 Translation of parallel fragment in sequence diagram into activity diagram for case a . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 G.2 Translation of parallel fragment sequence diagram into activity diagram for case b . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 G.3 Translation of parallel fragment sequence diagram into activity diagram for case c . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213 H.1 H.2 H.3 H.4
Translation Translation Translation Translation
of of of of
sequence sequence sequence sequence
diagram diagram diagram diagram
into into into into
xviii
activity activity activity activity
diagram diagram diagram diagram
for for for for
case case case case
a b c d
. . . .
. . . .
. . . .
218 218 219 220
List of Tables 1.1
Coverage and concurrency bug triggering analysis for the activity diagram shown in Figure 1.1 . . . . . . . . . . . . . . . . . . . . . . . .
5
Coverage and concurrency bug triggering analysis based on different intermediate representations and using different test scenario generators for the sequence diagram shown in Figure 1.2 . . . . . . . . . . . . .
8
2.1
Studies in tree-based test case generation . . . . . . . . . . . . . . . .
26
2.2
Studies in graph-based test case generation . . . . . . . . . . . . . . .
27
2.3
Studies in heuristic-based test case generation . . . . . . . . . . . . .
29
2.4
Gene encoding and fitness function for search based concurrency testing approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
34
Structural element coverage achieved by test cases for the activity diagram shown in Figure 3.2 . . . . . . . . . . . . . . . . . . . . . . . .
50
3.2
Structural element coverage achieved by the combination of test cases
51
3.3
Behavioral coverage achieved by test cases for behaviors B1 , B2 , and B31 of the activity diagram shown in Figure 3.2 . . . . . . . . . . . . .
56
3.4
Behavioral coverage achieved by the combination of test cases . . . .
56
3.5
Message and message sequence coverage achieved by test cases for the sequence diagram shown in Figure 3.3 . . . . . . . . . . . . . . . . . .
64
3.6
A summary of the proposed concurrency coverage criteria . . . . . . .
73
4.1
State transition table for ESMD . . . . . . . . . . . . . . . . . . . . .
83
4.2
UML diagrams’ attributes: application name, diagram type, data access type, and total interleaving paths for different applications . . . .
85
1.2
3.1
xix
List of Tables 4.3
4.4 4.5 4.6
Concurrent behavior classification using existing state machine by Lei et al. (2008) approach for test scenarios generated using DFS-BFS, EA, and DP-DFS algorithms . . . . . . . . . . . . . . . . . . . . . . . . .
86
Transition table of input data access trace Access-trace-1 for ESMD classifier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
88
Transition table of input data access trace Access-trace-2 for ESMD classifier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
88
Concurrent behavior classification using enhanced state machine diagram for test scenarios generated by DFS-BFS, EA, and DP-DFS algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
89
6.1
Interleaving activity path coverage (IAPC) of test scenarios generated by modified-DFS, DFS-BFS, CQS, and DFSLevelPermute algorithms for Figure 5.6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
6.2
Simple path coverage (SPC) and interleaving activity path coverage (IAPC) for test scenarios generated by modified-DFS, DFS-BFS, CQS, and DFS LevelPermute algorithms for Figure 5.7 . . . . . . . . . . . 119
6.3
Simple path (SPC), interleaving activity path (IAPC), and synchronization (SC) coverage for test scenarios generated by modified-DFS, DFS-BFS, CQS, and DFS LevelPermute algorithms for Figure 6.6 . . 121
7.1
Action scripts associated with control nodes to update node-list . . . 130
7.2
Coverage analysis of test scenarios generated for non-concurrent activity diagrams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
7.3
Basic path coverage (BPC), simple path coverage (SPC), interleaving activity path coverage (IAPC), and total activity path coverage (TAPC) analysis for concurrent activity diagrams . . . . . . . . . . . 138
7.4
Activity diagrams’ attributes: total number of activities, basic paths, simple paths, interleaving paths, and total paths for different applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
7.5
Basic path (BPC), simple path (SPC), interleaving activity path (IPC), and total interleaving activity path (TAPC) coverage analysis for test scenarios generated by DFS-BFS, DFS-LevelPermute, evolutionary algorithm, and hybrid evolutionary algorithms for applications listed in Table 7.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 xx
List of Tables 8.1 8.2 8.3 8.4 8.5
8.6 8.7
Mapping rules for node in MCFG construction procedure . . . . . . . Mapping for edges in MCFG construction procedure . . . . . . . . . . Efficacy analysis of intermediate representations for test scenario generation by respective test scenario generator for different applications Analysis of test suite size reduction by RMCFG representation over MCFG representation for different applications . . . . . . . . . . . . . Coverage analysis for approaches DFS:SCG by Nayak and Samanta (2010), DFS-BFS:CCG by Khandai et al. (2011), and proposed approaches CQS:ADG and DP-DFS:RMCFG algorithms . . . . . . . . . Description of mutation operators developed by Bradbury et al. (2006) for Java programs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mutation analysis for the Java programs developed from the designs for concurrent mutation operators RXTC, RSK, RSB, SHCR, and SKCR.
154 155 164 166
168 170 170
A.1 Modifications as per second review report . . . . . . . . . . . . . . . . 181 B.1 Modifications as per second review report . . . . . . . . . . . . . . . . 183
xxi
Abstract Concurrency is used for superior computational efficiency and/or better resource utilization. However, concurrent program execution is unpredictable due to random execution interleavings, which may give rise to different concurrency errors namely, data race, synchronization, starvation, and deadlocks. Concurrent system testing can be broadly grouped into: (i) code-based techniques, (ii) model checking based techniques, and (iii) UML design based techniques. UML model based testing not only provides early test case design and testing effort estimation, but also permits validation of the implementation. A few approaches have addressed test scenario generation for concurrency from UML models using graph theoretic approaches, but have failed to explore sufficient interleaving scenarios. Some of the difficulties in designing concurrency test cases from UML designs are: (i) unexplored interleavings amongst threads, (ii) increasing concurrency of behavioral coverage criteria, and (iii) incomplete utilization of concurrency constructs for test case design. Test scenario generation from a UML design is done using UML transition sequences, which could be (i) a state-sequence in a state machine diagram, (ii) an activity-sequence in an activity diagram, or (iii) a message-sequence in a sequence diagram. The contributions of this thesis addressing issues as described above are: (i) New Concurrency Coverage Criteria: Traditional structural coverage criteria are ineffective for concurrent designs. We have presented new concurrency criteria based on concurrency constructs of UML designs. (ii) Concurrent Behavior Analysis of Test Scenarios: Different classes of errors, such as data-race, synchronization, starvation, and deadlocks may occur in concurrent program execution. Any such error can be uncovered by a suitable test scenario, generated by an appropriate test scenario generator. We have developed a state machine based approach to analysis and classification of test scenarios for different classes of concurrency bugs. (iii) Test Scenario Generation from Non-nested Concurrent Activity Diagram: For UML concurrency constructs most of the test scenario generation algorithms do not explore adequate interleavings amongst threads, thereby do not uncover all concur-
rency bugs. We have developed two approaches. The first approach called concurrent queue search (CQS) generates interleaving test scenarios for uncovering bugs inside concurrency constructs. The second approach called DFS level permutation (DFS LevelPermute) is based on permutations that generate interleaving test scenarios to maximize concurrency coverage. (iv) Test Scenario Generation from Nested Concurrent Activity Diagrams: Nested concurrency increases the complexity of test scenario generators. Most algorithms fail to generate test scenarios for nested concurrent activity diagrams. We have developed a meta-heuristic based evolutionary approach for generating test scenarios for concurrency testing. (v) Test Scenario Generation from Sequence Diagrams with Concurrency Constructs: Existing test scenario generation algorithms have not adequately explored parallel fragments and critical regions of sequence diagrams for designing test scenarios. We have developed a dependency preserving depth first search (DP-DFS) algorithm to explore concurrent behaviors in sequence diagrams. Our proposed concurrency criteria ensure exploration of interleaving test scenarios, thereby helping the detection of concurrency errors. CQS generates bug triggering scenarios and DFS LevelPermute algorithms improves concurrency coverage for nonnested concurrent activity diagrams. Our evolutionary algorithm based techniques have generated test scenarios for both non-nested and nested concurrent activity diagrams. Our DP-DFS technique achieves 100% essential concurrent message path coverage, 50 to 100% synchronization coverage, and up to 20% interleaving message path coverage. Concurrent behavioral analysis of test scenarios significantly reduces test suite size. In a nutshell, the contributions of this thesis are focused on UML transition sequence based test scenario generation for concurrency testing. Our new UML-based coverage criteria for concurrency testing are satisfied by our proposed test scenario generation techniques. Keywords: Concurrency coverage criteria, Interleaving test scenario generation, Concurrent queue search, DFS level permutation, Evolutionary algorithm, DP-DFS algorithm.
Contents Title Page
iii
Certificate
viii
Acknowledgements
x
Declaration
xii
List of Abbreviations
xiv
List of Figures
xiv
List of Tables
xix
Abstract
xxv
1 Introduction 1.1 Motivation . . . . . . . . . . . . . . 1.2 Research Objectives . . . . . . . . . 1.3 A Brief Summary of Contributions 1.4 Thesis Organization . . . . . . . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
2 Literature Review 2.1 Code-Based Concurrency Testing . . . . . . . . . . . . . 2.1.1 Model Checking Techniques . . . . . . . . . . . . 2.1.2 Reachability Based Technique . . . . . . . . . . . 2.1.3 Thread Scheduling Techniques . . . . . . . . . . . 2.2 UML-Based Sequential Testing . . . . . . . . . . . . . . 2.2.1 Direct UML Specifications Processing Techniques 2.2.2 Formal Specification-Based Techniques . . . . . . 2.2.3 Graph-Theoretic Techniques . . . . . . . . . . . . 2.2.4 Meta-heuristic Techniques . . . . . . . . . . . . . 2.3 UML-Based Concurrency Testing . . . . . . . . . . . . . 2.3.1 Graph-Theoretic Techniques . . . . . . . . . . . . xxv
. . . .
. . . . . . . . . . .
. . . .
. . . . . . . . . . .
. . . .
. . . . . . . . . . .
. . . .
. . . . . . . . . . .
. . . .
. . . . . . . . . . .
. . . .
. . . . . . . . . . .
. . . .
1 4 8 9 12
. . . . . . . . . . .
15 15 16 18 19 22 22 23 25 28 31 31
Contents 2.3.2 Meta-Heuristic Techniques . . . . . . . . . . . . . . . . . . . . 2.3.3 Model Checking Techniques . . . . . . . . . . . . . . . . . . . Discussion and Summary . . . . . . . . . . . . . . . . . . . . . . . . .
33 34 36
3 Concurrency Coverage Criteria 3.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.1 Sequence Diagram . . . . . . . . . . . . . . . . . . . . . . . . 3.1.2 Activity Diagram . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.3 Coverage Criteria . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Test Adequacy Criteria for Activity Diagrams . . . . . . . . . . . . . 3.2.1 Existing Coverage Criteria . . . . . . . . . . . . . . . . . . . . 3.2.2 Concurrency Coverage Criteria for Activity Diagrams . . . . . 3.2.2.1 Basic path coverage criterion for concurrent activity diagrams . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2.2 Interleaving activity path coverage criterion for concurrent activity diagrams . . . . . . . . . . . . . . . 3.2.2.3 Total activity path coverage criterion . . . . . . . . . 3.3 Test Adequacy Criteria for Sequence Diagrams . . . . . . . . . . . . . 3.3.1 Existing Coverage Criteria . . . . . . . . . . . . . . . . . . . . 3.3.2 Concurrency Coverage Criteria for Sequence Diagrams . . . . 3.3.2.1 Interleaving message path coverage criterion . . . . . 3.3.2.2 Synchronization coverage criterion . . . . . . . . . . 3.3.2.3 Essential concurrent message path coverage criterion 3.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
39 40 40 42 44 48 48 57
4 Test Scenario Classification for Concurrent Behaviors 4.1 Data-Access Extensions to UML Models . . . . . . . . . 4.2 State Machine Based Test Scenario Analysis . . . . . . . 4.2.1 Existing State Machine Classifier . . . . . . . . . 4.2.2 Enhanced State Machine Classifier . . . . . . . . 4.3 Experimental Results . . . . . . . . . . . . . . . . . . . . 4.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . .
75 76 79 79 80 84 88
. . . . .
91 92 97 98 99 102
2.4
. . . . . .
. . . . . .
5 Translation of Sequence Diagrams into Activity Diagrams Classification of Activity Diagrams 5.1 Translation Method for Sequence Diagram . . . . . . . . . . 5.2 Classification of Activity Diagrams . . . . . . . . . . . . . . 5.2.1 Non-Concurrent Activity Diagrams . . . . . . . . . . 5.2.2 Concurrent Activity Diagrams . . . . . . . . . . . . . 5.3 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . xxvi
. . . . . .
. . . . . .
. . . . . .
. . . . . .
57 58 59 60 60 64 68 68 71 73
and . . . . .
. . . . .
. . . . .
. . . . .
Contents 6 Test Scenario Generation from Non-nested Concurrent agrams 6.1 Concurrent Queue Search Technique . . . . . . . . . . . 6.2 Permutation Based Techniques . . . . . . . . . . . . . . . 6.3 Experimental Results . . . . . . . . . . . . . . . . . . . . 6.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . .
Activity Di103 . . . . . . . 104 . . . . . . . 112 . . . . . . . 117 . . . . . . . 121
7 Test Scenario Generation from Nested Concurrent Activity Diagrams 7.1 Evolutionary Algorithm Basics . . . . . . . . . . . . . . . . . . 7.2 Evolutionary Algorithm Based Test Scenario Generator . . . . 7.3 Hybridization of Evolutionary Algorithm . . . . . . . . . . . . 7.4 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . 7.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . .
. . . . .
. . . . .
. . . . .
123 124 127 133 134 141
8 Test Scenario Generation from Sequence Diagrams with Concurrent Behavior 145 8.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 8.2 Dependency Preserving DFS Based Test Scenario Generator . . . . . 152 8.2.1 Intermediate Representation: MCFG . . . . . . . . . . . . . . 152 8.2.2 Dependency Preserving DFS Algorithm . . . . . . . . . . . . . 155 8.2.3 Elimination of Independent Operation Interleavings . . . . . . 157 8.2.4 Complexity Analysis . . . . . . . . . . . . . . . . . . . . . . . 162 8.3 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 8.3.1 Effect of Intermediate Representation on Test Suite Size . . . 164 8.3.2 Test Suite Size Reduction Due to RMCFG Representation . . 165 8.3.3 Coverage Analysis . . . . . . . . . . . . . . . . . . . . . . . . 167 8.3.4 Mutation Analysis . . . . . . . . . . . . . . . . . . . . . . . . 169 8.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 9 Conclusions and Future Work
175
Appendices
179
A First Review Report Modifications List
181
B Second Review Report Modifications List
183
C CQS Path Generator Implementation Details
187
D DP-DFS Path Generator Implementation Details
191
E State Machine Executor Implementation Details
197
F MCFG Representation for Different Sequence Diagrams
203
xxvii
Contents G Pseudo Code for Variations of Parallel Fragments
209
H More Examples for Translation of Sequence Diagram to Activity Diagram 217 References
223
xxviii
Chapter 1 Introduction With the advent of multi-core architectures, concurrency plays a significant role in the design and development of modern software systems. Concurrency is used for superior computational efficiency and/or better resource utilization. The difficulty of concurrent program testing is intensified due to features such as non-determinism, synchronization, and interprocess communication (Souza et al., 2011). Concurrent program execution is unpredictable due to random execution interleavings that may give rise to different kinds of concurrency errors like data race, synchronization, starvation, and deadlocks (Edelstein et al., 2002). Concurrency errors are difficult to uncover by traditional testing methods alone, specifically traditional structural testing. Traditional structural testing focuses on structural element coverage and sequential behavioral coverage criteria comprising of statement, branch and basic path coverage. Therefore, testing of concurrent systems to uncover concurrency errors is becoming increasingly important. Concurrent system testing can be broadly grouped into: (i) code-based techniques (Edelstein et al., 2002; Stoller, 2002; Sen, 2008), (ii) model checking based techniques (Godefroid, 1997; Havelund and Pressburger, 2000), and (iii) Unified Modeling Language (UML) design based techniques (Kim et al., 1999; Hartmann et al., 2005; Eshuis, 2006; Shousha et al., 2012). In code-based testing, test case design is done after the coding phase. Furthermore, the overall schedule in software development life cycle may be delayed since test case design is done in the later phase. On the contrary, model checking based test case generation is done early in the design cycle, whereby the entire state space is explored systematically. However, exponential state-space limits scalability of model checking approaches. On the other hand, UML 1
1. Introduction design based testing has dual benefits. It not only provides early test case design and testing effort estimation, but also permits validation of the implementation against UML designs. In this thesis, we study UML-based approaches for concurrency testing. UML has become the de-facto standard for the design and development of large software systems, where high level of abstractions can be used for representing concurrent behavior. Concurrent behavior is represented in terms of fork-join constructs in activity diagrams, parallel fragments in sequence diagrams, and concurrent states in state machine diagrams. Synchronization amongst threads is represented using critical region fragments and general ordering constructs in sequence diagrams. Given the design of concurrent software system in terms of activity, sequence, and state machine diagrams duly representing concurrent behavior within the system, it is promising to generate test cases for concurrency testing. UML-based concurrency testing can be done by applying different approaches such as search based techniques, model checking techniques, and graph theoretic techniques. In this thesis, we consider graph theoretic approaches for concurrency testing. Existing UML-based graph theoretic testing techniques contribute more to traditional structural testing aiming for coverage of structural elements and sequential behaviors. Since such techniques cover only sequential behavior of systems, it is necessary to develop techniques that also cover concurrent behavior. In order to cover concurrent behavior, it is necessary to develop behavioral test cases that consist of test data and test scenario (behavioral pattern at run time). Test data generation for UML-based concurrency testing can be done in a manner similar to that of code-based testing. A test scenario for sequential testing is a sequential behavior that consists of a sequence of actions representing a specified behavior (function or task). A test scenario is a particular sequence of steps to be tested, which does not contain any test data (i.e., input or expected outputs) (Reuys et al., 2003). However, a test scenario for concurrency testing is a concurrent behavior that includes a representation of the non-deterministic interleavings amongst threads in a concurrency setting. In UML-based testing, test scenario is represented using a UML transition sequence that can be (i) a state-sequence from a state machine diagram, (ii) an activity-sequence from an activity diagram, or (iii) a message-sequence from a sequence diagram. In this thesis, we focus on generation of transition sequences from activity and sequence diagrams for generating test scenarios for concurrency testing. In concurrent design, a critical section is assumed to be an atomic activity. However, a programmer has to enforce atomic execution of sequence of accesses to shared 2
data to avoid data inconsistency problem (Yamada et al., 2011). Generally programmers are used to sequential thinking and frequently assume code region to be atomic without proper enforcement this leads to atomicity-violation bugs (Lu et al., 2012). A program may lead to data race errors in absence of synchronization mechanism due to programmers mistake to obtain the appropriate locks, programmers fail to obey locking discipline, or every access to shared data is protected but a consistency property exists between multiple pieces of shared data (Vaziri et al., 2006). There are several research papers, addressing the issues of concurrency errors in the implementation using high-level programming language constructs, e.g., Dinning and Schonberg (1991); Vaziri et al. (2006); S¨ ußand Leopold (2008); Yamada et al. (2011); and Lu et al. (2012). Recent study by Lu et al. (2008) says that almost 97% non-deadlock bugs belong to atomicity violation, i.e., the desired serializability among multiple memory accesses is violated, and order violation, i.e., the desired order between two (groups of) memory accesses is flipped. Another study on the programming errors in parallel programming by S¨ ußand Leopold (2008) indicates that higher percentage of errors observed in a) access to shared variables not protected and b) Use of locks without flush. In such cases, predetermined interleaving test scenarios designed from UML concurrent design will help in uncovering some of the errors introduced by programmers. In testing concurrency errors, one may need test as large as n! execution interleavings to detect a concurrency error, where n is the degree of parallelism (Dinning and Schonberg, 1991). How to select representative interleavings from exponential interleaving space is an open problem (Lu et al., 2012). One of the possible ways to select representative interleavings for concurrency testing is to limit the search of interleaving around the dependent operations due to synchronization and locking mechanism. Therefore, testing correct implementation of critical region needs test scenarios to be designed specifically in interleaved fashion serialized and unserialized way inside and outside critical sections around the dependent operations (Hammer et al., 2008; Lu et al., 2012). Concurrency introduces the possibility of errors that do not exist in sequential programs. Concurrency errors, such as deadlock, race condition, and atomicity violation, that are not present in traditional sequential programming. Deadlock is perhaps the most common concurrency error that might occur in almost all parallel programming paradigms. A deadlock occurs when a chain of processes/threads are involved in a cycle in which each process is waiting for resources/locks that are held by some other processes (Naik et al., 2009). Data race happens when two or more accesses 3
1. Introduction from different threads access the same shared variable without proper synchronization and at least one of the accesses is a write to the variable (Savage et al., 1997). An atomicity violation occurs when an interleaved execution of a set of statements (expected to be atomic) by multiple threads is not equivalent to any serial execution of the same set of statements (Farzan and Madhusudan, 2006). The motivation is to develop ways to design interleaving test scenarios from UML concurrent designs that may uncover concurrency errors. Some of the difficulties in designing concurrency test cases from UML designs are: (i) unexplored interleavings amongst threads, (ii) lack of adequate concurrency behavioral coverage criteria, and (iii) limited utilization of concurrency constructs for test case design. Without addressing these issues, it is difficult to detect concurrency errors. The contributions of this thesis addressing the above issues are: (i) defining new concurrency behavioral coverage criteria based on UML concurrency constructs, (ii) analyzing test scenarios to classify them into concurrency behavior classes, (iii) enhancing concurrent behavioral coverage by introducing interleaving test scenario generators, (iv) developing meta-heuristic based approach to generate test scenarios for nested concurrent activity diagrams, and (v) exploring concurrency constructs of sequence diagrams for concurrency testing.
1.1
Motivation
We consider two examples to demonstrate (i) how traditional structural coverage criteria are inadequate for concurrency testing, and (ii) need of new test scenario generators to uncover concurrency errors. In the first example, we consider an activity diagram for an instance of geometric algorithm shown in Figure 1.1. Two threads say T hread1 and T hread2 are working on two separate processes and having shared data coordinates. T hread1 sets coordinates with actual parameters (x1 , y1 ). T hread2 sets coordinates with actual parameters (x2 , y2 ). Both threads work with shared coordinates(X,Y). Threads work with their local copy of coordinates in activity useCoordinates(). The concurrency construct fork-join in the diagram represents parallel execution of activities in two threads that may cause (3+5)!/3!*5!=56 execution interleaving scenarios. If one applies existing test scenario generators like standard DFS, Modified DFS by Chandler et al. (2005), and DFS-BFS by Kundu and Samanta (2009) 4
1.1. Motivation then they generate two test scenarios for the given example. These approaches achieve S Initialize()
a1 F
Process2()
a5 a2
Process1() a6
a3
SetCoordinates(x2,y2)
SetCoordinates(x1,y1)
a4
useCoordinates()
a7
getXCoordinate()
a8
getYCoordinate()
a9
useCoordinates()
J Process3()
a10 E
Figure 1.1: An activity diagram for an instance of geometric algorithm
100% structural element coverage like activity and transition, and 100% sequential behavioral coverage like basic path as shown in Table 1.1. On the contrary, these approaches should have explored concurrent behavior represented by concurrency constructs. To explore concurrency constructs a coverage requirement should be concurrent behavior, as oppose to the structural element and sequential behavioral coverage. This indicates structural coverage criteria are inadequate to explore concurrent behaviors, and they do not trigger concurrency bugs as shown in Table 1.1. Table 1.1: Coverage and concurrency bug triggering analysis for the activity diagram shown in Figure 1.1 Approach
Standard
# tests 2
DFS Modified DFS
2
DFS-BFS
2
Scenario Activity S → a1 → F a10 → E S → a1 → F a9 → J → a10 S → a1 → F a6 → a7 → a8 S → a1 → F a9 → a2 → a3 S → a1 → F a4 → a7 → a8 S → a1 → F a7 → a4 → a8
→ a2 → a3 → a4 → J → → → → → → → → → → →
a5 E a2 a9 a5 a4 a2 a9 a5 a9
Coverage (%) Transition
Basic path
Trigger bug?
Interleaving scenarios
100
100
100
No
0
100
100
100
No
2
100
100
100
No
2
→ a6 → a7 → a8 → → a3 → a4 → a5 → J → a10 → E → a6 → a7 → a8 → J → a10 → E → a5 → a3 → a6 → J → a10 → E → a2 → a6 → a3 → J → a10 → E
→ → → →
Moreover, above mentioned test scenario generators explore very limited approximately 4% interleavings of total execution interleavings. Due to this reason they may not generate execution interleavings that may uncover concurrency bugs. For 5
1. Introduction instance, an interleaving test scenario like I1 = S → a1 → F → a2 → a5 → a6 → a7 → a3 → a4 → a8 → a9 → J → a10 → E, may uncover data inconsistency error due to improper synchronization primitives. Example in Figure 1.1 demonstrates the data inconsistency in some of the execution interleavings such as I1 . Suppose the input test data for the first thread (x1 , y1 ) is (50, 50) and the second thread (x2 , y2 ) is (0, 0). Then test scenario I1 leads to non-existing coordinates (0, 50) at activity a9 thereby generating a data-race error. According to the equivalence class partitioning, one has to select representative test scenarios from every equivalence class, i.e., valid classes and invalid classes (Beizer, 1990). Test scenarios generated by modified-DFS and DFS-BFS represent a valid class, while test scenarios like I1 represents an invalid class. Test scenarios like I1 need to be explored for concurrency testing. This indicates that existing UML-based graph theoretic approaches fail to generate test scenarios that may uncover concurrency errors. In the second example, we consider a sequence diagram representing a money transfer bank transaction scenario in Figure 1.2. It is assumed the initial balance is of Rs. 1000 in each account. The parallel fragment in the diagram introduces two SD: MoneyTransfer
bank:Bank
thread1:Thread
thread2:Thread
account1:Account
account2:Account
m1: initialize() Par: transfer m2: transfer(account1, account2, 200) Critical: critical1
m3: withdraw(200) m4: deposit(200)
m5: transfer(account2, account1, 100) Critical: critical2
m6: withdraw(100)
m7: deposit(100)
m8: dispalyStatus()
Figure 1.2: A sequence diagram for a money transfer bank transaction representing concurrent behavior modeled using parallel and critical section fragments
threads. Parallel execution of these two: thread1 (m2 , critical region1 start, m3 , m4 , critical region1 end) and thread2 (m5 , critical region2 start, m6 , m7 , critical region2 end) threads, which may cause (5+5)!/5!*5! = 252 execution interleavings without re6
1.1. Motivation strictions. Execution interleaving calculation is based on MCFG representation in Figure 3.4 of Figure 1.2. Critical regions inside parallel fragments introduce synchronization between two threads that restricts some of the execution interleavings. In case of critical regions shown in sequence diagrams, there is a possibility of confusion that critical regions implementation is atomic. In the perspective of software testing, implementation of critical region for shared data should not be assumed to be always correct for the designs shown in sequence diagrams. Therefore, a tester should test valid as well as invalid possibilities of the implementation. So, test scenarios should have serialized as well as unserialized transition sequences. In Figure 1.2, two threads are shown concurrent and they transfer money between accounts account1 and account2 . The existing UML-based test scenario generators like DFS, DFS-like by Nayak and Samanta (2010), and DBF-BFS by Khandai et al. (2011) generate two test scenarios from the given sequence diagram. These approaches satisfy 100% message and message path coverage as shown in Table 1.2. They do not explore sufficient interleaving execution scenarios to uncover concurrency bugs. However, there are a few interleavings like I2 = sdStart → m1 → CF parStart → m2 → CF critical1Start → m3 → m5 → CF critical2Start → m6 → m7 → CF critical2End → m4 → CF critical1End → m8 → sdEnd, that either lead to a deadlock or a data-race error. Interleaving I2 is defined on MCFG representation in Figure 3.4 of Figure 1.2. The interleaving I2 is illegal and is not possible if the programmer has enforced atomic execution of the messages inside the critical region using proper synchronization primitives. However, if the programmer fails to enforce atomic execution by not obeying synchronization primitives or by applying inappropriate locking sequences, then interleaving I2 may lead to data-race or deadlock. The interleaving like I2 has been identified for testing based on the equivalence partitioning method. In this case, interleaving I2 represents an invalid partition. For example, a programmer chooses not to acquire locks at the critical region start in the implementation assuming that withdraw and deposit operations acquired locks. However, these locks are not released in the implementation. In such case, for scenario I2 say m3 acquires lock L1 on account1 then message m6 acquires lock L2 on account2 . After that, neither message m7 able to acquire lock L1 on account1 nor message m4 able to acquire lock L2 on account2 . Thus, this situation leads to deadlock. Generated test scenarios by the above approaches do not trigger concurrency errors because they do not generate test scenarios like I2 . Structural coverage criteria like message and message path are insufficient to satisfy concurrent behavior represented 7
1. Introduction by scenario I2 . Table 1.2: Coverage and concurrency bug triggering analysis based on different intermediate representations and using different test scenario generators for the sequence diagram shown in Figure 1.2 Approach, Intermediate representation
# tests
Scenario
Coverage (%) Message Message path
sdStart → m1 → CF parStart → m2 100 100 → CF critical1Start → m3 → m4 → CF critical1End → CF parEnd → m8 → sdEnd sdStart → m1 → CF parStart → m5 → CF critical2Start → m6 → m7 → CF critical2End→ CF parEnd → m8 → sdEnd DFS, SCG 2 sdStart → m1 → f ork → block1 → 100 100 join → m8 → sdEnd sdStart → m1 → f ork → block2 → join → m8 → sdEnd DFS-BFS, CCG 2 sdStart → m1 → f ork → block1 → 100 100 block2 → join → m8 → sdEnd sdStart → m1 → f ork → block2 → block1 → join → m8 → sdEnd CFG - control flow graph Figure 8.1, SCG - structured control flow graph Figure 8.2, CCG - composite control flow graph Figure 8.2. Standard CFG
DFS,
2
Trigger bug?
Interleaving scenarios
No
0
No
2
No
2
Both the above examples demonstrate that existing test scenario generation approaches are not sufficient to explore adequate interleaving test scenarios for concurrency testing. Though they achieve 100% coverage, they do not generate interleavings like I1 or I2 . At the same time, it is worth noting that traditional structural coverage criteria neither insist on concurrent behavior requirements nor sufficient for concurrency testing. The above analysis motivates us (i) to define a new set of concurrency coverage criteria based on interleaving, and (ii) to develop alternative approaches for generating test scenarios that may uncover concurrency bugs.
1.2
Research Objectives
After a comprehensive literature survey of UML-based graph theoretic approaches, we identified the following research gaps: (i) lack of adequate concurrent behavioral coverage criteria, (ii) need for test scenario generation algorithms that explore sufficient interleaving space for detecting concurrency errors, and (iii) limited use of UML concurrency constructs for designing test scenarios that trigger concurrency bugs. Based on these observations we set the following research objectives: (i) New concurrency coverage criteria: Traditional structural coverage criteria are ineffective for concurrent designs. It is necessary to propose new concurrency coverage criteria based on concurrency constructs of UML designs. (ii) Analysis and classification of test scenarios: Different classes of errors, such as 8
1.3. A Brief Summary of Contributions data-race, synchronization, starvation, and deadlocks may occur in concurrent program execution. Any such error can be uncovered by a suitable test scenario, which is generated by an appropriate test scenario generator. Analysis and classification of test scenarios for different classes of concurrency bugs can be achieved using a state machine based approach. The proposal is to select representative test scenarios for different classes of concurrency behaviors, thereby minimizing the size of the requisite test suite. We develop a state machine based approach to analyze and classify test scenarios for different classes of concurrency behaviors. (iii) Heuristic-based test scenario generation: Existing test scenario generation algorithms, such as modified DFS by Chandler et al. (2005) and DFS-like by Nayak and Samanta (2010), do not explore adequate interleavings amongst threads, these algorithms do not uncover most of the concurrency bugs. It is necessary to develop new heuristic based test scenario generators. (iv) Evolutionary approach for nested concurrent activity diagrams: Nested concurrency increases the complexity of test scenario generators. Most algorithms fail to generate test scenarios for nested concurrent activity diagrams. To circumvent this problem, it is necessary to develop a meta-heuristic based evolutionary approach for generating test scenarios for concurrency testing. (v) Exploring concurrency constructs in sequence diagrams: Existing test scenario generation algorithms have not adequately explored parallel fragments and critical regions of sequence diagrams for designing test scenarios. Such algorithms may not trigger all concurrency bugs. It is essential to develop a test scenario generator for exploring concurrency constructs in sequence diagrams. In this thesis, we explore concurrency due to i) f ork − join constructs in activity diagrams and parallel f ragments and critical regions in sequence diagrams; and ii) the asynchronous messages like send and receive signals in activity diagrams and asynchronous message and general ordering inside parallel fragments in sequence diagrams.
1.3
A Brief Summary of Contributions
This thesis presents a compilation of UML transition sequence based concurrency testing techniques. A brief summary of contributions of this thesis is presented below. 9
1. Introduction Concurrency Coverage Criteria- Behavioral coverage criteria are limited to sequential behaviors for UML based techniques in the literature. We have extended code-based concurrency coverage criteria to UML behavioral models such as activity and sequence diagrams. Our interleaving coverage criteria are inspired from threadpair-interleaving as in Lu et al. (2007). Our synchronization coverage criteria are based on the synchronization coverage model as in Bron et al. (2005). This work introduces UML transition sequence based concurrent behavioral coverage criteria. Proposed concurrency coverage criteria are based on intermediate graph representations of respective artifacts. For sequence diagrams, we have proposed interleaving message path coverage (IMPC), synchronization coverage (SC), and essential concurrent message path coverage (ECC) criteria. IMPC criterion explores interleaving inside parallel fragments of sequence diagrams. SC criterion explores only critical regions, and therefore, the test scenarios are restricted to a small region. ECC criterion gives a good trade-off between test suite size and concurrent behavior coverage by selecting SC coverage paths and sequentialized paths. Note that sequentialized path consists of thread sequences in the sequential way one after another without interleaving, and it is a subset of synchronized paths. For activity diagrams, we have defined interleaving activity path coverage (IAPC) and total activity path coverage (TAPC) criteria. We distinguish basic paths and interleaving paths for concurrent activity diagrams. IAPC criterion requires touring all the interleavings inside fork-join constructs. TAPC criterion combines all interleaving paths passing through fork-join constructs and non-interleaving basic paths of the activity diagram. In summary, the proposed concurrency coverage criteria ensure exploration of interleaving test scenarios, thereby helping in detection of concurrency errors. Classification of Test Scenarios for Concurrent Behaviors- We have proposed an enhanced state machine based approach to analyze concurrent behavior of test scenarios. The state machine analyzes test scenarios for concurrent behaviors such as serialize, blocking, data-race, and non data-race test scenarios. Analysis is useful for classification of test scenarios. Classification decides the type of concurrency error targeted by test scenarios, like data-race, synchronization, or deadlock. This approach accurately classifies test scenarios, thereby reducing the test suite size for the testing of a specific concurrency error. In conclusion, concurrent behavioral analysis significantly reduces test suite size. 10
1.3. A Brief Summary of Contributions Test Scenario Generation from Non-nested Concurrent Activity DiagramsWe have proposed two approaches. The first approach, called concurrent queue search (CQS), generates interleaving test scenarios for uncovering bugs inside concurrency constructs. In this approach, we first generate queue representation for activity diagrams. Concurrent queues are used to represent multiple threads. A simple heuristic is used to add a thread switch at specific locations amongst threads to generate bugtriggering test scenarios. The second approach, called DFS Level Permutation (DFS LevelPermute), is based on permutations that generate interleaving test scenarios to maximize concurrency coverage. In this approach, we first assign level numbers to activities. We generate sub-sequences using permutations of activities at respective levels. Finally, we combine sub-sequences to generate complete test scenarios. Both CQS and DFS LevelPermute algorithms achieve 100% basic path coverage and partial concurrency coverage for non-nested concurrent activity diagrams. However, these algorithms fail to generate test scenarios for nested concurrent activity diagrams. In summary, CQS algorithm generates test scenarios that may trigger bug and DFS LevelPermute algorithm generates interleaving test scenarios and improve interleaving coverage for non-nested concurrent activity diagrams. Test Scenario Generation from Nested Concurrent Activity Diagrams- We have proposed search-based evolutionary techniques to generate test scenarios from nested concurrent activity diagrams. In this approach, we first translate an activity diagram into an intermediate representation called activity control flow graph. A chromosome is designed appropriately to represent a test scenario. After that, we define a fitness for suitability of chromosomes. Subsequently, an evolutionary algorithm (EA) is run with a random population set. Finally, EA returns the fittest chromosomes after a sufficient number of iterations, when EA achieves convergence or achieves stopping criteria. Experimental results indicate that the evolutionary algorithm requires longer time to achieve convergence. Therefore, we have proposed hybrid evolutionary algorithms to improve convergence. In hybrid evolutionary algorithms, input is pre-populated, where some chromosomes are modified. Standard DFS, standard BFS and Level permutation algorithms are used to generate partial paths, which are seeded in the initial population to generate the required modified population. Our hybrid evolutionary algorithm improves convergence time up to 50% compared to our (non-hybrid) evolutionary algorithm. For concurrent activity diagrams, the hybrid evolutionary 11
1. Introduction algorithm achieves interleaving activity path coverage up to 50% and total activity path coverage up to 54%. In this manner, evolutionary algorithms have generated test scenarios for both non-nested and nested concurrent activity diagrams. Test Scenario Generation from Sequence Diagrams with Concurrency ConstructsSequence diagrams have a rich set of concurrency constructs like par, critical region and general ordering. We have proposed a dependency preserving depth-first search (DP-DFS) algorithm to explore concurrent behaviors in sequence diagrams. In this approach, we first represent a sequence diagram by a rich form of control flow graph called a message control flow graph (MCFG). Then, we apply DP-DFS algorithm to MCFG for generating interleaving concurrent test scenarios. DP-DFS algorithm preserves total order in each thread while interleave messages amongst the threads, thereby generating valid concurrent test scenarios. Finally, we carry out mutation analysis to assess the fault detection effectiveness of test suites. A reduced message control flow graph (RMCFG) representation is suggested to minimize the message interleaving space, thereby minimizing the test suite size. The RMCFG representation achieves 70 to 80% reduction in test suite size and achieves the same coverage as the MCFG representation for ECC and synchronization coverage. DP-DFS technique achieves 100% ECC coverage, 50 to 100% synchronization coverage, and up to 20% IMPC coverage. In a nutshell, the contributions of this thesis are focused on UML transition sequence based test scenario generation for concurrency testing. Proposed new UMLbased coverage criteria for concurrency testing are satisfied by our proposed test scenario generators.
1.4
Thesis Organization
The thesis is organized as follows: Chapter 2 presents the state of the art literature survey of concurrency testing techniques. First, we present a brief survey of code-based concurrency testing. Then, we discuss a survey of UML-based traditional structural testing techniques. After that, we present a survey of UML-based concurrency testing techniques. Finally, we conclude our views on research gaps and need for the proposed work. Chapter 3 discusses existing UML based coverage criteria and presents new proposed concurrent behavioral coverage criteria. First, we discuss concurrent behavior 12
1.4. Thesis Organization representation in sequence and activity diagrams. Then, we discuss existing structural and behavioral coverage criteria for activity diagrams. After that, we present new interleaving based concurrent behavioral coverage criteria for concurrent activity diagrams. We also discuss existing coverage criteria for sequence diagrams. Finally, we present new concurrent behavioral coverage criteria based on message interleaving inside parallel fragments and critical regions of sequence diagrams. Chapter 4 presents a translation of sequence diagrams into activity diagrams and classification of activity diagrams. First, we present rules to translate a sequence diagram into an activity diagram that unify the representations of dynamic behaviors thereby helping test scenario generation. The detailed procedure of translation is discussed for synchronous message, asynchronous message, and combined fragments such as alt, opt, loop, and par. Then, we discuss the classification of activity diagrams based on concurrency and nesting properties. Chapter 5 presents a state machine diagram based technique to analyze test scenarios for concurrent behavior. The proposed state machine diagram analyzes and classifies test scenarios into concurrent behavior, like serialize, blocking, data race, and non data-race test scenarios. Finally, results are presented to demonstrate the effectiveness of the proposed approach compared to the existing approaches for selection of the correct set of test scenarios for a specific type of error. Chapter 6 presents simple heuristic based test scenario generation techniques from activity diagrams. First, we present a heuristic based concurrent queue search (CQS) approach to generate test scenarios that may trigger bug. After that, we present a permutation-based DFS-LevelPermute technique to generate interleaving concurrent test scenarios. Finally, we discuss results for modified DFS, DFS-BFS, CQS and DFS-LevelPermute test scenario generators. Chapter 7 presents evolutionary approaches to generate interleaving test scenarios from concurrent activity diagrams. First, we present evolutionary algorithm technique to generate test scenarios. Subsequently, we present hybrid evolutionary algorithm approach to reduce convergence time of our (non-hybrid) evolutionary algorithm. Finally, we discuss results for evolutionary and hybrid evolutionary algorithms. Chapter 8 presents a graph theoretic approach to explore parallel fragments and critical regions of sequence diagrams for test scenario generation. We first discuss the motivating example elaborating intermediate representation and their effect on interleaving test scenario generation. Then, we discuss the proposed intermediate representation a message control flow graph (MCFG), and detailed procedure to translate 13
1. Introduction a sequence diagram into MCFG. We explain DP-DFS algorithm and its node selection technique. Subsequently, we present a reduced message control flow graph to minimize the size of interleaving space. Finally, we discuss results obtained by the proposed approach. Chapter 9 presents conclusions and directions for future research.
14
Chapter 2 Literature Review In last two decades, UML-based testing has evolved from static testing to dynamic testing as an automated test case generation platform. A widespread research has been focused on test case generation from structural and behavioral models, like class diagrams, activity diagrams, state machine diagrams, and sequence diagrams. Much of the early research in UML-based testing is on sequential systems. Test case generation from UML-designs is governed by the intermediate representation to which diagrams are transformed, test data generation techniques, and test scenario generation techniques. In this chapter, we briefly review different code-based and UML-based approaches, such as specification-based testing, graph-theoretic testing, heuristic testing, model checking techniques, and partial order based techniques. UML-based approaches focus on the goal of test case generation from UML models for functional testing, integration testing, and concurrency (reliability) testing of the software under test (SUT). This chapter is organized as follows. We first present a brief survey of code-based concurrency testing techniques. After that, a brief survey of UML-based sequential testing techniques is presented. Next, we present a survey of UML-based concurrency testing techniques. Finally, we present our opinions on research gaps in the discussion section.
2.1
Code-Based Concurrency Testing
The reality of multi-core hardware is making multi-threaded programs pervasive. However, concurrent programs are prone to bug due to the inherent complexity of 15
2. Literature Review concurrent execution. Concurrency bugs are difficult to detect and diagnose because of their non-deterministic execution. Unlike sequential bugs, concurrency bugs usually require a bug-triggering input data (similar to sequential testing) and a bug-triggering interleaving to expose rarely occurring concurrency bugs. A single execution of a concurrent program P with input X is insufficient to determine the correctness of P with input X. There is a possibility that most executions of P with input X produce correct results, but a few may cause concurrency errors. A common practice to expose concurrency bugs is to run a program with the same input for many times, which is referred as stress testing (Musuvathi and Qadeer, 2007). The rational behind stress testing is that the nondeterministic nature of concurrent programs will help to exercise different interleavings in different runs, and may cause concurrency error. Recently, several approaches (Bron et al., 2005; Edelstein et al., 2002; Musuvathi and Qadeer, 2008, 2007; Sen, 2008; Stoller, 2002) have improved stress testing. These all approaches try to select a small number of interleavings from the exponential-size interleaving space, and make testing focus on these representative interleavings. In this section we brief on model checking, thread scheduling, and reachability based code-based testing techniques.
2.1.1
Model Checking Techniques
Model checking is a technique for systematic searching the behaviors of a system for a specific type of errors (Corbett et al., 2000). First, the system is modeled as a finite state transition system. Each state represents an abstraction of the program’s state, and each transition represents the execution of one or more statements transforming this state. Then, the desired property of the system is expressed in temporal logic (Manna and Pnueli, 1992). The property describes some constraints on the permissible state/event sequences in the finite-state model. After that, a model checking tool algorithmically determines whether all paths through the finite-state transition system satisfy the property. Finally, model checkers display paths through the transition system violating the property, and such paths are interpreted as a faulty behavior of the system and are used to understand the type of error. Some of the most commonly checked properties include freedom from deadlock, simple assertions, state-sequencing properties, and the absence of null pointer dereferences. Model checking approaches used systematic state-space exploration techniques such as VeriSoft by Godefroid (1997), Bandera by Corbett et al. (2000), and Java pathfinder by Havelund and Press16
2.1. Code-Based Concurrency Testing burger (2000). These techniques systematically search the behaviors of a system for specific types of errors. Godefroid (1997) has first presented the VeriSoft tool for systematically exploring the state space of systems composed of several concurrent processes. Author has discussed model checking approach for concurrent programs implemented in C or C++. A partial-order based a new state-space exploration algorithm is presented. The algorithm combines stateless search with the persistent state and sleep-set techniques. Recently, Corbett et al. (2000) have presented a tool set, called Bandera, for model checking of Java programs. The tool automatically extracts finite-state model from Java programs, and also provides error trace interpretation techniques. The component-based model extractor includes major components: slicer, abstraction engine, back-end (intermediate representation constructor for different model checker), and graphical user interface to facilitate interaction with the various components and display counter-examples to the user. A specific modeling language based approach is presented by Havelund and Pressburger (2000) called Java PathFinder (JPF), which translates Java source code into PROMALA modeling language. PROMALA code is used for SPIN model checker, which detects deadlocks and violations for any assertions. Heuristic-based model checking- Model checking is a systematic exploration of the program behavior. However, it is difficult to perform a systematic search on large programs as the number of possible program behaviors grows exponentially with program size (Musuvathi and Qadeer, 2007). A model checker systematically executes each thread schedule, and verifies desired properties in that execution. In such case, a model checker may use some heuristic to check a subset of all thread schedules, which maximize the number of errors found. One such heuristic is limiting the number of preemptions (Musuvathi and Qadeer, 2007). It has many desirable consequences for systematic state-space exploration of multi-threaded programs. First, bounding the number of preemptions does not restrict the ability of the model checking to explore deep in the state space. Second, for a fixed number of preemptions, the total number of executions in a program is polynomial in the number of steps taken by each thread. Musuvathi and Qadeer (2007) have proposed an algorithm for effectively searching the state space of multi-threaded program, called iterative context bounding. Their algorithm gives priority to executions with less number of preemptions. Their approach is implemented in ZING and CHESS two different model checkers. ZING is 17
2. Literature Review an explicit-state model checker for concurrent programs specified in the ZING modeling language. CHESS is a stateless model checker that executes the program directly. CHESS dynamically partitions the set of program variables into data and synchronization variables which help to introduce context switches only at access to synchronization variables. An iterative context-bounding algorithm systematically enumerates program executions in increasing order of preemptions. Recently, Musuvathi and Qadeer (2008) have presented a tool called CHESS for finding and reproducing concurrency bugs. CHESS provides a platform for systematic and deterministic testing of concurrent programs. It uses systematic enumeration technique to force every run of the program along a different thread interleaving. CHESS explores thread schedules, where priority is given to schedules with less number of preemptions. For each bug, CHESS consistently reproduces an erroneous execution manifesting the bug, thereby making it significantly easier to debug the problem. CHESS is designed to be stateless, and it works on three different platforms: Win32, .NET, and Singularity.
2.1.2
Reachability Based Technique
Reachability testing is an approach that combines both nondeterministic and deterministic approaches (Hwang et al., 1994). One way to deal with nondeterministic behavior of concurrent programs during testing is to execute the program with same input many times and hope that faults will be exposed by at least one of the execution (stress testing). This type of uncontrolled testing, called non-deterministic testing, is easy to carry out, but it can be very inefficient. In non-deterministic testing, it is possible that some SYN-sequences of the program are executed many times whereas others are never exercised at all. An alternative approach, called deterministic testing, in which executions are controlled so that user-specified SYN-sequences can be exercised. Deterministic testing approach allows a program to be tested with carefully selected SYN-sequences. General idea of reachability testing is to start with one run of the program and discover all races in this run dynamically as described in the approaches by Hwang et al. (1994); Koppol et al. (2002); Carver and Lei (2004); Lei and Carver (2005, 2006); Souza et al. (2011). After that, these approaches keep exploring new interleavings for new races till all possible partial orders of synchronization events are over. They derive all testing targets on the fly by flipping race outcomes. The limitation is exponential testing complexity. 18
2.1. Code-Based Concurrency Testing Hwang et al. (1994) have presented a reachability testing approach for the concurrent program using Read-Write synchronization. Their approach generates psequences for exhaustive testing of concurrent programs. After that, Lei and Carver (2006) have presented a general model for concurrent programs testing using reachability technique. Their model can be applied to several commonly used synchronization constructs: asynchronous message passing, synchronous message passing, semaphores, and monitors. The approach extended traditional event time-stamping method for race detection. Each sequence and race variant generated during reachability testing is represented by a space-time diagram. Their approach presented a method for reachability testing, which guarantees that every partially ordered SYNsequence will be exercised exactly once. The proposed method works on the fly, i.e., without saving the history of SYN-sequences that have been exercised already. Next, a stateful reachability approach for testing multi-threaded monitors is presented by Carver and Lei (2010). In contrast to the stateless reachability approaches, stateful reachability testing approach uses stored states to reduce execution time significantly. They also presented state pruning technique to avoid storing of all the states of the monitor test program. Recently, Souza et al. (2011) have proposed a reachability approach for concurrent programs using synchronization coverage criteria on parallel control flow graph. The approach uses static analysis of the program under test. It combines both reachability testing and coverage testing to execute synchronization events. The combination of two testing techniques has significantly reduced the testing cost. Structural coverage criteria used to minimize the number of interleaving sequences in the reachability testing.
2.1.3
Thread Scheduling Techniques
Thread scheduling is another technique to alter execution flow, thereby generating a situation to happen concurrency errors. The size of the set of possible interleavings is exponential in the length of the program for concurrent programs. Only a few of interleavings actually produce concurrent faults. Therefore, some heuristic is required to alter the thread execution. Most commonly used technique in thread scheduling techniques are seeding the program with sleep(), yield(), and priority() primitives. ConTest is a tool for detecting synchronization faults in multi-threaded Java programs. It makes random or coverage based decision for executing seeded 19
2. Literature Review primitives. Random scheduling based approach is presented by Stoller (2002) called rstest (random scheduling test) to detect deadlocks and assertion violations in concurrent Java programs. The rstest is very similar approach to ConTest and ConTest-lite frameworks. In rstest, a Java program is instrumented by inserting calls to scheduling functions at selected points in the program. Their approach works with Java bytecode, because it is implemented as byte-code transformation using the Byte Code Engineering Library. The scheduling function either does nothing or causes a context switch. It makes a choice using a pseudo-random number generator or by some heuristics to weight the choice. Another feature is the use of a loop in the scheduling function to increase probabilistic completeness. Edelstein et al. (2002) have proposed a tool for detecting synchronization faults in multi-threaded Java programs, called as ConTest. In their technique, program under test is seeded with a sleep(), yield(), or priority() primitives at shared memory access and synchronization events. At run time, ConTest makes decision to whether the seeded primitive is to be executed or not. A replay algorithm facilitates debugging by saving the order of shared memory accesses and synchronization events. ConTest is used to functional and system testing. Authors have also created a light version of ConTest called ConTest-Lite. ConTest-Lite differs with ConTest in replay capability that increased performance of the tool. After that, Farchi et al. (2003) have presented taxonomy of concurrent bug patterns. The new taxonomy proposed to enhance ConTest’s ability to find concurrency bugs. The taxonomy suggested following categories: code assumed to be protected, interleaving assumed never to occur, and blocking or dead thread pattern. These categories are used to create new timing heuristics for ConTest. At run-time, ConTest is given control before and after a concurrent event is executed. At that time, the heuristics help to control concurrency event invocation. Active thread scheduling testing- Active thread scheduling testing (active testing) uses randomized thread scheduler that verifies warnings reported by a predictive program analysis (Joshi et al., 2009a). Active testing has been explored for concurrency errors like data-race, deadlock, and atomicity-violation in concurrent multithreaded programs. The technique works as follows. Active testing runs the given concurrent program under random schedules for specific potential bug. After that, active testing biases the scheduling by pausing the execution of any thread when the thread reaches a statement involved in potential concurrency bug. After pausing a 20
2.1. Code-Based Concurrency Testing thread, active testing checks the possibility of a real concurrency bug by the paused threads. Thus, active testing attempts to force the program to take a schedule in which the concurrency bug actually occurs. Recently, active testing is studied by Sen (2008); Park and Sen (2008); Joshi et al. (2009a); and Joshi et al. (2009b). A dynamic directed data race testing technique is proposed by Sen (2008) for concurrent multi-threaded programs using active testing. The new technique is called race-directed random testing (RACEFUZZER). RACEFUZZER combines race detection with randomized thread scheduling, thereby finding real data race conditions in a concurrent programs. Furthermore, it gives no false warnings about races because it actually creates a race condition by bringing two racing events temporally next to each other. Moreover, author’s approach determines an exception or error in the program due to the real races caused by RACEFUZZER. Remarkably, RACEFUZZER allows replay mechanism without recording of the events. After that, Park and Sen (2008) have proposed a random dynamic analysis technique to detect atomicity violations in real-world programs. They proposed a technique called ATOMFUZZER that performs a random execution of a multi-threaded program by choosing a random thread to execute at every program state. Their approach actively controls a randomized thread scheduler of a multi-threaded program to create real atomicity violations. Moreover, it allows the user to replay the concrete execution. However, it may not be able to create all atomicity violations that could happen with a given input. Subsequently, Joshi et al. (2009b) have presented a dynamic analysis technique to find real deadlocks in multi-threaded programs. The technique is called DEADLOCKFUZZER, which combines dynamic deadlock detection technique with a random thread scheduler to create real deadlocks with high probability. Their approach works in two stages. In the first stage, DEADLOCKFUZZER uses a simple variant of GoodLock algorithm by Havelund and Pressburger (2000) to find potential deadlocks in a multi-threaded program by observing the execution of the program. In the second stage, DEADLOCKFUZZER controls a random thread scheduler to create the potential deadlock with high probability. Joshi et al. (2009a) have presented an extensible framework for active testing of Java programs. The framework currently implements three active testers based on data races, atomic blocks, and deadlocks. In the code-based testing, a wide varieties of techniques are available. However, the size of the set of possible interleavings is exponentially proportional to the length of the concurrent program. On the contrary, design models express similar concurrency properties of the system at higher level of abstraction thereby reducing interleaving 21
2. Literature Review space. Although, code-based testing techniques are promising, test case generation is in the later phase of the development cycle that may delay the overall schedule. Very often products may have to be released with only structural coverage testing and/or with insufficient concurrency testing. On the contrary, model checking based test case generation is done early in the design cycle whereby state space explored systematically. UML model based testing has dual benefits. It not only provides early test case design and testing effort estimation, but also permits validation of the implementation.
2.2
UML-Based Sequential Testing
The unified modeling language has been widely adopted as a standard language for modeling the design of large software systems. Automated test case generation from UML models gives an edge on testing effort estimation and early test planning in the software development. The graphical nature of UML models enhances the readability and understandability of design, and hence eases the test case generation process. UML structural and behavioral models both contribute to the design of test cases. However, we focus on behavioral models only in this thesis. UML based sequential testing approaches can be broadly classified as specification-based testing, graph-theoretic testing, heuristic testing, and direct UML specification processing approaches, respectively. In subsequent subsections, we brief literature review of the above techniques.
2.2.1
Direct UML Specifications Processing Techniques
Some approaches process UML specifications without generating any intermediate model. These methods motivate direct processing of UML specifications available from visual diagram drawing tools. In recent years, Object Management Group (OMG) has defined a standard for exchanging metadata information via Extensible Markup Language (XML) called XML Metadata Interchange (XMI). Direct processing of UML models has become simpler due to XMI representation of UML models and parsers like DOM and SAX. Early research by Abdurazik and Offutt (2000) presented a model for static analysis and test input generation from collaboration diagram design specifications. The main contribution of their work is static checking rules for collaboration diagrams, 22
2.2. UML-Based Sequential Testing and code instrumentation algorithm for dynamic testing. A tool called SeDiTeC is presented by Fraikin and Leonhardt (2002) to generate test stubs from sequence diagrams for testing Java applications. SeDiTeC extracts testing information from the XMI representation of the UML sequence diagrams. Then, it combines multiple sequence diagrams into a testable sequence diagram. After that, it generates test stubs that validate implementation behavior of a sequence diagram. The tool is used for semi-automatic generation of test cases for integration level testing. Recently, a gray-box method approach is presented by Linzhang et al. (2004) to generate test cases from activity diagrams. Their approach first parses an activity diagram and derives the set of test scenarios using a retrospective DFS method to satisfy the basic path coverage criteria. Then, each test scenario is processed to generate test data using the category partition method. To process concurrent fork-join constructs Chandler et al. (2005) have presented an approach to generate usage scenarios from activity diagrams. Their approach focuses on the use of the XML meta-data of activity diagrams. It collects and stores data for usage scenario generation in a dynamic array. A modified DFS algorithm is proposed to generate usage scenarios. Their approach is suitable to process XML/XMI files. A modified DFS algorithm processes threads one after another inside fork-join construct thereby generating sequentialized test scenarios.
2.2.2
Formal Specification-Based Techniques
Specification-based testing obtains information from specifications of the SUT, rather than from implementation (code). In our discussion, we consider the specifications as the output of an analysis and/or design phase. Specifications are generated from the application’s textual information. These specifications represent a significant opportunity for testing because they precisely describe functional behaviors of the system, which are manipulated by automated means (Abdurazik and Offutt, 2000). Formal specification-based testing is an automated software testing method based on algebraic specifications, it is also called conformance testing. One of the most appealing advantages of algebraic testing is that formal specifications can be used to generate test cases automatically. Starting point for conformance testing is to write specifications in some formal notation followed by an implementation of the SUT. Test cases are derived from the specification and then applied to the implementation. Results obtained after applying test cases to the implementation conclude whether 23
2. Literature Review the implementation conforms to the specifications. If the equations of the specifications are satisfied then SUT implementation conforms to algebraic specifications. Conformance is defined by means of an implementation relation between the model’s implementation and the specification. Formal notation, like Petri Nets, colored Petri Nets, concurrent object-oriented Petri Nets, transition systems, and labeled transition systems are utilized in the literature as formal specifications for UML models. Specification-based testing in early phases presented state machine diagrams (statechart in UML 1.0) based approaches. Latella and Massink (2001) have proposed a formal testing framework for a behavioral subset of UML statechart diagrams (UMLSDs). Their approach presents testing pre-orders and equivalence for UMLSDs. Finally, automatic verification of statechart equivalence is done. Seifert et al. (2003) have described an approach to generate test cases automatically from statecharts. Their approach introduces a compact data structure for representing statechart semantics that allows efficient generation of test cases. A test case is a labeled transition system that checks whether a certain part of the observable behavior of an implementation under test conforms to the specifications described by the statechart. Formal semantics for dynamic behavior of UML models using the temporal OCL extension is presented by Flake and M¨ uller (2003). Their approach performs semantic integration of UML statecharts into OCL language concepts using the formal definition of statechart configurations. Gnesi et al. (2004) have presented a theoretical basis for testing and conformance theories for UML statecharts. An automatic test case generation algorithm is proposed using UML statecharts. A formal conformance testing relation is proposed for input-enabled transition systems having transitions labeled by input/output-pairs (IOLTSs). IOLTSs provide a suitable semantic model for a behavioral subset of UMLSCs. Subsequently, researchers have explored activity and sequence diagram for conformance testing. Eshuis and Wieringa (2004) have presented a tool that supports verification of workflow models specified in UML activity diagrams. First, activity diagrams are translated into transition systems. Then, linear temporal logic is used to specify functional requirements on transition systems. Propositional requirements are checked against the input model by the model checker. If a requirement fails to hold, then an error trace is returned by the model checker. Finally, the errors are highlighted in the activity diagram using graphical tools. For sequence diagrams, Lund and Stølen (2006) have presented an algorithm for deriving tests from sequence diagram specifications. Test cases are derived from formal specifications of sequence 24
2.2. UML-Based Sequential Testing diagrams, which may contain the operators neg and assert. Their approach uses STAIRS semantics to generate test cases based on conformance testing. Massink et al. (2006) have presented a formal approach to testing theory and equivalence, as well as conformance testing and test case generation from UMLSCs. Furthermore, their approach contributes to the investigation on the relationship between notions developed in the area of state-based, object-oriented programming (subtyping/ subclasses), and behavioral relations. Buchs et al. (2006) have presented an automatic test case generation approach to UML models using formal methods. First, UML models are translated into the concurrent object-oriented Petri Nets formal specifications. After that, test intentions for the system specification are expressed using the TestSet language. Finally, their tool generates test cases based on test intentions by exploring the model’s semantics. Recently, Chen et al. (2008) have presented a model checking based test generation approach from an activity diagram. Their approach defines fault models for the specifications that are used to generate directed test cases. NUSMV model checker is used to generate test cases. A formal verification approach using UML-B model is presented by Malik et al. (2010). UML-B is the formal modeling notation, which combines UML with the B-method. UML-B models are translated into Event-B specifications that are proved using theorem provers. Their approach generates test cases using qtronic modeling language from formally proved models.
2.2.3
Graph-Theoretic Techniques
Graph theory is an area of mathematics that helps deriving test cases from design models in many different ways. Graphs are the most widely used structures for abstraction. At a higher level of abstraction, graphs uniquely represent sequence diagrams, state machine diagrams, and activity diagrams. In the graph representation of artifacts, a test case is test data and test path that covers some specific system requirements. Test data satisfy certain constraints on the path while traversing nodes from a special node called the start node. Test case generation in graph-theoretic techniques comprises four steps: a) build a graph model, b) identify the test requirements, c) select test paths to cover the requirements, and d) derive test data. Graph-theoretic algorithms are well studied, and they allow easy manipulation of graphs. Though the UML models are graphical visual representations of the system specifications, they are not directly useful to derive test cases. Therefore, researchers 25
2. Literature Review initiated the use of formal representation of these informal artifacts such as trees and graphs for test case generation. In this section, we first review tree-based test case generation techniques after that graph-based test case generation techniques. Tree-based methods- A tree is a connected acyclic simple graph. An alternate sequence of vertices and edges beginning at the root and ending in a leaf node is of interest for a test scenario representation. Each such sequence is a potential test case. Test data is generated based on constraints along the test scenario. Graph-theoretic tree-based test case generation approaches are summarized in Table 2.1. Table 2.1: Studies in tree-based test case generation Author(s)
Briand et al. (2005) Li et (2007)
al.
Fan et al. (2009) Kansomkeat et al. (2010)
Input Model
Method
Intermediate model
Coverage Criteria
Statechart
Normalization and analysis of operation contracts and transition guards Traversal, function minimization
Invocation sequence tree (IST)
Round-trip path, full predicate
Scenario tree
Message, condition
Activity diagram
Bottom-up testing strategy
Activity diagram composition tree
Transition, activity
Activity diagram
Condition classification tree method, mutation analysis
Conditionclassification trees
Method
Sequence diagram
path,
An automated tool is presented by Briand et al. (2005) for deriving test requirements from statecharts. Their approach defines the invocation sequence tree to capture all possible sequences of actions triggered by a transition test sequence. Then, it performs normalization of OCL expressions and derives test constraints on the system state and event/action arguments. Normalization is based on the analysis of invocation conditions and operation contracts for events and actions. For sequence diagrams, Li et al. (2007) have presented an approach to generate test cases that augments OCL information with sequence diagrams to generate test data. First, the scenario tree is constructed from the sequence diagram, and then scenario paths are generated from the scenario trees. After that, for each path conditional predicates and OCL pre and post condition expressions are selected to generate test data. Test data is generated using a function minimization technique. Fan et al. (2009) have presented an approach to generate integrated test cases of the whole system through the analysis of sub-activity diagrams. Their approach generates test cases based on 26
2.2. UML-Based Sequential Testing activity diagrams’ composition trees using bottom-up testing strategy. A conditionclassification tree method is proposed by Kansomkeat et al. (2010) for generating test cases from activity diagrams. First, activity diagrams are analyzed to gather control flow information based on decision constructs. Then, guard conditions are used to generate condition-classification trees, which are used to build test case tables. Finally, test tables are translated into test cases. Graph-based methods- A graph is an ordered pair G = (V, E) consists of a set V of vertices (nodes) and a set E of edges. In the directed graph G, an edge e = (x, y) is considered to be directed from x to y. A path in the graph is a sequence of vertices such that, from each of its vertex, there is an edge to the next vertex in the sequence. From the testing perspective, two tasks are important: the first task is searching paths in a graph representation of the UML model, and the second task is solving constraints along the path to generate test data values. Table 2.2 presents different graph-based test case generation approaches using UML transition sequences. Table 2.2: Studies in graph-based test case generation Author(s)
Pilskalns (2003)
et
al.
Kansomkeat and Rivepiboon (2003) Dinh-Trong (2006)
et
al.
Cartaxo et al. (2007) Sarma et al. (2007) Lei and Lin (2008)
Zhou et al. (2008) Kundu and Samanta (2009) Nayak and Samanta (2010) Nayak and Samanta (2011)
Input Model
Method
Intermediate model
Coverage Criteria
Sequence diagram
DFS or BFS traversal
Object method directed acyclic graph
Statecharts
Parsing TFG, mutation analysis
Testing (TFG)
All message paths, full predicate State and transition
Sequence diagram
Symbolic execution, constraint solver
Variable graph
Sequence diagram Sequence diagram Sequence diagram
DFS traversal
Labeled transition systems Sequence diagram graph Message flow graph
Sequence diagram Activity diagram
DFS traversal, category partition DFS and BFS traversal
Polymorphic object method acyclic graph Activity graph
Sequence diagram
DFS traversal, symbolic execution
Structured graph
Activity diagram
DFS traversal
Intermediate testable model
BFS traversal DFS traversal
flow
graph
assignment
control
All message paths, condition All paths Message sequence path All nodes, edges, and paths Polymorphic method, edge Activity path All paths Selection, loop, concurrent
Initially, Pilskalns et al. (2003) have presented an integrated testing approach, 27
2. Literature Review which is based on combining structural and behavioral UML design representations of the system. Their approach transforms the behavioral information encapsulated in sequence diagrams into an object method directed acyclic graph (OMDAG). To generate test cases, test paths are generated by traversing OMDAG using DFS or BFS traversal algorithm. After that, appropriate attribute and parameter values for object instantiations are selected that would cause the desired test sequence to execute. Finally, all tests are defined in the object method execution table. A symbolic execution based approach is presented by Dinh-Trong et al. (2006) to generate test data. Test case information is collected from sequence and class diagrams in a directed graph called variable assignment graph. Test paths are generated using a traversing algorithm, and then for each path, a constraint solver generates test data. Recently, Kundu and Samanta (2009) have proposed an approach for generating test cases from activity diagrams, where an activity diagram is translated into an activity graph. Test cases are generated using a DFS-BFS algorithm. The generated test cases are based on an activity path coverage criterion, which is used to cover synchronization and loop faults. Nayak and Samanta (2010) have presented an approach to synthesize test data from UML models. A structured composite graph is constructed from sequence diagrams, class diagram and OCL constraints to generate test scenarios and test data. A synthesis method is presented by Nayak and Samanta (2011) for test scenarios generation from activity diagrams for system level testing. First, an activity diagram is transformed into an intermediate testable model (ITM), which maintains a simple control flow. Paths in an ITM are mapped onto test scenarios.
2.2.4
Meta-heuristic Techniques
Search-based software engineering transforms software engineering problems into optimization problems, which can be further solved by applying meta-heuristics. Several meta-heuristic techniques are available that can be utilized for test case generation, like genetic algorithms, hill climbing, alternating variable method, simulated annealing, genetic programming, particle swarm optimization, ant colony optimization, and tabu search. In UML transition sequences, the most commonly used meta-heuristic techniques are genetic algorithm (GA) and ant colony optimization. The application of meta-heuristic search techniques for testing is promising because exhaustive testing is infeasible considering the size and complexity of SUT. For UML transition 28
2.2. UML-Based Sequential Testing sequences, heuristic-based test case generation approaches are tabulated in Table 2.3. In this sub-section, we first discuss the genetic algorithm based approaches, and then ant colony optimization techniques. Table 2.3: Studies in heuristic-based test case generation Author(s)
Input Model
Method
Intermediate model
Coverage Criteria
Statechart
Ant colony optimization
–
All states
Li and (2005a)
Lam
Li and (2005b)
Lam
Activity diagram
Ant trace explore thin-thread tree
–
All edges
Xu et al. (2005)
Activity diagram
Adaptive agent
Scenario tree
State machine
Genetic algorithm
–
Simple and concurrent path Transition coverage
State diagram
Genetic algorithm
–
Transition coverage
Activity diagram
Random walk, exploratory search algorithm
Colored petri-net
Sequential and concurrency
Doungsa-ard et al. (2007) Lefticaru and Ipate (2008) Farooq et al. (2008)
activity
Genetic algorithm- A genetic algorithm mimics the process of natural evolution. It uses techniques inspired from biology, such as selection, recombination (crossover) and mutation. This heuristic is commonly used to generate useful solutions for optimization and search problems. A genetic algorithm evolves a population of candidate solutions, and a fitness function guides this search towards achieving coverage criteria. For state machine diagrams, Doungsa-ard et al. (2007) have proposed a framework for generating test data using a genetic algorithm. A sequence of triggers is taken as the test data, which is used to fire transitions in the state machine diagram. A sequence of triggers is represented by a chromosome. The fitness function is formulated to achieve transition coverage. Lefticaru and Ipate (2008) have presented an approach to generate test data using GAs from state diagrams. A genetic algorithm generates input data for a chosen path of the state machine. These input data values satisfy the corresponding guards on a test path and trigger the desired transitions. Their approach is empirically evaluated using three search techniques: simulated annealing, genetic algorithms and particle swarm optimization. 29
2. Literature Review Ant colony optimization- The ant colony optimization algorithm is a probabilistic technique for solving computational problems; it can be applied to find test paths through graphs. Ants initially wander randomly, and upon finding food they return to their colony. In this process, ants lay down pheromone trails. If other ants find such a trail, they are likely to follow the trail and reinforce it. A similar principle is used by Li and Lam (2005a), Li and Lam (2005b) and Xu et al. (2005) to find test paths in the graph representation of UML designs. An ant colony optimization technique is used for statechart and activity diagrams.
First, Li and Lam (2005a) have presented an ant colony optimization approach to generate test sequences form statechart diagrams. A statechart diagram is translated into a directed graph, and then a group of ants cooperatively explore the directed graph to generate test sequences. After that, Li and Lam (2005b) have proposed an ant colony optimization based approach to generate test threads (scenarios) from activity diagrams. Their approach uses ant-like agents to generate test threads. First, a group of ant-like agents is sent cooperatively to build test thread trees from a directed graph of the activity diagram. Then, ants frequently report their traces to test thread tree while exploring the graph. A redundant exploration of the test threads is avoided by using anti-ant-like ants. An adaptive agent-based approach is presented by Xu et al. (2005) to generate test scenarios from activity diagrams. Adaptive agents move around in a directed activity graph to search for nutrients. As these agents move, they also lay down a slime trail to mark their paths. A scenario tree is generated by using adaptive agents from an activity diagram. A path of the scenario tree from the root node to a leaf node is taken as a test scenario. Their approach emphasizes on test scenario generation from complex fork-join structures using agent cloning strategy. For activity diagrams, Farooq et al. (2008) have presented a test case generation approach by using exploratory testing technique. In their approach, an activity diagram is transformed into a colored Petri-net model. Test sequences are generated using a random walk method, which is based on the theory of probability. The generated test sequence suite is optimized for sequential as well as concurrent coverage criteria. 30
2.3. UML-Based Concurrency Testing
2.3
UML-Based Concurrency Testing
UML models are one of the sources of concurrency information at the design level in recent trend towards the model driven development. Designing of test cases to uncover concurrency errors is possible using UML design models because of the availability of concurrency constructs in behavioral models like activity diagrams, sequence diagrams, and state machine diagrams. Benefit of assessing the concurrency issues at a higher level of abstraction is generating test cases earlier in the software development cycle for concurrency testing. UML based concurrency testing approaches can be broadly classified as search based techniques, model checking techniques, and graph-theoretic techniques. We brief each the techniques and their research review in subsequent subsections.
2.3.1
Graph-Theoretic Techniques
Graph-theoretic techniques transform behavioral models into formal graphs as an intermediate representation. Test scenarios are generated using traversing algorithm to satisfy the specified coverage criteria. Earlier attempts by Sun (2008) presented a transformation based approach to generate test scenarios for testing concurrent applications. Their approach transforms an activity diagram into a set of trees, which is an intermediate model. A set of trees is normalized into extended binary tree. Test scenarios are generated using depth-first binary tree traversal technique that satisfy one of the three coverage criteria: weak concurrency coverage - the test scenario set is a one feasible sequence of concurrent activities without considering the interleaving of activities inside fork-join, moderate concurrency coverage - the test scenario set is a collection of all feasible sequences without considering interleaving of activities inside fork-join, and strong concurrency coverage - the test suite is a set of all feasible sequences of activities inside fork-join. However, permissible interleaving is moderately low. A simple way to identify data races is presented by Lei et al. (2008) based on state model that represent concurrent behavior. Basically their method is a postmortem approach to detect data-races in concurrent Java programs. Test scenarios are generated from extended activity diagrams with a retrospective DFS algorithm. For each scenario satisfying basic path coverage, test data is generated using a random method. Then instrumented Java programs are run using the random data to generate method call traces. The 31
2. Literature Review execution traces are used for data race analysis using the state transition technique shown in Figure 2.1. Data races are identified by searching the time overlaps of entering and exiting critical sections of different threads. To test data race, the shared data is expressed using read/write operation state transition behavior. When two concurrent threads access shared data and one access is write then participating threads execution is a potential data race execution. Figure 2.1 shows read/write behavior of shared data using state machine model. Most transitions in the figure are
Idle
Read / Out Read / In
Read / Out Write / In
Write
Read Write / Out
Read / In
Write / In
Figure 2.1: A state transition model for read/write behavior of a shared data for data race analysis (Lei et al., 2008)
W rite/In
W rite/In
safe and do not cause data race. However, the Read −−−−−→ Write, Write −−−−−→ Read/In Write, and Write −−−−−→ Write transitions cause potential data races. Recently, Kundu and Samanta (2009) have presented a test cases generation approach from activity diagrams at the use-case scope. First, an activity diagram is transformed into an intermediate format called an activity graph. Then test cases are generated using DFS-BFS algorithm from an activity graph. DFS algorithm is used for non-concurrent parts of the activity graph, and the BFS algorithm is used for concurrent parts of the activity graph inside fork-join to generate test cases. These generated test cases satisfy activity path coverage criterion. The proposed approach detects faults like synchronization and loop. In the case of nested activity diagrams, constructs like loop, decision, and fork-join inside any thread (fork flow) are replaced with higher abstraction level (i.e., a single node). Therefore, their approach suppresses interleavings inside nested fork-join concurrent structures. A synthesis based approach is presented by Nayak and Samanta (2010) for test data generation. First, 32
2.3. UML-Based Concurrency Testing a sequence diagram is transformed into a structured composite graph (SCG). Their approach includes fork node and join node to represent concurrency in the SCG for parallel fragment’s start and end in the sequence diagram, respectively. Furthermore, a sequence of messages within one operational fragment (operand) is represented as a block node in the SCG. Then, test scenarios are generated using is a DFS-like algorithm. However, the test scenario generation algorithm does not consider message interleaving, and test scenarios generated from their approach are not valid interleaving concurrent test scenarios. Khandai et al. (2011) have presented a concurrent composite graph (CCG) based test cases generation from sequence diagrams. Their approach uses intermediate model similar to Nayak and Samanta (2010) that includes fork and join nodes for parallel fragments to represent concurrency in CCG. Moreover, a sequence of messages within one operational fragment is represented as a block node in the CCG. DFS-BFS algorithm is used to generate test scenarios. Although, their approach generates valid interleaving concurrent test scenarios, it has not taken care of the join nodes, which require waiting for completion of all concurrent tasks in all threads before processing join and subsequent nodes. However, the test scenario generation algorithm does not generate interleaved concurrent test scenarios. Therefore, above approaches show limitation to detect concurrency errors due to insufficient interleaving of tasks.
2.3.2
Meta-Heuristic Techniques
Current trends in the field of search based software engineering are wide. They cover almost all the aspects of the development life cycle: requirement engineering, planning, testing, maintenance, and quality assessment Harman (2007). Shousha et al. (2008, 2009, 2012) have presented a genetic algorithm based approaches to detect concurrency problems such as data-race, starvation, and deadlocks. Their approaches are based on specialized UML design models expressed in SPT and MARTE profiles. Their approaches first collect all information relevant to testing form the system’s UML models. Then, extracted information is fed to a genetic algorithm. Execution of GA searches for interleaving sequences those exhibiting concurrency errors. Finally, when search finds the specific error, it is reported, and search ends. The details of the genetic algorithm, like gene encoding, fitness function, and concurrency error targeted are listed in Table 2.4. The solutions are applicable to the specialized embedded system design models, which comply with MARTE/SPT profile. Test case 33
2. Literature Review Table 2.4: Gene encoding and fitness function for search based concurrency testing approaches Author(s)
Shousha et al. (2008)
Shousha et al. (2009)
Gene Encoding (T,L,A)
Fitness Function
Concurrency Error
( #LE + T W, f (c) = #LE + T W + LC,
(T,A)
Data-race ( f (c) = minet i=st
Shousha et al. (2012)
Deadlock if T W < 2 if T W ≥ 2
|Wi − N (Wi ), et,
if #W ≥ 1 if #W = 0
(T,L,A)
Starvation
f (c) =
et X
1, i,
i=st
0,
if i = 0 and A ∈ / ETi and A ∈ W Ti if A ∈ / ETi and A ∈ W Ti otherwise
T - Thread ID, L - lock, A - access time, LC - summation of all lock capacities LE - total number of threads executing within all locks TW - total number of threads waiting on any lock Wi - time unit i during which a writer thread accesses the shared resource st - start time, et - end time N (Wi ) - time unit i of nearest executing thread to wi within the resources ETi - the set of threads executing within the target lock at time unit i, A - target thread W Ti - the set of threads waiting for access to the target lock at time unit i
generation problem is modeled in terms of threads, locks and access time using a genetic algorithm to search thread execution interleavings, which cause starvation or deadlocks. A deadlock is ensured by identifying cycle in resource allocation graph constructed from the chromosome. However, their approach needs prior information of execution time of each function. Therefore, it is not applicable to non-embedded applications.
2.3.3
Model Checking Techniques
Model checking has been used predominantly to address design based non-functional verification of concurrent applications. Model checking uses system models to detect automatically whether a given system meets its specification in terms of safety, concurrency, and other properties. Model checking properties are normally expressed in the form of temporal logic. For state machine diagrams, Kim et al. (2005) presented an approach to generate 34
2.3. UML-Based Concurrency Testing test sequences that cover all various possibilities of events and states using SAL model checker. Their approach first built Java class model of Object and T hread classes. Then, a separate composite state model is defined based on existing UML models, which describe shared object locking mechanism and thread blocking mechanism. For Java programs, basic business state model is designed, and then it is extended with Java concurrency features. After that, SAL model checker systematically generates test sequences from extended state model. Next, SAL model is combined with additional model used by concurrency analyzer (ConAn). This final model is used to generate counter-examples. Gagnon et al. (2008) presented a framework supporting a formal verification of concurrent UML models using Maude language. Concurrency aspects in class, state and communication diagrams are translated into Maude formal specifications. During the translation process of UML diagrams, three Maude modules: functional modules, system modules and object-oriented modules are generated. A functional module is associated with a state diagram. An object-oriented module is associated with each class in the class diagram. For internal concurrency, the class is modeled with more state attributes one for each of the orthogonal regions of its concurrent composite state. A specification integrates both static and dynamic features of the system. Maude allows simulation of each module by selecting a personalized initial configuration. Authors have defined some LTL properties about example system (producerconsumer problem). Verification of proposed properties is done using Maude’s integrated model checker. Partial-order methods have been developed as a means to avoid the state-explosion problem in the verification of concurrent programs. Seo et al. (2006) presented an approach to specification-based testing of concurrent programs. Their approach generates representative test sequences from statecharts based on the notion of a synchronization reduction. In order to generate representative interleavings, the proposed approach uses a state-space exploration technique with PMSs (possible macro steps in statecharts) based on the partial-order method. Stubborn sets technique is used to generate representative sequences. For each representative equivalent class, an automaton is constructed, which accepts all interleavings represented by an equivalent class. Thus, at least one interleaving for each equivalent class determine correct implementation of the specification. In their approach, a number of test executions are reduced from all possible sequences to the number of all automata. 35
2. Literature Review
2.4
Discussion and Summary
As a semi-formal modeling language, UML is widely used to describe analysis and design specifications by both academia and industry; thus, UML models have become prime sources for test case generation. Techniques proposed by researchers can be automated easily for innovative testing. They reduce the tedious manual procedure of deriving paths to comply with coverage criteria. This chapter reviews various techniques and algorithms that are used to address test case generation from UML transition sequences. Formal test case generation techniques used for conformance testing are explored with labeled transition systems, finite state machines, and Petri-nets. These formal notations have formal semantics to apply verification rules. However, state machine diagrams, sequence diagrams, and activity diagrams have semi-formal semantics. In UML 2.x, different new constructs are defined, not all of them are addressed using formal testing techniques. The semantics of these new constructs may be explored using formal methods for testing. Graph-theoretic algorithms have been extensively explored by researchers. These techniques transform UML graphical models into graph or tree abstractions. Different graph traversal techniques are used to generate test scenarios like DFS, BFS, and modified versions of DFS or BFS search algorithms. UML models are abstracted in many different types of directed graphs. For example, sequence diagrams are abstracted as a variable assignment graph, a scenario tree, a sequence diagram graph, a message dependency graph, or a structured control graph. State machine diagrams are abstracted as a specification graph, an invocation sequence tree or a testing flow graph. Activity diagrams are abstracted as a binary extended tree, an activity diagram composition tree, an activity graph, a flow dependency graph or a condition classification tree. This clearly indicates the use of directed graphs in a test case generation is significantly high. Meta-heuristic algorithms present a different approach of test case generation. Genetic algorithms are well proven techniques to generate test data (Sofokleous and Andreou, 2008). Genetic algorithms outperform the random search in terms of structural search (Ali et al., 2010). Ant colony optimization is a simple and elegant algorithm for test scenario generation. Meta-heuristic techniques are more easily deployable in the model based testing for more robust test cases generation. To generate test data, researchers have utilized techniques such as symbolic execu36
2.4. Discussion and Summary tions, category partitions, condition classification trees and evolutionary algorithms. Random testing has been taken as a comparison basis for most of the test data generation techniques. The presented techniques outperform random testing. Briand et al. (2004) have presented an empirical investigation of the cost-effectiveness of round trip strategies and category partitions. Their approach gives insights for using the category partition method with state based testing. Mouchawrab et al. (2011) have presented thorough and detailed assessment, comparison, and combination of structural and state testing. An experiment case study by Abdurazik and Offutt (2004) demonstrated the fault raveling capability of behavioral models. With growing research in UML transition based testing approaches, more experimental evidence is required to assess the effectiveness of model based testing. Nowadays, concurrent programs are widely used in the industry. Concurrent programs may be error-prone and non-deterministic if synchronization is not well taken care of by the developers. The UML models support concurrency design. Test cases generation for concurrency needs to focus on satisfying the given concurrency coverage criteria. UML provides concurrent execution behavior in activity diagrams using f ork-join constructs and in sequence diagrams using asynchronous message, par, and critical constructs. UML activity diagrams are widely used to model concurrent interactions among the multiple objects. Even though, there are a few approaches, which generates test cases for concurrency errors using UML transition sequences. In spite of several sequence diagram based test case generation techniques, interleaving of messages inside parallel fragments and synchronization behavior inside critical regions have not been studied in previous UML-based approaches. Concurrency test cases design remains an open issue for UML transition sequences. A test coverage criterion is a crucial factor in validating and analyzing the test adequacy of a test suite. The effectiveness of the coverage criterion is a measure of how likely that the test case satisfies criterion detects the presence of a fault. In the graph-theoretic testing, test scenario is a test path in the graph. Accordingly, graph based coverage criterion evaluates a test suite of the model under consideration. There are coverage criteria such as node, edge, simple path and complete path for a graph, and these criteria are used for the respective artifacts of the model. For example, a state machine diagram has state coverage, transition coverage, and roundtrip path coverage criteria. An activity diagram has activity coverage, transition coverage, simple-path coverage, and concurrent-path coverage criteria. A sequence diagram has message coverage, message-sequence path coverage, and all path coverage 37
2. Literature Review criteria. However, there is a lack of coverage criteria for concurrency testing using UML transition sequences. As a result, coverage criteria for concurrent systems need more attention in UML based testing. In summary, graph-theoretic approaches are extensively explored by researchers as compared to other approaches. Although graph-theoretic approaches generate test cases based on structural element and sequential behavioral coverage criteria, concurrency interleaving exploration and concurrency coverage criteria are explored limitedly by graph-theoretic approaches. Existing approaches have not fully explored UML concurrency constructs to design test cases. Therefore, applicability and scalability of existing UML-based graph-theoretic testing approaches to concurrency testing are limited. A few approaches addressed concurrent test scenario generation from UML models, but they fail to explore sufficient interleaving space because they comply with the structural element and sequential behavioral coverage rather than concurrent behavioral coverage. After a comprehensive literature survey of graph-theoretic approaches, we found following research gaps for UML-based concurrency testing. First, limited use of concurrency constructs to design concurrency test scenarios. Second, lack of concurrent behavioral coverage criteria. Third, need of intermediate representation and test scenario generation algorithm, which explore sufficient interleaving space to detect concurrency errors. These research gaps in UML based graph-theoretic approaches are instrumental to set our main objectives of the research for this thesis work. In subsequent chapters, we provide solutions to the above problems.
38
Chapter 3 Concurrency Coverage Criteria Software testing is an integral and important part of the software development process. Software testing is used for validation, reliability estimation and quality assurance of SUT (Harrold, 2000). Edward Dijkstra correctly said, “Testing can show the presence not the absence of errors.” Hence, one can say testing is successful if test case uncovers bugs. Testing is often incomplete for concurrent systems because testing cannot cover all possible system behaviors due to non-deterministic execution of concurrent programs. A test suite for concurrent system will be exponential if one tries to test/ explore every possible concurrent behavior. An effective test suite should have minimum test cases and should uncover maximum bugs. There are several heuristic means to measure the quality of test suites like fault detection, mutation analysis, or coverage criteria. Among these means, coverage criteria are commonly used and popular heuristic means to measure the quality of test suites (Weißleder, 2010). A coverage criterion sets a limit on the number of minimum test cases out of the infinite number of possible test cases to measure the quality of software. There are several kinds of coverage criteria that focus on data flow (Ammann and Offutt, 2008), control flow (Beizer, 1990), and transition sequences (Rountev et al., 2005), (Ferreira et al., 2010). In testing, achieving more coverage is a way to explore more problem space, so that, one can come closer to proving the absence of faults. As explained in motivating examples in Chapter 1, sometimes achieving 100% coverage may be confusing if required behavior is not completely assessed by the selected coverage metrics. Therefore, choosing the appropriate coverage metric is important to assess the specific behavior and thereby uncovering possible errors. Coverage requirements are specified using artifacts that represent some elements or 39
3. Concurrency Coverage Criteria behaviors of system. Coverage criteria are defined on any artifact in the software development such as specification, design, or code. One may design test cases as early as possible in the development process based on specifications or documents like UML designs. UML provides a variety of diagrams that can be used to present different views at different stages of the development life-cycle. UML-based testing techniques derive test requirements and coverage criteria from UML diagrams. This chapter presents UML-based concurrent behavioral coverage criteria. First, basic preliminaries of concurrent design in sequence and activity diagrams are discussed. Then, the notion of the coverage criteria is presented. After that, existing UML-based coverage criteria for activity diagrams are discussed, and their limitations towards concurrent behavior are illustrated. To overcome said limitation, concurrent behavioral coverage criteria for activity diagrams are presented. Next, existing UMLbased coverage criteria for sequence diagrams are discussed, and their limitations towards concurrent behavior are illustrated. Subsequently, concurrent behavioral coverage criteria for sequence diagrams are presented. Finally, conclusions are presented.
3.1
Background
Dynamic behaviors of systems are significant characteristics to design test cases (Micskei and Waeselynck, 2011). Sequence diagrams, state machine diagrams and activity diagrams represent dynamic behaviors of systems. These behavioral diagrams represent sequential and concurrent behaviors of systems. In this section, first an overview of sequence diagrams and concurrency modeling in sequence diagrams is described. After that, an overview of activity diagrams and concurrency modeling in activity diagrams is discussed. Finally, the notion of coverage criterion is presented.
3.1.1
Sequence Diagram
A sequence diagram is an interaction diagram. It represents the sequence of messages and control structures amongst the objects. Detailed semantics of sequence diagrams is described in the UML superstructure specifications (OMG, 2011). A sequence diagram consists of a group of objects and messages. Objects are represented by lifelines and messages are represented by directed lines amongst the objects. Messages show an association amongst the objects to complete the system functionality, and 40
3.1. Background they are exchanged in a natural order sequentially from the top to the bottom in sequence diagrams. For a given scenario, a sequence diagram is a representation of the successful and unsuccessful events ordering among the objects. Sequence diagrams are used to illustrate a possible scenario of a use-case, the execution of an operation, or simply an interaction scenario between objects of system. For each use-case, a sequence diagram can be drawn to understand the dynamic behavior of the system. In UML 2.0, a new notion is introduced to handle conditional, iterative and concurrent control behaviors for sequence diagrams. This new notion is called combined fragment, which is a logical grouping represented by a rectangle (Fowler, 2004). A combined fragment consists of conditional structures that affect the flow of messages. UML 2.0 introduced twelve kinds of combined fragments. A combined fragment contains interaction operands, and it is defined by the interaction operator. Interaction operator types are alt, opt, break, par, seq, strict, neg, assert, critical, loop, consider, and ignore. The operand can be a sub-sequence diagram, which constitutes the body of the combined fragment. Combined fragments can have one or more operands depending upon its type. Each operand has a guard, which is a boolean expression. A guard needs to be evaluated to true in order to execute the message sequence within the operand. Combined fragments may contain nested combined fragments that represent complex structures. One of the major advantages of sequence diagrams over communication diagrams is that one can specify sequential scenarios as well as specify parallel scenarios using asynchronous message and par combined fragments. The par and critical combined fragments allow modeling of concurrency in sequence diagrams (Micskei and Waeselynck, 2011). Concurrency in the actual system can be implemented in terms of multi-threaded programs. Consider the example of a money transfer bank transaction process illustrated using a sequence diagram representing concurrent behavior as shown in Figure 1.2. A sequence diagram is enclosed in a frame, which includes a pentagon in the upper left-hand corner with keyword sd followed by a label M oneyT ransf er identifying the sequence diagram. There are five objects participating in money transfer functionality. A bank object is the main thread of the application that creates two additional threads thread1 and thread2 represented by the parallel fragments Par: transfer. The thread1 is the first operand shown above the dotted line, and thread2 is the second operand shown below the dotted line in the parallel fragment. These two threads handle the money transfer between two accounts account1 and account2 . Money transfer operations are enclosed in the parallel combined fragment that represents 41
3. Concurrency Coverage Criteria concurrent execution of the messages/ events in fragments. There are three control flows in the figure. The first flow of control is associated with the bank object. The second and third flows of control, which are introduced by the first control flow, are associated with thread1 and thread2 objects, respectively. These three control flows are concurrent and create partial ordering of messages/ events to fulfill the complete functionality. The thread1 and thread2 access objects account1 and account2 simultaneously. Critical regions are included to protect the shared data of account objects in each thread, Critical: critical1 in thread1 and Critical: critical2 in thread2 . In case of improper implementation of critical sections, concurrent access of account1 and account2 may lead to concurrency errors in some of the interleavings executed by system. Therefore, it is interesting to find these interleaving sequences, and test the application for concurrency errors.
3.1.2
Activity Diagram
The basic idea of an activity diagram is to model activities and their possible order of execution. An activity diagram represents sequential, decisional, iterative, and concurrent flows of execution (Fowler, 2004). A sequential flow of activities results in a single flow of activities one after another. A decisional flow of activities results in more than one flows of activities based on result of the guard condition, however only one flow of activities is executed sequentially. An iterative flow of activities may result in zero or more number of flows of activities, which are executed sequentially based on the result of the iteration condition. A decision flow and an iterative flow is shown by a diamond symbol in activity diagrams, they have also corresponding merge diamond symbol or can use same decision diamond symbol as a merge symbol in some cases. A concurrent flow of activities start multiple flows of activities at a time, and all flows start simultaneously. A concurrent flow of activities is shown by the pair of two thick parallel lines. Activities in different flows may interleave in the concurrent flow. Execution of activities in sequential, decisional, and iterative flows is deterministic, whereas execution of activities in concurrent flow is non-deterministic. Concurrent execution behavior is represented using fork-join constructs in activity diagrams (Pilone and Pitman, 2005). The start and end of the concurrent threads are represented by fork and join constructs. A fork node divides an incoming control flow into multiple (concurrent) outgoing control flows. A join node combines multiple (concurrent) incoming flows into an outgoing flow. In addition, activity diagrams rep42
3.1. Background
Receive Signal ‘a’
Send signal ‘a’
Figure 3.1: Communication of activities inside concurrent flow using Send signal and Receive signal constructs
resent communication of activities inside concurrent execution using signals. Activity diagrams support signaling using send and receive activities as shown in Figure 3.1 . A send signal is shown by a convex pentagon that looks like a rectangle with a triangular point on one side. A send indicates that a signal or a message of some kind is sent to other activity or process. The name of the action indicates what kind of message it sends. Control is immediately passed to the next action in the control flow after send action. An receive signal is used to indicate that current activity (receive signal activity) waits for some external event or incoming message. The name of the action indicates the type of event that it waits. An receive signal is shown by a concave pentagon that looks like a rectangle with a triangular notch inside on one side. Consider an automatic teller machine money withdrawal functionality (Jorgensen, 2008) shown in Figure 3.2. The ATM system communicates with a bank for completing financial transactions. In this example, concurrent execution is shown by fork construct F 1 and join construct J1 thick parallel lines. System generates one more thread at fork, and then two threads start working simultaneously. The first thread completes ‘Debit amount’ and ‘Dispense cash’ activities. The second thread completes ‘Prepare printer’ activity. These two threads run parallel, and execution flow may have any permutation of activities to complete the task. If activities in both the threads are independent then all execution interleaving sequences do not cause 43
3. Concurrency Coverage Criteria a2
a1
S
Insert card
t1
t2
[invalid user] t4
Enter PIN
t3 D1 Authorize user t5 [valid user] a3
Select account type t6
Enter amount [amount > balance]
t8 Show error
a5
t9
a4
t7 D2 [amount , s2 =< a1 −b1 −a2 −b2 >, s3 =< b1 −a1 −b2 −a2 >, and s4 =< b1 −a1 −a2 −b2 > represent all same elements from threads T1 and T2 , and have partial order a1 a2 and b1 b2, where relation “ ” represents a partial order, then all paths s1 − s4 have same partial order. A selected any single representative path, say s1 , is a simple path for threads T1 and T2 . The value of simple path coverage is the ratio of the traversed simple paths to all simple paths in the activity diagram. The definition of simple path criterion presented in their approach is as follows: Definition 3.6. Simple Path Criterion: It requires that all simple paths in the activity diagram be covered. For any t ∈ T S we can get program execution trace ‘pet’. If simple path is correspondence with a ‘pet’ is not traversed in the activity diagram, we record the test case ‘t’ and put a traversed mark on the simple path (Mingsong et al., 2006). Example- Consider the example of a bank automated teller machine (ATM) shown in Figure 3.2. A path is represented as a sequence of nodes or a sequence of edges in the graph. Similarly, paths representing behaviors of activity diagrams can be represented as a sequence of activities, a sequence of transition or combination of both. For example, a basic path representing invalid user in Figure 3.2 can be represented as: i) a sequence of activities: S − a1 − a2 − D1 − M1 − a10 − E, ii) a sequence of transitions: t1 − t2 − t3 − t4 − t18 − t19 , and iii) a sequence of activities t1 t2 t3 t4 t18 t19 and transitions: S − → a1 − → a2 − → D1 − → M1 −→ a10 −→ E. For clear understanding and illustration purpose, we have chosen the third representation, i.e., sequence of activities and transitions, of behaviors in this section. Three different behaviors represented by the activity diagram shown in Figure 3.2 are: a) invalid user, b) valid user with insufficient balance, and c) valid user with sufficient balance. Among these behaviors third behavior, i.e., valid user with sufficient balance contains concurrency constructs, hence it is shown with three different interleavings of the partial order a6 a7 . Therefore, there are total five test scenarios representing sequential and concurrent behaviors. For non-concurrent programs, there is no notion of a context/thread switch (CS) among the parts of the same program. A context switch is the process of storing and restoring the state or execution context of a process or thread 54
3.2. Test Adequacy Criteria for Activity Diagrams so that execution can be resumed from the same point at a later time. In multithreaded environment, a thread is a unit of execution including program counter, stack counter, and registers. In this thesis, we use term context switch and thread switch interchangeably because at design level there is no much difference in these two terms. However, we used thread switch in further discussion. For concurrent programs, which are divided into different threads, there is the notion of a thread switch. Thread switches introduce non-determinism in program execution thereby generating interleaving. Therefore, concurrent behavior representation includes additional information about thread switches. A context/thread switch is represented t13 by keyword CS below the arrow, like a7 −− → a8 , in the B31 test scenario presentation. CS The behaviors are listed as follows: t1 t2 t3 t4 t18 t19 1) B1 = S − → a1 − → a2 − → D1 − → M1 −→ a10 −→ E. t1 t2 t3 t5 t6 t7 t8 t9 t18 t19 2) B2 = S − → a1 − → a2 − → D1 − → a3 − → a4 − → D2 − → a5 − → M1 −→ a10 −→ E. t1 t2 t3 t5 t6 t7 t10 t11 t12 t13 t15 1 → a1 − → a2 − → D1 − 3) B3 = S − → a3 − → a4 − → D2 −→ F1 −→ a6 −→ a7 −− → a8 −→ CS
t16
t17
t18
t19
t16
t17
t18
t19
t
t
t
t
J1 −→ a9 −→ M1 −→ a10 −→ E. t1 t2 t3 t5 t6 t7 t10 t11 t12 t15 t13 4) B32 = S − → a1 − → a2 − → D1 − → a3 − → a4 − → D2 −→ F1 −→ a6 −− → a8 −− → a7 −→ CS
CS
J1 −→ a9 −→ M1 −→ a10 −→ E. t1 t2 t3 t5 t6 t7 t10 t14 t15 t12 t13 → a1 − 5) B33 = S − → a2 − → D1 − → a3 − → a4 − → D2 −→ F1 −→ a8 −− → a6 −→ a7 −→ CS
16 17 18 19 J1 −→ a9 −→ M1 −→ a10 −→ E.
The third behavior has three interleavings like B31 , B32 , and B33 where superscript indicates interleaving number and subscript indicates behavior number. According to Mingsong et al.’s definitions, behaviors B31 , B32 , B33 represent basic paths that have the same set of activities and the same partial order relation a6 a7 among activities {a6 , a7 , a8 }. Therefore, one representative path among these paths is taken as a simple path, say B31 . For behavioral testing, test cases consist of a behavioral scenario and a set of data, which consists of a pair of input data and values that satisfy predicates along the scenario. Initial configuration for ATM withdrawal process is: Valid ATM card PAN = 1000, PIN=123, account type = savings, initial balance=1000, and minimum balance= 500. For example, initial test suite of five test cases for Figure 3.2 is as follows: T CS = { T1 ={B1 , {(CARD P AN, 1000), (P IN, 1234), (accountT ype, “savings”), (amount, 2000)}}, 55
3. Concurrency Coverage Criteria T2 ={B2 , {(CARD P AN, 1000), (P IN, 123), (accountT ype, “savings”), (amount, 2000)}}, T3 ={B31 , {(CARD P AN, 1000), (P IN, 123), (accountT ype, “savings”), (amount, 500)}}, T4 ={B32 , {(CARD P AN, 1000), (P IN, 123), (accountT ype, “savings”), (amount, 600)}}, T5 ={B33 , {(CARD P AN, 1000), (P IN, 123), (accountT ype, “savings”), (amount, 700)}} }. In a test case, there is a set of values for input data that satisfy all predicates along the path (test scenario). In other words, one test case (test scenario, test data) satisfies one behavior. There are two basic paths that cover two behaviors B1 and B2 . There are three simple paths that cover three behaviors B1 , B2 , and B31 . One representative path B31 is selected as a simple path for three interleaving order paths B31 , B32 , and B33 . Table 3.3 represents test cases and their individual basic path and Table 3.3: Behavioral coverage achieved by test cases for behaviors B1 , B2 , and B31 of the activity diagram shown in Figure 3.2
Test Case T1 T2 T3
%Coverage Basic path Simple path B1 50.00 B1 33.33 B2 50.00 B1 33.33 – – B31 33.33
simple path coverage. Different combinations of all test cases are checked to achieve 100% coverage shown in Table 3.4. We have identified two sets of test cases {{T1 , T2 }, Table 3.4: Behavioral coverage achieved by the combination of test cases
Test Cases combinations T1 , T2 T1 , T3 T2 , T3 T1 , T2 , T3
% Coverage Basic path Simple path 100.00 66.66 50.00 66.66 50.00 66.66 100.00 100.00
{T1 , T2 , T3 }} for testing that achieve 100% basic path coverage. A set {T1 , T2 , T3 } of test cases achieves 100% simple path coverage. A common smallest test set among these two sets is {T1 , T2 , T3 }, which is an adequate test set that achieve 100% basic path and simple path coverage. A test team can choose say T CS = {T1 , T2 , T3 } as 56
3.2. Test Adequacy Criteria for Activity Diagrams the final test case suite for behavioral testing. However, the basic path and simple path coverage do not explore most of the concurrent interleaving behaviors. Next subsection presents concurrency coverage criteria based on interleaving for activity diagrams.
3.2.2
Concurrency Coverage Criteria for Activity Diagrams
In activity diagrams, two types of execution paths are available: non-interleaving paths (which consists of activities, decisions, iterations, and merges) and interleaving paths (which consists of activities, decisions, iterations, merges, forks, and joins). Non-interleaving paths have a total order from the start activity to the end activity, and their execution represents single sequence of activities. Interleaving paths have a partial order inside fork-join from the start activity to the final activity; and their execution represents different interleaving sequences of activities inside fork-join. A basic path definition discussed by Linzhang et al. (2004) does not distinguish noninterleaving and interleaving paths. But, we distinguish non-interleaving paths inside concurrent activity diagrams. Hence, we redefine basic paths and introduce new interleaving activity path coverage criteria for concurrent activity diagrams. 3.2.2.1
Basic path coverage criterion for concurrent activity diagrams
Basic paths for non-concurrent designs cannot be directly applicable to concurrent activity diagrams. Concurrent activity diagrams include both non-interleaving and interleaving paths. To avoid confusion of basic paths, we redefined basic paths for concurrent activity diagrams. Distinction between paths and their respective coverage criteria help to effective effort estimation and test management for sequential and concurrency testing. Definition 3.7. Basic Path: A path from the start activity to the end activity through non-concurrent constructs of an activity diagram, and if each activity in the path occurs at the most twice then such path is called the basic path of the concurrent activity diagram. This new definition of basic path restricts path exploration to only non-interleaving paths through non-concurrent constructs of concurrent activity diagrams. Definition 3.8. Basic Path Coverage Criterion: Let P be the set of basic paths of an activity diagram, T S be the set of test scenarios generated from the activity diagram. 57
3. Concurrency Coverage Criteria The test set T CS satisfies basic activity path coverage, if and only if, ∀ p ∈ P, ∃ t ∈ T S such that the test scenario ‘t’ visits every activity and every transition of the path ‘p’ according to the total order specified by ‘p’. A basic path coverage (BPC) criterion for concurrent activity diagram requires that all the non-interleaving activity paths in an activity diagram be covered. The value of the basic path coverage criterion for concurrent activity diagram is the ratio of toured non-interleaving activity paths by a test suite T CS to all the non-interleaving activity paths present in the activity diagram. 3.2.2.2
Interleaving activity path coverage criterion for concurrent activity diagrams
For concurrent activity diagrams, there are many paths having the same set of activities and the same partial order relation. Simple path definition by Mingsong et al. (2006) selects one representative path among all concurrent interleaving paths having same partial order. A similar argument is made by Kundu and Samanta (2009) for activity path coverage that suggests selecting one representative path from paths generated by the BFS search traversal inside the concurrent fork-join structure. Although, both the approaches promote selection of single representative path for concurrent flow testing, it is good enough criteria to test interleaving of activities inside the fork-join constructs for parallel execution. But, these coverage criteria are not sufficient for testing concurrency errors for concurrent executions. In Chapter 1, we have shown that such a randomly selected single test path has very low capability of uncovering concurrency errors. A test suite must have more than one distinct interleaving test scenarios to uncover data races, data inconsistency and deadlocks. In the code based testing, Lu et al. (2007) have proposed interleaving coverage criteria hierarchy for concurrent programs to systematically explore the interleaving space, and thereby effectively expose concurrency bugs. Similarly, we have proposed interleaving activity path coverage and total activity path coverage criteria for UML-based testing based on Lu et al. (2007)’s coverage criteria hierarchy. Definition 3.9. Interleaving Activity Path: A path from the start activity to the end activity through concurrent constructs of an activity diagram, if the path includes concurrent activity interleaving due to the fork-join construct and each activity in the path occurs at the most twice then such path is called the interleaving activity path. 58
3.2. Test Adequacy Criteria for Activity Diagrams Definition 3.10. Interleaving Activity Path Coverage Criterion: Let IP be the set of interleaving activity paths of an activity diagram, T S be the set of test scenarios generated from the activity diagram. The test set T CS satisfies interleaving activity path coverage, if and only if, ∀ p ∈ IP, ∃ t ∈ T S such that the test scenario ‘t’ visits every activity and every transition of the path ‘p’ according to the order specified by ‘p’. An interleaving activity path coverage criterion requires that all activity interleavings due to the fork-join construct of a concurrent activity diagram be covered. The value of the interleaving activity path coverage is the ratio of toured interleaving paths by the test suite T CS to all the interleaving paths present due to the fork-join construct in the concurrent activity diagram. 3.2.2.3
Total activity path coverage criterion
A non-concurrent activity diagram consists of non-interleaving activity paths. A concurrent activity diagram consists of either interleaving activity paths and noninterleaving activity paths or only interleaving activity paths. Therefore, the total activity path coverage criterion is proposed for concurrent activity diagrams that encompasses non-interleaving activity paths and interleaving activity paths. Definition 3.11. Total Activity Path Coverage Criterion: Let T P be the union of the set of basic paths and set of interleaving activity paths of an activity diagram, T S be the set of test scenarios generated from the activity diagram. The test set T CS satisfies total activity path coverage, if and only if, ∀ p ∈ T P, ∃ t ∈ T S such that the test scenario ‘t’ visits every activity and every transition of the path ‘p’ according to the order specified by ‘p’. A total activity path coverage criterion is calculated as the ratio of all the noninterleaving activity paths and interleaving activity paths toured by the test suite T CS to the summation of non-interleaving activity paths and interleaving activity paths present in a concurrent activity diagram. In the case of non-concurrent activity diagrams, number of total activity paths is same as the number of basic paths. Example- According to the new definitions, there are two basic paths which cover two behaviors B1 and B2 of the system shown in Figure 3.2. There are three interleaving activity paths that cover three concurrent behaviors B31 , B32 , and B33 . Total 59
3. Concurrency Coverage Criteria activity paths are five that satisfy all non-concurrent behaviors B1 and B2 and concurrent behaviors B31 , B32 , and B33 . Therefore, test case T1 and T2 achieve 100% non-interleaving activity path coverage. Test cases T3 , T4 , and T5 achieve 100% interleaving activity path coverage. Test cases T1 , T2 , T3 , T4 , and T5 achieve 100% total activity path coverage.
3.3
Test Adequacy Criteria for Sequence Diagrams
In this section, first we discuss existing sequence diagram based coverage criteria. After that, new concurrency coverage criteria are introduced. They are based on interleaving of message sequences, i.e., message transitions. Interleaving message path coverage criterion is specifically designed for parallel fragments of sequence diagrams. Synchronization coverage (SC) criterion utilizes the information available in the critical region fragment to define blocking thread and blocked thread. Finally, a suitable subset of test scenarios from IMPC and SC are combined to form essential message path coverage criterion.
3.3.1
Existing Coverage Criteria
Coverage criteria of sequence diagrams are based on the comparison of paths in sequence diagrams or intermediate representations of sequence diagrams and paths generated by a graph traversing algorithm or program execution traces of an implementation of the design. Several sequence diagram based test adequacy criteria are presented in the literature by Abdurazik and Offutt (2000); Rountev et al. (2005); and Dinh-Trong et al. (2006). Coverage criteria presented in these works are message coverage, message sequence coverage, and message path coverage. These criteria are based on a graph representation of sequence diagrams, and they are generalization of code based coverage criteria like node coverage, edge coverage, and path coverage derived from control flow graph. A message coverage criterion requires that all the messages in the sequence diagram be covered (visited). The value of the message coverage criterion is the ratio of messages visited by the test suite T CS to all the messages in the sequence diagram. Definition 3.12. Message Coverage Criterion: Let M be the set of messages of a sequence diagram, T S be the set of test scenarios generated. The test set T CS 60
3.3. Test Adequacy Criteria for Sequence Diagrams satisfies message coverage, if and only if, ∀ m ∈ M, ∃ t ∈ T S such that the test scenario ‘t’ visits the message ‘m’. In sequence diagrams, messages are ordered from the top to bottom in the order of their occurrence. Message occurrence order defines the message sequence from the start to the end of the sequence diagram. The message sequence is a path in the intermediate-graph representation of the sequence diagram that represents the behavior of the system. Message sequence coverage criterion requires a message sequence path must be covered/ visited by the test scenario. Message sequence coverage is used for plain sequence diagrams, which do not include any combined fragments. Definition 3.13. Message sequence Coverage Criterion: Let M S be the set of message sequence of a sequence diagram, T S be the set of test scenarios generated. The test set T CS satisfies message sequence coverage, if and only if, for ∀ ms ∈ M S, ∃ t ∈ T S such that the test scenario ‘t’ visits all messages in the order specified by ‘ms’. Message sequence path by Abdurazik and Offutt (2000) and all message paths by Pilskalns et al. (2003) criteria are defined for collaboration diagrams and subsequently adopted for sequence diagrams. A message sequence path is a path that includes all messages in the order specified by the collaboration diagram. This definition is defined for collaboration diagram in UML 1.0 specification, and it does not address combined fragments. UML 2.0 specification has introduced combined fragments that represent control flow information in sequence diagrams. With the introduction of combined fragments, message sequence path may be used for fragments or operands of the combined fragments. In other words, a message sequence path is a path that includes all messages in a non-nested fragment or non-nested operands of the combined fragment in the order specified by the order specification. We adopt this extended meaning of message sequence path in this thesis. All message path coverage criterion is satisfied by traversing CFG-like intermediate representation such as variable assignment graph (VAG) by Dinh-Trong et al. (2006) or object-method directed acyclic graph (OMDAG) by Pilskalns et al. (2003). A complete path generated by traversing algorithm like DFS or BFS from the start node to the end node are used for satisfying all message path coverage criterion for sequence diagrams. We adopt the same notion of all message paths coverage in this thesis. We consider all message paths as path generated by DFS and BFS traversal from CFG-like representation of sequence diagrams. 61
3. Concurrency Coverage Criteria In UML 2.0, a combined fragment is used to group a set of messages together to show conditional flow of sequence diagrams. A combined fragment is defined by an interaction operator and the corresponding interaction operands. Due to combined fragments, there are more than one message sequences, which represent different behaviors of the system. A message path coverage criterion must tour all such message sequences due to combined fragments in sequence diagrams. Definition 3.14. Message Path: A path from the start to the end of a sequence diagram through combined fragments that represent a system behavior. Definition 3.15. Message Path Coverage Criterion: Let M P be the set of message paths of a sequence diagram, T S be the set of test scenarios generated from the sequence diagram. The test set T CS satisfies message path coverage, if and only if, ∀ p ∈ M P, ∃ t ∈ T S, such that the test scenario ‘t’ visits every message in the path ‘p’ according to the order specified by ‘p’. Example- Consider a non-concurrent sequence diagram for money withdrawal process from automated teller machine (ATM) shown in Figure 3.3. This diagram is used to illustrate the structural element coverage. Customers have ATM card containing a personal authorization number (PAN) in digital form. A customer requests withdrawal of money from ATM by inserting ATM card and input PIN and the amount through messages m1 − m3 . If the user is invalid then system displays invalid user message in message m11 and ejects ATM card in message m12 . Otherwise, system checks balance of the customer’s account through messages m4 − m5 . The teller object sends message m6 : bank.Debit(accno, amount) message to bank object and in turn the bank object sends the message m7 : account.Debit(amount) to the specific instance of account object. System either dispenses cash through messages m6 − m9 or gives an error in message m10 based on the balance available in the customer’s account and the money requested by the customer. Finally, system ejects ATM card in message m12 . Initial configuration and assumptions are required to start execution of test cases. For ATM withdrawal process, initial configuration is: Valid ATM card PAN = 1000, PIN=123, account type = savings, initial balance=1000, and minimum balance= 500. Test case is a set of the pair of input data and its value. For example, initial test case suite of five test cases for Figure 3.3 is as follows: T CS = {T1 ={(CARD P AN, 1000), (P IN, 1234), (amount, 2000)}, T2 ={(CARD P AN, 1000), (P IN, 123), (amount, 2000)}, 62
3.3. Test Adequacy Criteria for Sequence Diagrams
SD: ATM Money Withdrawal cust : Customer
teller : ATM
bank : Bank
m1: insertATMCard m2: enterPIN Alt: alt1 m3: withdraw(amount)
[valid user] m4: getBalance(accno) balance
Alt: alt2
[amount balance] m10: showError(“insufficient balance”)
[invalid user]
m11: showError(“Invalid user or password”)
card
m12: ejectCard()
Figure 3.3: A non-concurrent sequence diagram for ATM money withdrawal process
T3 ={(CARD P AN, 500), (P IN, 123), (amount, 500)}, T4 ={(CARD P AN, 1000), (P IN, 123), (amount, 500)}, T5 ={(CARD P AN, 1000), (P IN, 123), (amount, 1500)} }. For sequence diagrams with combined fragments, a message sequence criterion is considered for each operand of the combined fragments. Thus, there are six different message sequences: M S1 =< m1 , m2 >, M S2 =< m3 , m4 , m5 >, M S3 =< m6 , m7 , m8 , m9 >, M S4 =< m10 >, M S5 =< m11 >, and M S6 =< m12 >. Table 3.5 shows the message and the message sequence coverage by each test case. From Table 3.5, we have identified four test cases set for the testing {{T1 , T2 , T4 }, {T1 , T4 , T5 }, {T2 , T3 , T4 }, {T3 , T4 , T5 }} that achieve 100% message and message sequence coverage. In sequence diagrams, behavior can be represented as a sequence of messages, i.e., message transition sequence representing a complete/partial path. For example, a basic path representing invalid user in Figure 3.3 can be represented as a sequence of messages: m1 → m2 → m11 → m12 . Sequence diagrams are converted into CFG-like intermediate representation for coverage analysis purpose. For simplicity, we consider only message nodes for representing behaviors excluding control nodes 63
3. Concurrency Coverage Criteria Table 3.5: Message and message sequence coverage achieved by test cases for the sequence diagram shown in Figure 3.3
Test Case T1 T2 T3 T4 T5
Message m1 , m2 , m11 , m12 m1 , m2 , m3 , m4 , m5 , m10 , m12 m1 , m2 , m11 , m12 m1 , m2 , m3 , m4 , m5 , m6 , m7 , m8 , m9 , m12 m1 , m2 , m3 , m4 , m5 , m10 , m12
Coverage % 33.33 58.33 33.33 83.33
Message sequence M S1 ,M S5 ,M S6 M S1 ,M S2 ,M S4 , M S6 M S1 ,M S5 ,M S6 M S1 ,M S2 ,M S3 , M S6
% 33.33 66.66 50.00 66.66
58.33
M S1 ,M S2 ,M S4 , M S6
66.66
present in the intermediate representation. Three different behaviors represented by the sequence diagram shown in Figure 3.3 are: a) invalid user, b) valid user with insufficient balance, and c) valid user with sufficient balance. The behaviors are listed as follows: 1) B1 =m1 → m2 → m11 → m12 , 2) B2 =m1 → m2 → m3 → m4 → m5 → m10 → m12 , and 3) B3 =m1 → m2 → m3 → m4 → m5 → m6 → m7 → m8 → m9 → m12 . For behavioral testing, a test case consists of a test scenario (behavior) and a set of data, which is a pair of input data and value that satisfy predicates along the scenario. For example, initial test case suite of three test cases for behavioral coverage of Figure 3.3 is as follows: T CS = { T1 ={B1 , {(CARD P AN, 1000), (P IN, 1234), (amount, 2000)}}, T2 ={B2 , {(CARD P AN, 1000), (P IN, 123), (amount, 2000)}}, and T3 ={B3 , {(CARD P AN, 500), (P IN, 123), (amount, 500)}} }. Test cases T1 , T2 , and T3 achieve 100% message path coverage, hence TCS is the final test set for testing behavioral coverage.
3.3.2
Concurrency Coverage Criteria for Sequence Diagrams
A test adequacy criterion specifies the requirements for software testing. Several sequence diagrams based test adequacy criteria are presented by Abdurazik and Offutt (2000); Andrews et al. (2003); and Rountev et al. (2005). These criteria are based on graph representations of sequence diagrams. Nevertheless, these coverage criteria do not take into account the potential for concurrency in the design. In spite of sev64
3.3. Test Adequacy Criteria for Sequence Diagrams eral sequence diagram based coverage criteria presented in the literature, interleaving of messages in concurrent section and synchronization behavior inside parallel fragments have not been studied in previous UML-based approaches. In this section, we present new additional coverage criteria namely interleaving message path coverage, synchronization coverage (SC), and essential concurrent message path coverage for sequence diagrams. First, some useful definitions are presented for the process of defining coverage criteria. Subsequently, concurrent behavioral coverage criteria are presented. Consider an example of the bank transaction for illustration purpose shown in Figure 1.2 of a sequence diagram. For testing purpose, it is necessary to translate sequence diagram into some form of intermediate representation. In this chapter, a sequence diagram is translated into a message control flow graph (MCFG) for testing purpose. MCFG complete construction details are given in Chapter 8. Here we only explain MCFG definition in the context of subsequent details. Definition 3.16. A message control flow graph is a tuple G (N, E, n0 , nf ), where N − a set of nodes, a node n ∈ N is represented as a tuple (l, nt), ‘l’ is a label associated with node and ‘nt’ is a type of node; E− a set of edges, an edge e ∈ E is represented as a tuple (n1 , n2 , g, et), n1 , n2 ∈ N , ‘g’ is a guard associated with the edge, and ‘et’ is a type of the edge; n0 − the initial node of G, n0 ∈ N ; and nf − the final node of G, nf ∈ N. In MCFG, nodes are broadly classified into two types: message nodes (M N ) and control nodes (CN ), i.e., N = (M N ∪ CN ). Message nodes are associated with messages of sequence diagrams, and control nodes are associated with fragments, combined fragments, and the start (n0 ) and the end (nf ) of sequence diagrams. There are total 28 control nodes: sdStart, sdEnd, Signal, Wait, and two nodes for each twelve combined fragments: alt, opt, break, par, seq, strict, neg, assert, critical, loop, consider, and ignore. The details are given in Table 8.1. The mapping of the sequence diagram elements to MCFG edges is given in Table 8.2. It includes sequence diagram element mapped to edges in MCFG, guard condition and type of edges. Edges are classified as order-specification edges and message-interleaving edges. Order-specification edges shown by dark solid lines represent order specification in the sequence diagram. Message-interleaving edges shown by dotted lines that represent virtual edges added in the construction MCFG for exploring message interleavings inside parallel frag65
3. Concurrency Coverage Criteria ments. All edges are unidirectional in MCFG representation. The Figure 3.4 shows
m2
sd Start
m1
CF critical 1 Start
m3
m4
CF critical 1 End
CFpar End
CFpar Start
m5
CF critical 2 Start
m6
m7
m8
sd End
CF critical 2 End
Figure 3.4: A message control flow graph representation of the bank transaction sequence diagram shown in the Figure 1.2
an MCFG graph of the bank transaction sequence diagram shown in Figure 1.2. In parallel fragments, some actions in different threads may be related that are represented by general ordering constructs. They need to be executed in a certain order. This is expressed by a dashed line with an arrow in the middle of the messages as shown in Figure 8.3. For example, a general ordering shown by a directed dashed line sends a signal to another thread that wait for the signal in Figure 8.3. The directed dashed line starts after message m3 and ends before message m8 in Figure 8.3 This construct is similar to send and receive signal for synchronization. In MCFG construction, these nodes are modeled as Signal and W ait control nodes as shown in Figure 8.4. A path in an MCFG is used to represent concurrent behavior of the system. In MCFG, a node nji inside the parallel fragment of sequence diagrams is the node at ith location in j th thread (fragment interaction operand). Definition 3.17. MCFG Path : A path of MCFG is an alternative sequence of nodes eq e1 e2 e3 and edges < n11 − → n12 −−→ n21 − → . . . nkm−1 − → nkm > or simply a sequence of nodes CS
< n11 , n12 , n21 , . . . , nkm−1 , nkm >, where node nji is the ith node of the j th thread and CS is a thread switch, if an edge e is associated with CS then it is a message-interleaving edge; otherwise, it is an order-specification edge. Definition 3.18. MCFG Test Path : A path in MCFG is a test path p = < n11 , n12 , . . . , nji , nji+1 , . . . , nkm−1 , nkm >, if p starts from the initial node (i.e., n11 = n0 ) 66
3.3. Test Adequacy Criteria for Sequence Diagrams and terminates in the end node (i.e., nkm = nf ) that represents a system functionality for non-concurrent execution and serialized or non-serialized schedule for concurrent execution. Definition 3.19. Fragment Sequence : A fragment sequence of MCFG is a path f = < nj1 , nj2 , . . . , njm−1 , njm >, where nodes nj1 is a fragment start node and njm is a fragment end node and nj1 , njm ∈ CN and ‘f ’ represents a path through combined fragment’s j th interaction operand in a sequence diagram. A sequentialized message path inside parallel fragments of sequence diagrams is a test path that has parallel fragments concatenated one after another (messages inside fragments do not have any interleaving). If an edge e = (npi , nqj ) from fragment ‘p’ to fragment ‘q’ then p = q =⇒ no thread switch and p 6= q =⇒ thread switch. Definition 3.20. Sequentialized Message Path : A sequentialized message path of MCFG is a test path tp, if tp consists of a permutation of fragment sequences f s of parallel fragment in a sequence diagram, and if an edge e = (npi , nqj ) ∈ f s, fragment p has m messages 1 ≤ i ≤ m and fragment q has k messages 1 ≤ j ≤ k, then either p = q and j = i + 1 (no thread switch) or p 6= q and i = m − 1, j = 2 (thread switch at the end of the thread). For example, in Figure 3.4, there are two sequentialized message paths. Consider a path p1 =< sdStart → m1 → CF parStart → m2 → CF critical1Start → m3 → m4 → CF critical1End −−→ m5 → CF critical2Start → m6 → m7 → CF critical2End→ CS CF parEnd → m8 → sdEnd >. The edge (m3 , m4 ) has node m3 and node m4 belong to same fragment sequence and the edge ( CF critical1End, m5 ) has nodes CF critical1End and m5 belong to different fragment sequences. Node CF critical1End is the last node of the first fragment sequence before fragment end and node m5 is the first node of another fragment sequence after fragment start of MCFG in Figure 3.4. Definition 3.21. Interleaved Message Path : An interleaved message path of MCFG is a test path tp, if tp consists of interleave of messages in fragment sequences f s of the parallel fragments in a sequence diagram, and if an edge e = (npi , nqj ) ∈ f s, fragment p has k messages 1 ≤ i ≤ k and fragment q has l messages 1 ≤ j ≤ l, then either p = q and j = i + 1 (no thread switch) or p 6= q and 1 ≤ i ≤ k and 1 ≤ j ≤ l (thread switch anywhere). For example, a path p2 =< sdStart → m1 → CF parStart→ m5 → CF critical2Start → m6 −−→ m2 −−→ m7 → CF critical2End −−→ CF critical1Start → m3 → m4 CS
CS
CS
67
3. Concurrency Coverage Criteria → CF critical1End → CF parEnd → m8 → sdEnd > is an interleaved message path of Figure 3.4. 3.3.2.1
Interleaving message path coverage criterion
An interleaving message path includes messages from all operands of the parallel fragment. There are possibilities of data-race and synchronization issues in the some of the executions inside parallel fragments. To address these errors, concurrent test case suite must contain more than single interleaving path. We have proposed Interleaving Message Path Coverage criterion for sequence diagrams. This coverage criterion is extended from thread-pair-interleaving by Lu et al. (2007). Definition 3.22. Interleaving Message Path Coverage Criterion: Let CP be the set of interleaving message paths of a sequence diagram, T S be the set of test scenarios generated. The test set T CS satisfies interleaving message path coverage, if and only if, ∀p ∈ CP ∃t ∈ T S. An IMPC criterion requires that all concurrent message interleaving paths due to the parallel fragments of a sequence diagram to be covered. IMPC criterion selects a large number of message interleaving paths to test concurrent behaviors effectively, thereby exposing concurrency errors. The coverage value of the IMPC is the ratio of toured interleaving message paths by the test suite T CS to all interleaving message paths present due to parallel fragments of the sequence diagram. 3.3.2.2
Synchronization coverage criterion
Critical region and general ordering constructs inside parallel fragments in sequence diagrams restrict some of the interleaving paths. There are possibilities of synchronization and data inconsistency errors if these constructs are not implemented properly (Vaziri et al., 2006). It is necessary to address synchronization behavior for concurrency testing. The synchronization coverage for code based testing is studied by Bron et al. (2005) and Kim et al. (2011). We define the synchronization coverage criterion for UML-based testing that is extended from code based synchronization coverage model of Bron et al. (2005). Most of the times, the thread switch inside a critical region does not occur, because a number of statements in it are less. In other words, the execution time frame is 68
3.3. Test Adequacy Criteria for Sequence Diagrams very small. Synchronization is an important behavior of concurrent systems. Concurrency errors like data race and data inconsistency are associated with critical region. Therefore, it is necessary to test synchronization behavior inside the critical region. Unattended concurrency behavior due to lack of synchronization, if not tested may lead to data inconsistency in some future runs. The synchronization coverage model by Bron et al. (2005) forces a thread switch inside the critical region. This will cause an execution to show the effect of synchronization primitives. Synchronization primitives acquire a lock on shared data and obtain mutual exclusion among the executing threads. A lock leads to two types of behaviors called blocked and blocking. The blocked behavior causes current executing thread to stop the execution till it gets an access to the shared data. The blocking behavior acquires a lock on shared data and stops other threads that try to access the shared data. Synchronization primitives ensure that critical section code to be executed by only one thread at a time and protect shared information from other threads. Thus, synchronization primitives guarantee the data race free execution of the implementation of the model. Each interaction operand of the parallel fragment is termed as a thread. Consider a parallel fragment has two threads T1 and T2 , and each thread includes a critical region. The concurrent execution of these threads may lead to data race. To avoid the data-race situation an implementation needs to include proper synchronization primitives (locks) to have race free execution. Synchronization primitives insist that every thread may sometime execute as a blocked thread and sometime as a blocking thread. Therefore, to test proper implementation of the critical region, each thread should be exercised as a blocking thread and a blocked thread. We assume start of a critical region in parallel fragment acquires required locks, and the end of a critical region releases the obtained locks in the implementation of the design. For sequence diagrams, blocking thread and blocked thread are defined as follows. These definitions are based on Bron et al. (2005) synchronization coverage model. Definition 3.23. Blocking thread: Let a parallel fragment has two interaction operands called thread T1 and T2 . Consider the thread T1 acquires a lock ‘l’ and another thread T2 tries to acquire the same lock ‘l’, which is currently held by the thread T1 . In this case, thread T2 gets blocked by thread T1 ; and thread T1 acts as a blocking thread. Definition 3.24. Blocked thread: Let a parallel fragment has two interaction operands called thread T1 and T2 . Consider the thread T1 tries to acquire a lock ‘l’, which is already acquired by the thread T2 . In this case, thread T1 gets blocked by thread T2 ; 69
3. Concurrency Coverage Criteria and thread T1 acts as a blocked thread. Definition 3.25. Synchronized Message Path: An interleaving message path is a synchronized message path, if one of the parallel fragment operand is either getting blocked or blocking another parallel fragment operand due to thread switch inside the critical region construct or thread switch due to the general ordering construct. Definition 3.26. Synchronized Message Path Coverage Criterion: Let the set of synchronized message paths SP of a sequence diagram, T S be the set of test scenarios generated. The test set T CS satisfies synchronized message path coverage, if and only if, ∀p ∈ SP, ∃t ∈ T S. A synchronized message path coverage criterion requires that a path passing through a critical region of the parallel fragment should be exercised as either a blocked task or a blocking task. The value of the synchronized message path coverage is the ratio of toured synchronized message paths by a test suite T S to all synchronized message paths present due to a critical region(s) or a general ordering inside the parallel fragment in a sequence diagram. A Reduced MCFG graph is a sub-graph of MCFG generated by considering all interleaving edges inside critical section and ordered-specification edges of a sequence diagram shown in Figure 3.5. A reduced MCFG graph is discussed in detail in Chapter 8. There are two threads m2
Sd Start
m1
CF critical 1 Start
m3
m4
CF critical 1 End
CFpar End
CFpar Start
m5
CF critical 2 Start
m6
m7
m8
Sd End
CF critical 2 End
Figure 3.5: An RMCFG representation of the bank transaction sequence diagram shown in the Figure 1.2
executing concurrently in Figure 3.5. Each thread has a critical region CR1 = {CF critical1Start, m3 , m4 , CF critical1End} and CR2 = { CF critical2Start, m6 , m7 , CF critical2End} with threads say T1 and T2 , respectively. In the implementation, when execution reaches CF critical1Start node in the CR1 , thread T1 acquires 70
3.3. Test Adequacy Criteria for Sequence Diagrams a lock(s) from the locking mechanism to protect the shared information and release the lock(s) when execution reaches CF critical1End node. Similarly, when execution reaches CF critical2Start node in the CR2 , thread T2 acquires a lock(s) from the locking mechanism to protect the shared information and releases the lock(s) when execution reaches CF critical2End node. The most of times, execution happen in serialized manner, i.e., either thread T1 execution is followed by thread T2 or vice versa. However, synchronization coverage ensures execution scenarios to exercise thread switch inside the critical region. Even though, two critical regions run concurrently, they do run into each other, but do not modify shared data in the presence of proper synchronization primitives. Synchronization coverage checks the above property by executing non-serialized execution schedules.
Total synchronized paths Consider a parallel fragment, which has two interaction operands, say T1 and T2 with n1 and n2 number of messages inside the critical regions, respectively. Then every path, which has a thread switch(s) inside a critical region, is a synchronized message path. Therefore, the total number of synchronized concurrent message paths (TSYMP) is calculated as T SY M P = (n1 + n2 )!/(n1 !) ∗ (n2 !).
3.3.2.3
Essential concurrent message path coverage criterion
Concurrent programs have exponential interleaving space. However, for testing purpose only a few representative interleaving sequences are sufficient. Therefore, the most crucial subset of total interleaving space is adequate to detect the concurrency errors. Cost and bug-exposing capability are two most important metrics of a test suite that need to be satisfied by a coverage criterion (Lu et al., 2007). For a given coverage criterion, a number of test cases in the test suite is used to measure the cost of the test suite. Bug exposing capability of a test case is defined by type of requirement imposed by coverage criterion, which is satisfied by the test case execution. Therefore, we define ECC criterion as a trade-off between cost and bug-exposing capability of test scenarios for concurrency testing. ECC suggests selecting a representative set of test scenarios among all interleaving scenarios that cover different concurrent behavioral aspects of systems like serialization, sequentialization, and synchronization. Definition 3.27. Essential Concurrent Message Paths: A set of interleaved message paths of MCFG that include all sequentialized message paths and two synchronized 71
3. Concurrency Coverage Criteria blocked and/ or blocking path(s) per thread for parallel fragments of a sequence diagram. Essential concurrent message paths include serialize paths as a set of sequentialized paths and non-serialized paths as synchronized blocked and blocking paths. Therefore, essential concurrent message paths consists of a set of valid and invalid complete concurrent test scenarios. Definition 3.28. Essential Concurrent Message Path Coverage Criterion: Let the set of essential concurrent message paths ECP of a sequence diagram, T S be the set of test scenarios generated from the sequence diagram. The test set T CS satisfies essential concurrent message path coverage, if and only if, ∀p ∈ ECP, ∃t ∈ T S. Total ECC paths Consider a parallel fragment has T interaction operands (threads) in a sequence diagram, and |T | represents the total number of threads. Essential number of sequentialized concurrent message paths is a permutation of a number of threads, and it is calculated as ESQM P = (|T |!). Let CSt is a number of the critical regions associated with a thread t and GOt is a number of general ordering associated with a thread t. Then essential number of synchronized concurrent message paths is the summation of the twice number of critical regions associated with each thread and number of general ordering associated with each thread. Therefore, ESY M P = P P 2 ∗ ( nt=1 CSt ) + ( nt=1 GOt ). Inside a critical region, a thread may act as a blocking thread and/ or blocked thread; hence, for each critical region associated with thread two test scenarios are selected per thread. For general ordering, a thread act as a blocking thread or blocked thread; therefore, one test scenario is selected per thread for each general ordering associated with the thread. Total number of essential concurrent message paths ECM P = ESQM P + ESY M P . The coverage value of the ECC criterion is the ratio of toured essential concurrent message paths by a test suite T CS to all essential concurrent message paths present inside the parallel fragment in the sequence diagram. For example, the bank transaction example in Figure 1.2 having MCFG intermediate representation Figure 3.4, has ESQM P = 2! = 2 and ESY M P = 2 *(2)+(0) = 4, therefore, ECM P = 2 + 4 = 6. IMPC test case suite size, of the bank transaction example using MCFG intermediate representation, is 10!/(5!.5!) = 252 and ECC test case suite size is 6. The ECC test suite is comparatively small to ICMP test suite size, and it has the capability of detecting concurrency errors. 72
3.4. Conclusions
3.4
Conclusions
In this chapter, we have described the structural element coverage and sequential behavioral coverage criteria for activity and sequence diagrams. Suitable examples are discussed explaining coverage achievement. Concurrent behavioral coverage criteria are presented for UML activity and sequence diagrams to explore the concurrency errors. Moreover, it will give sufficient test scenarios, which can be classified into different concurrent behaviors as discussed in Chapter 4. For activity diagrams, we point out a clear distinction between sequential and concurrent execution behaviors of concurrent activity diagrams. To take account of distinct execution behaviors, we heave redefined basic path coverage criterion and presented new coverage criteria: interleaving activity path and total activity path coverage for concurrent activity diagrams. Interleaving activity path coverage criterion includes more interleaving test scenarios compared to the basic path and simple path coverage criteria. Due to exponential space of interleaving paths, it is not possible to always achieve 100% interleaving activity path coverage. Concurrency errors are probabilistically very rare, hence higher the interleaving activity path coverage more the possibility of uncovering errors. Table 3.6: A summary of the proposed concurrency coverage criteria UML diagram
Coverage Scenario required
Intermediate representation
Maximum interleavings
IMPC
Message control flow graph
Qm Pm ( i=1 ni )! j=1 (nj !)
Interleaving inside parallel fragment
Error types uncovered
Data-race, synchronization, deadlocks Sequence Qm Pm Diagram SC Interleaving inside crit- Message con- ( i=1 ni )! Data-race, syn(n !) j j=1 ical region trol flow graph chronization Qm Pm ECC Synchronized and se- Message con- [( i=1 ni )! (n !)]+ Data-race, synj j=1 quentialized interleav- trol flow graph n! chronization, ings deadlocks Qm Pm IAPC Interleaving inside fork- Activity con- ( i=1 ni )! (n !) Data-race, synj j=1 Activity join trol flow graph chronization, Diagram deadlocks Qm Pm TAPC Interleaving inside fork- Activity con- [( i=1 ni )! (n !)]+ Data-race, synj j=1 join and basic paths trol flow graph basicpaths chronization, deadlocks n - number of threads, m - number of nodes (messages/ activities) in a given concurrency construct (thread) 0 < m
New additional UML-based concurrency coverage criteria are presented for sequence diagrams, namely interleaving message path coverage, synchronization cover73
3. Concurrency Coverage Criteria age and essential concurrent message path coverage to address concurrency errors. Interleaving message path coverage criterion suggests exploring all messages interleavings due to parallel fragments. However, IMPC is very difficult and costly to achieve 100% because an exponential increase in test scenarios in proportion to messages inside parallel fragments. Therefore, we further analyze the behavior of message interleavings like sequentializaton, serialization, and synchronization. This analysis subsequently results in synchronization coverage, which is based on code based coverage criteria. Finally, we have introduced ECC coverage criterion to select a smaller subset of all message interleavings to test concurrent execution behaviors. A test suite satisfying ECC is capable of detecting concurrency errors like synchronization, data-race, and deadlock. In summary, proposed concurrency coverage criteria ensure exploration of interleaving test scenarios, thereby helping detection of concurrency errors. A summary of proposed coverage criteria is presented in Table 3.6. Proposed concurrency coverage criteria are based on intermediate graph representations of respective artifacts. The proposed interleaving based concurrency coverage criteria are evaluated in subsequent chapters. In the next chapter, we present a state machine based analyzer to classify test scenarios into different execution behaviors.
74
Chapter 4 Test Scenario Classification for Concurrent Behaviors
Concurrent programs split themselves into multiple smaller tasks and then run these small tasks in separate threads to enhance the performance of systems. All these different tasks run parallel, but in some situations a few tasks need synchronization and interprocess communication among them thereby increasing the difficulty of testing. Concurrent program execution is non-deterministic that occasionally results in concurrency errors (Lei and Carver, 2006). When simultaneous executing threads do not share data or do not wait for any action in another thread then such threads are parallel executions, and they do not cause any concurrency errors. On the contrary, when simultaneous executing threads share data or wait for some action to be completed by another thread then such threads are concurrent executions, and they may lead to concurrency errors. Recent study by Lu et al. (2008) says that almost 97% non-deadlock bugs belong to atomicity violation, i.e., the desired serializability among multiple memory accesses is violated, and order violation, i.e., the desired order between two (groups of) memory accesses is flipped. In concurrent systems, there are different classes of errors like synchronization, data-race, data inconsistency, starvation, and deadlock (Souza et al., 2011). These errors are results of specific execution interleaving scenarios. Insufficient interleaving test scenarios may cause concurrency errors slip through testing process, thereby a few future runs of the system can cause concurrency errors (Musuvathi et al., 2008). Concurrency errors need systematic analysis and study of execution interleaving scenarios. 75
4. Test Scenario Classification for Concurrent Behaviors UML behavioral models, such as activity diagram and sequence diagram, express concurrency in designs. UML models are also used to generate test cases for concurrency problems in the early stages of software life cycle. In implemented code, concurrent execution interleaving has exponential space (Ball et al., 2011); however, its miniature abstract representation can be studied for analysis in terms of UML transition sequences. Implementation of concurrency constructs is a prime source of concurrency errors (Hammer et al., 2008). Hence, analysis of test scenarios generated from concurrency constructs should guide selection of appropriate test scenarios for concurrency testing. Accordingly, we classified interleaving space of UML transition sequences into four classes: sequentialize interleavings, serialized interleavings, blocking interleavings, and blocked interleavings based on the concurrency coverage criteria discussed in Chapter 3. In this chapter, we mapped interleaving classes to concurrency behaviors like non data-race, serialized, blocked, and data-race. Note that, sequentialized interleavings are a subset of serialized interleavings where all threads execute one after another. Test scenario generator algorithms generate a set of test scenarios from given UML artifacts. Testing all interleaving test scenarios is impossible for error detection in tight delivery schedules. Testing coverage criteria enforce selecting specific interleaving scenarios for test case design. We have proposed a state machine based approach to analyze and classify the test scenarios generated from UML concurrent designs. The rest of the chapter is organized as follows: First, we present extension to UML models to represent data access-tags, which are not a part of standard UML specification. Then, we study existing state machine model based approach for data race analysis. After that, we present our extended state machine diagram based approach for test scenario analysis and classification. Finally, we discuss the experimental results and conclude the chapter.
4.1
Data-Access Extensions to UML Models
Although concurrency is well supported by UML behavioral models, there is no mechanism available to provide information regarding shared data and their access type in the standard UML 2.0 (Lei et al., 2008). In concurrent applications, thread objects and shared objects play an important role. Note that, shared objects mean objects whose data is accessed by more than one thread in concurrent run of the system. A 76
4.1. Data-Access Extensions to UML Models pattern of data access to the shared object decides possibility of data race and other concurrency errors. There is a necessity to assign access types to shared objects to analyze test scenarios. We have adopted data access-tag technique as explained in Lei et al. (2008) approach and assign a data access-tag to every operation only for shared objects in UML designs. Data access-tags are of two types: read-access (r) and write-access (w). One may specify access-tags to the data members of a shared object to increase the granularity of analysis and hence improve results. An operation may have more than one access-tag either on different data members of a shared object or different shared objects. To include all such possibilities an access-tag is represented as operationName/(r/w):[dataMemeber] for single access and can be repeated with comma for multiple accesses. The operation name and access type fields are compulsory, and data member field is optional in data-access tags. If data member is not specified then access is assumed to all data members of the shared object. Example- Consider an example of ATM money withdrawal banking functionality modeled in sequence diagram shown in Figure 4.1. We have selected this new diagram for illustration of data-access tags, here read-access-tag and write-access-tag are on separate messages. Figure represents successful flow of events considering sufficient user account, i.e., amount includes shared object access in messages m7 and m9 that help in generating data-access trace. For test scenario S, a data-access trace is < bankT hread.read − bankT hread.write > for the data member balance of account object of bankThread. Data access-tags can be used to represent an interaction with shared objects in sequence and activity diagrams. Data-access traces play a crucial role in analyzing test scenarios for concurrency testing. Different interleaving scenarios generate different data-access traces that need to be analyzed for the pattern of a specific behavior. Some traces are serialized access to shared data, some traces are blocking other threads, some traces lead to data race errors, and other traces are non-blocking non data-race. 78
4.2. State Machine Based Test Scenario Analysis
4.2
State Machine Based Test Scenario Analysis
In the code based testing, data race detection is done using dynamic method, which uses lock-based approaches (Savage et al., 1997). These methods run the program and maintain lock status to analyze lock-set for data race problems. Whereas, data race analysis can be done using data-access traces of UML transition sequences. In this section, we first discuss state machine based data race detection approach presented by Lei et al. (2008). After that, we present an enhanced state machine based approach to analyze concurrent behaviors. Our approach reduces false classification of test scenarios, thereby improving accuracy of analysis and reducing test suite size for concurrency testing.
4.2.1
Existing State Machine Classifier
Lei et al. (2008) have proposed state machine diagram based approach for data race detection. Their approach used a state machine diagram of shared variables as shown in Figure 2.1. The state machine diagram consists of three states {Idle, Read/Taken, Write/Taken} and four events {Rd/In, Rd/Out, W r/In, W r/Out}. When a data-access trace of a given test scenario is given input to the state machine then trace analysis starts in Idle state. State machine changes its state based on the input, it consumes a complete data-access trace and then stops. When a readaccess or write-access to shared variable is followed by another write-access then a W r/In race condition happens. In their approach, the Read/T aken −−−−→ W rite/T aken, W r/In Rd/In W rite/T aken −−−−→ W rite/T aken, and W rite/T aken −−−→ W rite/T aken transitions are illegal in the given state machine diagram, and all other transitions are legal. If state machine traverses illegal transitions then test scenario violates the mutual exclusion rule, and thereby causing a data-race error. In Lei et al.’s approach, access traces are generated by instrumenting a Java program. Their instrumentation program adds statements in original Java program for all synchronized methods to generate traces for two events namely enter event: Rd/In or W r/In and exit event: Rd/Out or W r/Out. An execution of the instrumented program returns the data-access trace. The generated data-access traces are given input to state machine diagram. If the access trace traverses illegal transitions then data race is reported. However, the state machine diagram shown in Figure 2.1 does not consider thread switches among threads while generating data-access traces. 79
4. Test Scenario Classification for Concurrent Behaviors Therefore, some of the traces may mis-classify test scenarios as data race, particularly when the same thread accesses shared data for both read and write operations. Moreover, their sate machine diagram is ambiguous and may generate two results for some data access-traces. For example, a test scenario S from Figure 4.1 generates access-trace DAT =< Rd/In − Rd/Out − W r/In − W r/Out >. When DAT is given input to the state machine diagram then there are two possible outputs: one does not traverse illegal transition and another traverses illegal transition. For examRd/In Rd/Out W r/In ple, state machine execution for DAT is Idle −−−→ Read/Taken −−−−→ Idle −−−−→ W r/Out Write/Taken −−−−−→ Idle, and thus does not report data-race. Another execution is Rd/In Rd/Out W r/In W r/Out Idle −−−→ Read/Taken −−−−→ Read/Taken −−−−→ Write/Taken −−−−−→ Idle, and reports data-race. In a nutshell, the above example clearly indicates ambiguity in state machine diagram of Lei et al.’s approach shown in Figure 2.1. Moreover, the state machine reports data-race errors even if read and write operations belong the same thread. In other words, the state machine results in false classification of some test scenarios. Hence, there is a necessity to enhance the state machine diagram to mitigate the above problems.
4.2.2
Enhanced State Machine Classifier
An enhanced state machine diagram (ESMD) is proposed to analyze test scenarios. In this approach, we have analyzed test scenarios for non data-race, serialize, blocking, and data-race behaviors of concurrent applications. Serialize concurrent behaviors are useful to check the correct implementation of synchronization primitives in all concurrent tasks. Some of the serialize test scenarios are sequentialize test scenarios that consist of message/ task sequences of all threads in a sequential way, i.e., one thread sequence followed by another thread sequence without interleaving of messages/ tasks among threads. Sequentialize concurrent behaviors are useful to check the priority of all threads to access shared data. Blocking concurrent behaviors are useful to check proper synchronization primitives in all concurrent tasks. Blocking test scenarios are also useful to analyze starvation and deadlock properties of applications. Datarace behavior leads to unexpected results such as data inconsistency. Non data-race scenarios are useful to analyze proper synchronization primitives. Test scenarios classified as a data-race behavior are useful to check synchronization primitives and to detect possible data races, which are discussed in Section 4.1. The selected test sce80
4.2. State Machine Based Test Scenario Analysis narios from each class and test data are used to carry out deterministic concurrency testing (Carver and Lei, 2004). Proposed enhanced state machine diagram is shown in Figure 4.2. It takes input data-access traces. ESMD execution starts at Initial state. The input data-access trace decides transitions of ESMD and thereby changing machine states. At the end, when input data-access trace finishes the machine stop. The last state decides the behavior of the test scenario thereby classifying test scenario into one of the concurrent behavior like serialize, blocked, data-race, or non data-race. Thus, data-access traces generated from the activity and sequence diagrams are used to analyze test scenarios for concurrency behavior. ESMD considers only two concurrent threads for analysis purpose. Next, we formally describe the ESMD. read/NOCS
csEnd
read/CS, write/CS, csEnd, read/NOCS, write/NOCS
Safe Read Safe Thread
Blocked
write/NOCS Safe Write
Initial
read/NOCS
Unsafe Read
read/NOCS, write/NOCS read/CS csEnd
Shared Read
read/CS, read/NOCS
write/CS, write/NOCS
write/NOCS Unsafe Write
write/CS, read/CS
read/NOCS, write/NOCS
Data Race (Shared Modify) read/CS, write/CS, csEnd, read/NOCS, write/NOCS
States=>Behavior: Initial =>serialized, Blocked-=>blocking, Data Race=> data-race behavior. Any other state => non-data-race behavior
Figure 4.2: An enhanced state machine diagram to analyze and classify test scenario’s concurrency behavior
States: ESMD has nine states {Initial, SafeThread, SafeRead, SafeWrite, UnsafeR81
4. Test Scenario Classification for Concurrent Behaviors ead, UnsafeWrite, SharedRead, DataRace, Blocked}. These states are described as follows: 1) Initial: The Initial state is a virgin state, which indicates the shared data are new and have not yet been accessed by any thread. 2) SafeThread: Indicates safe access to shared data inside a critical region by acquiring the required locks. 3) SafeRead: Indicates safe read access to shared data inside a critical region. 4) SafeWrite: Indicates safe write access to shared data inside a critical region. 5) UnsafeRead: Indicates unsafe read access to shared data outside a critical region. 6) UnsafeWrite: Indicates unsafe write access to shared data outside a critical region. 7) SharedRead: Indicates unsafe read access to shared data by two threads outside a critical region. 8) DataRace: Indicates unsafe write access to shared data by two threads outside a critical region. 9) Blocked: Indicates thread is blocked to acquire lock on shared data, which is already locked by another thread.
Events: ESMD has six events {csStart, csEnd, read/CS, read/NOCS, write/CS, write/NOCS}. We consider two conditions on read and write events: i) Thread Switch - CS and ii) No Thread Switch (NOCS) - NOCS. Events are described as follows: 1) csStart: Represents critical region start. 2) csEnd: Represents critical region end. 3) read/CS: Indicates read access by another thread with a thread switch. 4) read/NOCS: Indicates read access by the same thread. 5) write/CS: Indicates write access by another thread with a thread switch. 6) write/NOCS: Indicates write access by the same thread. Read/write on non-shared data are not considered as events in the above mentioned read/write events. Transitions: State transition table for ESMD is shown in Table 4.1, where X indicates do not care event. 82
4.2. State Machine Based Test Scenario Analysis Table 4.1: State transition table for ESMD XX
XXX Event csStart XXX State XX Initial SafeThread SafeThread Blocked SafeRead Blocked SafeWrite Blocked UnsafeRead X UnsafeWrite X SharedRead X Data-race X Blocked X X - do not care event
csEnd
read/CS
read/NOCS write/CS
write/NOCS
X Initial Initial Initial X X X DataRace Blocked
UnsafeRead UnsafeRead SharedRead DataRace SharedRead DataRace SharedRead DataRace Blocked
UnsafeRead SafeRead SafeRead SafeWrite UnsafeRead UnsafeWrite SharedRead DataRace Blocked
UnsafeWrite SafeWrite SafeWrite SafeWrite UnsafeWrite UnsafeWrite DataRace DataRace Blocked
UnsafeWrite UnsafeWrite DataRace DataRace DataRace DataRace DataRace DataRace Blocked
There is a transition from Saf eW rite state to Saf eW rite state for the event read/N OCS because both read and write operations belong to the same thread. The transition from U nsaf eRead state to SharedRead state for event read/CS says that read operations by more than one thread without acquiring locks on shared data are possible. Inside a parallel fragment, a critical-section is used to access shared data. In order to realize a critical section, it is the responsibility of a programmer to access shared data in isolation. Isolation is achieved in implementations usually with locks on shared data items. Therefore, a programmer shall enforce the synchronization primitives to shared data access inside a critical region, i.e., a proper locking mechanism shall be implemented for instructions inside the critical region for atomic access to shared data in the implementation. Shared data access without using critical region is unsafe. A write-access, or read-access by one thread followed by a write-access by another thread leads to data race. ESMD considers only two concurrent threads because interleaving coverage criteria considered in the thesis are for thread-pair-interleaving. For example, If a test scenario has four concurrent threads, where threads T1 and T2 share the only data d12 and thread T3 and T4 share the only data d34 then threads T1 and T2 and threads T3 and T4 work separately in ESMD as they share separate data. Mixing of two separate data items by four different threads is not considered in the figure. As discussed in Section 4.1, UML designs shall include read-tags and write-tags for shared object access. These tags are used to generate data-access traces by test 83
4. Test Scenario Classification for Concurrent Behaviors scenario generators. There are two ways to get data-access traces: the first method, where data-access traces are generated by analyzing test scenarios, and the second method, where data-access traces are generated simultaneously with test scenarios generation. We have used the second method. In the second method, data-access traces are generated by test scenario generation algorithms at the time of generating test scenarios. The generated traces are given input to ESMD executor. The execution of the state machine diagram starts at Initial state. After processing the data access trace the executor returns the final state of the execution. If the final state is Data Race then a test scenario is returned with a data-race tag. If the final state is Blocked then a test scenario is returned with a blocking tag. If the final state is Initial then a test scenario is returned with a serialized tag. Otherwise, a test scenario is returned with a non data-race tag. The mapping of test scenario interleaving classes and concurrent behaviors is as follows: Sequentialized interleaving test scenarios exhibit serialized concurrent behaviors. Blocked interleaving test scenarios exhibit blocked concurrent behaviors. A few blocking interleaving test scenarios exhibit data-race concurrent behaviors. Serialized interleaving and some blocking interleaving test scenarios exhibit non data-race concurrent behaviors.
4.3
Experimental Results
We have implemented three algorithms for test scenario generation. The first algorithm is DFS-BFS by Kundu and Samanta (2009), and other two algorithms are DP-DFS and evolutionary algorithms. The DP-DFS algorithm is proposed by us to generate test scenarios from sequence diagrams in Chapter 8. The evolutionary algorithm is proposed by us to generate test scenarios from activity diagrams in Chapter 7. These algorithms handle concurrency constructs and generate interleaving test scenarios. We have implemented state machine executor (SMExecutor) in Java that loads and executes a state machine from the text file. The SMExecutor accepts input in terms of a sequence of events like data-access trace. For each event of the input trace the SMExecutor checks matching transitions from the current state. If matching transition is found and guard condition is satisfied then state is changed, otherwise not. Finally, current state is returned by the SMExecutor when the input sequence finishes. Test scenario generation algorithms utilize SMExecutor to classify the gener84
4.3. Experimental Results ated test scenario. SMExecutor implementation details are given in the Appendix E that describes SMExecutor class diagram and input text file format. All experiments are conducted using 3.00 GHz Intel Core 2 Quad CPU with 4 GB RAM desktop computer. To carry out a detailed study of the proposed approach, we have chosen six sequence diagrams and four activity diagrams. These diagrams contain concurrency constructs to represent concurrency in applications. All applications depict different aspects of design. Their attribute details are listed in Table 4.2. We have generated test scenarios DFS-BFS, DP-DFS, and evolutionary algorithms. Table 4.2: UML diagrams’ attributes: application name, diagram type, data access type, and total interleaving paths for different applications Application number App 1 App 2 App 3 App 4 App 5 App 6 App 7 App 8 App 9 App 10
Application
UML diagram type
Data access
System startup Map rendering Multi-view data display ATM money withdraw Graphics utility Bank money transfer Airport check in ATM Order processing Graphics Utility
Sequence diagram Sequence diagram Sequence diagram Sequence diagram Sequence diagram Sequence diagram Activity diagram Activity diagram Activity diagram Activity diagram
– read read read, read, read, – read read, read,
write write write
write write
# Interleaving paths 070 010 070 070 056 252 007 011 032 056
The diagrams are transformed into intermediate representations required by the respective test scenario generator. We have used a control flow graph-like intermediate representation for DFS-BFS test scenario generator, a message control flow graph intermediate representation for DP-DFS test scenario generator, and activity control flow graph intermediate representation for evolutionary test scenario generator. DFS-BFS and DP-DFS approaches are used to generate test scenarios from sequence diagrams. DFS-BFS and evolutionary algorithms are used to generate test scenarios from activity diagrams. Results in Table 4.3 show the classification of test scenarios using existing approach by Lei et al. (2008). The table includes the total number of test scenarios generated, classification of test scenario: non-data race, serializing, blocking and data race classes, and false classification count. Results indicate correct classification of test scenarios for applications App1, App2, and App3. This signifies Lei et al.’s approach is robust for shared read operation and does not miss-classify test scenarios as data race. However, it miss-classifies some of 85
4. Test Scenario Classification for Concurrent Behaviors Table 4.3: Concurrent behavior classification using existing state machine by Lei et al. (2008) approach for test scenarios generated using DFS-BFS, EA, and DP-DFS algorithms
Application Algorithm number App 1 App 2 App 3 App 4 App 5 App 6 App 7 App 8 App 9 App 10
DFS-BFS DP-DFS DFS-BFS DP-DFS DFS-BFS DP-DFS DFS-BFS DP-DFS DFS-BFS DP-DFS DFS-BFS DP-DFS DFS-BFS EA DFS-BFS EA DFS-BFS EA DFS-BFS EA
Total
2 61 2 10 2 64 2 64 2 50 2 146 3 7 6 9 2 3 0 13
# Scenarios Serializing Blocking
Non Datarace 2 61 2 10 2 64 – – – – – – 3 7 6 9 2 3 – –
– – – – – – – – – – – – – – – – – – – –
– – – – – – – – – – – – – – – – – – – –
Datarace – – – – – – 2 64 2 50 2 146 – – – – – – – 13
Falseclassification – – – – – – 2 64 – 5 2 146 – – – – – – – 2
the test scenarios for applications App4, App5, and App6 as shown in the Table 4.3. For App4, the state machine diagram miss-classifies all the scenarios. Generated test scenarios are all valid non data-race scenarios because their read and write operation belongs to the same thread. But, they are miss-classified. For App5, the state machine diagram miss-classifies some of the scenarios as data race scenarios. For example, five test scenarios generated by DP-DFS algorithm are mis-classified as data-race out of 50 test scenarios. For App6, the state machine diagram miss-classifies all test scenarios as data race. But, these scenarios are blocking and serialize scenarios. For activity diagrams, it is difficult to find data-race and blocking scenarios because it does not have sufficient concurrency constructs to specify all concurrent behaviors. Apart from classification result, we come to know that DFS-BFS algorithm generates fewer interleavings. On the contrary, DP-DFS algorithm generates a sufficient number of interleavings test scenarios so as to select representative test scenarios from different 86
4.3. Experimental Results classes. After these results, we have applied our proposed ESMD classifier to analyze same test scenarios. Here, we first illustrate the sample input trace and the transitions for ESMD classifier shown in Figure 4.2 for Graphics Utility application (Figure 1.1). In order to complete the analysis, we have inserted critical region start and end activities around the activities which write shared data, i.e., setCoordinates(x1,y1) and setCoordinates(x2,y2) activities. Consider following two test scenarios: Test Scenario-1 and Test Scenario-2. Test Scenario-1: sdStart → initialize → par1Start → t2.process2() → CriticalRegion2Start −−→ t1.process1() −−→ t2.setCoordinates(x2, y2) −−→ CS CS CS CriticalRegion1Start −−→ CriticalRegion2End → t2.getXCoordinate() → CS t2.getY Coordinate() −−→ t1.setCoordinates(x1, y1) → CriticalRegion1End → CS t1.useCoordinate() −−→ t2.usecoordinates() → par1End → process3 → sdEnd, and CS Test Scenario-2: sdStart → initialize → par1Start → t2.process2() → CriticalRegion2Start → t2.setCoordinates(x2, y2) −−→ t1.process1() −−→ CS CS CriticalRegion2End → t2.getXCoordinate() −−→ CriticalRegion1Start −−→ CS CS t2.getY Coordinate() −−→ t1.setCoordinates(x1, y1) → CriticalRegion1End −−→ CS CS t2.usecoordinates() −−→ t1.useCoordinate() → par1End → process3 → sdEnd. CS Their corresponding data-access traces are access-trace-1 and access-trace-2, where each event is separated by a ’#’ symbol. Access-trace-1: csStart/NOCS#write/NOCS#csStart/CS#csEnd/CS#read/NOCS# read/NOCS#write/CS#csEnd/NOCS, and Access-trace-2: csStart/NOCS#write/NOCS#csEnd/NOCS#read/NOCS# csStart/CS#read/CS#write/CS#csEnd/NOCS. Table 4.4 shows detailed transitions from initial sate to the last state on each input event. ESMD processes input data access trace Access-trace-1 of test scenario Test Scenario-1 to output Blocked State. Similarly, Table 4.5 shows detailed transitions from initial sate to the last state on each input event. ESMD processes input data access trace Access-trace-2 of test scenario Test Scenario-2 to output Data-Race State. The results are shown in Table 4.6 for ESMD classifier for 10 applications listed in Table 4.2. The table includes the total number of test scenarios generated, classification of test scenarios: non-data race, serializing, blocking, and data race test scenarios, and false classification count. For all applications, ESMD correctly classifies the test scenarios as shown in Table 4.6. The ESMD is capable of classifying test 87
4. Test Scenario Classification for Concurrent Behaviors Table 4.4: Transition table of input data access trace Access-trace-1 for ESMD classifier Sr. No. 1 2 3 4 5 6 7 8
Event csStart/NOCS write/NOCS csStart/CS csEnd/CS read/NOCS read/NOCS write/CS csEnd/NOCS
Current State Initial SafeThread Safe Write Blocked Blocked Blocked Blocked Blocked
New State SafeThread Safe Write Blocked Blocked Blocked Blocked Blocked Blocked
Table 4.5: Transition table of input data access trace Access-trace-2 for ESMD classifier Sr. No. 1 2 3 4 5 6 7 8
Event csStart/NOCS write/NOCS csEnd/NOCS read/NOCS csStart/CS (Invalid Event) read/CS write/CS csEnd/NOCS
Current State Initial SafeThread SafeWrite Initial UnsafeRead UnsafeRead SharedRead DataRace
New State SafeThread SafeWrite Initial UnsafeRead UnsafeRead SharedRead DataRace DataRace
scenarios as non data-race, serialize, blocking, and data race. It provides a correct set of test scenarios for concurrent behavioral testing for the different type of concurrency errors. Proposed ESMD classifier takes care of misclassification of the test scenarios. In summary, ESMD correctly classifies all test scenarios in all applications. Existing state machine by Lei et al. (2008) is not able to classify test scenarios as blocking and serializing. The experiment also suggests the use of interleaving exploring algorithms, such as DP-DFS and evolutionary, is important to generate sufficient test scenarios to get the correct set of test scenarios for each class of concurrency behaviors.
4.4
Conclusions
This chapter has presented an approach to analyze and classify of test scenarios for concurrent behaviors. First, we have presented a way to represent data-access tags to 88
4.4. Conclusions Table 4.6: Concurrent behavior classification using enhanced state machine diagram for test scenarios generated by DFS-BFS, EA, and DP-DFS algorithms
Application Algorithm number App 1 App 2 App 3 App 4 App 5 App 6 App 7 App 8 App 9 App 10
DFS-BFS DP-DFS DFS-BFS DP-DFS DFS-BFS DP-DFS DFS-BFS DP-DFS DFS-BFS DP-DFS DFS-BFS DP-DFS DFS-BFS EA DFS-BFS EA DFS-BFS EA DFS-BFS EA
Total
2 61 2 10 2 64 2 64 2 50 2 146 3 7 6 9 2 3 0 13
Non Datarace
# Scenarios Serializing Blocking
2 61 2 10 2 64 2 64 – 5 – – 3 7 6 9 2 3 – –
– – – – – – – – – – – 8 – – – – – – – 2
– – – – – – – – – – 2 138 – – – – – – – –
Datarace – – – – – – – – 2 45 – – – – – – – – – 11
Falseclassification – – – – – – – – – – – – – – – – – – – –
UML diagrams. These tags are used to analyze and classify test scenarios for concurrent behaviors. Then, we have proposed enhanced state machine based approach to classify test scenarios for concurrency testing to overcome limitations of the existing state machine based classifier. Experimental results show that the proposed enhanced state machine diagram correctly classifies test scenarios. The experiment also suggests the use of interleaving exploring algorithms play an important role in generating the enough number of test scenarios for concurrency testing. In the next chapter, we present translation of sequence diagrams into activity diagrams and classification of activity diagrams.
89
Chapter 5 Translation of Sequence Diagrams into Activity Diagrams and Classification of Activity Diagrams
UML design models consist of a number of diagrams describing static and dynamic views of systems Fowler (2004). Dynamic views of systems are represented by the behavioral diagrams such as sequence diagrams, activity diagrams, and state machine diagrams. Sequence diagrams represent a flow of messages among objects that participate in functionality in a timely order. However, the visual and internal representation of sequence diagrams is not similar to a graph. Graph theoretic algorithms find difficulty for generating test cases from sequence diagrams. On the other hand, activity diagrams depict flows of activities that include decisions, iterations and concurrent actions. Activity diagrams represent a control flow like structure, and they are easy to process by graph theoretic algorithms. Therefore, we thought to have common representation of sequence and activity diagrams by translating sequence diagrams into activity diagrams. The translation eases the process of test scenario generation from both behavioral diagrams using graph theoretic approaches. It also explores relationship between different control structures of both the activity diagrams and sequence diagrams. Activity diagrams have a comprehensive structure for sequential and concurrent control flows to represent sequence diagrams. Activity diagrams consist of sequential and concurrent flows. A sequential flow is a simple sequence, decisions and iterations, or nesting of sequential flows. Similarly, 91
5. Translation of Sequence Diagrams into Activity Diagrams and Classification of Activity Diagrams concurrent flows are parallel flows of simple or nested sequential flows, or nesting of concurrent flows. Nested concurrent flows increase the complexity of activity diagrams for most of the traversing algorithms, thereby increase the difficulty for test scenario generators (Xu et al., 2005). There is a necessity to classify activity diagram to understand the complexity of processing them, and thereby planning test efforts. As a result, we have proposed classification of activity diagrams on the basis of nesting of sequential structural constructs and concurrency constructs. We classify activity diagrams as (i) non-nested activity diagrams, (ii) nested activity diagrams, (iii) nonnested concurrent activity diagrams, and (iv) nested concurrent activity diagrams. Note that concurrent activity diagrams means activity diagrams consisting of forkjoin constructs. The classification is helpful for assessing test scenario generation algorithms, selecting an appropriate test scenario generator and coverage criteria. The terms event/task at requirement stage, message/activity at design stage, and function/method at implementation stage are used in same reference. We use these terms interchangeably in this chapter. This chapter is organized as follows. First, we describe an informal procedure to translate sequence diagrams into activity diagrams. The translation process is described in detail for sequence diagrams’ objects, messages, sequential structural combined fragments, and parallel combined fragments. Finally, we present the classification of activity diagrams based on nesting of sequential and concurrent structures/ constructs.
5.1
Translation Method for Sequence Diagram
In this section, we discuss informal description of the translation process that converts sequence diagrams into activity diagrams. Sequence diagrams consist of lifelines that denote the existence of objects during the system execution (Jacobson, 1992). Activity diagrams represent existence of objects by using partitions. In the translation process, first objects represented by lifelines of sequence diagrams are converted into swimlane partitions representing objects in activity diagrams. A message in sequence diagrams represents an execution of a function/ method by the object that receives it. In activity diagrams, activities represent a function/ method. In the translation process, messages in sequence diagrams are translated into activities in activity diagrams. Sequence diagrams represent control flow structures using combined fragments. In the 92
5.1. Translation Method for Sequence Diagram translation process, different combined fragments of sequence diagrams are translated into control nodes of activity diagrams. The detailed procedure of translation for synchronous message, asynchronous message, and combined fragments such as alt, opt, loop, and par is discussed as follows. Interaction Frame- An Interaction Frame is a set of lifelines and a sequence of events among the lifelines representing a sequence diagram. Interaction frames are translated into activity diagrams. The start of an interaction frame is represented by start node, and end of the interaction frame is represented by the end of the activity diagram. All messages and combined fragments of the sequence diagram are translated into activity nodes and control nodes of the activity diagram. The first message invocation of the sequence diagram depicts the control flow between the start node and the first activity of the activity diagram. The last reply message of the sequence diagram depicts the control flow between the last activity node and the end node of the activity diagram. The Figure 5.1 shows the translation of a sequence diagram interaction frame into the corresponding activity diagram. Synchronous message - A synchronous message is represented by two arrows: the message invocation arrow and the message reply dotted arrow. The first arrow invokes the action (operation, function), while the second arrow is the reply action, which returns data and control back to the invoking object. Both the arrows are represented with filled arrowhead arrows. A synchronous message is represented by an activity in the activity diagram as shown in Figure 5.1. Message invocation in a sequence diagram depicts the control flow between the sender object and the receiver object, so in the activity diagram a message invocation is represented as an edge between the previous activity and the current activity. When synchronous message is executed, there is no delay for reply message. Therefore, only one activity is shown for a pair of arrows representing synchronous messages. Next message on the sequence diagram decides the control flow in the activity diagram after current activity. Asynchronous message - An asynchronous message does not wait for a reply from the called object. The calling object starts its next message immediately. Asynchronous calls are present in multithreaded applications. An asynchronous message and it’s reply are represented by an open arrowhead in sequence diagrams. An asynchronous message introduces concurrent execution of the calling and the called object processes. Therefore, asynchronous message in sequence diagrams is translated as a f ork − join construct in activity diagrams. An arrow representing the asynchronous message in the sequence diagram is modeled as the f ork node in the activity dia93
5. Translation of Sequence Diagrams into Activity Diagrams and Classification of Activity Diagrams
O1
Sd: synchronousMessage
O1: Object1
O2:Object2
O2
Sd: asynchronousMessage
m1
O1
O2
O2:Object2
O1: Object1
1: m1()
1: m1() 2: m1
2: m2()
m1
m2
3: m2 3: m2() 4: m2
4: m1
m2
Figure 5.1: A translation method for synchronous messages of sequence diagrams
Figure 5.2: A translation method for asynchronous messages of sequence diagrams
gram. An arrow representing the reply message of the asynchronous message in the sequence diagram is modeled as the join node in the activity diagram. There are two threads. One thread shows the control flow of the called object process and another thread shows the control flow of the calling object process as shown in Figure 5.2. In the sequence diagram of Figure 5.2, message m1 is an asynchronous message shown with open arrowhead; it will complete either before or after m2. If a synchronous message is sent, then the calling object must wait until the message is done. If an asynchronous message is sent, then the calling object can continue processing and does not have to wait for a response. Message m1 will complete either before m2 or after m2 based on the execution time of m2. This is modeled in the activity diagram as shown in the figure. Alternative Fragment - An alternative combined fragment in sequence diagrams represents a choice that has multiple operands, where each operand is guarded by a guard condition. In alternative fragment, one operand is executed, whose guard condition is satisfied first. If guard condition is not satisfied then default else operand is executed. An alternative fragment in the sequence diagram is represented as a decision − node in the activity diagram. Figure 5.3 shows the translation of an alternate combined fragment of the sequence diagram into the decision construct of the activity diagram. All messages of operands in the alternative fragment are modeled as sequential flows, and these sequential flows are attached to each branch of 94
5.1. Translation Method for Sequence Diagram
O1
Sd: AlternativeFragment
O1: Object1
O2:Object2
O3:Object3
O2
O3
m1
1: m1() 2: m1
O1
Sd: OptionalFragment
O1: Object1
O2:Object2
O3:Object3
O2
m1
1: m1()
[true]
O3
[true]
2: m1 [false] Alt [true]
m3
[false]
m2
Opt [true]
3: m2()
3: m2()
4: m2
4: m2
m2
[false]
5: m3()
5: m3()
6: m3
6: m3
Figure 5.3: A translation method for alternative fragments of sequence diagrams
m3
Figure 5.4: A translation method for option fragments of sequence diagrams
the decision-node. All sequential flows begin in the decision-node and end in the merge-node. A condition expression associated with each operand in the alternative fragment is attached to the respective branch of the decision-node. Option Fragment -An option fragment has one operand, and it is executed if the condition is satisfied. Option fragment is similar to the alternative fragment with a single choice and empty else operand. The translation process is similar to the alternative fragment. Figure 5.4 shows the translation of the optional fragment in the sequence diagram into the decision-node in the activity diagram. Condition’s true expression is attached to the optional fragment’s sequential flow between decisionnode and merge-node. Condition’s false expression is attached to the decision-node branch directly entering into merge-node. Loop Fragment - A loop fragment represents iterative execution of the operand in the fragment. This fragment has two parameters minint and maxint. The operand is processed at least minint times and at most maxint times. The operand is processed when an operand condition is satisfied, and it is not processed when an operand condition is not satisfied, or minint is zero. A loop fragment in sequence diagrams is represented as a decision in activity diagrams. The loop operand’s sequential flow is placed in the true branch of the decision-node, and outflow from last activity of the sequential flow is merged back into the decision-node. A message sequence after loop fragment is attached to the f alse branch of the decision-node in the activity diagram 95
5. Translation of Sequence Diagrams into Activity Diagrams and Classification of Activity Diagrams
O1
Sd: loopFragment
O1: Object1
O2
m1
O2:Object2 1: m1()
O1
Sd: ParallelFragment
O1: Object1
O2:Object2
O3:Object3
O2
O3
m1
1: m1()
[false]
2: m1 [true]
2: m1
m2
Par
m2
3: m2()
Loop
m3
4: m2
3: m2() m3 4: m2
5: m3() 6: m3
5: m3()
m4
7: m4() 8: m4
Figure 5.5: A translation method for loop fragments of sequence diagrams
Figure 5.6: A translation method for parallel fragments of sequence diagrams
as shown in Figure 5.5. The loop operand’s condition expression is attached to the decision-node’s true branch having sequential flow, and the negation of condition expression is attached to the decision-node’s false branch. Parallel Fragment - A parallel fragment is used for concurrent/parallel execution of operands. Messages in each operand keep their order of execution, while messages among the operands interleave in any order. A parallel fragment in sequence diagrams is translated into a fork-join construct in activity diagrams. The start of the parallel fragment is represented as the f ork construct, and the end of the parallel fragment is represented as the join construct in activity diagrams. For each operand, one thread is created. Operands’ message flows of the parallel fragment are translated into respective threads’ flows of the fork-join construct as shown in Figure 5.6. General Ordering - In parallel fragments, a causal order between messages is shown by general ordering construct in sequence diagrams. A general ordering is represented by a dotted-line from one operand to another operand in sequence diagrams. Some messages in different operands may be related; therefore, they should execute in the specific order. This causal relationship is expressed by a dashed line with an arrow inside parallel fragments in sequence diagrams. In an activity diagram, general ordering construct is translated into send-signal and receive-signal constructs as shown in Figure 5.7. First the translation of parallel fragment is done as discussed above in the parallel fragment. Then, the start of general ordering dashed line is translated 96
5.2. Classification of Activity Diagrams
O1
Sd: General Ordering O2:Object2
O1: Object1
O3:Object3 m1
Par
3: m2() 4: m2
O2
1: m1() 2: m1
5: m3() 6: m3
7: m4()
8: m4
m2
m4
s1
r1
m3
m5
Signal: S1
O3
9: m5() 10: m5
m6
11: m6() 12: m6
Figure 5.7: A translation method for general ordering inside the parallel fragment of sequence diagrams
into the send-signal, and the end of general ordering dashed line is translated into the receive-signal. Sequence diagrams are directly translated into activity diagrams with the above process. After that, test scenarios can be generated easily using graph theoretic approaches. However, combined fragments like critical region are difficult to represent using activity diagrams, but with additional locking activities for the critical region translation will be easy. Activity diagrams include different types of nesting inside sequential and concurrent structures. Classification of activity diagrams need to be understood, so that, one can selects an appropriate test scenario generator algorithm for generating test cases.
5.2
Classification of Activity Diagrams
Activity diagrams define logic of complex operations and describe work-flow of the system (Fowler, 2004). They support sequential as well as concurrent flows for system design. Exploring execution interleavings in nested activity diagrams is harder than non-nested (sequential/serial) activity diagrams. Xu et al. (2005) have classified concurrent activity diagrams on the basis of nesting inside the fork-join structure as atomic fork-join, simple fork-join, simple nested fork-join diagrams and branch nested fork-join diagrams. An atomic fork-join diagram contains only two threads, and a 97
5. Translation of Sequence Diagrams into Activity Diagrams and Classification of Activity Diagrams simple fork-join diagram contains more than two threads. A simple nested fork-join diagram consists of nesting of fork-join inside simple fork-join diagrams. A branch nested fork-join diagram consists of nesting of decision structures inside the fork-join. Similarly, in this chapter we classify activity diagrams as 1) non-nested activity diagrams, 2) nested activity diagrams, 3) non-nested concurrent activity diagrams, and 4) nested concurrent activity diagrams. Our classification may be used for the purpose of analyzing the efficacy of test scenario generation algorithms. In next subsections, we describe the classification of activity diagrams with suitable examples.
5.2.1
Non-Concurrent Activity Diagrams
Activity diagrams (non-concurrent) do not include fork-join structures. On the basis of the combination of sequential structures, these diagrams are further classified as non-nested activity diagrams and nested activity diagrams. Test scenarios generated from these activity diagrams satisfy activity coverage, basic path coverage, and total activity path coverage criteria. Non-nested activity diagrams consist of decision and iteration structures in a sequential way. For example, a decision structure is followed by another decision structure, or a decision structure is followed by an iteration structure and vice versa as shown in the Figure 5.8 and Figure 5.9. The non-nested S a1 D1 a3
a2 M1 a4 D2/ M2 a5 a6 a7 E
Figure 5.8: SDD: Non-nested (serial) decision-decision activity diagram
Figure 5.9: SDI: Non-nested (serial) decision-iteration activity diagram
decision-decision activity diagram (SDD) is shown in Figure 5.8, which includes 16 98
5.2. Classification of Activity Diagrams
Figure 5.11: NDI: Nested decisioniteration activity diagram
Figure 5.10: NDD: Nested decisiondecision activity diagram
nodes: 10 activities and 6 control nodes, six basic paths, six simple paths, and six total activity paths. The non-nested decision-iteration activity diagram (SDI) is shown in Figure 5.9, which includes 12 nodes: 7 activities and 5 control nodes, four basic paths, four simple paths and four total activity paths. Nested activity diagrams consist of nesting of decision and iteration structures, for example, a decision structure is a part of another higher level decision structure, or iteration is a part of another higher level decision structure and vice versa as shown in Figure 5.10 and Figure 5.11. The nested decision-decision activity diagram (NDD) is shown in Figure 5.10, which includes 18 nodes: 10 activities and 8 control nodes, four basic paths, four simple paths, and four total activity paths. The nested decisioniteration activity diagram (NDI) is shown in Figure 5.11, which includes 17 nodes: 10 activities and 7 control nodes, four basic paths, four simple paths and four total activity paths.
5.2.2
Concurrent Activity Diagrams
Concurrent activity diagrams include fork-join structures. On the basis of the combination of fork-join and decision structures, these diagrams are classified as non-nested concurrent activity diagrams and nested concurrent activity diagrams. Test scenarios generated from these activity diagrams satisfy activity coverage, basic path coverage, simple path coverage, interleaving activity path coverage, and total activity path 99
5. Translation of Sequence Diagrams into Activity Diagrams and Classification of Activity Diagrams coverage criteria. Non-nested concurrent activity diagrams consist of decision and fork-join structures in a sequential way. For example, a decision/ iteration structure is followed by a fork-join structure, or a fork-join structure is followed by another fork-join structure and vice versa as shown in Figure 5.12 and Figure 5.13. The non-
Figure 5.12: SDF: Non-nested (serial) decision-fork concurrent activity diagram
Figure 5.13: SFF: Non-nested (serial) fork-fork concurrent activity diagram
nested decision-fork activity diagram (SDF) is shown in Figure 5.12, which includes 17 nodes: 11 activities and 6 control nodes, two simple paths, 180 interleaving activity paths and 180 total activity paths. The non-nested fork-fork structure diagram (SFF) is shown in Figure 5.13, which includes 18 nodes: 12 activities and 6 control nodes, one simple path, 60 interleaving activity paths, and 60 total activity paths. Nested concurrent activity diagrams consist of nesting of decision/iteration and fork-join structures; for example, a decision/iteration structure is a part of a higher level fork-join structure, or a fork-join structure is a part of another higher level fork-join structure and vice versa as shown in Figure 5.14 and Figure 5.15. The nested decision-fork-decision activity diagram (NDFD) is shown in Figure 5.14, which includes 17 nodes: 9 activities and 8 control nodes, three simple paths, 12 interleaving activity paths, and 13 total activity paths. The nested decision-fork-fork activity diagram (NDFF) is shown in Figure 5.15, which includes 18 nodes: 10 activities and 8 control nodes, two simple paths, 24 interleaving activity paths, and 25 total activity paths. Proposed classification can be used as a benchmark to assess the new test scenario 100
5.2. Classification of Activity Diagrams
Figure 5.14: NDFD: Nested decisionfork-decision concurrent activity diagram
Figure 5.15: NDFF: Nested decisionfork-fork concurrent activity diagram
generation algorithms. It gives a rough estimation for testing efforts. Number of test scenarios are calculated as follows: For a decisions construct, there are two exit entries, so to achieve decision coverage at least two test cases are required. For a loops construct, there are two exit entries and many possible execution paths, but for loop coverage two test cases are sufficient, looping and not looping (Beizer, 1990). For a fork-join construct, from fork there are more than one exit entry and each represents a parallel execution up to join. For a fork-join structure, a number of possible execution paths result in an exponential function of the number of activities in each flow up to join. Consider a fork-join structure has m threads. Let |T | is a P number of activities in thread T . Then total activities in all m threads are m i=1 (|Ti |). Total execution interleaving sequences (TEIS) for fork-join is P ( m (|Ti |))! T EIS = Qi=1 m i=1 (ni !) At this moment, we do not know how many minimum interleavings may uncover concurrency errors. Existing test case generation approaches, such as Mingsong et al. (2006) and Kundu and Samanta (2009), promote single interleaving path for testing. However, a single interleaving path is insufficient for concurrency testing as demonstrated in the motivating examples in Chapter 1. 101
5. Translation of Sequence Diagrams into Activity Diagrams and Classification of Activity Diagrams
5.3
Conclusions
Our sequence diagram to activity diagram UML model translation procedure is simple and straightforward to convert sequence diagrams into activity diagrams. The proposed classification is useful for assessment of test scenario generator algorithms. Examples presented in the classification can be used as a testbed for test scenario generator algorithms to carry out a detailed study. After knowing the different classes of activity diagrams, we propose test scenario generation algorithms from UML concurrent designs. Next chapter presents two techniques for generating interleaving test scenarios for non-nested concurrent activity diagrams.
102
Chapter 6 Test Scenario Generation from Non-nested Concurrent Activity Diagrams
Concurrent defects are difficult to uncover and often remain undetected even after the product deployment (Edelstein et al., 2002). Concurrent programs have a degree of non-determinism that sequential programs do not have. As a result, testing concurrency errors using traditional techniques is a challenging task. A test case should not only provide a bug fixing input, but also provide an execution interleaving (test scenario) that expose concurrency bugs (Kundu et al., 2010). Concurrency errors such as data-race, synchronization, and deadlock need to be exposed by exploring appropriate interleavings from concurrent designs. Concurrency constructs available in activity diagrams, sequence diagrams, and state machine diagrams should be utilized for test scenario generation for concurrency testing. Generally programmers are used to sequential thinking and frequently assume code region to be atomic without proper enforcement, this leads to atomicity-violation bugs (Lu et al., 2012). Recent study on the programming errors in parallel programming by S¨ ußand Leopold (2008) indicates that higher percentage of errors observed in a) access to shared variables not protected and b) use of locks without flush. In such cases, predetermined interleaving test scenario designed from UML concurrent design will help in uncovering some of the errors introduced by programmers. Recent approaches presented by Chandler et al. (2005); Kim et al. (2007); and Sun et al. (2009) generate test case from activ103
6. Test Scenario Generation from Non-nested Concurrent Activity Diagrams ity diagrams that use concurrent fork-join constructs. Despite utilizing concurrency constructs, the above approaches do not address on concurrency errors. We have proposed two approaches, which explore concurrent behavior present in the design to generate test scenarios. Such scenarios may trigger concurrency bugs and maximize concurrent behavioral coverage. The first approach is called concurrent queue search, which generates interleaving test scenarios for uncovering concurrency bugs. A simple heuristic is used to add a thread switch at specific locations amongst threads to generate test scenarios. The second approach is called DFS-LevelPermute, which is based on constrained permutations that generate interleaving test scenarios to maximize concurrency coverage. Both CQS and DFS LevelPermute achieve 100% basic path coverage and partial (i.e., average 12.5%) concurrency coverage for non-nested concurrent activity diagrams. Note that non-nested concurrent activity diagrams do not consist of any type of nesting inside fork-join constructs. In summary, CQS and DFS LevelPermute algorithms generate test scenarios, which may trigger bug, and improve concurrency coverage for non-nested concurrent activity diagrams. This chapter is organized as follows. Section 6.1 presents the concurrent queue search test scenario generation approach. Section 6.2 describes constrained permutation based test scenario generation approach. Section 6.3 presents results of the test scenario generators. Finally, Section 6.4 concludes the chapter.
6.1
Concurrent Queue Search Technique
In graph theoretic approaches, test scenario generator approaches have proposed algorithms like modified-DFS by Chandler et al. (2005), DFS-like by Nayak and Samanta (2010), and DFS-BFS by Kundu and Samanta (2009) to handle concurrency in UML designs. Test scenarios generated by the above approaches are suitable for structural coverage like activity and transition; and sequential behavioral coverage like basic path. Some of the approaches achieve very low (i.e., average 10%) concurrent behavioral coverage like interleaving activity path. Hence, generated test scenarios mostly do not address the concurrency errors. We have proposed a DFS-Like concurrent queue search algorithm to mitigate above the limitations. CQS algorithm is designed for non-nested concurrent activity diagrams for generating interleaving test scenarios. Implementation detials of CQS algorithm are given in Appendix C that describes a class diagram and an activity diagram graph (ADG) text file format. The detailed 104
6.1. Concurrent Queue Search Technique procedure of the CQS algorithm is described as follows. CQS algorithm has two phases that are listed in Algorithm 6.1. In each run, CQS algorithm generates scenarios from the start to the end of the activity diagram in two phases. Algoritm accepts input non-nested activity diagram’s, with two way decision, ADG and number of test scenarios. In the first phase (line numbers 1–34), CQS algorithm constructs a queue structure for each thread of execution, i.e., for main program and each forked thread. A queue structure consists of a mainQueue for the main thread of execution and multiple subQueues for threads generated by the fork construct. Some control nodes, such as decision (iteration) and fork, are processed differently as explained below. Activity nodes and other control nodes are enqueued from the start to the end of an activity diagram in respective queues. Loop nodes are shown by decision nodesin activity diagrams, where decision node may act as merge node for loops. This algorithm does not support more than two way decisions. For decision node (line numbers 8–18), activities in each conditional flow are appended one after another in the mainQueue. First, activities of the decision’s true control flow are enqueued up to the merge node including merge node, and after that activities of the decision’s f alse control flow are enqueued up to the merge node including merge node. Function getOutF lows(node) returns number of outflows associated with the node. Function getN ode() returns the next node in the graph. Function getN ode(N ode, i) returns the next node associated with ith outflow of the node. For fork node (line numbers 19–31), separate subQueues are generated for each thread (branch going out of the fork node). Unlike decision nodes that represent alternative flows, fork-join constructs represent every flow as a seperate execution therefore subQueues are generated for processing fork-join constructs as oppose to decision nodes. Then each subQueue is enqueued with activities in the respective thread up to the join node including join node. A separate array of queue is associated with each f ork node indentified by f orkId and outf low count. All other types of nodes are enqueued in the mainQueue (line numbers 32–34). The process is repeated up to end of the activity diagram. In the above process, every control flow of a fork-join construct represents a separate thread therefore subQueues are constructed for them; this is not true for decision constructs. Therefore, for thenpart and else-part of decision nodes subQueues are not created but their respective part is inserted in the main queue In the second phase, algorithm generates test scenarios. CQS algorithm dequeues activities and control nodes from active queues to generate a test scenario. CQS al105
6. Test Scenario Generation from Non-nested Concurrent Activity Diagrams Algorithm 6.1 Concurrent queue search (CQS) Input: ADG: Activity Diagram Graph, n: Number of test scenarios Output: T S: Array of test scenarios 1: mainQueue = empty-queue ; . First phase: create concurrent queues 2: subQueues[][] = null ; 3: Integer f orkId = 0 ; 4: Let T S be a string array of size n ; 5: repeat 6: node = ADG.getN ode(); 7: switch (node.type) do 8: case : “decision” . covers both if-test and loop-test 9: mainQueue.enqueue(node); 10: numOutf lows = ADG.getOutF lows(node); 11: tempN ode = node; 12: for (i = 0; i < numOutf lows; i + +) do . i=0 - corresponds to true branch 13: node = ADG.getN ode(tempN ode, i); 14: repeat 15: mainQueue.enqueue(node); 16: node = ADG.getN ode(); 17: until (node.type 6= “merge”) 18: end for 19: case : “f ork” 20: mainQueue.enqueue(node); 21: numOutf lows = ADG.getOutF lows(node); 22: tempN ode = node; 23: for (i = 0; i < numOutf lows; i + +) do 24: subQueues[f orkId][i] = empty − queue; 25: node = ADG.getN ode(tempN ode, i); 26: repeat 27: subQueues[f orkId][i].enqueue(node); 28: node = ADG.getN ode(); 29: until (node.type 6= “join”) 30: end for 31: f orkId + + ; 32: case : Default . handles activity and other control nodes 33: mainQueue.enqueue(node); 34: until (node.type 6= “end”) 35: for (i = 0; i < n; i + +) do . Second phase: generate test scenarios 36: T S[i] = getTestScenario(mainQueue, subQueues); 37: end for 38: return T S 106
6.1. Concurrent Queue Search Technique 1: function getTestScenario(mainQueue, subQueues) 2: Let path be an empty string; 3: Integer f orkId = 0 ; 4: repeat 5: node = mainQueue.dequeue(); 6: switch (node.type) do 7: case : “decision” 8: path.append(node); 9: if (node.decision == true) then 10: path.append(node) ; 11: repeat 12: node = mainQueue.dequeue(); 13: path.append(node) ; 14: until (node.type = 6 “merge”) 15: else . skip nodes upto merge node 16: repeat 17: node = mainQueue.dequeue(); 18: until (node.type 6= “merge”) 19: repeat 20: node = mainQueue.dequeue(); 21: path.append(node) ; 22: until (node.type = 6 “merge”) 23: end if 24: case : “f ork” 25: f orkF lows = getOutF lows(node); 26: path.append(node) ; 27: repeat 28: randomSubQueue = random(f orkF lows); 29: repeat 30: node = subQueues[f orkId][randomSubQueue].dequeue(); 31: if (node.type == releaseLock||sendSignal|| activity after receiveSignal ) then 32: if (node.visited() == true) then 33: path.append(node) ; 34: else 35: node.setV isited(true); 36: path.append(”threadSwitch”) ; 37: subQueues[f orkId][randomSubQueue].enqueue(node); 38: break; 39: end if 40: end if 41: until (node.type 6= “join”) 42: allQueuesOver = true ; 43: for (i = 0; i < f orkF lows; i + + do 44: if (!subQueues[f orkId][i].isEmpty()) then 45: allQueuesOver = f alse ; 46: end if 47: end for 48: until (allQueuesOver 6= true) 49: f orkId + + ; 50: case : Default 51: path.append(node) ; 52: node.setV isited(true); 53: until (node.type 6= “end”) 107 54: return path 55: end function
6. Test Scenario Generation from Non-nested Concurrent Activity Diagrams gorithm calls getT estScenario() function in a loop for generating specified number of test scenarios (line numbers 35–37). The getT estScenario() function in CQS algorithm does a special processing for some control nodes. For decision nodes (line numbers 7–23), first the algorithm evaluates a guard condition associated with the decision. A guard condition is a logical expression associated with decision nodes that determine flow of control. The algorithm dequeues either true control flow activities or f alse control flow activities based on the result of the guard condition, and CQS appends dequeued activities to the test scenario. For fork nodes (line numbers 24–49), the algorithm assigns one of the subQueues to the active queue. After that, nodes are dequeued from the active queue up to a specific node called as switch-point or the end of that subQueue. When first time switch-point node visited algorithm switches thread and appends threadSwitch tag in a path (line numbers 34–39); and enqueues switch-point node back in the respective queue with node’s visited status true. At this point, algorithm stops processing of selected queue and selects new queue randomly, thus introduce interleaving in test scenarios. After that, when second time switchpoint node visited with visited status true then algorithm appends the node in the path (line numbers 32–33). The dequeued nodes are appended to the test scenario. The algorithm check if all nodes in all threads are processed for the fork (line numbers 43-47). Subsequently, algorithm processes other subQueues until all the subQueues associated with the fork node are processed. Then, other types of nodes are processed from the mainQueue up to the end node. Finally, getT estScenario() function returns a path. Function getT estScenario() uses randomsubQueue = random(f orkF lows) in line 28 to select a random queue to append nodes in a test path, thereby generating different interleaving test scenarios. The process is repeated until n test scenarios are generated. If n is so large that there are no n distinct test scenarios then algorithm returns n test scenarios that are not all distinct. The generated test scenarios are used for testing concurrency errors of the implementation. Selection of an adequate switch-point is more crucial for generating test scenarios that uncover bugs. In this work, we say a thread switch among the threads as a switch-point. Generally, concurrency errors happen among the threads that share common data or need a specific causal-order between activities of different threads; these threads are called concurrent threads. Threads that neither share common data nor need a specific causal-order are called parallel threads. Parallel threads do not cause concurrency errors; hence a single test scenario that represents any interleaving is sufficient to validate the execution of all activities inside the fork108
6.1. Concurrent Queue Search Technique join construct. In CQS algorithm, there are no switch-points in parallel threads; hence activities in each thread are appended sequentially, one thread followed by another thread. For concurrent threads, a test scenario should have thread switch at specific locations to uncover concurrency errors. A proper selection of switchpoint(s) will introduce thread switch and generates a test scenarios that uncover concurrency errors. Shared data among different threads need synchronized access. It is common practice that shared data are accessed inside critical sections. A critical section is implemented using a locking mechanism. Before entering the critical section, the locking mechanism acquires a lock(s) and immediately after exiting the critical section, the locking mechanism releases the lock(s). Unserialised interleaving is only capable of testing proper implementation of synchronization primitives. Therefore, to capture data-race errors of synchronized threads, a switch-point can be placed: (i) before acquiring a lock, (ii) after acquiring a lock, (iii) before releasing a lock, or (iv) after releasing a lock. CQS algorithm selects switch-point before releasing the lock. This switch-point may capture data-race errors in the synchronized thread implementation of concurrency constructs. Interprocess communication among the activities in different threads is implemented using a send-signal construct from one thread and receive-signal construct of another thread. Interprocess communication represents a causal-order between activities of different threads. To capture errors due to causal-order, a switch-point can be placed: (i) before a send signal, (ii) after a send signal, (iii) before a receive signal, or (iv) after a receive signal. In CQS algorithm, switch-points are selected before send signal and after receive signal. These switch-points may uncover causal ordering errors in the concurrent thread implementation. Test scenarios generated by CQS algorithm are negative interleaving test scenarios for testing synchronization primitive errors. CQS algorithm will not interleave activities inside the fork-join construct unless and until there is a causal order among the activities, or they share any data. Consider an example as shown in Figure 6.1 to illustrate the working of CQS algorithm. This figure is a combination of two figures Figure 5.9 and Figure 5.13 from Chapter 5. Figure 6.1 is a non-nested decision iteration followed by non-nested fork-fork activity. The running instance of the queue generation for this example is shown in Figure 6.2. One main queue and two separate queues for each fork-join are shown in Figure 6.2 for Figure 6.1. This example illustrates the process of processing queues for control nodes, like decision and fork. The decision construct works with conditional flows like if..endif, if..else..endif , and loops. In this example, decision 109
6. Test Scenario Generation from Non-nested Concurrent Activity Diagrams D1 represents if..else..endif and decision D2 represents loop. Also, two fork-join constructs generate two subqueue structures for fork F 1 and F 2 each having two subqueues as shown in Figure 6.2. In the first phase, CQS algorithm reads ADG of Figure 6.1. Variables are initialized; and mainQueue is generated. Nodes are read from ADG; and they are enqueued in the mainQueue up to the decision D1. Then the decision node is processed to enqueue nodes a2 and M 1 followed by nodes a3 and M 1 (where M 1 is the merge node) in mainQueue as discussed in lines 8-18 in CQS algorithm. Similarly, nodes a4 , . . . , a8 are processed to insert nodes in mainQueue. When algorithm reaches at fork node F 1, then it is enqueued in the mainQueue, and the two subQueues are generated for two threads of F 1. Then nodes a9 , a10 , and J1 and a11 , a12 , and J1 are enqueued in the first and the second subQueues of F 1 as described in lines 9-31 of CQS algorithm. Similarly, nodes a13 , . . . , E are processed and enqueued in the appropriate queues.
In the second phase, test scenarios are generated by dequeuing nodes from mainQueue and subQueues. Two sample test scenarios are as follows: S1 = S → a1 → D1 → a2 → M 1 → a4 → D2 → a5 → a6 → M 2 → a7 → a8 → F 1 → a9 → a10 −−→ a11 → a12 → J1 → a13 → F 2 → a14 → a15 → a16 −−→ a17 → CS CS a18 → J2 → a19 → E. S2 = S → a1 → D1 → a3 → M 1 → a4 → D2 → M 2 → a7 → a8 → F 1 → a11 → a12 −−→ a9 → a10 → J1 → a13 → F 2 → a14 → a15 → a16 −−→ a17 → a18 → J2 → CS CS a19 → E. In scenario S1, true branches of decision nodes D1 and D2 are processed. The first subQueue followed by the second subQueue are processed for fork nodes F 1 and F 2 in scenario S1. In scenario S2, f alse branches of decision nodes D1 and D2 are processed. The second subQueue is processed followed by the first subQueue for fork node F 1 in scenario S2. The first subQueue is processed followed by the second subQueue for fork node F 2 in scenario S2. In this example, there is no shared data access or send-receive signals hence fork sub queues are processed by inserting threadswitch at the end of each thread thereby generating sequentialized test scenarios. As described in the above example, CQS algorithm enqueues all activity and control nodes of the ADG in the first phase and dequeues those nodes in second phase to generate test scenarios. Thus CQS generates the test scenarios. 110
6.1. Concurrent Queue Search Technique
S
a8 F1
a1 D1 a3
a2
a9
a11
a10
a12
J1 M1
a13 F2
a4
a14 a17
D2/ M2
a15 a18
a16 a5 J2 a6
a19 E
a7
Figure 6.1: An illustration example of non-nested decision iteration and fork-fork activity diagram
Main Queue S a1 D1 a2 M1 a3 M1 a4 D2 a5 a6 M2 M2 a7 a8 F1 a13 F2 a19 E
Subqueues of F1 a9 a10 J1
a11 a12 J1
Subqueues of F2 a14 a15 a16 J2
a17 a18 J2
Figure 6.2: A queue structure representing process of queue generation of main queue and sub queues for the Figure 6.1
111
6. Test Scenario Generation from Non-nested Concurrent Activity Diagrams
6.2
Permutation Based Techniques
Permutation is a simple technique to generate all possible interleaving paths, but it is exponential. Permutation is a sequence containing each element from a finite set and each element is to be included in the sequence exactly once. A valid test scenario must start from the start activity and stop at the end activity in an activity diagram. If a permutation technique is used to generate test scenarios then some permutations are valid test scenarios, and some permutations are invalid test scenarios. Consequently, it is necessary to check every permutation to accept it as a valid test scenario. Number of permutations on a set of n elements is given by n!. This clearly indicates a permutation technique is exhaustive and exponential task. In order to generate the valid concurrent test scenarios, a permutation technique can be applied to a subset of activities inside the fork-join structure. In this section, we present two permutation techniques: a) level permutation and b) DFS level permutation. Both techniques use restricted permutation to avoid exponential task and invalid test scenario generation.
Level Permutation- Level permutation (LevelPermute) algorithm takes input ADG and returns set of test scenarios. Algorithm 6.2 lists LevelP ermute algorithm. This algorithm works for one set of fork join in activity diagrams. In the level permutation approach, all the activities inside the fork-join are divided into different levels. Function getF orkJoinActivities() returns activities inside fork join in line number 2. Activities, which are the immediate neighbors of the fork, are in the first level, the immediate neighbors of the first level are in the second level, and so on up to the join node. Function assignLevelN umbers() assigns level numbers to activities and returns highest level number in line number 3. Then, activities at each level are permuted to generate the subsequences. Function permutation() permutes activities and returns all permutations. After that, current level each subsequence is augmented with the previous level subsequences. The process of adding and appending subsequences to test scenarios set T S is given in line numbers 7–26. This includes adding first level subsequences in first iteration. Then, algorithm increases size of T S to accommodate all possible combinations of T S and new subsequences in each level. After that, sub-sequences of each level are appended to the T S. Finally, this process returns a set of interleaving test scenarios inside the fork-join structure. This approach maintains the happen before relationship (Lamport, 1978) between activities of threads. 112
6.2. Permutation Based Techniques
Algorithm 6.2 Level Permutation algorithm (LevelPermute) Input: ADG: Activity Diagram Graph Output: T S: Set of test scenarios 1: T S = empty − set; 2: f orkJoinActivityList = getF orkJoinActivities(ADG); 3: levels = assignLevelN umbers(f orkJoinActivityList); 4: for (i = 1; i < levels; i + +) do 5: levelActivities = f orkJoinActivityList.getActivitiesAtLevel(i); 6: subSequencesList = permutation(levelActivities); 7: if (T S == null) then . first time add paths to TS 8: for (j = 1; j < subSequencesList.size(); j + +) do 9: T S.add(subSequencesList.get(j)); 10: end for 11: else . increase size of T S in multiple of subsequence size 12: tSize = T S.size(); 13: sSize = subSequencesList.size(); 14: tsN ewSize = (tSize ∗ sSize) − tSize; 15: for (p = 1; p < tsN ewSize; p + +) do 16: sequence = T S.get(p mod sSize); 17: T S.add(sequence); 18: end for 19: . append every element in subSequenceList to every element in T S 20: for (p = 1; p < T S.size(); p + +) do 21: for (q = 1; q < subSequencesList.size(); q + +) do 22: index = q + (p − 1) ∗ T S.size(); 23: T S.append(index, subSequencesList.get(q)); 24: end for 25: end for 26: end if 27: end for 28: return T S
113
6. Test Scenario Generation from Non-nested Concurrent Activity Diagrams Consider two examples Figure 6.3 and Figure 6.4 illustrating LevelPermute algorithm working instance for test scenario generation. In the first example, Figure 6.3 consists of two threads and six activities three in each thread. There are three levels each consisting two activities generating two sub-sequences at each level. Concatenation of all subsequences generates eight sequences that all are valid test scenarios inside the fork-join construct. Level permutation has generated 2!.2!.2! = 8 sequences as compared to permutation that might have generated 6! = 720 sequences. In the
Test Test Test Permutation Scenarios Scenarios Scenarios Subsequences Iteration-0 Iteration-1 Iteration-2 a1
a2
a3
Level1
Level2
Level3
a4
a5
a6
{a1-a4, a4-a1}
{a1-a4, a4-a1}
{}
{a2-a5, a5-a2}
{a1-a4-a2-a5, a1-a4-a5-a2, a4-a1-a2-a5, a4-a1-a5-a2}
Test Scenarios Intration-3 {a1-a4-a2-a5-a3-a6, a1-a4-a2-a5-a6-a3, a1-a4-a5-a2-a3-a6, a1-a4-a5-a2-a6-a3, a4-a1-a2-a5-a3-a6, a4-a1-a2-a5-a6-a3,
{a3-a6, a6-a3}
a4-a1-a5-a2-a3-a6, a4-a1-a5-a2-a6-a3}
Figure 6.3: A level permutation algorithm illustration for example 1, which is having equal number of activities per level
second example, Figure 6.4 consists of three threads and six activities. First thread has three activities, second thread has two activities and third thread has single activity as shown in the figure. There are three levels each consisting three, two, and one activities, respectively, generating six, two, and one subsequences at each level, respectively. Concatenation of all subsequences generates twelve sequences that all are valid test scenarios inside the fork-join construct. Level permutation has generated only 3!.2!.1! = 12 sequences as compared to permutation that might have generated 6! = 720 sequences. Test scenario calculation The total number of test scenarios generated by the LevelPermute algorithm is calculated as follows. Let us assume there are n levels. If a level Li , 1 ≤ i ≤ n, has |Li | activities then the level Li has |Li |! subsequences. Let T S be the total number of test scenarios. Then, the total number of valid concurrent 114
6.2. Permutation Based Techniques
Permutation Subsequences Level 1
a1
b1
a2
b2
a3
c1
{a1-b1-c1, a1-c1-b1, b1-a1-c1, b1-c1-a1, c1-a1-b1, c1-b1-a1}
Level 2
{a2-b2, b2-a2}
Level 3
{a3}
Test Test Test Scenarios Scenarios Scenarios Iteration-0 Iteration-1 Iteration-2 {}
{a1-b1-c1, a1-c1-b1, b1-a1-c1, b1-c1-a1, c1-a1-b1, c1-b1-a1}
Test Scenarios Iteration-3
{a1-b1-c1- a2-b2, {a1-b1-c1- a2-b2-a3, a1-c1-b1- a2-b2, a1-c1-b1- a2-b2-a3, b1-a1-c1- a2-b2, b1-a1-c1- a2-b2-a3, b1-c1-a1- a2-b2, b1-c1-a1- a2-b2-a3, c1-a1-b1- a2-b2, c1-a1-b1- a2-b2-a3, c1-b1-a1- a2-b2, c1-b1-a1- a2-b2-a3, a1-b1-c1- b2-a2, a1-b1-c1- b2-a2-a3, a1-c1-b1- b2-a2, a1-c1-b1- b2-a2-a3, b1-a1-c1- b2-a2, b1-a1-c1- b2-a2-a3, b1-c1-a1- b2-a2, b1-c1-a1- b2-a2-a3, c1-a1-b1- b2-a2, c1-a1-b1- b2-a2-a3, c1-b1-a1- b2-a2} c1-b1-a1- b2-a2-a3}
Figure 6.4: A level permutation algorithm illustration for example 2, which is having different number of activities per level
test scenarios generated by level permutation technique inside the fork-join with Q n levels is T S(LP ) = ni=1 |Li |!. One of the most commonly used techniques to generate interleaving test scenarios is the BFS technique inside the fork-join construct for activity diagrams. In comparison with BFS algorithm, LevelPermute algorithm generates more number of concurrent test scenarios. Total number of valid concurrent test scenarios generated by BFS technique inside fork-join with m threads and n levels is T S(BF S) = m!. Following equation holds true for permutation, level permutation and BFS algorithms. T S(BF S) ≤ T S(LevelP ermute) ≤ T S(P ermutation) Thus, LevelPermute technique achieves better interleaving activity path coverage than BFS technique.
DFS-LevelPermute- BFS and LevelPermute algorithms generate valid concurrent test scenarios inside fork-join structures. In the most of applications, fork-join structure is surrounded by other structures. A test scenario is complete if it has first activity is the start activity, and last activity is the end activity of the activity diagram. In other words, a complete test scenario also includes activities outside the fork-join structure for testing purpose. Therefore, we have proposed a DFS-LevelPermute algorithm to generate complete valid test scenarios. It is a combination of DFS and 115
6. Test Scenario Generation from Non-nested Concurrent Activity Diagrams LevelPermute algorithms. A DFS algorithm is used for non-concurrent parts of activity diagrams. A LevelPermute algorithm is used for concurrent parts of activity diagrams inside fork-join. In this algorithm, first LevelPermute algorithm is executed to generate concurrent test scenarios (CTS) inside the fork-join structure. Then standard DFS algorithm traverses the activity diagram from the start node to the end node to generate paths. After that, for each path, which includes fork-join, a non-concurrent prefix (a portion from the start node to the fork node) and a suffix (a portion from the join node to the end node) are separated. Finally, each sequence of CTS is concatenated (sandwiched) between the suffix and the prefix of the path to generate complete valid concurrent test scenarios. Figure 6.5 represents an illustration of LevelPermute algorithm. LevelPermute algorithm generates four sequences from activities a2 , a3 , a4 , a5 and DFS algorithm generates two paths both including fork-join pair. Then their prefix S → a1 → F and suffix J → a6 → E are separated. Finally, all sequences from CTS are concatenated between prefix and suffix to generate four distinct test scenarios. Level permutation algorithm generates test scenarios
Assign Level Numbers
S
L1 = {a2,a4}
L2 = {a3,a5}
Permute activities in the levels
a1
Receive Order
L1 = F
a2-a4, a4-a2
L2 =
a3-a5, a5-a3
Concatenate lists of all levels Send Invoice
Fill Order
L1 CTS =
a4
a2
Receive Payment
Deliver Order
a3
L2
a2-a4-a3-a5 a4-a2-a3-a5 a2-a4-a5-a3 a4-a2-a5-a3
Generate DFS paths
a5
1. S-a1-F- a2-a3-J-a6-E 2. S-a1-F- a4-a5-J-a6-E
J a6
Append DFS Prefix and Suffix
Close Order
Prefix:S-a1-F- Suffix:-J-a6-E 1. S-a1-F- a2-a4-a3-a5-J-a6-E ... 4. S-a1-F- a4-a2-a5-a3-J-a6-E
E
Figure 6.5: An illustrative example of DFS level permutation algorithm
for causal-order, they may trigger bug if send and receive signals are at same level. 116
6.3. Experimental Results
6.3
Experimental Results
In this study, test scenario generators for both sequence and activity diagrams are presented to test concurrency errors in systems. We have proposed the concurrent queue search algorithm to generate test scenarios that may trigger bug and permutationbased DFS LevelPermute algorithm to improve concurrency coverage. We have implemented modified-DFS by Chandler et al. (2005), DFS-BFS by Kundu and Samanta (2009), CQS, and DFS LevelPermute algorithms in Java. All experiments are conducted using 3.00 GHz Intel Core 2 Quad CPU with 4 GB RAM desktop computer. We have compared our results with modified-DFS and DFS-BFS algorithms. The following examples illustrate test scenarios generation by modified-DFS, DFS-BFS, CQS, and DFS LevelPermute algorithms. The implementation details are presented in Appendix C. Example 1- Consider the example shown in Figure 5.6, which has parallel threads. The figure neither includes any causal ordering among the activities nor do threads share any common data. Table 6.1 lists test scenarios generated by modified-DFS, DFS-BFS, CQS, and DFS LevelPermute algorithms for Figure 5.6. Table 6.1: Interleaving activity path coverage (IAPC) of test scenarios generated by modified-DFS, DFS-BFS, CQS, and DFSLevelPermute algorithms for Figure 5.6
Algorithm Modified-DFS
Test scenarios S → m1 → F → m2 −−→ m3 → J → m4 → E CS
% IAPC 100
S → m1 → F → m3 −−→ m2 → J → m4 → E CS
DFS-BFS
S → m1 → F → m2 −−→ m3 → J → m4 → E CS
100
S → m1 → F → m3 −−→ m2 → J → m4 → E CS
CQS
S → m1 → F → m2 −−→ m3 → J → m4 → E CS
100
S → m1 → F → m3 −−→ m2 → J → m4 → E CS
DFS LevelPermute
S → m1 → F → m2 −−→ m3 → J → m4 → E CS
100
S → m1 → F → m3 −−→ m2 → J → m4 → E CS
Test scenarios generated by above algorithms are all valid test scenarios. The scenarios assess execution of all the activities of parallel processes, and they achieve 100% activity, transition, and interleaving activity path coverage. Modified-DFS algorithm test scenario generator first includes a sequence of activities from the start activity to 117
6. Test Scenario Generation from Non-nested Concurrent Activity Diagrams fork node. After that, it includes a sequence of activities from each thread of execution one after another until the join node. Finally, it includes a sequence of activities up to the end of the activity diagram. DFS-BFS algorithm test scenario generator first includes a sequence of activities from the start activity to fork node. After that, it lists activities breadth-wise i.e. one activity from each thread until the join node. Finally, it includes a sequence of activities up to the end of the activity diagram. For this example, all algorithms generate similar test scenarios because each thread includes one activity, and there are only two threads. When number of activities in each thread or number of threads increases then DFS-BFS and DFS LevelPermute algorithms generate a more number of test scenarios as compared to modified-DFS and CQS algorithms. Example 2- Consider a general ordering example shown in Figure 5.7. General ordering in a sequence diagram is transformed as send and receive signals in an activity diagram. Both the constructs represent causal-order among the tasks (messages/activities) in respective diagrams. For causal-order, a test scenario should uncover synchronization errors. The execution of the system must respect the causal-order represented in design models. One of the methods to uncover synchronization errors is by executing a negative test scenario, which violates the causal-order. If proper synchronization primitives are not implemented then negative test passes, otherwise test fails. In other words, if a negative test scenario is executed, and it exercises all the tasks in the order specified by the test scenario order then the code has not implemented causal-order in terms of synchronization primitives. If a negative test scenario is executed, and it does not exercise all the tasks in the order specified by the test scenario then the code has implemented causal-order in terms of synchronization primitives. For Figure 5.7, test scenarios generated by the modified-DFS, DFS-BFS, CQS, and DFS LevelPermute algorithms are as given in Table 6.2. Modified-DFS and DFSBFS algorithms generate two test scenarios as shown in Table 6.2 out of that one is a positive test scenario, and another is a negative test scenario. However, it is difficult to conclude whether the causal ordering is implemented correctly by single negative test scenario. CQS algorithm generates two test scenarios, and both are negative test scenarios that increase the confidence in the implementation of causalorder. Scenarios generated by CQS algorithm have high capability to uncover causal ordering error compared to modified-DFS and DFS-BFS algorithms; because, it forces a thread switch at specific locations. DFS LevelPermute algorithm achieved maxi118
6.3. Experimental Results Table 6.2: Simple path coverage (SPC) and interleaving activity path coverage (IAPC) for test scenarios generated by modified-DFS, DFS-BFS, CQS, and DFS LevelPermute algorithms for Figure 5.7
Algorithm Test scenarios Modified- S → m1 → F → m2 → Ss1 → m3 −−→ m4 → Rs1 → m5 → CS DFS J → m6 → E S → m1 → F → m4 → Rs1 → m5 −−→ m2 → Ss1 → m3 → CS
DFS-BFS
J → m6 → E S → m1 → F → m2 −−→ m4 −−→ Ss1 −−→ Rs1 −−→ CS
CS
m3 −−→ m5 → J → m6 → E
CS
CS
Trigger % coverage bug ? SPC IAPC no 100 10 yes no
100
10
100
10
100
40
CS
S → m1 → F → m4 −−→ m2 −−→ Rs1 −−→ Ss1 −−→ CS
CS
m5 −−→ m3 → J → m6 → E
CS
CS
yes
CS
CQS
S → m1 → F → m2 −−→ m4 → Rs1 −−→ Ss1 −−→ m5 −−→
yes
m3 → J → m6 → E S → m1 → F → m4 → Rs1 −−→ m2 → Ss1 −−→ m5 −−→
yes
m3 → J → m6 → E S → m1 → F → m2 −−→ m4 −−→ Ss1 −−→ Rs1 −−→
no
CS
CS
CS
CS
CS
m3 −−→ m5 → J → m6 → E DFS Level Permute
CS
CS
CS
CS
CS
CS
CS
S → m1 → F → m4 −−→ m2 → Ss1 −−→ Rs1 −−→ m3 −−→ CS
CS
CS
CS
no
m5 → J → m6 → E S → m1 → F → m2 −−→ m4 → Rs1 −−→ Ss1 → m3 −−→
yes
m5 → J → m6 → E S → m1 → F → m4 −−→ m2 −−→ Rs1 −−→ Ss1 → m3 −−→
yes
CS
CS
CS
CS
CS
CS
CS
m5 → J → m6 → E S → m1 → F → m2 → m4 → Ss1 → Rs1 → m5 → m3 → J → m6 → E S → m1 → F → m4 −−→ m2 → Ss1 −−→ Rs1 → m5 −−→ CS
CS
CS
no no
m3 → J → m6 → E S → m1 → F → m2 −−→ m4 → Rs1 −−→ Ss1 −−→ m5 −−→
yes
m3 → J → m6 → E S → m1 → F → m4 −−→ m2 −−→ Rs1 −−→ Ss1 −−→
yes
CS
CS
CS
m5 −−→ m3 → J → m6 → E
CS
CS
CS
CS
CS
CS
mum IAPC coverage, i.e., 40% indicating improvement in interleaving activity path coverage. Moreover, DFS LevelPermute algorithm generates total eight test scenarios and four scenarios are having bug triggering capability. The above observation says that DFS LevelPermute algorithm increases interleaving activity path coverage. On the contrary, CQS algorithm achieves less interleaving activity path coverage but generates test scenarios that may uncover concurrency bugs. 119
6. Test Scenario Generation from Non-nested Concurrent Activity Diagrams Example 3- Consider a bank transaction example that deposit and withdraw amount from a common/ shared object A of Account class shown in Figure 6.6. In this example, a sequence diagram accesses object A of Account class for depositing and withdrawing amount. When converting a sequence diagram into an activity diagram, locks are introduced for shared access as shown in Figure 6.6 to achieve isolation of access to the shared object. Test scenario generator should generate test scenario that
Sd: bank_transaction S
t1:Thread
bank: Bank
t2:Thread
A: Account
F
Par
Lock(A)
Lock(A)
Deposit(A,100)
Withdraw(A,50)
UnLock(A)
UnLock(A)
1: deposit(A,100) 2: deposit(100)
3: withdraw(A,50)
J
4: withdraw(50) E
Figure 6.6: A bank transaction example showing shared object access by two threads for read access and write access
uncover data-race errors and detect inconsistent state of shared data due to specific interleaving of tasks. The test scenarios generated by modified-DFS, DFS-BFS, CQS, and DFS LevelPermute algorithms are shown in Table 6.3 for Figure 6.6. Results include the classification of test scenarios based on the ESMD classifier Figure 4.2 discussed in Chapter 4. Modified-DFS algorithm generates two test scenarios, which are not capable of finding data-race errors because they do not have interleavings among concurrent tasks. Therefore, they ensure implementation of synchronization primitives and do not target data-race error, because they are serialized test scenarios. Test scenarios achieve 100% SPC, 10% IAPC, and 0% synchronization coverage. DFS-BFS and CQS algorithms generate two test scenarios, which are capable of finding data-race errors, as they are well interleaved to uncover data-race error. These test scenarios achieve 100% SPC, 10% IAPC, and 100% synchronization coverage; all are blocking 120
6.4. Conclusions type test scenarios because synchronization mechanism is in design. However, DFSTable 6.3: Simple path (SPC), interleaving activity path (IAPC), and synchronization (SC) coverage for test scenarios generated by modified-DFS, DFS-BFS, CQS, and DFS LevelPermute algorithms for Figure 6.6 Algorithm
Test scenarios
Concurrent Behavior Type *
Trigger
Bugs? S → F → T1 .Lock(A) → T1 .Deposit(A, 100) → T1 .U nlock(A) → ModifiedT2 .Lock(A) → T2 .W ithdraw(A, 50) → T2 .U nlock(A) → J → E DFS S → F → T2 .Lock(A) → T2 .W ithdraw(A, 50) → T2 .U nlock(A) → T1 .Lock(A) → T1 .Deposit(A, 100) → T1 .U nlock(A) → J → E S → F → T1 .Lock(A) → T2 .Lock(A) → T1 .Deposit(A, 100) → T2 .W ithdraw(A, 50) → T1 .U nlock(A) → T2 .U nlock(A) → J → E DFS-BFS S → F → T2 .Lock(A) → T1 .Lock(A) → T2 .W ithdraw(A, 50) → T1 .Deposit(A, 100) → T2 .U nlock(A) → T1 .U nlock(A) → J → E S → F → T1 .Lock(A) → T1 .Deposit(A, 100) → T2 .Lock(A) → T2 .W ithdraw(A, 50) → T1 .U nlock(A) → T2 .U nlock(A) → J → E CQS S → F → T2 .Lock(A) → T2 .W ithdraw(A, 50) → T1 .Lock(A) → T1 .Deposit(A, 100) → T2 .U nlock(A) → T1 .U nlock(A) → J → E S → F → T1 .Lock(A) → T2 .Lock(A) → T1 .Deposit(A, 100) → T2 .W ithdraw(A, 50) → T1 .U nlock(A) → T2 .U nlock(A) → J → E S → F → T2 .Lock(A) → T1 .Lock(A) → T1 .Deposit(A, 100) → T2 .W ithdraw(A, 50) → T1 .U nlock(A) → T2 .U nlock(A) → J → E S → F → T1 .Lock(A) → T2 .Lock(A) → T2 .Deposit(A, 100) → T1 .W ithdraw(A, 50) → T1 .U nlock(A) → T2 .U nlock(A) → J → E S → F → T2 .Lock(A) → T1 .Lock(A) → T2 .Deposit(A, 100) → DFS Level T1 .W ithdraw(A, 50) → T1 .U nlock(A) → T2 .U nlock(A) → J → E Permute S → F → T1 .Lock(A) → T2 .Lock(A) → T1 .Deposit(A, 100) → T2 .W ithdraw(A, 50) → T2 .U nlock(A) → T1 .U nlock(A) → J → E S → F → T2 .Lock(A) → T1 .Lock(A) → T1 .Deposit(A, 100) → T2 .W ithdraw(A, 50) → T2 .U nlock(A) → T1 .U nlock(A) → J → E S → F → T1 .Lock(A) → T2 .Lock(A) → T2 .Deposit(A, 100) → T1 .W ithdraw(A, 50) → T2 .U nlock(A) → T1 .U nlock(A) → J → E S → F → T2 .Lock(A) → T1 .Lock(A) → T2 .Deposit(A, 100) → T1 .W ithdraw(A, 50) → T2 .U nlock(A) → T1 .U nlock(A) → J → E *-scenarios’ concurrent behavior type as per ESMD classifier discussed in Chapter 4.
Serialized
no
Serialized
no
Blocking
yes
Blocking
yes
Blocking
yes
Blocking
yes
Blocking
yes
Blocking
yes
Blocking
yes
Blocking
yes
Blocking
yes
Blocking
yes
Blocking
yes
Blocking
yes
% coverage
SPC
IAPC
SC
100
10
0
100
10
100
100
10
100
100
40
100
BFS algorithm introduces switch-point after every task in the test scenario thereby increasing difficulty to analyze it. On the other hand, CQS algorithm introduces at most two switch-points in the test scenario. DFS LevelPermute algorithm generates eight test scenarios that are capable of uncovering data-race and deadlock errors. These test scenarios achieve 100% SPC, 40% IAPC, and 100% synchronization coverage; all are blocking type test scenarios because synchronization mechanism is in the design. If programmer does not implement the synchronization primitives then these test scenarios may uncover data-race type of errors. Experimental results indicate that CQS and DFS LevelPermute algorithms have high capability to uncover synchronization errors as compared to modified-DFS and DFS-BFS algorithms.
6.4
Conclusions
In this chapter, we have presented two test scenario generation algorithms from both sequence and activity diagrams in the presence of concurrency. We found that vir121
6. Test Scenario Generation from Non-nested Concurrent Activity Diagrams tually in all the cases test scenarios generated by CQS algorithm may uncover synchronization (causal ordering) and data-race errors. CQS algorithm has more synchronization error uncover capability as compared to modified-DFS and DFS-BFS algorithms. In the case of data-race errors, DFS-BFS algorithm forces thread switch after every task while CQS algorithm introduces two thread switches. Empirical study indicates an overall improvement of CQS algorithm over modified-DFS and DFS-BFS algorithms to generate test scenarios for concurrency testing. Results clearly indicate that DFS-LevelPermute test scenario generation algorithm achieved better interleaving activity path coverage for concurrent activity diagrams. DFS-LevelPermute algorithm is good for the (non-nested) concurrent activity diagrams. However, these algorithms fail generating test scenarios for nested concurrent activity diagrams. In the next chapter, we present test scenario generation technique for nested concurrent activity diagrams.
122
Chapter 7 Test Scenario Generation from Nested Concurrent Activity Diagrams
Concurrency in application systems is designed and visualized using UML behavioral models. For concurrent applications, the main objective of designing test cases is to design valid test scenarios, which may trigger bugs in the system. As discussed in Literature review Chapter 2, test scenario generation from UML design models for concurrency testing is little explored. In the past, UML based graph theoretic testing has focused on structural testing, thereby restricting exploration of sufficient execution interleaving. This may ship concurrency errors in the final software even after achieving specified structural coverage on UML models. A few researchers (Chandler et al., 2005; Kim et al., 2007; Kundu and Samanta, 2009; Sun et al., 2009) have focused on generating test scenarios from concurrent activity diagrams. Most of the graph theoretic algorithms face difficulty in generating test scenarios inside concurrency constructs, and it is hard for these approaches to explore nested concurrency constructs (Xu et al., 2005). Also, our proposed CQS and DFS LevelPermute algorithms in Chapter 6 fail to generate test scenarios for nested concurrent activity diagrams. Therefore, there is a necessity to explore nested concurrency using non-trivial way. Test scenario generation inside nested concurrency constructs can be formulated as a search problem (Ali et al., 2010). Recent UML based testing studies have adopted search based techniques like Li and Lam (2005a); Xu et al. (2005); Farooq et al. 123
7. Test Scenario Generation from Nested Concurrent Activity Diagrams (2008); Ali et al. (2010); Shousha et al. (2011). However, hardly any evolutionary approach has applied UML-based testing for nested concurrent activity diagrams. In this chapter, interleaving test scenario generation problem is formulated as a search based optimization problem for nested concurrent activity diagrams. This chapter presents two meta-heuristic search based solutions. In the first solution, an evolutionary algorithm is presented to generate test scenarios form nested concurrent activity diagrams. In the second solution, a hybrid evolutionary algorithm is presented that try to improve convergence of our non-hybrid evolutionary approach. These proposed approaches satisfy better concurrency coverage criteria presented in the Chapter 3. The rest of the chapter is organized as follows: Section 7.1 describes evolutionary algorithm basics. Section 7.2 discusses the proposed evolutionary algorithm to generate test scenarios. Section 7.3 presents the hybrid evolutionary algorithm to improve convergence. Experimental results are presented in Section 7.4. Finally, Section 7.5 concludes the chapter.
7.1
Evolutionary Algorithm Basics
An evolutionary algorithm is a stochastic search optimization technique based on the principles of natural evolutions (Jones, 1995). EA maintains a collection of potential solutions to the problem known as a population. Evolutionary algorithm starts with a random population. It requires a suitable representation of candidate solutions for a particular problem. EA represents the problem in terms of solution encoding known as a chromosome. Generally, chromosomes are encoded as binary encoding, permutation encoding, or value encoding (Hanrahan, 2011). A chromosome consists of genes that encode traits. A fitness merits a chromosome by comparing the similarity between the chromosome and the actual solution (Goldberg, 1989). Some of these possible solutions are selected to create new potential solutions. Various mechanisms, such as Boltzmann selection, tournament selection, rank selection, and steady state selection, are used to select the individuals that create an offspring based on its fitness value (Yang and Stacey, 2001). After the selection step, recombination takes place to produce the next generation. A crossover operation is used for recombination. The most commonly used technique is a single-point crossover, which is illustrated in Figure 7.1. This illustrative example is based on the chromosome encoding presented in Section 7.2. It takes two random chromosomes and randomly chooses a locus for 124
7.1. Evolutionary Algorithm Basics recombination. The crossover operation generates two new chromosomes. The first chromosome contains nodes from the beginning of the first parent chromosome up to crossover locus, and rest is from the second parent chromosome. We consider geometric algorithm activity Figure 1.1 for illustration of evolutionary operators on paths. For example, consider two parent chromosomes C1 and C2 for Figure 1.1. Locus point of crossover 0 |0 is at the seventh position. The crossover operation produces two new chromosomes C10 and C20 as shown in Figure 7.1. Parent Chromosomes S
a1
F
a2
a5
Locus point 7th node
a3
a6
a7
a8
a4
a9
J
a10
C1 S C2
E Crossover
a1
F
a5
a6
a2
a3
a4
a7
a8
a9
J
a10
E
New Child Chromosomes S a1 F a2 a5 a3 a6 C1‘ S a1 C2‘
F
a5
a6
a2
a3
a4
a7
a8
a9
J
a10
E
a7
a8
a4
a9
J
a10
E
Figure 7.1: An illustration of a crossover operation for paths represented by chromosomes in evolutionary algorithm
In order to maintain population diversity and avoid local minima in the search process (Goldberg, 1989), new characteristics are infrequently injected by applying mutation operation as illustrated in Figure 7.2. The mutation operator randomly flips some nodes in the chromosome. For example, chromosome C3 is mutated at the sixth and seventh position to yield a new chromosome C30 as shown in Figure 7.2. One Mutation point 6th & 7th location S
a1
F
a2
a5
Parent Chromosome C3
a3
a6
a7
a8
a4
a9
J
a1
E
S
0
a1
F
Mutation
a2
a5
a6
a3
a7
a8
a4
a9
J
a1
E
0
New Child Chromosome C3‘
Figure 7.2: An illustration of a mutation operation for a path represented by the chromosome in evolutionary algorithm
generation of EA is complete after completion of a cycle of reproduction, crossover, and mutation operations to the whole population. The algorithm repeats this process to generate new collections of potential solutions until a stopping criterion is met. A stopping criterion may be the maximum number of iterations or a desired accuracy of the solution (Safe et al., 2004). The desired accuracy of the solution is calculated using fitness. The basic outline of the evolutionary algorithm (Ipate and Lefticaru, 2008) is as follows: 1. Start- Create a random population of n chromosomes
125
7. Test Scenario Generation from Nested Concurrent Activity Diagrams 2. New Population- Generate new populations by repeating following steps. • Crossover- Combine (met) parent chromosomes with crossover probability to generate new offspring. • Mutation- Mutate a parent chromosome with mutation probability at specific locus to generate a new offspring. • Accepting- Place new offspring into the current population. 3. Fitness- Evaluate fitness f (x) for each chromosome x in the population. 4. Selection- Select fit chromosomes from the population according to the fitness threshold. 5. Test- If stopping criteria is satisfied then stop the algorithm and return the best solutions in the current population. 6. Replace- Use new selected chromosomes population for further iterations. 7. Loop- Go to Step 2.
A well constructed fitness not only improves the likelihood of searching a solution, but also it consumes less time and fewer resources Baresel et al. (2002). The previous studies Sthamer (1995), Baresel et al. (2002), Doungsa-ard et al. (2007), and Ipate and Lefticaru (2008) have done research in designing the fitness function for structure-oriented testing that can be divided into two types: (i) calculating predicate distances to determine how far the test data are from the actual value, and (ii) computing difference between the target path and execution path. We have done several experiments to generate test scenarios and test data using the evolutionary algorithm to get confidence. First, we have presented a genetic algorithm based test case generation approach for sequence diagrams that exploit sequence diagram to search method sequences leading to usable behavior in the application domain. This approach generates both valid and invalid flows of message sequences. After that, we have proposed a search-based approach to find feasible transition sequences and test data. An extended control flow graph (ECFG) representation of a state machine diagram is used to specify the control and data flow. Genetic algorithm is used to generate the feasible paths and test data from ECFG. Where, test data satisfy guard condition on each transition. A valid concurrent test scenario starts from the start activity and stops at the end activity, and it must include interleaving of activities inside the fork-join structures. It is a difficult task to generate valid concurrent test scenarios by graph theoretic traversal algorithms in the case of nested concurrent activity 126
7.2. Evolutionary Algorithm Based Test Scenario Generator diagrams (Xu et al., 2005). Evolutionary algorithms have the ability of simultaneous stochastic search in different regions of the solution space thereby generating a diverse set of valid concurrent test scenarios.
7.2
Evolutionary Algorithm Based Test Scenario Generator
Nested concurrent activity diagrams, explained in Chapter 5, include nesting of decision, iteration, and concurrency constructs. A suitable intermediate representation is required for test case design from activity diagrams. For an evolutionary algorithm, first one needs to translate a problem in terms of suitable chromosome encoding, and then define fitness. Fitness is a numeric value that measures the distance between the expected outcome and chromosome’s decoded outcome. Rest of the evolutionary process is same as discussed in the previous section. Finally, fittest chromosomes are translated back into the problem domain to get solutions. This section discusses proposed evolutionary algorithm by describing an activity diagram graph, chromosome encoding, fitness and evaluation of chromosome’s fitness.
Activity diagram graph- Activity diagrams provide the list of activities, decisions, merges, forks, joins, transitions and guard condition associated with each transition. We translate activity diagram into intermediate model called an activity diagram graph. An activity diagram graph is a graph representation of an activity diagram’s control and data structures suitable for test scenario generation. Definition 7.1. Activity diagram graph G = (N, E) is a set of nodes N = {a1 , a2 , . . . , am } and a set of edges E = {e1 , e2 , . . . , en }, where ei is an ordered pair of nodes (aj , ak ) such that 1 ≤ i ≤ n and 1 ≤ j, k ≤ m; j 6= k. A set N is a union of two types of nodes: activity nodes (Na ) and control nodes (Nc ). An activity node is represented by a data structure, which includes the name of the activity and variables associated with that activity. Variables play an important role in evaluating the guard condition associated with transitions. A control node is represented by a data structure, which includes the control node type, associated guard condition, and action script. 127
7. Test Scenario Generation from Nested Concurrent Activity Diagrams Chromosome encoding- A chromosome is an ordered collection of genes. Chromosome encoding should represent a path in the given intermediate representation of activity diagram. A sequence of nodes is chosen as a chromosome in this problem. Consecutive nodes in the chromosome represent an edge of the graph. A gene is represented as a node of intermediate representation of activity diagram that plays as a source node, Snode, or a target node, T node, of an edge. A sequence of genes forms a chromosome, which defines a path in an activity diagram graph. The first node of a chromosome is the start node of the path. The last node of a chromosome is the end node of the path. Any two consecutive nodes in a chromosome represent an edge. Figure 7.3 represents a chromosome encoding for a path S −a1 −D1 −a2 −· · ·−a10 −E the NDD: nested decision-decision activity diagram shown in Figure 5.10. Each node is represented by a binary representation of node number. A chromosome with fitness SNode
S/TNode
S/TNode
S/TNode
101010001 101010001 101010001 101010001 S
a1
D1
a2
S/TNode
TNode
101010001 101010001 a10
E
Figure 7.3: A chromosome encoding of a test scenario for the NDD activity diagram shown in Figure 5.10
value near to zero is treated as the fittest chromosome. The fittest chromosome with fitness value zero represents a valid interleaving path in an activity diagram graph. Fitness- A fitness measures the distance between the expected outcome and chromosome’s decoded outcome. This is usually done by analyzing the chromosome, which holds data about a particular solution to the problem. The fitness computes the fitness value by utilizing chromosome information, and makes some qualitative assessment of the chromosome. Every chromosome in the population represents a set of nodes representing a path. Consider a target path represented by a chromosome p = a1 − a2 − a3 − · · · − ak , where a1 is the start node of the path, ak is the end node of the path, and edges represented by (ai , ai+1 ) ∈ E of activity diagram graph. For a given path p decoded from a chromosome, fitness is calculated as follows: Consider three constants c1 , c2 , and c3 such that c1 ≥ c3 ≥ c2 , and node-list, which is a set containing activity nodes. Nodes in the node-list indicate a set of valid nodes available as the next possible node 128
7.2. Evolutionary Algorithm Based Test Scenario Generator (immediate neighbor) of the current node in an interleaving path. In other words, there is an edge from the current node to the nodes in the node-list, or they may be traversed (in an interleaving path) in case of a thread switch. The fitness of a chromosome is calculated using the equation 7.1. F itness =
k X
w(ai ),
(7.1)
i=1
where, a node ai ∈ p and w(ai ) is the weight associated with the node. Weight w of a node ai is calculated as follows:
w(ai ) =
0,
c , 1 0, w(ai ) = c , 1 0, w(ai ) = c , 2 0, w(ai ) = c , 2 0, w(ai ) = c , 2 0, w(ai ) = c , 2
if (i == 1 and ai is initial node)
(7.2)
otherwise if (i == k and ai is end node and ak ∈ node list)
(7.3)
otherwise if (ai ∈ Na and ai ∈ node list)
(7.4)
otherwise if (ai ∈ Nc and ai .type == “decision” and ai ∈ node list and
(7.5)
(ai−1 , ai ).guard == true) otherwise if (ai ∈ Nc and ai .type == “merge” and ai ∈ node list)
(7.6)
otherwise if (ai ∈ Nc and ai .type == “f ork” and ai ∈ node list)
(7.7)
otherwise (7.8) 129
7. Test Scenario Generation from Nested Concurrent Activity Diagrams
w(ai ) =
0,
if (ai ∈ Nc and ai .type == “join”
and ai ∈ node list) c , otherwise 3
(7.9)
Evaluation process of the fitness- A fitness of each chromosome is calculated by decoding the chromosome from the first node to the last node by using equations (7.1) through (7.8) and Table 7.1. Fitness evaluation algorithm is discussed in Algorithm 7.1. This algorithm takes two inputs a chromosome and an ADG; and it returns the fitness-value of the given chromosome. A chromosome is a binary string as shown in Figure 7.3. Implementation details of the ADG and text file format are given in Appendix C. The objective function is to minimize the fitness. A calculation of the fitness starts with an empty node-list and zero initial fitness. It is required to update the node-list in the process of evaluating the chromosome. Updated node-list guides the evaluation process to search possible nodes that are next node in the valid interleaving path. When a node is evaluated, action scripts are executed to update the node-list. The action scripts are listed in the Table 7.1. The type of the current node and its associated action script decide the successor node suitable for the interleaving path. For each node in the path, fitness is updated as explained below. Table 7.1: Action scripts associated with control nodes to update node-list Sr. No. 1 2 3 4
Current node type start node end node action node decision
5 6 7
merge fork join
Action scripts node list ∪ {neighbors of start node} node list = ∅, stop and return F itness node list \ {current node}, node list ∪ {neighbor of action node} node list \ {current node}, node list ∪ {neighbors of decision node} node list \ {current node}, node list ∪ {neighbor of merge node} node list \ {current node}, node list ∪ {neighbors of f ork node} node list\{current node}, if node list ∩ {immediate predecessors of join node} = ∅ then node list ∪ {neighbor of join}
For the first node in the chromosome (line numbers 7-13), equation (7.2) decides weight contribution to the fitness. If the first node of the decoded chromosome is the start node of the activity diagram then weight contribution is zero, otherwise weight contribution is c1 . When this node is evaluated, the action script adds an immediate 130
7.2. Evolutionary Algorithm Based Test Scenario Generator Algorithm 7.1 Fitness Input: cr: Chromosome, ADG: Activity Diagram Graph Output: f itness: Fitness of chromosome 1: double f itness = 0.0; 2: nodeList = empty-list; 3: decompose cr into gene array gn; 4: previousN ode = actGraph.getRootN ode(); 5: for (i = 0; i < gn.size; i + +) do 6: currentN ode = actGraph.getN ode(gn[i]); 7: if (i = 0) then 8: if (currentN ode.type == “start”) then 9: add adjacent nodes of currentN ode into nodeList; 10: else 11: f itness+ = C1; 12: end if 13: else if (currentN ode ∈ nodeList) then 14: if (i == gn.Size − 1 and currentN ode.type 6= “end”) then 15: f itness+ = C1; 16: return f itness; . chromosome ends 17: end if 18: switch (currentN ode.type) do 19: case : “join” 20: if (All Join predecessors visited) then 21: remove all occurrences of join node from nodeList; 22: add adjacent nodes of currentN ode into nodeList; 23: else 24: f itness+ = C3; 25: end if 26: case : “end” 27: if (f itness == 0) then 28: return f itness; . stop further processing of chromosome 29: else 30: f itness+ = C1; 31: remove currentN ode from nodeList; 32: get randomly tempCurrentN ode from adjacent nodes of previousN ode; 33: add adjacent nodes of tempCurrentN ode into nodeList; 34: currentN ode = tempCurrentN ode; 35: end if 36: case : Def ault . “action”, “decision”, “merge”, “fork”, other types 37: remove currentN ode from nodeList; 38: add adjacent nodes of currentN ode into nodeList; 39: else 40: f itness+ = C2; 41: remove currentN ode from nodeList; 42: get randomly tempCurrentN ode from adjacent nodes of previousN ode ; 43: add adjacent nodes of tempCurrentN ode into nodeList; 44: currentN ode = tempCurrentN ode; 45: end if 46: previousN ode = currentN ode; 47: currentN ode.V isited(T rue) 48: end for 49: return f itness
131
7. Test Scenario Generation from Nested Concurrent Activity Diagrams neighbor of the start node in the node-list. For the last node in the chromosome (line numbers 14-17), equation (7.3) decides weight contribution to the fitness. If the last node of the decoded chromosome is the end node of the activity diagram and the node-list includes that node then weight contribution is zero, otherwise weight contribution is c1 . When the end node occurs before the last gene in the chromosome (line numbers 26-35), then it indicates a short length path or an invalid path. In such case, if the fitness of a chromosome is zero up to this node then the path is a valid path, and action script stops further processing and returns the valid path. Otherwise, chromosome’s fitness is penalized with weight c1 . If the current node is an activity node (line numbers 36-38), then equation (7.4) decides weight contribution from the node. If the current node of the decoded chromosome is an activity node and the node-list includes that node then weight contribution is zero, otherwise weight contribution is c2 . When an activity node is evaluated, action script removes the current node from the node-list, and adds immediate neighbor of the current node in the node-list. If the current node of the decoded chromosome is a decision node (line numbers 36-38), then if the node-list includes that node and guard condition on the edge is true then weight contribution is zero, otherwise weight contribution is c2 as shown in equation (7.5). When a decision node is evaluated, action script removes the current node from the node-list, and immediate neighbors of the current node are added in the node-list. If the current node of the decoded chromosome is a merge node (line numbers 36-38), and the node-list includes that node then weight contribution is zero, otherwise weight contribution is c2 as shown in equation (7.6). When a merge node is evaluated, action script removes the current node from the node-list and immediate neighbor of the current node is added in the node-list. If the current node is a fork node (line numbers 36-38), then equation (7.7) decides weight contribution. When the current node is a fork node and the node-list includes that node then weight contribution is zero, otherwise weight contribution is c2 . When a fork node is evaluated, action script removes the current node from the node-list and all immediate neighbors of the current node are added in the node-list. If the current node of the decoded chromosome is a join node (line numbers 19-25), and the node-list includes that node then weight contribution is zero, otherwise weight contribution is c2 as shown in equation (7.8). When a join node is evaluated, and if all predecessor nodes of the join node have processed then action script removes the current node from the node-list and immediate neighbor of the join node is added in 132
7.3. Hybridization of Evolutionary Algorithm the node-list. Otherwise, such chromosome’s fitness is penalized by weight c3 . If node is not available in node-list (line numbers 40-45), then weight contribution is c2 . Action script removes the current node from the node-list. Select randomly chosen adjacent node of previous node. Add adjacent nodes of newly selected node in the node-list. If the current node’s parent node is a decision node then one of the out sequences of the decision node is selected. Therefore, remove remaining adjacent nodes representing start of other sequences (adjacent nodes of the decision node) of the parent node from the node-list. The fittest chromosomes having fitness zero represent valid test scenarios of the activity diagram. Evolutionary algorithm returns a set of fittest chromosomes. After that, chromosome data is retranslated into the problem domain that represents a path in the activity diagram graph. Thus, the evolutionary algorithm generates valid interleaving test scenarios. However, experimental results indicate that the evolutionary algorithm takes a longer time to converge. To overcome this problem, we have proposed hybridization of EA with graph theoretic algorithms as explained in the next section.
7.3
Hybridization of Evolutionary Algorithm
In recent years, complex problems have been solved by the combination of metaheuristic and other techniques. The idea of hybridization refers to taking advantage of the potentialities of some technique and overcoming their weaknesses using other mechanisms that make them more robust. Hybrid approaches can provide more efficient behavior, higher flexibility, and improved solutions (Kumar and Rockett, 1998; Kumar and Banerjee, 2011; Saha et al., 2013). This subsection presents hybridization of graph search and permutation techniques with evolutionary algorithm. Hybridization suggested in this section accelerates the convergence of our non-hybrid evolutionary algorithm. Hybridization framework is shown in Figure 7.4, which is used to generate concurrent test scenarios from activity diagrams. First, the framework extracts the required information from the activity diagram and generates an activity diagram graph. Then, a graph search (or permutation) technique is applied to generate paths from the activity diagram graph. These paths may not be valid interleaving paths, but these paths may be the sub-paths of the valid (complete) interleaving paths, which accelerate the 133
7. Test Scenario Generation from Nested Concurrent Activity Diagrams
Activity control flow graph
Apply graph search or permutation technique to generate paths (sub-paths)
Concurrent Test Scenarios
Evolve the population to generate valid concurrent test scenarios
Initial population
Populate EA initial population from sub-paths
Sub- paths
Activity diagram
Extract information and transform into activity control flow graph
Figure 7.4: A hybrid evolutionary algorithm framework for generating interleaving test scenarios
process of generating the valid interleaving paths using an evolutionary algorithm from the activity diagram graph. After that, above generated paths are seeded in the initial population of the evolutionary algorithm, and the remaining population is filled as usual randomly. Finally, the evolutionary algorithm generates valid test scenarios using preprocessed population. Hybridization benefits the evolutionary algorithm to generate the valid interleaving paths with improved convergence time. The main motivation of using this mechanism is to increase the probability of obtaining interleaving path affixes (prefix or suffix) from the preprocessed population.
7.4
Experimental Results
Based on the approaches discussed in the above sections, we have developed algorithms in Java that automate the process of test scenario generation. For evolutionary 134
7.4. Experimental Results algorithm, we have used Java Genetic Algorithms Package (JGAP) as evolutionary algorithm base tool. Our implementation takes input an activity diagram graph in the form of text data file, and it generates a set of test scenarios. Activity diagrams are obtained in the XML Metadata Interchange (XMI) files from UML-tools. Our parser-program parses the XMI file, and it returns required information to construct an activity diagram graph. Parser-program also collects data types and guard condition information required for test data generation. Test scenarios are generated using the evolutionary algorithm based on the activity diagram graph. Later, test data is generated using category partition method for each test scenario. A test scenario and test data together form a test case for integration level and system level testing. We have set population size 200 and number of iteration 2000 for each run. To generate test suite, each algorithm is run multiple times, and distinct test scenarios are collected as a final test suite. All experiments are conducted using 3.00 GHz Intel Core 2 Quad CPU with 4 GB RAM desktop computer. To carry out a detailed study of the test scenario generation algorithms, a testbed of eight different types of activity diagrams is chosen. The testbed is a collection of activity diagrams according to classification discussed in the Chapter 5. Proposed algorithms are evaluated using non-concurrent and concurrent activity test diagrams from the testbed, and application scenarios as listed in Table 7.4. Non-concurrent Activity Diagrams- Non-concurrent activity diagrams are simple to traverse. Traditional graph search algorithms generate paths, which are not necessarily valid test scenarios. A valid test scenario must start from the start activity and stop at the end activity in an activity diagram. To generate valid test scenario, a search should start from the start node and stop at the end node, and this small modification in standard algorithms is used to generate valid test scenarios. We have developed two variants of DFS algorithm: (i) DFS test scenario generator, which statically (chronological order) selects next neighbor in data structure, and (ii) random DFS test scenario generator, which selects a next neighbor randomly from the current node’s neighbor list. We have applied DFS test scenario generator, random DFS test scenario generator, and evolutionary algorithm to generate test scenarios. Table 7.2 shows results for non-concurrent activity diagrams, and lists total generated paths, activity coverage, basic path coverage, simple path coverage, and time taken to complete search by each algorithm. These algorithms run on non-nested activity diagrams SDD and SDI, and nested activity diagrams NDD and NDI as shown in 135
7. Test Scenario Generation from Nested Concurrent Activity Diagrams Figure 5.8, Figure 5.9, Figure 5.10, and Figure 5.11. Table 7.2: Coverage analysis of test scenarios generated for non-concurrent activity diagrams Activity diagram
SDD
SDI
NDD
NDI
Algorithm
Total paths
DFS Path Generator Random DFS Path Generator Evolutionary Algorithm DFS Path Generator Random DFS Path Generator Evolutionary Algorithm DFS Path Generator Random DFS Path Generator Evolutionary Algorithm DFS Path Generator Random DFS Path Generator Evolutionary Algorithm
1 6 6 1 4 4 1 4 4 1 4 4
Coverage (%) Activity Basic Simple path path 075.00 016.66 016.66 100.00 100.00 100.00 100.00 100.00 100.00 075.00 025.00 025.00 100.00 100.00 100.00 100.00 100.00 100.00 083.33 025.00 025.00 100.00 100.00 100.00 100.00 100.00 100.00 070.58 033.33 025.00 100.00 100.00 100.00 100.00 100.00 100.00
Time (sec) 0.00 0.00 4.29 0.00 0.00 4.29 0.00 0.00 7.62 0.00 0.00 6.26
Results show that random DFS and evolutionary algorithm achieved 100% activity, basic path, and simple path coverage for non-nested (serial) non-concurrent activity diagrams and nested non-concurrent activity diagrams. Whereas, DFS test scenario generator algorithm generates single test scenario in all activity diagrams. The static selection procedure always selected first neighbor in the current node neighbor list that leads to the generation of the same path in each run. On the contrary, random DFS algorithm generated different paths in different run leading to achieve 100% coverage. As shown in Figure 7.5 and Figure 7.6, random DFS and evolutionary algorithms generated the maximum number of valid test scenarios. Time required to generate test scenarios for DFS and random DFS algorithms is very little, whereas evolutionary algorithm took up to 4.29 seconds in the case of non-nested non-concurrent activity diagrams and up to 7.62 seconds in the case of nested non-concurrent activity diagrams. Evolutionary algorithm starts with a random population and improves population through generations that are guided by the fitness evaluation process. EA generated all valid test scenarios and achieved 100% coverage. Due to evolutionary processes, evolutionary algorithm needs longer time. In conclusion, random DFS algorithm is the best choice for non-concurrent activity diagrams to generate test scenarios. 136
7.4. Experimental Results S D D S D I N D D N D I
S D D S D I N D D N D I 7
6
9 8 7
5
P a th s G e n e ra te d
T im e in S e c o n d s
6 4
3
2
5 4 3 2 1
1 0 0
-1 1
2
A lg o r 1 . D S F P a th 2 . R a n d o m D 3 . E v o lu tio n a
3
ith G F ry
m e n S A
1
2
A lg o r 1 . D S F P a th 2 . R a n d o m D 3 . E v o lu tio n a
s e ra to r P a th G e n e ra to r lg o r ith m
Figure 7.5: A graph representing a number of test scenarios generated by DFS, Random DFS, and Evolutionary algorithms for non-concurrent activity diagrams
3
ith G F ry
m e n S A
s e ra to r P a th G e n e ra to r lg o r ith m
Figure 7.6: A graph representing time required to generate test scenarios by DFS, Random DFS, and Evolutionary algorithms for non-concurrent activity diagrams
Concurrent Activity Diagrams- Concurrent activity diagrams are complex for generating test scenarios (Xu et al., 2005). Traditional graph search algorithms generate paths that are not necessarily valid concurrent test scenarios. A valid concurrent test scenario starts from the start activity and stops at the end activity, and it must include interleaving of activities inside the fork-join structures. We have developed DFS-BFS, DFS-Level permute, evolutionary algorithm, and hybrid evolutionary algorithms to generate valid concurrent test scenarios. For concurrent activity diagrams, in addition to basic path coverage and simple path coverage; we have used interleaving activity path coverage and total activity path coverage criteria defined in the Chapter 3. Table 7.3 shows results for concurrent activity diagrams. Table contains total generated path, valid paths, basic path coverage, simple path coverage, interleaving activity path coverage, total activity path coverage, and time for each algorithm. To generate interleaving test scenarios, we have applied DFS-BFS test scenario generator by Kundu and Samanta (2009), DFS-LevelPermute test scenario generator, evolutionary algorithm, hybrid DFS-EA, hybrid BFS-EA, and hybrid LevelPermuteEA algorithms. Algorithms run on non-nested concurrent activity diagrams SDF and SFF, and nested concurrent activity diagrams NDFD and NDFF as shown in Figure 5.12, Figure 5.13, Figure 5.14, and Figure 5.15 from the classification presented in the Chapter 5. Figure 7.7 and Figure 7.8 show that simple path coverage criterion is achieved 100% by all the algorithms. In non-nested concurrent activity diagrams, 137
7. Test Scenario Generation from Nested Concurrent Activity Diagrams Table 7.3: Basic path coverage (BPC), simple path coverage (SPC), interleaving activity path coverage (IAPC), and total activity path coverage (TAPC) analysis for concurrent activity diagrams Activity Algorithm Diagram DFS-BFS DFS-LevelPermute Evolutionary Algorithm SDF Hybrid DFS-EA Hybrid BFS-EA Hybrid LevelPermute-EA DFS-BFS DFS-LevelPermute Evolutionary Algorithm SFF Hybrid DFS-EA Hybrid BFS-EA Hybrid LevelPermute-EA DFS-BFS DFS-LevelPermute Evolutionary Algorithm NDFD Hybrid DFS-EA Hybrid BFS-EA Hybrid LevelPermute-EA DFS-BFS DFS-LevelPermute Evolutionary Algorithm NDFF Hybrid DFS-EA Hybrid BFS-EA Hybrid LevelPermute-EA
Total paths 12 72 39 30 60 17 16 16 02 01 09 01 10 – 05 03 07 03 12 – 01 02 05 02
Valid paths 12 72 39 30 60 17 02 16 02 01 09 01 01 01 05 03 07 03 01 01 01 02 05 02
BPC – – – – – – – – – – – – 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00
Coverage (%) SPC IAPC 100.00 06.66 100.00 40.00 100.00 21.66 100.00 16.66 100.00 33.33 100.00 09.44 100.00 03.33 100.00 26.66 100.00 03.33 100.00 01.66 100.00 15.00 100.00 01.66 033.33 00.00 033.33 00.00 100.00 33.33 100.00 16.66 100.00 50.00 100.00 16.66 050.00 00.00 050.00 00.00 050.00 00.00 100.00 04.16 100.00 16.66 100.00 04.16
TAPC 06.66 40.00 21.66 16.66 33.33 09.44 03.33 26.66 03.33 01.66 15.00 01.66 07.69 07.69 38.46 23.07 53.84 23.07 04.00 04.00 04.00 08.00 20.00 08.00
Time (sec) 0.03 0.02 1230.00 501.92 195.02 513.36 0.02 0.00 499.00 196.00 35.81 198.60 0.02 – 84.27 5.61 22.33 31.86 0.02 – 90.17 3.58 36.70 48.14
any single interleaving activity path will achieve 100% simple path coverage, and results in Table 7.3 show that all algorithms have generated at least one concurrent test scenario. Interleaving activity path and total activity path coverage overlap in the case of non-nested concurrent activity diagrams. This coverage overlap is due to non-availability of basic paths in non-nested concurrent activity diagrams. For nested concurrent activity diagrams, all hybrid evolutionary algorithms achieve 100% simple path coverage as shown in Figure 7.9 and Figure 7.10. Whereas, DFSBFS and DFS-LevelPermute algorithms are unable to achieve 100% simple path coverage criteria because they fail to generate valid concurrent test scenarios inside nested fork-join and decision structures. Hybrid evolutionary algorithms have achieved 5 to 50% interleaving activity path coverage and 8 to 55% total activity path coverage. Test scenario generation of all algorithms is plotted in Figure 7.11 that compares 138
7.4. Experimental Results S im p le P a th C o v e r a g e In tr e le a v in g P a th C o v e r a g e T o ta l P a th C o v e ra g e
1 1 0
1 0 0
1 0 0
9 0
9 0
8 0
8 0
7 0
7 0
C o v e ra g e
1 1 0
6 0 5 0 4 0
6 0 5 0 4 0
3 0
3 0
2 0
2 0
1 0
1 0 0
0 1
2
3
4
A lg o r ith m s 1 . D F S _ B F S 3 . E v o lu tio n a r y A lg o r ith m 5 . H y b r id L e v e lP e r m u te _ E A
5
6
1
3
4
5
S im p le P a th C o v e r a g e In te r le a v in g P a th C o v e r a g e T o ta l P a th C o v e ra g e
1 2 0 1 1 0
1 0 0
1 0 0
9 0
9 0
8 0
8 0
7 0
7 0
C o v e ra g e
1 1 0
6 0 5 0
%
4 0
6 0 5 0 4 0
3 0
3 0
2 0
2 0
1 0
1 0 0
6
2 . D F S _ L e v e lP e r m u te 4 . H y b r id D F S _ E A 6 . H y b r id B F S _ E A
Figure 7.8: A graph representing coverage analysis for SFF: Sequential fork-fork activity diagram
S im p le P a th C o v e r a g e In te r le a v in g P a th C o v e r a g e T o ta l P a th C o v e ra g e
1 2 0
C o v e ra g e
2
A lg o r ith m s 1 . D F S _ B F S 3 . E v o lu tio n a r y A lg o r ith m 5 . H y b r id L e v e lP e r m u te _ E A
2 . D F S _ L e v e lP e r m u te 4 . H y b r id D F S _ E A 6 . H y b r id B F S _ E A
Figure 7.7: A graph representing coverage analysis for SDF: Sequential decisionfork activity diagram
%
S im p le P a th C o v e r a g e In tr e le a v in g P a th C o v e r a g e T o ta l P a th C o v e ra g e
1 2 0
%
%
C o v e ra g e
1 2 0
0 1
2
3
4
5
6
1
A lg o r ith m s 1 . D F S _ B F S 3 . E v o lu tio n a r y A lg o r ith m 5 . H y b r id L e v e lP e r m u te _ E A
2
3
A lg o r ith m s 1 . D F S _ B F S 3 . E v o lu tio n a r y A lg o r ith m 5 . H y b r id L e v e lP e r m u te _ E A
2 . D F S _ L e v e lP e r m u te 4 . H y b r id D F S _ E A 6 . H y b r id B F S _ E A
Figure 7.9: A graph representing coverage analysis for NDFD: Nested decisionfork-decision activity diagram
4
5
6
2 . D F S _ L e v e lP e r m u te 4 . H y b r id D F S _ E A 6 . H y b r id B F S _ E A
Figure 7.10: A graph representing coverage analysis for NDFF: Nested decisionfork-fork activity diagram
the number of test scenario generated by all the algorithms for concurrent activity diagrams. It shows a significant improvement by hybrid evolutionary algorithms for nested concurrent activity diagrams. Whereas, DFS-LevelPermute algorithm generated the maximum number of test scenarios for non-nested concurrent activity diagrams. Hybrid-BFS-EA algorithm generated the maximum number of test scenarios for nested concurrent activity diagrams. A significant improvement in convergence time is observed by hybrid evolutionary algorithms shown in Figure 7.12. For nonnested and nested concurrent activity diagrams, Hybrid-BFS-EA algorithm requires comparatively less time to converge in all hybrid algorithms. Results clearly demonstrate that DFS-LevelPermute test scenario generator achieved better interleaving activity path coverage and total activity path coverage for non139
7. Test Scenario Generation from Nested Concurrent Activity Diagrams S F D S F F N D F D N D F F
6 4
S F D S F F N D F D N D F F
1 2 0 0
3 2
1 0 0 0
1 6
T im e in S e c o n d s
P a th s g e n e ra te d
8 0 0 8
4
6 0 0
4 0 0
2 2 0 0 1 0 0 .5 1
2
3
4
5
6 1
A lg o r ith m s 1 . D F S _ B F S 2 . D F S _ L e v e lP e r m u te 3 . E v o lu tio n a r y A lg o r ith m 4 . H y b r id D F S _ E A 5 . H y b r id L e v e lP e r m u te _ E A 6 . H y b r id B F S _ E A
2
3
4
5
6
A lg o r ith m s 1 . D F S _ B F S 2 . D F S _ L e v e lP e r m u te 3 . E v o lu tio n a r y A lg o r ith m 4 . H y b r id D F S _ E A 5 . H y b r id L e v e lP e r m u te _ E A 6 . H y b r id B F S _ E A
Figure 7.11: A graph representing a number of test scenarios generated for concurrent activity diagrams by different algorithms (log scale)
Figure 7.12: A graph representing time required to generate test scenarios for concurrent activity diagrams by different algorithms
nested concurrent activity diagrams. On the contrary, DFS-LevelPermute and DFSBFS algorithms do not achieve interleaving activity path coverage for nested concurrent activity diagrams. In the case of nested concurrent activity diagrams, HybridBFS-EA has achieved the better interleaving activity path coverage and total activity path coverage as compared to other algorithms. Moreover, it has taken minimum time to generate interleaving test scenarios. In conclusion, Hybrid-BFS-EA algorithm is suitable to generate test scenarios for the nested concurrent activity diagrams, and DFS-LevelPermute algorithm is suitable for the non-nested concurrent activity diagrams.
Application Scenarios- Results in the previous sub-section have demonstrated significant improvement in test scenario generation for non-concurrent and concurrent activity diagrams with proposed new approaches such as DFS-LevelPermute, evolutionary algorithm, and Hybrid evolutionary algorithms. To verify the effectiveness of these algorithms for real application scenarios, five application scenarios are chosen: Automatic Teller Machine (ATM) money withdraw, Airport check in, on-line purchase, order processing, and graphics utility. All these activity diagrams depict different aspects of design, and their attributes are listed in Table 7.4. Application 3 is non-concurrent application, and Application 5 does not include any basic paths (non-interleaving paths). Table 7.5 shows total test scenarios generated, basic path coverage, simple path coverage, interleaving activity path coverage, and total activity path coverage by DFS-BFS, DFS-LevelPermute, evolutionary al140
7.5. Conclusions Table 7.4: Activity diagrams’ attributes: total number of activities, basic paths, simple paths, interleaving paths, and total paths for different applications App. No. App 1 App 2 App 3 App 4 App 5
Activity diagram Airport check in ATM On-line purchase Order processing Graphics Utility
# Activities 13 21 20 18 14
Basic 01 03 06 02 00
# Simple 03 05 06 04 01
Paths Interleaving 06 08 00 30 56
Total 07 11 06 32 56
gorithm, and hybrid evolutionary algorithms. As shown in Table 7.5 total activity path coverage and interleaving activity path coverage are in decreasing order for algorithms: evolutionary algorithm, Hybrid-BFS-EA, Hybrid-LevelPuermute, HybridDFS-EA, DFS-LevelPermute, and Random DFS-BFS respectively. For Application 2, evolutionary algorithm has achieved 81.81% total activity path coverage as compared to 63.63% by Hybrid-BFS-EA algorithm. In all other cases, both algorithms have achieved same total activity path coverage, but Hybrid-BFS-EA has achieved significant improvement by achieving the same coverage in approximately 50% less time as compared to the (non-hybrid) evolutionary algorithm. For applications App1-App5 except on-line purchase App3 classification of test scenarios is shown in Table 4.6 result rows of applications App7-App10, respectively.
7.5
Conclusions
In this chapter, we have proposed test scenarios generation approaches for concurrent activity diagrams to uncover concurrency errors. We have presented the new test scenario generation approaches that achieve higher coverage of interleaving activity path criterion for concurrent activity diagrams. At the design level, test scenarios exhibit a high-level abstraction; as a result, they represent comparatively fewer interleavings than implementation level. To detect concurrency errors, exploration of more execution interleavings is required and it is achieved through the new approaches: evolutionary algorithm and hybrid evolutionary algorithms. Evolutionary algorithms do parallel search for multiple interleaving sequences with the help of randomness injected by reproduction operators. Experimental results show that Random DFS test scenario generation algorithm 141
7. Test Scenario Generation from Nested Concurrent Activity Diagrams Table 7.5: Basic path (BPC), simple path (SPC), interleaving activity path (IPC), and total interleaving activity path (TAPC) coverage analysis for test scenarios generated by DFS-BFS, DFS-LevelPermute, evolutionary algorithm, and hybrid evolutionary algorithms for applications listed in Table 7.4
App. No. App App App App App
App App App App App
DFS-LevelPermute Coverage (%) SPC IAPC TAPC 066.66 33.33 42.85 060.00 00.00 27.27 050.00 – 50.00 050.00 00.00 06.25 100.00 14.28 14.28
time (sec) 0.00 0.00 0.00 0.00 0.02
1 2 3 4 5
Path BPC 06 100.00 09 100.00 06 100.00 03 100.00 13 –
Evolutionary Algorithm Coverage (%) SPC IAPC TAPC 100.00 83.33 085.71 100.00 75.00 081.81 100.00 – 100.00 075.00 03.33 009.38 100.00 23.21 023.21
time (sec) 050.84 104.59 138.97 069.16 110.33
Path BPC 05 100.00 04 100.00 04 100.00 02 100.00 11 –
Hybrid DFS-EA Coverage (%) SPC IAPC TAPC 100.00 66.66 71.42 080.00 12.50 36.36 066.66 – 66.66 050.00 00.00 06.25 100.00 19.64 19.64
time (sec) 25.19 06.90 04.75 00.52 99.78
1 2 3 4 5
Path BPC 06 100.00 07 100.00 06 100.00 03 100.00 13 –
Hybrid BFS-EA Coverage (%) SPC IAPC TAPC 100.00 83.33 085.71 100.00 50.00 063.63 100.00 – 100.00 075.00 03.33 009.38 100.00 23.21 023.21
time (sec) 020.34 055.97 051.69 015.45 103.80
Hybrid LevelPermute-EA Path Coverage (%) BPC SPC IAPC TAPC 05 100.00 100.00 66.66 071.42 07 100.00 080.00 50.00 063.63 06 100.00 100.00 – 100.00 03 100.00 075.00 03.33 009.38 08 – 100.00 14.28 014.28
time (sec) 25.39 59.50 58.97 35.52 01.28
App. No.
time TAPC (sec) 42.85 0.02 54.54 0.02 66.66 0.00 06.25 0.02 00.00 0.02
Path BPC 03 100.00 03 100.00 03 100.00 02 100.00 02 –
1 2 3 4 5
App. No. App App App App App
DFS-BFS Coverage (%) SPC IAPC 100.00 33.33 100.00 37.50 066.66 – 050.00 00.00 000.00 00.00
Path BPC 03 100.00 06 100.00 04 100.00 02 100.00 00 –
is better for non-concurrent activity diagrams. This algorithm achieves 100% activity coverage, basic path coverage, and simple path coverage. DFS-LevelPermute algorithm is good for the non-nested concurrent activity diagrams. Hybrid BFS-EA algorithm is good for nested concurrent activity diagrams to generate test scenarios, and it has improved interleaving activity path coverage significantly and improved convergence time. In summary, interleaving activity path coverage is hard to achieve compared to basic path and simple path coverage criteria. Due to exponential space of interleaving paths, it is not possible to always achieve 100% interleaving activity path coverage. For concurrency errors, which are probabilistically very rare, higher the interleaving activity path coverage more the possibility of uncovering errors. DFS, DFS-BFS, and 142
7.5. Conclusions DFS-LevelPermute test scenario generators generate a same set of test scenarios in every run. On the contrary, evolutionary algorithms generate a different set of test scenarios in every run thereby introducing new test scenarios in successive runs to the test suite. Hybridization of the evolutionary algorithm incrementally strengthened the test suite. In the next chapter, we describe an approach for exploring sequence diagrams to generate bug-triggering test scenarios, which will achieve better concurrency behavioral coverage.
143
Chapter 8 Test Scenario Generation from Sequence Diagrams with Concurrent Behavior In contemporary computing, concurrency has crucial importance not only in highend servers, but also in personal computers and mobiles, more so with the advent of multi-core hardware. Concurrent program execution is unpredictable. This may give rise to a different set of concurrency errors in concurrent systems like data race, deadlock, starvation and synchronization (Shousha et al., 2009). Therefore, testing of concurrent systems is becoming increasingly important. UML sequence diagrams support parallel fragment and critical region constructs to elaborate concurrency in system designs OMG (2011). Critical region exploration for concurrent test case generation remains unaddressed in sequence diagrams. A critical section in an application at design stage, though an atomic activity, could be implemented using a variety of high-level programming language constructs. There could be errors in its implementation (Vaziri et al., 2006). Existing UML-based graph theoretic test scenario generation algorithms have not adequately explored parallel fragments and critical regions of sequence diagrams for test scenario generation. Therefore, it is essential to find alternative techniques to generate test scenarios that may uncover concurrency errors. In this chapter, our main aim is to generate test scenarios from sequence diagrams for concurrency testing. Sequence diagrams that support constructs such as parallel fragment, critical region, and general ordering are explored to generate test 145
8. Test Scenario Generation from Sequence Diagrams with Concurrent Behavior scenarios. A rich form of CFG, which we call a message control flow graph, is used as an intermediate model to generate interleaving test scenarios. A dependency preserving depth-first search algorithm is proposed to generate valid test scenarios for concurrency testing. Mutation analysis is carried out to investigate the fault detection effectiveness of the test suite generated by the proposed technique satisfying concurrent coverage criteria discussed in the Chapter 3. This chapter is organized as follows. Section 8.1 provides a motivating example and preliminaries. Section 8.2 discusses test scenario generation method using DP-DFS algorithm. Section 8.3 includes results and analysis. Finally, Section 8.4 concludes the chapter.
8.1
Preliminaries
A sequence diagram is an interaction diagram, which focuses on message exchange among the participating objects. Consider the bank transaction scenario to understand concurrent system design using a sequence diagram shown in Figure 1.2. The bank money transfer process consists of a series of messages sent between banks directing each to make debit and credit entries to complete the transaction. In UML sequence diagrams, combined fragments are used to group a set of messages together to show conditional flows. A combined fragment is defined by an interaction operator and the corresponding interaction operands. There is a set of interaction operators that define semantics of the combined fragments (OMG, 2011). Here, we consider semantics of only three combined fragments that express concurrency in sequence diagrams. • Parallel fragment- The interaction operator par defines parallel execution behavior of the combined fragment operands. Messages in different operands may interleave in any way as long as the ordering imposed by each operand is preserved. For example, in Figure 1.2 the parallel fragment transf er has two operands; they contain messages {m2 , m3 , m4 } and {m5 , m6 , m7 } in the first and second operands, respectively. • Critical fragment- The interaction operator critical defines the critical region combined fragment. Critical region is treated atomic by the enclosing fragment, i.e., messages inside the critical region should not be interleaved. Even if interleaving of messages happen inside the critical region, other fragments do not get 146
8.1. Preliminaries access to shared objects’ data until the end of the critical region. Critical regions are enclosed inside operands of parallel fragments. For example, Figure 1.2 has two critical regions critical1 and critical2 inside the parallel fragment transf er. This construct enforces atomic execution of messages {m3 , m4 } in the first interaction operand and messages {m6 , m7 } in the second interaction operand. • General orderings- In parallel fragments, a causal ordering between messages is shown by the general ordering construct in sequence diagrams. Some messages in different operands may be related, and this is expressed by a directed dashedline with an arrow. Such messages need to be executed in the specified order. For example, Figure 8.3 represents general ordering construct by dashed line, which starts after message m3 in the first interaction operand and ends before message m8 in the second interaction operand. This signifies the message m8 must execute after the message m3 in any execution. Sequence diagram in Figure 1.2 uses the parallel and critical region fragments to express the concurrent behavior of the system. The bank object bank initiates the transfer of money from the account account1 to account account2 and from the account account2 to account account1 . Two transaction thread objects thread1 and thread2 participate to complete fund transfer functionality. Threads thread1 and thread2 receive transfer(sourceAccount, destinationAccount, amount) messages. These two messages start interaction inside parallel fragment that means the two interaction operands run parallel. Source and destination accounts in both operands are accessed by thread1 and thread2 , and they represent shared data; hence both operands run concurrently. Critical region fragment in both operands of parallel fragment enforces atomic execution of both transfer functions. It is necessary to represent the sequence diagram in a suitable intermediate representation to generate test cases. Graph theoretic approaches traverse the intermediate representation from the start of the sequence diagram to the end of the sequence diagram generating test paths. Intermediate representation- The most commonly used intermediate representation is a control flow graph. Concurrency constructs like parallel fragment and fork-join are not supported by CFG representation; therefore, a CFG-like representation may be used to represent sequence diagrams having parallel fragments. A CFG-like representation for Figure 1.2, the bank transaction sequence diagram, is 147
8. Test Scenario Generation from Sequence Diagrams with Concurrent Behavior shown in Figure 8.1. It has 16 nodes in three threads, one main thread and two forked threads. Each forked thread has five nodes (three message nodes, two control nodes - critical section start and critical section end). In this representation, we have used message names to normal nodes and operand names to control nodes. Combined fragments represent control flow in sequence diagrams. Combined fragments are represented by start node and end node with labels CF xxStart and CF xxEnd (xx- operand name), respectively. In this chapter, we say a path generated by the test scenario generator algorithm from the start node to the end node is the valid concurrent path if it represents a serialized schedule for concurrent execution. A path from the start node to the end node is the invalid concurrent path if it represents a non-serialized schedule for concurrent execution. A valid concurrent path represents a positive test scenario, and an invalid concurrent path represents a negative test scenario. A path generated by a test scenario generation algorithm from the start node to the end node is the complete concurrent path if it is either a valid or an invalid concurrent test path. A path from the start node to the end node is the incomplete concurrent path if it represents neither serialize nor non-serialize schedule for concurrent execution. For example, in Figure 8.1 a path represented by sequence of nodes < sdStart → m1 → CF parStart → m2 → CF critical1Start → m3 → m4 → CF critical1End → CF parEnd → m8 → sdEnd > is an incomplete concurrent path.
m2
sd Start
m1
CF critical 1 Start
m3
m4
CF critical 1 End
CFpar End
CFpar Start
m5
CF critical 2 Start
m6
m7
m8
sd End
CF critical 2 End
Figure 8.1: A CFG-like representation for the money transfer bank transaction example shown in the Figure 1.2
Previous researchers have proposed CFG-like graph representations for exploring sequence diagrams for generating test cases. Lei and Lin (2008) have proposed 148
8.1. Preliminaries message flow graph, which is a control flow graph of a sequence diagram. However, their approach processes only three combined fragments, namely alternative fragment, optional fragment, and loop fragment. Their approach uses depth-first search (DFS) algorithm, thereby generating sequential test scenarios. We have applied standard graph search algorithms on CFG-like representation shown in Figure 8.1 to understand the applicability of standard graph search algorithms for generating test scenarios in the presence of concurrency constructs. Application of the DFS scenario generation algorithm to the CFG-like representation generates two sequences: < sdStart → m1 → CF parStart → m2 → CF critical1Start → m3 → m4 → CF critical1End → CF parEnd → m8 → sdEnd > and < sdStart → m1 → CF parStart → m5 → CF critical2Start → m6 → m7 → CF critical2End→ CF parEnd → m8 → sdEnd >. These both scenarios are not complete concurrent test scenarios because a concurrent path inside the parallel fragment is lost. Application of the BFS scenario generation algorithm to the CFG-like representation generates two sequences: < sdStart → m1 → CF parStart → m5 −−→ m2 −−→ CF critical2Start −−→ CF critical1Start −−→ m6 −−→ m3 −−→ m7 −−→ m4 CS CS CS CS CS CS CS −−→ CF critical2End −−→ CF critical1End → CF parEnd → m8 → sdEnd > and CS CS < sdStart → m1 → CF parStart → m2 −−→ m5 −−→ CF critical1Start −−→ CF critical2Start CS CS CS −−→ m3 −−→ m6 −−→ m4 −−→ m7 −−→ CF critical1End −−→ CF critical2End → CS CS CS CS CS CS CF parEnd → m8 → sdEnd >. BSF generates complete concurrent test paths inside the parallel fragments, but it may lead to incomplete test paths either due to conditional flows or unequal messages in the fragment operands. The above illustration clearly indicates the necessity of a suitable intermediate representation of sequence diagrams and test scenario generation algorithm for concurrency testing. To circumvent the above problems, Nayak and Samanta (2010) have proposed construction of structured composite graph as an intermediate form and used DFS approach to generate test scenarios. Khandai et al. (2011) have proposed construction of concurrent composite graph as an intermediate form and used DFS-BFS approach to generate test scenarios. Construction procedures of both SCG and CCG graphs are similar. Both approaches abstract all messages inside parallel fragment operands as a single block node in their intermediate representation to curb the exponential increase in interleaving paths. We have translated Figure 1.2, bank transaction example, into SCG (or CCG) representation, which includes two block nodes block1 and block2 as shown in Figure 8.2. In their representation, f ork and join nodes are used to represent parallel fragment start and end nodes. 149
8. Test Scenario Generation from Sequence Diagrams with Concurrent Behavior Application of the DFS scenario generation algorithm to SCG generates following
m2
CF critical 1 Start
m3
m4
CF critical 1 End
block1
sd Start
m1
join
fork
m8
sd End
block2
m5
CF critical 2 Start
m6
m7
CF critical 2 End
Figure 8.2: A structured composite graph / concurrent composite graph representation for the money transfer bank transaction example shown in the Figure 1.2
two sequences: < sdStart → m1 → f ork → block1 → join → m8 → sdEnd > and < sdStart → m1 → f ork → block2 → join → m8 → sdEnd >. These both scenarios are incomplete concurrent test scenarios because a concurrent path inside parallel fragment is lost. Application of DFS-BFS algorithm to CCG generates following two sequences: < sdStart → m1 → f ork → block1 −−→ block2 → join → m8 → sdEnd > CS and < sdStart → m1 → f ork → block2 −−→ block1 → join → m8 → sdEnd >. CS Although, this technique generates complete concurrent test paths inside parallel fragment. Both approaches process parallel combined fragments, but they do not explore message interleaving and do not process critical fragment and general ordering constructs. A test scenario generation algorithm should explore maximum number of execution interleavings inside interleaving space of the sequence diagram. Interleaving space- A sequence diagram can express the possibility that a certain scenario may occur. In other words, a sequence diagram models behavior of one of the possible interactions. The scenario needs to be explored to generate different message interleaving sequences to elaborate all interactions. Generating all possible message interleaving sequences is a major difficulty to test scenario generators. An interleaving sequence is obtained by merging together two or more sequences. Suppose that a sequence diagram with concurrent constructs P consists of two operands p1 and p2 in parallel fragment. The implementation of P executes any single permutation of messages (tasks) inside the parallel fragment. This permutation can be obtained 150
8.1. Preliminaries by interleaving messages (tasks) of the two operands. If the operand p1 consists of m messages and the operand p2 consists of n messages then there are not less than ((m + n)!/(m!) ∗ (n!)) TEIS. The interleaving space is exponential with respect to the number of messages in each operand. Concurrency testing poses a challenge due to exponential interleaving space in the concurrent programs/ designs. For example, consider Figure 1.2, where parallel combined fragment’s each operand has three messages and its intermediate representation has five nodes from which test scenarios are generated. Thus there are in total ((5 + 5)! / (5!) * (5!)) = 252 T EIS. Some of the interleavings amongst T EIS may lead to concurrency errors, and that small subset of interleaving sequences is useful for concurrency testing. Partial order- The partial order amongst tasks plays an important role to find a smaller subset of total execution interleaving sequences in the presence of a causality relation of concurrent systems. Causality is the relationship between the first event (the cause) and a second event (the effect), where the second event is understood as a consequence of the first event. In causal ordering past events influence future events. Partial ordering of tasks in a series is an ordering that does not specify the exact order of every task, but only defines the order between certain key tasks that depend on each other. Partial orders are often used to reflect causal relationships between tasks. For sequential systems, such as single-threaded programs, events can be observed in the order they occur. In other words, the order in which events happen is a total order in a single-threaded application, because the CPU executes instructions sequentially. Whereas, in concurrent systems, such as multi-threaded programs, events of an individual thread still execute in the order they occur, but there are typically many possible ways in which events of different threads can be interleaved (Campbell et al., 2005). The only concern is a causal relationship between tasks of different threads. In other words, it is meaningful to consider only interleaving of those tasks that have a dependency in a multi-threaded application. For example, consider three tasks {p, q, r}, if they always have to happen in order p → q → r then they are in the total order. However, if p must happen before r, but one does not care when q happens, then the sequence, {< p − q − r >, < p − r − q >, < q −p−r >}, satisfies the partial order imposed by the dependency relationship. For these three events, there are total six (3!) possible total order interleavings and three partial order interleavings, i.e., 50% reduction in the interleaving scenarios. In multithreaded programs, partial order of events is defined by the access of shared resources. 151
8. Test Scenario Generation from Sequence Diagrams with Concurrent Behavior For sequence diagrams, message dependency can be inferred from critical regions and general ordering constructs. These constructs force to use locking mechanism for mutual exclusion or semaphores for signaling. Therefore, for testing purpose violations of these mechanisms needs to be tested. Partial order sequences from critical regions and general ordering may help to detect concurrency errors. In the presence of causality relationship between two operands of parallel fragments in a sequence diagram, a set of partial order execution interleaving sequences (PEIS) is a subset of total execution interleaving sequences T EIS. Therefore, P EIS significantly reduces the total testing effort.
8.2
Dependency Preserving DFS Based Test Scenario Generator
The purpose of this approach is to search the most of the interleavings from parallel fragments to satisfy the behavioral concurrency coverage criteria defined in the Chapter 3. First, we present a technique to translate a sequence diagram into an intermediate representation a message control flow graph. An MCFG is a suitable representation to generate the maximum number of complete concurrent test scenarios with the help of extra added message-interleaving edges inside parallel fragments. After that, we present the special version of the DFS algorithm called dependency preserving DFS algorithm, which avoids cycles that are introduced during the construction of the MCFG. A DP-DFS algorithm enables generating complete concurrent paths from an MCFG that represent positive and negative concurrent test scenarios. Finally, we introduce a reduced MCFG, which reduces message interleavings generated by DP-DFS algorithm thereby reducing the test suite size. Specifically, reduced MCFG representation allows generating only partial order interleavings for concurrent designs.
8.2.1
Intermediate Representation: MCFG
A sequence diagram is a collection of lifelines, messages, and fragments in a specific order. A suitable intermediate representation is required to generate test cases from sequence diagrams. We represent a sequence diagram as a directed graph called message control flow graph. Formally a sequence diagram is represented as follows. 152
8.2. Dependency Preserving DFS Based Test Scenario Generator Definition 8.1. A sequence diagram is a tuple {L, M, F, CF, IOP, OS}, where L is a set of lifelines, M is a set of messages, F is a set of fragments, CF is a set of combined fragments, IOS is a set of interaction operands, and OS is an order specification that defines the order for F , M , and CF . An order specification is an order of fragments, combined fragments, and open messages (messages not associated with any fragment operands) from the start to the end of a specified artifact. Fragment includes fragment name and the order specification. Combined fragment is a triplet (operator name, combined fragment name, order specification). Interaction operand is an ordered list of messages and fragments. Each interaction operand is associated with a guard condition, if not specified then it is true. A sequence diagram may be represented using the formal tuple defined above. For example, the information collected from the bank transaction sequence diagram shown in Figure 1.2 is represented as follows. SD={L, M, F, CF, IOP, OS} Lifelines: L = {bank, thread1 , thread2 , account1 , account2 } Messages: M = {m1 , m2 , m3 , m4 , m5 , m6 , m7 , m8 } Fragments: F = {F1 = (M oneyT ransf er,{m1 ,CF1 , m8 })} Combined Fragments: CF = {CF1 = (par, transf er, {IOP1 , IOP2 }); CF2 = (critical, critical1 , {IOP3 }); CF3 = (critical, critical2 , {IOP4 })} Interaction Operands: IOP = {IOP1 = (true, { m2 , CF2 }); IOP2 = (true, {m5 , CF3 }); IOP3 = (true, {m3 , m4 }); IOP4 = (true, {m6 , m7 })} Order specification: OS = {F1 } An MCFG is a richer form of control flow graph where nodes and edges have some extra information useful for test scenario generation algorithms. An MCFG is defined in Chapter 3 Definition 3.16. In sequence diagrams, some actions in different threads may be related in parallel fragments. They need to be executed in a specific order. This is expressed using general ordering construct represented by a directed dashed-line with an arrow in the middle of the messages in sequence diagrams. The general ordering construct is similar to send and receive signals for synchronization. In MCFG construction, these nodes are modeled as Signal and W ait control nodes. When execution reaches Signal node, test scenario generator algorithm will set a binary semaphore value. When execution reaches W ait node, test scenario generator algorithm will check a binary semaphore value if value is set then the algorithm will continue further, otherwise wait till the value is set. For parallel fragment, when all 153
8. Test Scenario Generation from Sequence Diagrams with Concurrent Behavior predecessor nodes of CF parEnd node visited then further processing after CF parEnd node will continue by test scenario generator algorithm. To translate a sequence diagram into an MCFG graph, we have developed a procedure that uses two types of mapping rules. The first type mapping rules are used to map sequence diagram elements to MCFG nodes as shown in Table 8.1. The second type mapping rules are used to map sequence diagram elements to MCFG edges as shown in Table 8.2. MCFG graphs are constructed from sequence diagrams by applying transformation rules given in Table 8.1 and Table 8.2. Guard condition considered true for each edge; in the case of interaction operands, interaction operand (IOP) guard condition is considered.
Table 8.1: Mapping rules for node in MCFG construction procedure MCFG element Node name Node type message message name message node Start of sequence diagram sdStart control node End of sequence diagram sdEnd control node Combined fragment start CFxxStart* control node Combined fragment end CFxxEnd* control node General Ordering Start Signal control node General Ordering End Wait control node * xx- combined fragment name Sequence diagram element
The Figure 3.4 shows the MCFG graph of the bank transaction sequence diagram. First, according to mapping rules defined in Table 8.1 all message nodes and control nodes are generated. Then, all edges among nodes are generated according to mapping rules defined in Table 8.2. Edges, which are added inside parallel fragment among different operands, are message-interleaving edges because these edges do not represent any specified order of nodes inside interaction operands as per order specification. Message interleaving edges are represented as dotted lines and order-specification edges are represented as dark lines in MCFG diagrams. After graph is constructed, paths are generated using traversing algorithms. A path in the MCFG is an abstract representation of the execution interleaving of the designs implementation. 154
8.2. Dependency Preserving DFS Based Test Scenario Generator Table 8.2: Mapping for edges in MCFG construction procedure Sequence diagram element
MCFG element Edge
Sequence diagram
connect sdStart node and a first node (message or CFxxxStart) connect a last node (message or CF xxEnd) to SdEnd node interaction operand connect nodes as per order specification Combined fragment connect CFxxStart* node to the first message of each IOP connect IOP last node to the CFxxEnd * node General Ordering connect signal node to the corresponding wait node Loop fragment connect CFloopEnd node to a CFloopStart node connect CFloopStart node to a next node after CFloopEnd node Break fragment connect CFbreakEnd node to the sdEnd node connect CFbreakStart node to a next node after CFbreakEnd node Parallel fragment connect IOP nodes, such that n1 ∈ IOPi and n2 ∈ IOPj , i 6= j, add edges (n1 , n2 ) and (n2 , n1 ) * xx- combined fragment name
8.2.2
Guard condition true true true IOP guard true true true ¬IOP guard true ¬IOP guard true
Edge type orderspecification orderspecification orderspecification orderspecification orderspecification orderspecification orderspecification orderspecification orderspecification orderspecification messageinterleaving
Dependency Preserving DFS Algorithm
Major factors that contribute generating complete concurrent test scenarios are orderspecification messages, interleaving messages, and dependency preservation of messages inside parallel fragments in sequence diagrams. In non-concurrent design, a test scenario is represented by an order of messages as a sequence of nodes (messages) from the start node to the end node traversed by DFS or BFS algorithm. On the contrary, concurrent design imposes a condition that all predecessor nodes of CF parEnd node must be visited inside parallel fragment before processing the CF parEnd node, and they should maintain order-specification of messages in each interaction operand. Therefore, traditional DFS technique needs a modification. In a traditional CFG-like representation, the flexibility of choosing any message inside 155
8. Test Scenario Generation from Sequence Diagrams with Concurrent Behavior parallel fragment is not available because each interaction operand is represented as a sequence of messages from the start to the end of an operand. This causes traversal algorithm to generate a path without interleaving of messages. Therefore, a rich form of CFG is required such as MCFG, where additional message-interleaving edges help to traverse from any node to any other node inside the parallel fragment. The MCFG facilitates generating interleaving of messages inside parallel fragments. There is a possibility of re-visiting the same node more than once by the DFS traversal process for MCFG representation due to interleaving edges. This may lead generating unrealistic (invalid) concurrent test paths. This necessitates modifying DFS algorithm with dependency preserving ability inside the operand. In short, an MCFG and modified version of DFS algorithm are required to generate complete and realistic (valid) concurrent test paths. MCFG graph facilitates the construction of interleaving message sequences. In order to generate complete concurrent test paths, a traversal algorithm should traverse an MCFG by preserving the total order (order specification) in each interaction operand. It should also ensure to traverse a single side of a conditional flow in the selection interaction fragments such as alt and loop. We present a modified DFS algorithm, which we call as DP-DFS that includes the above restrictions. The DPDFS algorithm explores edges out of the most recently discovered node u. While exploring the edges, the algorithm searches for an unvisited node v from the node u, where node v is in the order-specification of previously visited nodes in the same interaction operand, i.e., all the nodes up to node v are visited in operand IOPi such that v ∈ IOPi . For example, consider nodes m5 , CF critical2Start and m6 of Figure 3.4 are visited inside parallel fragment by the DP-DFS traversal algorithm. Now the algorithm is processing node m2 , and it has a choice of selecting nodes that are neighbors of m2 , i.e., {CF critical1Start, m5 , CF critical2Start, m6 , m7 , CF critical2End}. The DP-DFS algorithm should select either a CF critical1Start node or m7 node, which preserves the order specification dependency in their respective interaction operands. In other words, order specification parents are visited for nodes CF critical1Start and m7 ; therefore, they are suitable to select as a new node. A current node’s order specification parent is a predecessor of that node in the same thread (interaction operand of the parallel fragment). It is mandatory to process all predecessor nodes of the CF parEnd node to avoid visiting non-concurrent messages before processing all concurrent messages inside the parallel fragment. For example, CF critical1End and CF critical2End nodes must be visited before processing the 156
8.2. Dependency Preserving DFS Based Test Scenario Generator CF parEnd node in Figure 3.4. This node selection procedure preserves the order specification inside the interaction operands, and it generates a sequence of interleave messages among the different interaction operands of the parallel fragment. The detailed procedure for DP-DFS algorithm is presented in Algorithm 8.1. The above suggested modifications are implemented in the get dependencyPreservingNode(u) procedure. Let a node u is the current node input to get dependencyPreservingNode(u) procedure and a node v is an adjacent node of the current node u. The procedure first obtains a set of the order specification parents of a node v to select the node v as a new node. If the order specification parent(s) of the node v is (are) visited, then algorithm returns the node v as a dependency preserving node, else it checks other adjacent nodes. In the case of a parallel fragment end, the procedure checks for visit of all order specification parents of the CF parEnd node. If all order specification parents of the CF parEnd node are visited, then algorithm selects the CF parEnd node as a next dependency preserving node. DP-DFS algorithm generates different interleaving message paths, which are complete concurrent test scenarios. Implementation details of the DP-DFS algorithm are given in Appendix D that describes the class diagram and MCFG graph text file format.
8.2.3
Elimination of Independent Operation Interleavings
Even for a small parallel fragment (with two interaction operands and five nodes in each interaction operand) results in a test scenario explosion ( 10!/(5!.5!)=252 interleaving message paths). Most of the scenarios are representing a specific class of the concurrent execution behaviors as discussed in Chapter 4. So, one can reduce the test scenarios, which represent similar concurrent behavior having less number of thread switches rather than with a large thread switches from that class. In this subsection, we present a reduced MCFG construction to achieve reduction in interleaving message paths, and thereby reducing the test suite size. The reduced MCFG representation allows generating only partial order interleavings for concurrent designs by eliminating interleaving around independent operations. A concurrent system design is implemented as concurrent programs in a language that supports concurrency. A concurrent program consists of a finite set of threads. Each thread executes a sequence of operations. These operations are either dependent or independent. The operations that use a shared object(s) or wait for some task to happen are called dependent operations, and other operations are called independent 157
8. Test Scenario Generation from Sequence Diagrams with Concurrent Behavior Algorithm 8.1 Dependency Preserving Depth-First Search (DP-DFS) Input: mcf g: MCFG of sequence diagram, n: Number of test scenarios Output: T S: Array of test scenarios 1: Let T S be a string array of size n ; 2: for (i = 0; i < n; i + +) do 3: S = empty-stack; 4: path = “ ”; 5: mcf g.resetF lags(); 6: u = mcf g.getRoot(); 7: u.setV isited(true); 8: path.append(u); 9: S.push(u); 10: while (S 6= null) do 11: u = S.peek(); 12: v = get dependencyP reservingN ode(u); 13: if (v 6= null) then 14: v.setV isited(true); 15: path.append(v); 16: S.push(v); 17: else 18: S.pop(); 19: end if 20: if (v.type == “sdEnd”) then 21: break; 22: end if 23: end while 24: T S[i] = path; 25: end for 26: return T S; 1: function get dependencyPreservingNode(u) 2: for all (vertexvadjacent[u]) do 3: if (v.visited() 6= true) then 4: if (v.type == “CF parEN D”) then 5: parents = isAllT otalOrderP arentsV isietd(v); 6: if (parents == true) then 7: return v; 8: end if 9: else 10: parent = isT otalOrderP arentV isited(v); 11: if (parent == true) then 12: return v; 13: end if 14: end if 15: end if 16: end for 17: return null; 18: end function
operations. Independent threads, which contain independent operations, may execute 158
8.2. Dependency Preserving DFS Based Test Scenario Generator parallel without affecting the execution of other threads. Operations such as acquire a lock, release a lock, send a signal, and receive a signal are dependent operations. Dependent threads, which contain dependent operations, may affect other threads while executing if not properly implemented. A critical region inside the parallel fragment represents dependent operations in sequence diagrams. Interleaving caused due to independent operations in dependent and independent threads does not add more value for concurrency testing. Therefore, interleaving sequences induced by independent operations can be eliminated to reduce the number of test scenarios. There are two approaches to reduce the domain of complete concurrent test scenarios: a) add a filter in DP-DFS algorithm to generate only complete scenarios for dependent operations, or b) restrict construction of the MCFG graph to explore the graph around only dependent operations. In this chapter, we have adopted the second approach to reduce the domain of complete concurrent test scenarios. An MCFG graph that explores around the dependent operations inside parallel fragments is called as a RMCFG graph. It is constructed using MCFG construction procedure with small modifications. For the construction of RMCFG graph, mapping rules of MCFG construction are adopted as it is except for parallel fragments and general ordering constructs. First, we discuss the modification for critical regions inside parallel fragments. Then, we discuss the modification for general ordering inside parallel fragments. To construct an RMCFG graph, mapping rules for parallel fragments add edges in two steps. The first step adds bidirectional edges only inside dependent operations, i.e., inside critical regions of parallel fragments. The second step adds unidirectional edges from every critical section’s start node to all the other interaction operands’ start node. For example, Figure 3.5 shows RMCFG representation for the bank transaction example. It has the critical section in the first interaction operand that starts at node CF critical1Start and end at node CF critical1End, and in the second interaction operand has the critical section that start at node CF critical2Start and end at node CF critical2End. As per the construction of the RMCFG graph, the first step adds bidirectional edges only inside critical sections, i.e., for nodes CF critical1Start, m3 , m4 , CF critical1End, CF critical2Start, m6 , m7 , and CF critical2End. Second step adds two unidirectional edges CF critical1Start to m5 and CF critical2Start to m2 . A RMCFG graph significantly reduces the test scenarios generated by the DP-DFS algorithm as compared to the test scenarios generated from MCFG. For example, RMCFG graph construction algorithm introduces 16 bidirectional edges plus 159
8. Test Scenario Generation from Sequence Diagrams with Concurrent Behavior two unidirectional edges for RMCFG graph in Figure 3.5. These edges are less compared to 25 bidirectional edges in MCFG graph as shown in Figure 3.4. This new construction of RMCFG graph of the bank transaction sequence diagram reduces the domain of interleaving sequences to 72 as compared to 252 interleaving sequences in the MCFG graph. SD: General Ordering O1 : Object
O2 : Object
O3 : Object
1 : m1() Par: par1
2 : m2() 3 : m3() 4 : m4() 5 : m5() 6 : m6() 7 : m7() 8 : m8() 9 : m9() 10 : m10()
Figure 8.3: A sequence diagram illustrating general order construct inside the parallel fragment
A causal order between messages is represented by a general ordering construct in a sequence diagram as shown in Figure 8.3. In RMCFG construction, start of the general ordering is represented as Signal control node, and end of the general ordering is represented as W ait control node. General ordering contains two dependent operations inside parallel fragments. Therefore, RMCFG construction process significantly reduces the interleaving edges inside parallel fragments by a few fix numbers of edges. In RMCFG construction process, general ordering adds five bidirectional edges as follows: i) an edge from the Signal node to the W ait node, ii) an edge from a message before the W ait node to the message after the Signal node, iii) an edge from a message before the Signal node to the message after the W ait node, and iv) two edges are added, the first from the Signal node to the start node of W ait node’s interaction operand, and the second from the W ait node to the start node of Signal node’s interaction operand. For example, message m8 should be exe160
8.2. Dependency Preserving DFS Based Test Scenario Generator cuted after execution of message m3 , this order is shown by dashed-line in Figure 8.3. Figure 8.4 shows the RMCFG graph for the above example. In the case of general
sd Start
m1
m7
wait
m8
m9
General Ordering
m6
CFpar Start
m2
m3
sign al
CFpar End
m4
m10
sd End
m5
Figure 8.4: An RMCFG representation of the Figure 8.3, which has general order construct inside the parallel fragment
ordering, an RMCFG graph considerably reduces the interleaving message sequences. For example, RMCFG representation in Figure 8.4 has five bidirectional edges inside the parallel fragment and MCFG representation has 25 directional edges inside the parallel fragment for the design represented in Figure 8.3. DP-DFS algorithm can generate maximum 20 interleaving message sequences from RMCFG representation compared to 252 interleaving message sequences from MCFG representation. The DP-DFS algorithm with input RMCFG graph generates concurrency test scenarios that may trigger bug. In order to detect data-race errors in the implementation of a given design, a test case should have thread switch inside critical section, and it should achieve synchronization coverage. Test scenarios generated from RMCFG satisfy these requirements. RMCFG graph reduces the number of complete concurrent paths; thus, the total test scenario generation and testing time is reduced. DP-DFS algorithm with an RMCFG graph generates test scenarios that are synchronous valid concurrent test scenarios, i.e., execution of scenarios force the thread switch inside the critical section. If the implementation does not guard shared data inside critical section, then there is a possibility of overwriting data or reading stale values in the execution of the implementation. In order to detect synchronization errors due to general ordering in the implementation of a given design, a test suite should have one valid test scenario and one invalid test scenario. A valid test scenario checks an 161
8. Test Scenario Generation from Sequence Diagrams with Concurrent Behavior implementation execute the signal and wait operations as represented by the design. An invalid test scenario checks either an implementation waits at the end of general ordering message, if it misses the signal message. This ensures the correct implementation of the signal-wait primitives. Thus, the test suite generated from these scenarios detects synchronization errors.
8.2.4
Complexity Analysis
Complexity analysis is carried out to estimate the efficacy of the DP-DFS algorithm and storage requirement of the MCFG graph. Analysis determines the amount of resources (time and space) required to execute the DP-DFS algorithm. Space complexity is a function describing the amount of memory (space) required by the algorithm. The algorithm requires space to store input MCFG graph and other variables. We calculate the space complexity in terms of number of nodes and edges required for MCFG representation for a given sequence diagram. Consider a sequence diagram S has m messages and k combined fragments. Let the sequence diagram has parallel fragment P F with n interaction operands. Interaction operands of parallel fragment are termed as threadi , 2 ≤ i ≤ n. Threads in MCFG representation contains nodes tmi , 1 ≤ tm and 2 ≤ i ≤ n. Let number of interaction operands for a combined fragment cf are IOP (cf ), then nodes = m + 2 × k k n X Y edges = (IOP (cfi ) × 2) + m − 2 + 2 × (tmi ) i=1
i=1
Pk
where, (IOP (cfi ) × 2) + m − 2 are order-specification edges in MCFG and Qn i=1 2 × i=1 (tmi ) are interleaving edges in MCFG. Time complexity is a function describing the amount of time the algorithm takes in terms of the amount of input to the algorithm. DP-DFS algorithm is an extension of DFS algorithm for concurrent message interleaving path generation. Therefore, complexity is similar to DFS algorithm. MCFG graph has extra interleaving edges, which increase the node selection time in proportion to the number of adjacent edges. Consider M CF G = (V, E) be a directed graph. V is a set of vertices (nodes), and E is a set of directed edges. Let n = |V | and m = |E|, then time complexity of DP-DFS algorithm is O n + m . However, MCFG representation includes interleaving edges 162
8.3. Experimental Results in the order of O n2 ; therefore complexity of DP-DFS algorithm is O n2 .
8.3
Experimental Results
Based on the proposed approach and the coverage criteria defined in the Chapter 3, we have conducted experiments on a set of sequence diagrams. We have analyzed the effectiveness of our approach on five representative sequence diagrams. Each sequence diagram depicts a different nature of the interactions inside the parallel fragment. The first example is a desktop startup sequence diagram (Pilone and Pitman, 2005). This application includes parallel fragment with two interaction operands, where all tasks are independent, and they do not access any shared data. Therefore, there is no possibility of concurrency errors. The second example is a map rendering sequence diagram (Pilone and Pitman, 2005), this application sequence diagram has the parallel fragment with two interaction operands of which one interaction operand includes a critical section. Third and fourth examples are the bank money transfer sequence diagram and the dining philosophers problem (2 philosophers), respectively. These two examples have parallel fragment with two interaction operands, and each interaction operand includes a critical section. These two examples access shared data in the both threads and may lead to concurrency errors like data race. Fifth example is on-line train ticket reservation application. This example includes a general ordering construct inside the parallel fragment and may lead to concurrency errors like starvation (missing signal). We compared our approach with the three contemporary approaches by Nayak and Samanta (2010) and Khandai et al. (2011) available in the literature, and our previous CQS approach presented in the Chapter 6, which deal with parallel fragments in sequence diagrams. First, we have analyzed the relationship between intermediate representations and the number of test scenarios generated from them. Then, we have analyzed the test suite size reduction due to RMCFG representation with DP-DFS algorithm. After that, we have done the coverage analysis of the proposed approach with contemporary approaches. Finally, we have carried out mutation analysis to investigate defect detection capability of test suites generated by different coverage metrics. 163
8. Test Scenario Generation from Sequence Diagrams with Concurrent Behavior
8.3.1
Effect of Intermediate Representation on Test Suite Size
Intermediate representation plays an important role in generating test scenarios and subsequently analyzing behaviors of the system. We have considered intermediate representations from existing approaches such as CFG-like graph, SCG, CCG, ADG, and proposed representations MCFG and RMCFG for the study of the effect on test suite size. To generate complete concurrent test paths from an intermediate representation, we have used test scenario generation algorithms like DFS, BFS, DFS-BFS, CQS, and proposed DP-DFS. For the CFG-like representation, we have used BFS, DFS and DFS-BFS algorithms to generate test scenarios. For other representations, we have used the respective test scenario generator algorithm presented in the literature, such as DFS-like algorithm for SCG graph, DFS-BFS algorithm for CCG graph, CQS algorithm for ADG. The DP-DFS algorithm is used for MCFG and RMCFG graphs. In this experiment, first sequence diagrams are transformed into different intermediate forms in the text file format. Then test scenario generator algorithms are applied to generate test paths from the respective intermediate representation. Table 8.3 shows TEIS for each sequence diagram and the total number of message paths generated by BFS, DFS, DFS-BFS, CQS, and DP-DFS algorithms. Results indicate Table 8.3: Efficacy analysis of intermediate representations for test scenario generation by respective test scenario generator for different applications
Number of paths generated BFS DFS DFS-BFS CQS DP-DFS CFG CFG SCG CFG CCG ADG MCFG RMCFG Desktop startup 70 2 0 0 2 2 2 70 2 Map Rendering 21 0 0 0 0 2 2 15 3 Bank Transaction 252 2 0 0 2 2 2 237 42 Dining Philosopher 12870 2 0 0 2 2 2 6832 1686 Online Ticket Booking 252 2 0 0 2 2 2 21 4 TEIS-total execution interleaving sequences Applications
TEIS
that BFS and DFS-BFS algorithms generate complete concurrent test scenarios using a CFG-like representation for Desktop Startup, Bank Transaction, Dining Philosopher, and On-line Ticket Booking sequence diagrams. But these algorithms do not generate complete concurrent test scenarios for the Map Rendering sequence diagram 164
8.3. Experimental Results because interaction operands do not contain equal number of messages. For the Map Rendering application, BFS explores nodes in both branches in an alternate way, and it does not wait at CF parEnd node until processing of all other interaction operands message nodes. Then, it reaches sdEnd node and terminates without generating a complete concurrent test scenario. DFS algorithm does not generate complete concurrent test scenarios using CFG and SCG representations for all examples. DFS-BFS algorithm using a CCG representation and CQS algorithm using an ADG representation generated two test scenarios each. This indicates a limitation of these approaches toward exploring message interleaving space. DFS-BFS algorithm generated sequentialized message scenarios using a CCG representation. The generated test scenarios do not interleave inside the critical section, and hence they are not capable of uncovering data race errors. CQS algorithm generated interleaving message scenarios with two thread switches inside the critical section, and hence they are capable of uncovering data race errors. DP-DFS algorithm with MCFG and RMCFG graph representations generated complete concurrent test scenarios for all application sequence diagrams. These generated test scenarios are capable of catching data-race errors and synchronization errors due to incorrect implementation of the design. DFS-BFS and CQS algorithms generated negative test cases, i.e., invalid test paths; therefore, they are suitable to test synchronization coverage. DP-DFS algorithm generated both positive and negative test cases, i.e., valid test paths as sequentialized test scenarios and invalid test paths as synchronized test scenarios. Therefore, the test suite generated by DP-DFS algorithm is capable of detecting synchronization, data-race, and deadlock errors. Moreover, generated test scenarios are well classified into different concurrent behaviors as presented in Chapter 4.
8.3.2
Test Suite Size Reduction Due to RMCFG Representation
Independent operations in dependent and independent threads do not cause any concurrency errors. Test scenarios contributed due to independent operations are suppressed in the RMCFG representation, thereby generating only partial order interleavings. For critical region fragments, the reduction factor is exponentially proportional to the difference between the number of messages of the thread and the number of messages inside the critical region. For general ordering constructs, only a constant number of test scenarios are generated from RMCFG. To analyze the bug triggering 165
8. Test Scenario Generation from Sequence Diagrams with Concurrent Behavior capability of the test scenarios, we segregated generated paths into different concurrent behavioral classes, such as non-data-race, serializing, blocking, and data-race as discussed in Chapter 4. The results shown in Table 8.4 indicate that RMCFG rep-
Table 8.4: Analysis of test suite size reduction by RMCFG representation over MCFG representation for different applications Applications
TEIS
Desktop Startup
70
Map Rendering
21
Bank Transaction
252
Dining Philosopher
12870
Online Ticket Booking
252
Type of path non-data-race serialize blocking data-race non-data-race serialize blocking data-race non-data-race serialize blocking data-race non-data-race serialize blocking data-race non-data-race serialize blocking data-race
# paths generated MCFG RMCFG 0 0 70 2 0 0 0 0 15 3 0 0 0 0 0 0 0 0 12 2 240 40 0 0 0 0 18 2 10818 1456 0 0 0 0 21 2 1 1 0 0
resentation generates test scenarios of the required type, and they are sufficient to satisfy the essential concurrent message path coverage criterion. The results indicate a significant reduction in the number of test scenarios generated using the RMCFG graph as compared to the MCFG graph. A set of test scenarios generated from the RMCFG graph is a subset of the total scenarios generated from the MCFG graph. The RMCFG representation achieves 70 to 80% reduction in the test suite size. The test scenarios generated from the RMCFG graph may uncover concurrency bugs. As shown in the coverage analysis sub-section, the test scenarios generated from RMCFG are effective for the concurrency testing. 166
8.3. Experimental Results
8.3.3
Coverage Analysis
Message sequence path (Abdurazik and Offutt, 2000) and all message paths (Pilskalns et al., 2003) criteria are defined for collaboration diagrams and subsequently adopted for sequence diagrams. A message sequence path is a path that includes all messages in collaboration in the order specified by the collaboration diagram. With combined fragments in UML 2.0, message sequence path may be used for fragments or operands of the combined fragments, where a message sequence path is a path that includes all messages in a fragment in the order specified by the order specification. We adopt this extended meaning of message sequence path for the evaluation purpose. All message paths coverage criterion satisfied by traversing CFG-like intermediate representations. We have considered all message paths as paths generated by DFS and BFS traversal of a CFG-like representation of sequence diagrams. Table 8.5 shows UML-based coverage analysis of results obtained from proposed approach and related research contributions for sequence diagrams. For SCG and CCG representations of sequence diagrams, block nodes are included inside parallel fragments for each thread. Therefore, thread messages are abstracted at a higher level as a single block node. This causes exceptionally less message and message sequence path coverage for chosen application sequence diagrams. Hence, we consider messages inside block nodes for coverage analysis purpose. For the approach presented by Nayak and Samanta (2010), DFS algorithm has generated two basic paths from the SCG representation, and generated paths are not complete concurrent paths. Their approach achieves 100% message and message sequence path coverage, and 50% all message path coverage. However, zero percentages of IMPC, SC, and ECC coverage because of no interleaving paths. For the approach presented by Khandai et al. (2011), DFSBFS algorithm has generated sequentialized paths from CCG representation, which are complete concurrent paths. Their approach achieves 100% message coverage and message sequence path coverage. Even though their approach use BFS technique inside parallel fragments, BFS treats all messages inside an interaction operand as a block node due to the intermediate representation, thereby generating sequentialized paths rather than interleaving paths. Therefore, their approach does not achieve all message paths coverage. Their approach achieved up to 10% IMPC coverage and up to 3% synchronization coverage, except in the case of general ordering construct. It achieved 100% ECC in the first example, which does not include critical section or general order construct inside the parallel combined fragment. The second example 167
8. Test Scenario Generation from Sequence Diagrams with Concurrent Behavior Table 8.5: Coverage analysis for approaches DFS:SCG by Nayak and Samanta (2010), DFS-BFS:CCG by Khandai et al. (2011), and proposed approaches CQS:ADG and DPDFS:RMCFG algorithms
Applications Desktop startup Map Rendering Bank Transaction Dining Philosopher Ticket booking
Applications
Percentage coverage (%) # DFS:SCG Nayak and Samanta (2010) # DFS-BFS:CCG Khandai et al. (2011) paths MC MSP AMP IMPC SC ECC paths MC MSP AMP IMPC SC ECC 2
100.00 100.00 50.00 0.00
NA
0.00
2
100.00 100.00 0.00
2.85
NA
100.00
2
100.00 100.00 50.00 0.00
NA
0.00
2
100.00 100.00 0.00
9.25
NA
100.00
2
100.00 100.00 50.00 0.00
0.00
0.00
2
100.00 100.00 0.00
0.79
2.85
33.33
2
100.00 100.00 50.00 0.00
0.00
0.00
2
100.00 100.00 0.00
0.02
0.06
33.33
2
100.00 100.00 50.00 0.00
0.00
0.00
2
100.00 100.00 0.00
9.52
100.00 50.00
# paths MC
MSP
Percentage coverage (%) CQS:ADG # AMP IMPC SC ECC paths MC
MSP
DP-DFS:RMCFG AMP IMPC SC
Desktop 2 100.00 100.00 0.00 2.85 NA 100.00 2 100.00 100.00 startup Map Ren2 100.00 100.00 0.00 9.25 NA 100.00 3 100.00 100.00 dering Bank 2 100.00 100.00 0.00 0.79 2.85 33.33 42 100.00 100.00 Transaction Dining 2 100.00 100.00 0.00 0.02 0.06 33.33 1686 100.00 100.00 Philosopher Ticket 2 100.00 50.00 0.00 9.52 100.00 50.00 4 100.00 100.00 booking MC- Message coverage, MSP- message sequence path coverage, AMP- all message paths coverage, IMPC- interleaving message path coverage, SC- synchronization coverage, and ECC- essential concurrent message path coverage
ECC
0.00
2.85
NA
100.00
0.00
14.28 NA
100.00
0.00
16.66 60.00 100.00
0.00
13.10 49.12 100.00
0.00
19.04 100.00 100.00
does include a critical region in one thread, but synchronization coverage does not have any test case requirement. Therefore, their approach achieved 100% ECC coverage for the second application. In other cases, their approach achieved 33 to 50% ECC coverage. For our previous approach CQS:ADG presented in the Chapter 6, CQS algorithm has generated two synchronized message paths. This approach achieved 100% message coverage. However, it does not achieve 100% message sequence path coverage in all cases because thread switches introduced while generating test path leads to break of message sequence path in some cases. Test paths generated by this approach are not similar to that of generated either by DFS or BFS; therefore, it does not achieve all message path coverage. This approach has achieved up to 10% IMPC 168
8.3. Experimental Results coverage and up to 3% synchronization coverage, except in the case of Ticket Booking Application. This approach has achieved 33 to 100% ECC coverage. Approaches DFS-BFS:CCG by Khandai et al. (2011) and CQS:ADG algorithms generated equal number of test scenarios for all applications leading to the same IMPC, SC, and ECC coverage values. However, they target on the different type of behaviors such as the DFS-BFS:CCG approach generates serialized test scenarios and the CQS:AD approach generates blocking test scenarios. As compared to the above approaches, DP-DFS algorithm generated complete concurrent test scenarios using the RMCFG graph representation and achieved 100% ECC coverage in addition to message coverage and message sequence path coverage. Moreover, it has achieved high IMPC and SC coverage for all applications. However, this approach did not achieve all message path coverage because RMCFG representation and DP-DFS algorithm did not generate test scenarios similar to either DFS or BFS paths for all applications. Results indicate that DP-DFS algorithm using the RMCFG representation achieves improved IMPC, SC, and ECC coverage for independent and dependent thread applications.
8.3.4
Mutation Analysis
Mutation analysis is a fault-based test suite assessment technique (Andrews et al., 2006). In mutation testing, different mutant programs are generated to mimic typical syntactic errors that a programmer can write in the code. Faults are intentionally inserted into original programs so that test cases detect them while testing (Offutt and Untch, 2001). When a test case detects inserted faults then the test case is strong enough to detect errors similar to inserted errors. Test cases kill mutants, thereby assess the test suite. In this experiment, we have done mutation analysis on the programs developed from models. For generating mutants, we have used concurrency bug patterns presented by Farchi et al. (2003) such as losing a notify bug pattern, no lock bug pattern and a blocking critical section bug pattern. Bradbury et al. (2006) presented a set of mutation operators for Java concurrent programs that are related to above bug patterns. We have selected RTXC, RSB, RSK, SHCR, and SKCR mutation operators to generate mutant programs. The description of the mutation operators is given in Table 8.6. We have not used the non-concurrent mutation operators for the analysis purpose. Mutant programs are generated manually by deleting or modifying the required keywords. First, we instrument the original program by inserting 169
8. Test Scenario Generation from Sequence Diagrams with Concurrent Behavior Table 8.6: Description of mutation operators developed by Bradbury et al. (2006) for Java programs. Operator Name RTXC RSB RSK SHCR SKCR
Description Remove Thread Method-X Call (wait(), join(), sleep(), yield(), notify(), notifyAll() Methods) Remove Synchronized Block Remove Synchronized Keyword from Method Shift Critical Region Shrink Critical Region
the required sleep method calls and trace generator calls. The modified program generates method call sequence that is prescribed by the test scenario. We call this instrumented program as a test program. Then, we generate different mutants from the test program. After that, we run the test program and the mutant program to generate the execution traces, which are method call sequences. Finally, we compare the execution traces to find the differences in the method call sequences. For positive test scenarios, if the method call sequences of the test program and a mutant program are similar then the mutant is not killed by the test scenario; otherwise it is killed by the test scenario. Table 8.7 shows the mutation analysis results. The results show the Table 8.7: Mutation analysis for the Java programs developed from the designs for concurrent mutation operators RXTC, RSK, RSB, SHCR, and SKCR. Applications Desktop startup Map Rendering Bank Transaction Dining Philosopher Online Ticket Booking
Mutant operator RTXC RSK/ RSB SHCR/SKCR RTXC RSK/ RSB SHCR/SKCR RTXC RSK/ RSB SHCR/SKCR RTXC RSK/ RSB SHCR/SKCR RTXC RSK/ RSB SHCR/SKCR
# mutants 1 0 0 1 1 4 1 2 4 1 2 4 3 0 0
170
% mutant killed MC MSP ECC 0 100 100 0 100 100 0 0 100 0 0 100 0 100 100 0 0 100 0 0 100 0 100 100 0 0 100 0 0 100 0 33.33 100 -
8.3. Experimental Results percent mutants killed by the test cases generated that satisfy coverage criteria like message coverage, message sequence coverage, and essential concurrent coverage. To evaluate the mutation analysis, we have considered the test scenarios generated by the DFS:SCG by Nayak and Samanta (2010) and DFS-BFS:CCG by Khandai et al. (2011) for message and message path coverage, and DP-DFS:RMCFG for ECC coverage. An intermediate representation plays an important role in evaluating mutants, which are based on the design artifacts. Mutation operators modify or delete some keywords, which are the control nodes in the intermediate representation. Therefore, while evaluating mutation analysis, we consider the message and control nodes in traces. As shown in Table 8.7, ECC coverage test suite has killed 100% mutants as compared to MSP coverage test suite, which killed average 40% mutants. However, message coverage test suite did not kill any mutant. In our approach of mutation analysis, it is observed that only serialized schedules only able to detect faults due to the mutants. Because, original test program generates the complete trace of message sequence including the control nodes, and mutant test program generates the faulty trace of message sequence. Although, DFS:SCG by Nayak and Samanta (2010) and DFS-BFS:CCG by Khandai et al. (2011) approaches include the fork node and join node in their intermediate representation, these approaches consider every thread inside the fork-join construct as a single block node. Therefore, test scenarios generated by these approaches are found ineffective towards RSK and RSB operators. SCG and CCG intermediate representations do not consider the critical section. Hence, test scenarios generated by their approach do not detect faults due to SHCR and SKCR operators. There is no possibility of killing mutants using unserialized schedules because test program execution traces are either incomplete or same as that of the serialized schedules. This causes difficulty in comparison of test case scenario traces and test program execution traces. However, unserialized schedules are useful for blocking and blocked behavior of threads for synchronization coverage.
Related works Recently researchers have been utilizing UML design models for concurrency testing. For UML behavioral models, concurrent test scenario generation from activity diagrams is presented in (Chandler et al., 2005; Mingsong et al., 2006; Kim et al., 2007). An activity diagram has been chosen by the system designers for modeling concurrency in the system, but activity diagrams do not support critical sections, which are 171
8. Test Scenario Generation from Sequence Diagrams with Concurrent Behavior key notion for concurrent execution. One can get an idea of dependent operations in the thread with the help of critical sections, thereby targeting interleavings around dependent operations for concurrency testing. Therefore, test scenario generators need full interleaving space exploration of activity diagrams for concurrency testing. Recently, Shousha et al. (2012) have presented an approach to detect concurrency errors using UML Modeling and Analysis of Real-Time and Embedded Systems (MARTE) profile sequence diagrams. In their approach, a genetic algorithm is used to search for execution sequences exhibiting deadlock or starvation problem. Their approach needs execution time of tasks, which is very rare in non-embedded designs, to search error-prone execution schedules. Our approach is used to general designs (nonembedded) and may be extended easily to the embedded designs. A few researchers have addressed the problem of concurrent test scenarios generation from sequence diagrams in (Nayak and Samanta, 2010; Khandai et al., 2011), where parallel fragments are processed to generate test scenarios. Although, their approaches generate complete concurrent test scenarios, they are not interleaving concurrent test scenarios to uncover concurrency bugs. This limits the concurrency defect detection capability of Nayak and Samanta (2010) and Khandai et al. (2011) approaches. Our approach generates interleaved concurrent test scenarios and enhances the concurrency defect detection capability.
8.4
Conclusions
In this chapter, we have presented test scenario generation from UML sequence diagrams. Concurrency constructs, such as parallel fragment, critical region and general ordering in sequence diagrams, are elaborated to generate test scenarios. The test scenarios are designed to uncover data race and synchronization errors due to incorrect implementation of the design. The experimental results justify the proposed intermediate representation and DP-DFS algorithm to explore the sufficient message interleavings to satisfy the adequate coverage of concurrent behaviors. The RMCFG representation achieves 70 to 80% reduction in test suite size and achieves the same coverage as the MCFG representation for ECC and synchronization coverage criteria. DP-DFS technique achieves 100% ECC coverage, 50 to 100% synchronization coverage, and up to 20% IMPC coverage. In mutation analysis, ECC coverage test suite has killed 100% mutants as 172
8.4. Conclusions compared to MSP coverage test suite, which killed average 40% mutants. In a nutshell, the contributions of this thesis are focused on UML transition sequence based test scenario generation for concurrency testing. Proposed new UML-based coverage criteria for concurrency testing are satisfied by the proposed test scenario generation techniques. In the next chapter, we present our conclusion and directions for future work.
173
Chapter 9 Conclusions and Future Work This thesis is focused on UML transition sequence based test scenario generation for concurrency testing, presenting new UML-based coverage criteria for concurrency testing and test scenario generation techniques for the proposed coverage criteria. The new UML-based concurrency coverage criteria for interleaving test scenarios focus on concurrent behavior of systems. These behavioral coverage criteria are suitable for concurrent designs represented by activity diagrams and sequence diagrams for concurrency testing. Interleaving Activity Path Coverage and Total Activity Path Coverage criteria are proposed for activity diagrams. Interleaving Message Path Coverage, Synchronization Coverage, and Essential Concurrent Message Path Coverage criteria are proposed for sequence diagrams. These new coverage criteria enforce generation of interleaving test scenarios for different concurrent behaviors such as serialized scenarios, blocking scenarios, and data-race scenarios. Test scenarios generated from activity diagrams and sequence diagrams are classified and analyzed based on the state machine approach. This approach accurately classifies test scenarios into serialized scenarios, blocking scenarios, data-race scenarios, and non-data-race scenarios. Our state machine based approach attempts to optimize the test suite for testing each type concurrency error. We have proposed a unique intermediate representation for both sequence and activity diagrams. We have proposed two test scenario generation algorithms that generate interleaving test scenarios. The first algorithm is called a concurrent queue search algorithm. This algorithm is designed specifically to generate test scenarios that uncover data-race and synchronization errors. Experimental results show that CQS algorithm generates test scenarios that may trigger bugs with the minimum 175
9. Conclusions and Future Work number of thread switches. The second algorithm is called a DFS Level Permutation algorithm that generates a larger number of interleaving test scenarios compared to DFS-BFS approach. Our proposed approaches improve concurrency coverage. These algorithms however have a few limitations for nested concurrent designs, where they fail to explore interleaving space. We have used a non-trivial meta-heuristic approach to generate interleaving test scenarios for nested concurrent designs. We have proposed two solutions: evolutionary algorithm and hybrid evolutionary algorithm to generate test scenarios for nested concurrent activity diagrams. The first solution is an evolutionary algorithm where test scenario generation problem is modeled as an optimization problem. Test scenarios are represented in terms of chromosomes, and evaluated using fitness that assesses correctness of interleaving paths for the nested concurrency. Experimental results show better achievement of interleaving coverage by the evolutionary algorithm, even though it requires longer time to converge. The second solution is a hybrid evolutionary approach that blends initial random population by paths generated using standard DFS, standard BFS, and level permutation techniques. Hybrid evolutionary algorithm is aimed at faster convergence for test scenario generation. Experimental results show improvement in convergence time up to 50% by hybrid evolutionary algorithms compared to our (non-hybrid) evolutionary algorithm. The Hybrid BFS-EA algorithm is good for concurrent nested activity diagrams. Evolutionary algorithms generate interleaving test scenarios for both nonnested and nested concurrent activity diagrams with improved interleaving activity path coverage. We have presented a new intermediate representation and test scenarios generation algorithm for sequence diagrams with concurrency constructs. The proposed algorithm explores synchronization behavior inside parallel fragments represented by critical regions and general ordering to generate test scenarios for concurrency testing. We have proposed an intermediate representation called a message control flow graph to generate message interleaving paths. We have presented a dependency preserving DFS algorithm to explore parallel fragments represented using an MCFG that avoid unnecessary looping inside the MCFG. DP-DFS algorithm preserves total-order in each thread while interleaving amongst the threads, thereby generating complete concurrent test scenarios. Redundant interleavings due to independent operations are reduced by using a reduced MCFG representation of sequence diagrams. Experimental results show 70 to 80% reduction in the test suite size generated by DP-DFS algorithm using RMCFG representation compared to MCFG representation. DP176
DFS algorithm achieves 100% ECC coverage, 50 to 100% synchronization coverage, and up to 20% interleaving message path coverage using RMCFG representation of sequence diagrams.
Directions for Future Work We have presented the evolutionary algorithm for interleaving test scenario generation for nested concurrent activity diagrams. We have tested the algorithms on testbed activity diagrams and small examples. Still there is need of scalability of the algorithm for large designs. This can be done in terms of defining new evolutionary operators, which will work for large designs with less convergance time. There is a need for a comparative study of model checking techniques and evolutionary techniques on parameters like test suite size, time, and other relevant parameters. We have proposed new interleaving coverage criteria based on code based testing. There is a necessity to evaluate empirically the effectiveness of concurrency coverage criteria that explore the relationship between concurrent coverage and fault detection effectiveness for large applications. We have designed a state based classifier to analyze and classify test scenarios into serialized scenarios, blocking scenarios, data-race scenarios, and non-data-race scenarios classes. This classifier can be extended further to explore blocking scenarios for detailed investigation of deadlock and starvation. We have proposed DP-DFS algorithm for non-nested concurrency constructs. It would be beneficial to work on this algorithm for nested constructs like parallel, alternate, and option fragments. Also, a comparative study of model checking and DP-DFS algorithm techniques on parameters, like test scenario generation, error uncover rate, convergence time, and state space exploration would be useful. For nested concurrent designs, unfolding of nesting inside concurrency may help in designing interleaving test cases. Therefore, different techniques of unfolding of nested-concurrent designs need to be explored. For DFS level permutation algorithm, current data structure used to test sequences is list. It would be beneficial to use a tree-structure for efficient storage and management of test sequences. There is need to do empirical study on the storage of exponential test scenarios using different data structures.
177
Appendices
179
Appendix A First Review Report Modifications List This appendix gives the details of the modifications carried out in the thesis based on the first review report comments of the examiners. Table A.1: Modifications as per second review report Sr. No.
Modification
Page No.
1.
A paragraph is added in the revised thesis for clarification on need of interleaving test scenarios inside a critical section.
2
2.
Eliminated multiple copies of figures. Caption of the Figure 1.2 is changed.
6
3.
Modified the typo mistake the word ”propose” to ”adopt”.
77
4.
Figure 4.1 is modified to show parameters to methods.
78
5.
Algorithm 6.1 is modified to separate two phases of the algorithm.
106
6.
Figure 6.1 and Figure 6.2 are included to illustrate working instance of queues in CQS algorithm as suggested by the examiner.
111
7.
Text is modified to clarify CQS does not process three-way decision.
105
8.
Presentation of Algorithm 6.1 is modified i) to specify input and output in the beginning instead of parameters, ii) to keep the same processing order of elements in both the phases, and iii) to change if . . . else . . . endif statements with case statements.
106
9.
Figure 1.2 is modified to change message numbers to m1 , m2 , . . . instead of 1, 2, . . . .
6
10.
Algorithm 6.2 is modified to show detailed programming steps in the algorithm.
113
11.
Figure 6.4, with threads having different number of activities, is included to illustrate working instance of queues in LevelPermute algorithm as suggested by the examiner.
115
12.
Figure 3.2 is modified to label all activities and links.
44
13.
Explanation of the same partial order is illustrated with suitable example.
54
Continued on next page
181
A. First Review Report Modifications List Table A.1 – Continued from previous page Sr. No.
Modification
Page No.
14.
Chapter 6 title is modified to clear that it handles concurrent threads for only nonnested activity diagrams.
103
15.
Figure 7.1 and Figure 7.2 are modified to represent crossover and mutation on actual paths.
125
16.
Thesis is modified to avoid repeatedly use of term ”bug triggering scenario” by the term ”scenario that may trigger bug”.
–
17.
Thesis is modified to change use of term ”fitness function” by the term ”fitness”.
–
18.
Thesis is modified to change use of term ”context-switch” by the term ”threadswitch”.
–
182
Appendix B Second Review Report Modifications List This appendix gives the details of the modifications carried out in the thesis based on the second review report comments of the examiners. Table B.1: Modifications as per second review report Sr. No.
Modification
Page No.
1.
Page numbers are included in the abbreviations list
xiii
2.
A paragraph is added in Chapter 1 to list concurrency constructs considered in this thesis
9
3.
Expression signal to another message is changed to signal to another thread
66
4.
Sequence diagrams Figure 1.2, Figure 6.6, and Figure 8.3, are modified to add return lines.
6, 120, and 160
5.
Sequence diagrams Figure 3.3, Figure 5.1, Figure 5.2, Figure 5.3, Figure 5.4, Figure 5.5, Figure 5.6, and Figure 5.7, are modified to add “live-time” of each method call.
63, 94, 95, 96, and 97
6.
Def. 3.18 is updated to include non-concurrent paths
66
7.
Figure 3.4 caption is modified.
66
8.
Figure 3.4, Figure 3.5, and Figure 8.4, are modified to change node names Cf parStart with CF parStart and Cf parEnd with CF parEnd.
66, 70, and 161
9.
Thesis text is changed to Receive-Signal instead of Accept-Signal to keep consistency with Figure 3.1.
43
10.
Thesis text is modified to change ”total-order-edge” with ”order-specification edge”.
154
11.
Figure 5.7 is modified to change receive-signal notation.
97
12.
Thesis text is modified to express ESMD Figure 4.2 handles only two threads.
81
Continued on next page
183
B. Second Review Report Modifications List Table B.1 – Continued from previous page Sr. No.
Modification
Page No.
13.
Modal verb“can” is added in the sentence “Insufficient interleaving ....”.
75
14.
Description of the ESMD is re-written in-line with the suggestion given by the examiner also state transition description is consolidated in Table 4.1.
82
15.
Thesis text is changed to add sentence Read/write on non-shared data are not considered as events.
82
16.
Figure 4.2 name of states Write and Read are changed to safeWrite and safeRead.
81
17.
Two sample input data-access traces to ESMD and their transitions are included to illustrate transitions of Figure 4.2.
87
18.
Caption of Figure 4.1 is changed to ”ATM money withdrawal banking functionality modeled in sequence diagram using parallel fragment to illustrate data access-tags”.
78
19.
Test scenario S representation is modified for consistency of notations.
78
20.
Figure 4.1 is modified to show updateBalance() self method call to atm object
78
21.
Method name Withdraw is changed to withdraw in Figure 3.3 and Figure 4.1 to maintain naming consistency
63 and 78
22.
Java code is included for Figure 5.6 and Figure 5.7 of the thesis in Appendix G.
209
23.
Figure 5.5 is modified to indicate the loop-test is done by object O1 .
96
24.
Figure 5.6 is modified to remove f alse label.
96
25.
Figure 6.1 is modified to change second level caption F 1 to F 2.
111
26.
Variables value notations are changed i) “new Queue” with “empty queue” and ii)replace with null in Algorithm 6.1.
106
27.
Statement TS: Set of test scenarios is changed to TS: ArrayList of test scenarios in Algorithm 6.1.
106
28.
Algorithm 6.1 is title updated to remove comment “Modified Detailed”.
106
29.
Activity diagram graph (ADG) is used instead of actGaph throughout thesis for consistency.
106
30.
Added comment “covers both if-test and loop-test” in line 8 of Algorithm 6.1.
106
31.
Variable name “outf lows” is changed to “numOutf lows” in Algorithm 6.1.
106
32.
The comment “i=0 true control flow” is changed to “i=0 corresponds true control branch” in line 12 of Algorithm 6.1.
106
33.
Variable names “main queue” and “sub queue” are changed to “mainQueue” and “subQueue” in Algorithm 6.1.
106
34.
Added missing statement “F orkId + +” at line 49 of function getT estScenario.
107
35.
Changed the article “the” with an article “a” in the sentence “This clearly indicates the permutation technique is exhaustive and exponential task.”
112
36.
Explanation of asynchronous messages is elaborated with example for Figure 5.2
94
37.
Explanation is given in the thesis regarding critical-section for access to shared data needs testing of serialized and serialized interleavings.
7
Continued on next page
184
Table B.1 – Continued from previous page Sr. No.
Modification
Page No.
38.
Explanation regarding interleaving I2 is given and elaborated with example for Figure 1.2
7
39.
Explanation regarding different kinds of control nodes of MCFG is given in the text.
65
40.
Explanation regarding mapping of interleaving classes and concurrent behaviors is given in the text.
84
41.
Explanation regarding ESMD for some transitions is given in the text.
83
42.
Explanation regarding “what happens if the activity diagram is too small and n is so large” is given in the text.
108
43.
Explanation regarding not using subQueues for decision nodes is given in the text.
105
44.
Comment is added in Line 32 in Algorithm 6.1.
106
45.
Appendix H is added to illustrate some cases of sequence diagrams to convert them into activity diagrams.
217
185
Appendix C CQS Path Generator Implementation Details This appendix gives implementation details of the concurrent queue search path generator. First we describe the architecture of the path generator and then explain CQSPathGenerator class. Subsequently we describe an ADG class. Finally, the text file format of an ADG is explained.
CQS Path Generator: The architecture of the path generator is shown in Figure C.1. PathGenerator is a super class and it provides basic functionalities to path generators. CQSPathGenerator and DFSLevelPermuteGenerator are two implementations of the path generator for non-nested activity diagrams. A path generator needs an ADG for generation of the test scenarios (paths).
1
1
ActivityDiagramGraph
PathGenerator
1 2..n Activity
CQSPathGenerator
DFSLevelPermuteGenerator
Figure C.1: An architecture of path generator for CQS
187
C. CQS Path Generator Implementation Details P athgenerator class diagram is shown in Figure C.2. It has three attributes graph, startT ime, and endT ime. It has following functionalities: i) getPath(): this is an abstract function. Its derived classes implement this function to return a path. ii) getPathSet(int n): this function returns the set of n paths. This function calls getP ath() function n times. iii) getDistinctPathSet(String[] paths): this function returns the distinct paths from input string array paths. iv) loadActivityDiagramGraph(String fname): this function loads ADG from a text file. v) setStartTime(): this function sets the start time of path generation. vi) setStartTime(): this function sets the end time of the generation. Start and end times are used for the calculation of the time required to generate paths. CQSP athGenerator and DF SLevelP ermuteGenerator classes extends pathGenerator
PathGenerator -graph : ActivityDiagramGraph -startTime : long -endTime : long +pathGenerator() +setStartTime():void +setEndTime():void +loadActivityDiagramGraph(String fname):void +getPath():String (abstract) +getPathSet(int num):String[] +getDistinctPathSet(String[] paths):ArrayList
Figure C.2: A path generator class diagram
class. They implement getP ath() method of pathGenrator class. The detailed algorithm of CQS is given as Algorithm 6.1. The detailed algorithm of DFSLevelPermute is given as Algorithm 6.2. 188
Activity Diagram Graph Class Diagram: Activity diagram graph is input to the path generator. The class diagram of an ADG is as shown in Figure C.3. ADG is described by attributes nodes, edges, numberOf V ertices, and rootN ode. Member
Activity
2..n
ActivityDiagramGraph -nodes : Map -edges : LinkedList -activities : HashTable -numberOfVertices : int -rootNode : int
-id : int -name : String -umlId : String -type : int -preCondition : String -postCondition : String -visited=false : boolean -visitCount=0 : int -levelNumber : int -Predecessors : LinkedList +Activity(String umlId, String Name, int Type) +setVisited(boolean flag):void +getVisited():boolean +setVisitedCount():void +getVisitedCount():int +clearVisitCount():void +getType():String +setPreCondition(String condition) :void +getPreCondition():String +setPostCondition(String condition):void +getPostCondition(): String +addPredecessor(int nodeId):void +getPredecessor():LinkedList +toString():String +setLevelNumber(int num):void +getLevelNumber():int
+ActivityDiagramGraph() +getRootNode():int +addNode(String umlid, Activity node):void +addEdge(String source, String destination):void +initializeEdgeSet():void +getUnvisitedChildNode(Activity node):Activity +getRandomUnvisitedChildNode(Activity node):Activity +clearNode():void +readActivityGraphFromFile(String fname):void +getAdjacencyCount(Activity node):int +getActivity(int num):Activity +getAdjacentNodes(int activity):LinkedList +toString():String
Figure C.3: An activity diagram graph class diagram
functions facilitate the functionalities as follows: a)getRootNode(): this function returns an integer value indicating the root node of the graph. b)addNode(String umlId, Activity node): this function adds the given activity in nodes. c)addEdge(String source, String destination): this function adds an edge from the source node to the destination node. e)initializeEdgeSet(): this function initializes the linked list of edges. f)getUnvisitedChild(Activity node): this function returns the unvisited adjacent 189
C. CQS Path Generator Implementation Details activity node of activity node. g)clearNode(): this function assigns the visit count of nodes to zero. h)readActivityDiagramGraphFromFile(String fname): this function reads the ADG stored in the text file f name. i)getAdjacencyCount(Activity node): this function returns the integer count of the number of adjacent nodes to node. Implementation of ADG uses Java collections like Map, Linked List, and Hash Table as shown in the figure. ADGs are stored in text files. Activity Diagram Graph Text File Format: A structure of ADG text file is sown in Figure C.4. There are a total 11 types of nodes as shown in the figure. The first line has the number n of activities and ADG name separated by the symbol ‘#’. The second line onwards, all the n activities are listed. Each line describes activity by umlId, name, and type separated by the symbol ‘#’. After all activities are listed, Activity Node umlId#Name#type
n # activity nodes # graph name a1#Start#0 a2#Enter Customer No#1 .
Types 0-Start 1-Activity 2-End 3-FlowEnd 4-Decision 5-Merge 6-Fork 7-Join 8-Send Signal 9-Receive Signal 10-Lock 11-Unlock
. . . an#End#2 m # edges a1#a2 a2#a3 .
.
Edge sourceNode#destinationNode
.
a13#an
Figure C.4: An activity diagram graph text file format
the next line has the number m of edges. Thereafter, all the m edges are listed. An edge with source node a and destination node b is written as a#b.
190
Appendix D DP-DFS Path Generator Implementation Details This appendix gives implementation details of DP-DFS path generator. First we describe the architecture of the path generator and then explain the DP-DFS path generator class. Subsequently, we describe the MCFG class and finally describe a text file format for MCFG graph.
DP-DFS Path Generator: The architecture of the path generator is shown in Figure D.1. The path generator needs an MCFG for the generation of test scenarios.
1
1
MCFG
PathGenerator1
1 2..n
DP-DFSPathGenerator +DP-DFSPathGenerator(MCFG) +getPath(): String +getPathSet(int numOfPaths) : String []
Node
Figure D.1: An architecture of path generator for DP-DFS path generator
191
D. DP-DFS Path Generator Implementation Details P athGenerator1 class is the superclass of DP-DFSPathGenerator class. An MCFG graph is read from a text file using member function loadMCFG(String fname). The path generator maintains start time and end time of each execution. It has getPath() abstract method, which is implemented by the specific path generator. The getDinstictPathSet(String [] paths) returns distinct paths from input string array paths.
PathGenerator1 -graph : MCFG -startTime : long -endTime : long +pathGenerator1() +setStartTime():void +setEndTime():void +loadMCFG(String fname):void +getPath():String (abstract) +getPathSet(int num):String[] +getDistinctPathSet(String[] paths):ArrayList
Figure D.2: A class diagram of path generator
DP-DFS scenario (path) generator extends the P athGenerator1 class. It implements the abstract method getPath() of P athGenerator1 class to realize the DP-DFS algorithm. Details of the DP-DFS algorithm are given in Algorithm 8.1 in Chapter 8. MCFG Graph Class Diagram: An MCFG graph data structure is shown in Figure D.3. It is described by attribute vertices, edges, nodes, type, numberOf vertices, and rootN ode. We used map collection for vertices to store node objects. A linked list is used for edges. We have also used nodes hash table for string and integer mapping of node objects. MCFG provides the following functionalities: i) addVertex(String id, Node node): this function adds a node to vertices. If node type is Start then this node is set as rootN ode. This function increments the numberOf V ertices count. It also, updates nodes with umlId of the node. 192
MCFG - vertices : Map - edges[] : LinkedList - nodes : HashTable - type : int - numberOfVertices=0 : int -rootNode : int +MCFG() +getRootNode() : int +addVertex(String id, Node node) : void +initializeEdgeSet() : void +addEdge(String source, String destination) : void +addEdge(String source, String destination, int edgeType) : void +readMCFGFromFile(String fname) : void
Figure D.3: A class diagram of MCFG graph
ii) addEdge(String Source, String Destination, Int Type): this function adds an edge to edges. iii) getRootNode(): this function returns an integer value for the root node of the graph. iv) initializeEdgeSet(): this function initializes edges linked list array based on numberOf V ertices. v) MCFG(): this is a constructor of the class. It initializes variables vertices, nodes, and numberOf V ertices. vi) readMCFGFromFile(String fname): this function reads an MCFG graph from text file f name; the structure of the text file is given in Figure D.5. MCFG is a collection of nodes and edges between nodes. N ode data structure is shown in Figure D.4. It is described by attributes Id, name, umlId, type, visited, and parents. P arents is a set containing parent nodes of the node. Member function addParent(int parent) adds parent node id to P arents set of the node. Member function getParents() returns the set of all parents of the node. MCFG Graph Text File Format: MCFG graph is stored in text file format. The structure of the text file is shown in Figure D.5. There are twelve types of 193
D. DP-DFS Path Generator Implementation Details
Node - id : int - name : String - umlId : String - type : int - visited=false : boolean -Parents : Set +Node(int id, String Name, int type) +Node(String umlId, String Name, int type) +getId() : int +setVisited(boolean flag): void +getVisited(): boolean +getName(): String +getType():String +addparent(int Parent):void +getParents(): Set +toString():String
Figure D.4: A class diagram of message node of MCFG graph
Node information umlId # name # type Types 0-start 1-end 2-message 3-parallel fragment Start 4-parallel fragment End 5-critical region start 6-critical region end 7-send signal 8-receive signal 9-lock 10-unlock 11-decision 12-merge
n#nodes# Bank account transfer sds#SdStart#0 par1s#par1Start#3 cs1s#CritialRegion1Start#5 . .
. m13#deposit(100)#2 sde#SdEnd#1 m#edges Sds#par1s#1 .
Edge Information Source node # destination node # type Types 0- message interleaving edge 1-order specification edge
. . m#m6#0
Figure D.5: A text file format of the MCFG and RMCFG graphs
194
message nodes as shown in the figure. There are two types of edges: 0-interleaving edge, and 1-order specification edge. The first line of the file has the number n of nodes and name of the graph separated by the symbol ‘#’. The second line onwards, message nodes are listed. They are described by umlId, name, and type of the node separated by the symbol ‘#’. The next line has the number m of edges in the graph. Thereafter, each line lists edges, one edge in each line. They are described by source node, destination node, and type of the edge separated by the symbol ‘#’.
195
Appendix E State Machine Executor Implementation Details This appendix provides implementation details of the state machine executor used in Chapter 4. First we describe a class diagram of the sate machine executor and then an input file format of the state machine. Finally we described the data structure used by the test scenario generator to generate data access traces.
State Machine Executor: State machine executor (SMExecutor) simulates an execution of the state machine. SMExecutor loads a specific state machine from a file. It processes input data access trace to give the final state of the machine. A class diagram of SMEexecutor is shown in Figure E.1. There are three main classes: State, T ransition, and StateM achne. Class State describes states of a state machine by attribute sateId, satename, stateT ype, and exitT ransitions. A state machine has a minimum of two and a maximum of n states. Class T ransitions of a state machine are described by attributes transitionId, transitionN ame, eventN ame, f romState, toState, and guard. A state machine has a minimum of one and a maximum of n transitions. Class StateM achine is described by attributes statemachineN ame, initialState, activeState, inputEvent, states, and transitions. StateM achine provides the following functionalities: i) addState(State s): this function adds state s to the state machine. ii) addTransition(Transition tr): this function adds transition tr to the state machine. iii) ExecuteStatemachine(String events): this function reads events and start 197
E. State Machine Executor Implementation Details execution of stateM achine from initialState and returns the last state in which execution stops. iv) ExecuteTransition() :this function changes a state of the state machine based on the guard on the transition. v) getState(String name): this function returns the state object having statename name. vi) isLoaded(): this function returns the boolean value indicating whether state machine information is read from file or not. vii) readSMFromFile(String fname): this function reads state machine from a text file. viii) setInitialState(State is): this function sets the initial state of the state machine.
StateMachine
-satemachineName : String -initialState: State - activeState: State - inputEvent: String -states: ArrayList -Transitions :ArrayList +stateMachine(String name) +addState(State s) +setInitialState(State is) +isLoaded(): boolean +addTransition(Transition tr) +executeTransition() +readSMFromFile(String fname) +getState(String name):State +executeStateMachine(String events):String 1 1
State - stateID: int - statename: String - stateType: int - exitTransition: ArrayList +State(int ID, String Name, int Type) + State(int ID, String Name) +entryAction():String +exitAction():String +getName():String +getType():int +addExitTransition(Transition tr) +getTransition(int i): Transition +getExitTransitionCount():int
2..n
Transition - transitionID: int - tranictionName : String - eventName: String - fromState: State - toState: State - guard: String
1..n
2
0..n
+Transition(int id, String name, State fromstate, State tostate) +setEvent(String event) +setGuard(String guard) +getFramState(): State +getToState():State +getEvent():String +evaluateGuard():boolean +toString():String
Figure E.1: A class diagram of the State Machine Executor
198
State Machine Text File Format: State machines are stored in text files. SMExecutor uploads state machines from a text file. The format of a text file is as shown in Figure E.2. A state machine file contains information about states and transitions. Information is separated by the symbol ‘#’. The first line gives the number N of states in the state diagram. Subsequent lines give information about the states. Each state is described by state name and the type of state that is separated by the symbol‘#’. After states, the next line gives the number M of transitions in the state diagram. Subsequent lines give information of transitions in the state diagram. Transitions are described by transition name, source sate, destination state, event, and guard. As shown in Figure E.1, there are three types of the states: i) 0 - start state, ii) 1 - normal state, iii) 2 - accept state. N # number of state # State Machine Name Start#0 Read#1 . . . Blocked#2 M # number of transitions T1#Start#Read#read/CS#true . . Tm#safeThread#Blocked#csStart#true
State Name # type state type : 0-start state, 1- normal state, 2-accept state
Transition Name # source state # destination state # event # guard
Figure E.2: A text file format of the state machine
Test Scenario Generator Information: Test scenario generators need additional information to generate data access traces than the information given in Appendix D. Therefore, we modify the class M CF G to define class M CF G1 and provide text file format for class M CF G1 to store graph information. A class diagram of MCFG1 is shown in Figure E.3. M CF G1 class contains additional information of shared objects as shown in the figure. In addition, nodes of M CF G1 require information of threads, locks, shared accesses and their counts. Accordingly, two more classes are added: myLock for lock information and SharedAccess for shared access information. A class diagram in Figure E.4 shows the relation between N ode, myLock, and SharedAccess classes. 199
E. State Machine Executor Implementation Details
MCFG1 - vertices : Map - edges [] : LinkedList -type : int -numberOfVertices=0 : int -rootNode : int -objects : Map -numberOfObjects : int +MCFG() +getRootNode() : int +addVertex(String id, Node node) : void +addObject(ObjectClass myObject) : void +initializeEdgeSet() : void +addEdge(String source, String destination) : void +addEdge(String source, String destination, int edgeType) : void +readMCFG1FromFile(String fname) : void +toString(): String
Figure E.3: A class diagram of MCFG graph for access trace generation
Node
0. .n
-id : int -name : String -type : int -threadId : int -Visited=false:boolean -Parents : Set -Locks : Map -NumberOfLocks=0 : int -Saccess : Map -numberOfSharedAccess : int +Node(String umlId, String Name, int type, int threadId) +getId() : int +getThreadId(): int +setVisited(boolean flag): void +getVisited(): boolean +getName(): String +getType():String +addparent(int Parent):void +getParents(): Set +addLock(myLock lock): void +addSharedAccess(SharedAccess maccess) +getAccessInfo(): String +toString():String
myLock -lockType=0 : int -lockName : String -ClassName : String -objectName : String -dataMember : Sting -lockOn=0 : int +myLock(String Name, int Type) +setLockData(String ClassName, String ObjactName) :void +setLockData(String ClassName, String ObjectName, String Field):void +getLockType():String +toString():String
SharedAccess
0. .n -accessType : int -className : String -ObjectName : String -DataMember : String
+SharedAccess(int Type, String ClassName, String ObjectName, String Field) + getAccessType() : String +toString() : String
Figure E.4: A class diagram of message node for access trace generation
200
A text file format of M CF G1 information is shown in Figure E.5. Information about critical regions is important for data access traces generation. A text file includes information of objects, threads, critical region nodes, and data-access nodes. Objects information is included in the beginning of the file, followed by nodes information. Finally edges information is given in the file. Here objects are of three types: normal objects, thread objects, and shared data objects. Each object information includes object name, class name, and type of object. Nodes are described by node name, node type and thread number. There are thirteen types of nodes. The node type 5 is of critical region start that includes additional information about the number of locks and lock details for each lock. Lock details include the name of lock, object on which lock is held, and details of class name of the object and fields if there are any for granular locks on fields. The node type 13 is of shared access that includes additional information about the number of shared accesses and details for each shared access. Shared access details include access type of field name, the object name and the class name. Finally, edges are included in the text file. Edge information includes source node name, destination node name and type of the edge. There are two types edges: message interleaving edges and order specification edges. This file structure is applicable for both the types of graphs MCFG and RMCFG.
201
E. State Machine Executor Implementation Details
5#objects O1#account#Account#2 . . 16#nodes sds#SdStart#0#0 par1s#par1Start#3#0 cs1s#CritialRegion1Start#15#1#1#lock1@1@Account@account1@ balance . . m13#deposit(100)#13#1#1#0@Account@account1@balance 66#edges Sds#par1s#1 . . . m2#m6#0 Extra information for Critical Start Node (type=5) umlId # ObjectName # ClassName # type # number of locks#Lock1#... Lock Information LockName @ Class Name @ Object Name @ Field Name Extra information for shared access operation node (type=13) umlId # ObjectName # ClassName # type # number of shared Access # SharedAccess1 #... Shared Access Information Access type @Class Name @ Object Name @Field Name Access type : 0-write, 1-read
Object Information umlId # ObjectName # ClassName # type Types 0-noraml object 1- thread object 2-shared data object
Node information umlId # name # type # threadId Types 0-start 1-end 2-message 3-parallel fragment Start 4-parallel fragment End 5-critical region start 6-critical region end 7-send signal 8-receive signal 9-lock 10-unlock 11-decision 12-Merge 13-shared access Edge Information Source node # destination node # type Types 0- message interleaving edge 1-order specification edge
Figure E.5: A text file format of the MCFG and RMCFG graphs for access trace generation
202
Appendix F MCFG Representation for Different Sequence Diagrams This appendix defines MCFG for different types of sequence diagrams. This will illustrate which features of sequence diagram lead to what kind of nodes, edges, and guard conditions in MCFG. The complete procedure of MCFG construction is given in Section 8.2.1 that clarifies all type of nodes and constraints on the number of edges. Table 8.1 lists different types of nodes in MCFG for their respective constructs in sequence diagrams. Table 8.2 lists different types of edges and guards associated with edges. MCFG construction includes control nodes for twelve types of combined fragments: alt, opt, break, par, seq, strict, neg, assert, critical, loop, consider, and ignore. A set sequence diagram figures 5.1 to 5.7 from Chapter 5 is used to illustrate MCFG representation. Synchronous message, asynchronous message, and combined fragments such as alt, opt, loop, and par are processed to generate MCFG in the following diagrams. In all the diagrams the start of a sequence diagram is shown by sdStart control node and the end is shown by sdEnd node. The message node is generated in MCFG for a message and its reply in sequence diagrams. The name of a message node is the same as message name in sequence diagrams. A combined fragment start in sequence diagrams is shown by CF xxStart control node in MCFG, where xx indicates the name of the combined fragment. For example, the start of alternative combined fragment in Figure F.3 is shown by CF altStart node. A combined fragment end in sequence diagrams is shown by CF xxEnd control node in MCFG, where xx insidcates the name of the combined fragment. For example, the 203
F. MCFG Representation for Different Sequence Diagrams end of alternative combined fragment in Figure F.3 is shown by CF altEnd node. A general ordering start in sequence diagrams is shown by Signal control node as shown in Figure F.7. A general ordering end in sequence diagrams is shown by W ait control node as shown in Figure F.7. Edges in MCFG are drawn as per the mapping given in Table 8.2. There are two types of edges: order specification edges and message interleaving edges. Order specification edges are shown by dark solid lines. Messages interleaving edges are shown by dashed lines. Sd: synchronousMessage sdStart O1: Object1
O2:Object2 1: m1() m1
2: m1
3: m2()
m2 4: m2
sdEnd
Figure F.1: An MCFG graph representation of a synchronous message for sequence diagrams
The MCFG graph representation of a synchronous message for sequence diagrams is shown in Figure F.1. Two message nodes m1 and m2 are created for two messages in addition to sdStart and sdEnd control nodes. All edges are of the type order specification; and they are drawn as per the mapping given in Table 8.2. The MCFG graph representation of an asynchronous message for sequence diagrams is shown in Figure F.2. Two message nodes m1 and m2 are created for two messages in the sequence diagram. Two control nodes CF parStart and CF parEnd are created to fork threads that represent the asynchronous behavior of messages m1 and m2. These threads representation allows message m2 to complete independently without waiting for the completion of message m1. Two message interleaving edges are created between messages m1 and m2. All edges are of the type order specification; and they are drawn as per the mapping given in Table 8.2. The MCFG graph representation of an alternative combined fragment for sequence 204
Sd: asynchronousMessage
sdStart
O2:Object2
O1: Object1
CFpar Start
1: m1() 2: m2()
m2
m1
3: m2
CFpar End
4: m1
sdStart
Figure F.2: An MCFG graph representation of an asynchronous message for sequence diagrams
sdStar t
Sd: AlternativeFragment
O1: Object1
O2:Object2
m1
O3:Object3
1: m1() 2: m1
CFalt Start Alt
[true]
m3
m2
3: m2() 4: m2
CFalt End
[false]
5: m3() 6: m3
sdEnd
Figure F.3: An MCFG graph representation of an alternative combined fragment for sequence diagrams
diagrams is shown in Figure F.3. Three message nodes m1, m2, and m3 are created for three messages in the sequence diagram. Two control nodes CF altStart and 205
F. MCFG Representation for Different Sequence Diagrams CF altEnd are created to represent the alternative combined fragment in the sequence diagram. The edge from CF altStart to m2 is created; and it has assigned interaction operand guard true from which message m2 belongs. Similarly, the edge from CF altStart to m3 is created; and it has assigned interaction operand guard f alse from which message m3 belongs. All edges are of the type order specification; and they are drawn as per the mapping given in Table 8.2. sdSta rt
Sd: AlternateFragment
O1: Object1
O2:Object2
m1
O3:Object3
1: m1()
CF opt Start
2: m1 Opt
[true]
3: m2() m2
4: m2
CF opt End
5: m3() m3
6: m3
sdEnd
Figure F.4: An MCFG graph representation of an option combined fragment for sequence diagrams
The MCFG graph representation of an option combined fragment for sequence diagrams is shown in Figure F.4. Three message nodes m1, m2, and m3 are created for three messages in the sequence diagram. Two control nodes CF optStart and CF optEnd are created to represent the option combined fragment in the sequence diagram. The edge from CF optStart to m2 is created; and it has assigned interaction operand guard true from which message m2 belongs. The edge from CF optStart to CF optEnd is created. All edges are of the type order specification; and they are drawn as per the mapping given in Table 8.2. The MCFG graph representation of a loop combined fragment for sequence diagrams is shown in Figure F.5. Three message nodes m1, m2, and m3 are created for three messages in the sequence diagram. Two control nodes CF loopStart and CF loopEnd are created to represent the loop combined fragment in the sequence diagram. The edge from CF loopStart to m2 is created; and it has assigned the loop interaction operand’s guard, i.e., true. The edge from CF loopEnd to CF loopStart is 206
sdSta rt
Sd: loopFragment
O1: Object1
m1
O2:Object2 [True]
1: m1()
2: m1
m2
CF loop Start [False]
Loop
3: m2() CF loop End
4: m2
m3
5: m3() sdEnd
Figure F.5: An MCFG graph representation of a loop combined fragment for sequence diagrams
created. The edge from CF loopStart to m3 is created; and it has assigned negation of the loop interaction operand’s guard, i.e., f alse. All edges are of the type order specification; and they are drawn as per the mapping given in Table 8.2.
sdStart
Sd: parallelFragment
O1: Object1
O2:Object2
m1
O3:Object3
1: m1() CFpar Start
2: m1 Par
m3
m2
3: m2() 4: m2
CFpa rEnd
5: m3() 6: m3
m4
7: m4() 8: m4 sdStart
Figure F.6: An MCFG graph representation of a parallel combined fragment for sequence diagrams
207
F. MCFG Representation for Different Sequence Diagrams The MCFG graph representation of a parallel combined fragment for sequence diagrams is shown in Figure F.6. Four message nodes m1, m2, m3, and m4 are created for four messages in the sequence diagram. Two control nodes CF parStart and CF parEnd are created to represent the parallel combined fragment in the sequence diagram. Two message interleaving edges are created between messages m1 and m2. All other edges are of the type order specification; and they are drawn as per the mapping given in Table 8.2.
Sd: General Ordering O2:Object2
O3:Object3
1: m1() 2: m1 Par
3: m2() 4: m2
m4
5: m3()
sd Start
m1
CFpar Start
6: m3 7: m4()
8: m4
wait
m5
General Ordering
O1: Object1
CFpar End
m6
sd End
Signal: S1 m2
9: m5()
sign al
m3
10: m5
11: m6() 12: m6
Figure F.7: An MCFG graph representation of a general ordering inside the parallel fragment for sequence diagrams
The MCFG graph representation of a general ordering inside the parallel combined fragment for sequence diagrams is shown in Figure F.7. Six message nodes m1, m2, m3, m4, m5, and m6 are created for six messages in the sequence diagram. Two control nodes CF parStart and CF parEnd are created to represent the parallel combined fragment in the sequence diagram. Two control nodes Signal and W ait are created to represent the general ordering inside the parallel combined fragment in the sequence diagram. The nine (3*3=9) bidirectional message interleaving edges are created inside parallel fragment represented by nodes CF parStart and CF parEnd. All other edges are of the type order specification; and they are drawn as per the mapping given in Table 8.2.
208
Appendix G Pseudo Code for Variations of Parallel Fragments This appendix provides variations of the parallel fragment and its relevant activity diagram for Figure 5.6. The proposed variations in the sequence diagram leads to the same activity diagram. However, their implementations are different. The figure has three objects: O1, O2, and O3, four messages: m1, m2, m3, and m4, and a parallel fragment. Messages m1 and m4 are called from object O1 on object O2. Messages m2 and m3 are called from Object O2 on object O3 inside parallel fragment in two operands (threads). The parallel fragment in the sequence diagram is translated into the fork-join construct in the activity diagram as shown in Figure 5.6. We discuss the following three variations to the figure: a) both the messages m2 and m3 are called from object O1 and both are on object O3, b) message m2 is called from object O2 and message m3 is called from object O3, both are on object O3, and c) message m1 is called from object O1 on object O2, but instead of returning to O1 it first does somework (call it m11) on O2, then it makes calls m2 and m3 on object O3. We have shown the translation and Java like pseudo code for each of the above cases. Finally, we have presented Java like pseudo code for general ordering construct inside parallel fragment as in Figure 5.7.
209
G. Pseudo Code for Variations of Parallel Fragments Case a: In this case, the calling object of messages m2 and m3 is changed to O1 for Figure 5.6. Activity diagram is same as that of the original sequence diagram as shown in Figure G.1. Java like pseudo code for the design in Figure G.1 is as follows:
O1:Object1
O2:Object2
O3:Object3
m1()
O1
O2
O3
m1
Par
m2
m2()
m3
m3()
m4()
m4
Figure G.1: Translation of parallel fragment in sequence diagram into activity diagram for case a
Three classes Object1, Object2, and Object3 are generated. Object O1 of class Object1 forks two threads t1 and t2 where methods m2 and m3 are called. Class: Object1 : Case a 1: Object2 O2; 2: Object3 O3 ; 3: 4: 5: 6: 7: 8: 9: 10: 11: 12:
O2.m1( ); Fork t1 O3.m2( ); Fork t2 O3.m3( ); t1.start(); t2.start(); Join(t1, t2); O2.m4( );
210
Class: Object2 : Case a 1: 2: 3: 4:
m1() print “m1”; m4() print “m4”;
Class: Object3 : Case a 1: 2: 3: 4:
m2() print “m2”; m3() print “m3”;
Case b: In this case, the calling objects of messages m2 and m3 are changed for Figure 5.6. Message m2 is called from object O2 and message m3 is called from object O3 and both are on object O3. Activity diagram is same as that of the original sequence diagram as shown in Figure G.2. Java like pseudo code for the design in
O1:Object1
O2:Object2
O3:Object3
O1
m1()
O2
O3
m1
Par m2
m2()
m3 m3() m4()
m4
Figure G.2: Translation of parallel fragment sequence diagram into activity diagram for case b
Figure G.2 is as follows: Three classes Object1, Object2, and Object3 are generated. Class Object1 has two object instances O2 and O3 of classes Object2 and Object3, respectively. Class Object1 forks two threads t1 and t2 where methods m2 and m3 are called. In this case, object O2 needs an instance of class Object3 say O30 to call method m2. Object O1 of class 211
G. Pseudo Code for Variations of Parallel Fragments Object1 forks two threads t1 and t2 where methods m2 and m3 are called. Class: Object1 : Case b 1: Object2 O2; 2: Object3 O3 ; 3: 4: 5: 6: 7: 8: 9: 10: 11: 12:
O2.m1( ); Fork t1 O2.O30 .m2( ); Fork t2 O3.m3( ); t1.start(); t2.start(); Join(t1, t2); O2.m4( );
Class: Object2 : Case b 1: Object3 O30 ; 2: 3: 4: 5: 6:
m1() print “m1”; m4() print “m4”;
Class: Object3 : Case b 1: 2: 3: 4:
m2() print “m2”; m3() print “m3”;
212
Case c: In this case, the calling object of messages m2 and m3 is changed for Figure 5.6. Messages m2 and m3 are called from object O2 on object O3. Activity diagram is same as that of the original sequence diagram as shown in Figure G.3. In this case, object O2 has instance of object O3 of class Object3 that calls methods m2 and m3 on O3. Class Object2 has method m10 that is called inside method m1. Object O2 forks two threads: t1 and t2 calling method m2 and m3 on object O3. Object O1 will call methods m1 and m4 on object O2. Java like pseudo code for the design in
O1:Object1
O2:Object2
O3:Object3
O1
O2
O3
m1 m1’
m1() m1’() Par
m2
m2()
m3
m3()
m4() m4
Figure G.3: Translation of parallel fragment sequence diagram into activity diagram for case c
Figure G.3 is as follows:
Class: Object1 : Case c 1: Object2 O2; 2: 3: 4:
O2.m1( ); O2.m4( );
213
G. Pseudo Code for Variations of Parallel Fragments Class: Object2 : Case c 1: Object3 O3 ; 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16:
m1() print “m1”; m10 ( ); Fork t1 O2.O3.m2( ); Fork t2 O3.m3( ); t1.start(); t2.start(); Join(t1, t2); m4() print “m4”; m1 () print “m10 ”; 0
Class: Object3 : Case c 1: 2: 3: 4:
m2() print “m2”; m3() print “m3”;
General ordering inside parallel fragment: Java like pseudo code for a general ordering construct inside the parallel fragment as in Figure 5.7 as follows. Three classes Object1, Object2, and Object3 are generated. Class Object1 has two object instances O2 and O3 of classes Object2 and Object3, respectively. Object2 has three methods m1, m2, and m4. Object3 has three methods m3, m5, and m6. Class Object1 forks two threads t1 and t2 where methods m2 and m4 of object O2 and methods m3, and m5 of object O3 are called. The general ordering construct in design ensures that method m5 in thread t2 shall wait for method m2 in thread t1 to complete.
214
Class: Object1 - general ordering 1: Object2 O2; 2: Object3 O3 ; 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16:
O2.m1( ); Fork t1 O2.m2( ); N otif y m2 is complete...; O2.O3.m3( ); Fork t2 O2.m4( ); W ait for m2 to complete...; O2.O3.m5( ); t1.start(); t2.start(); Join(t1, t2); O2.m6( );
Class: Object2 - general ordering 1: 2: 3: 4: 5: 6:
m1() print ”m1”; m2() print ”m2”; m4() print ”m4”;
Class: Object3 - general ordering 1: 2: 3: 4: 5: 6:
m3() print ”m3”; m5() print ”m5”; m6() print ”m6”;
215
Appendix H More Examples for Translation of Sequence Diagram to Activity Diagram This appendix provides some more of sequence diagram and their conversion to activity diagrams. The proposed variations in the sequence diagram is translated into different activity diagrams, but their test scenarios are same. We discuss the following four sequence diagrams: a) A sequence diagram has one object and three messages called in a sequence. b) A sequence diagram has one object and three messages called in a nested way. c) A sequence diagram has three objects and three messages called in a sequence. d) A sequence diagram has three objects and three messages called in a nested way.
Case a: In this case, sequence diagram has one object O1 and three messages m1, m2, and m3 in order as shown in Figure H.1. All three message are called from object O1 on itself. The corresponding activity diagram is sequential structure; all messages are called in sequence as shown in Figure H.1. The test scenario generated from this design is Sa1 = S → m1 → m2 → m3 → E.
217
H. More Examples for Translation of Sequence Diagram to Activity Diagram
O1:Object1
O1
m1 m1 m2 m2 m3
m3
Figure H.1: Translation of sequence diagram into activity diagram for case a
O1
O1:Object1
m1
m1
m2 m2 m3 m3
Figure H.2: Translation of sequence diagram into activity diagram for case b
Case b: In this case, sequence diagram has one object O1 and three messages m1, m2, and m3 in order as shown in Figure H.2. All three message are called from object O1 on itself in the nested way. The corresponding activity diagram is as shown in Figure H.2. In this case the test scenarios need will be two. If the messages m2 and m3 are public methods then the test scenario generated is Sb1 = S → m1 → m2 → m3 → E. If the messages m2 and m3 are private methods 218
then the test scenario generated is Sb2 = S → m1 → E. Case c: In this case, sequence diagram has three object O1, O2, and O3 and three messages m1, m2, and m3 as shown in Figure H.3. All three message m1, m2, and m3 are called from three object O1, O2, and O3 on there self. The corresponding activity diagram is sequential structure; all messages are called in sequence as shown in Figure H.3. The test scenario generated from this design is Sc1 = S → m1 → m2 → m3 → E.
O1:Object1
O2:Object2
O1
O3:Object3
m1
O2
O3
m1 m2 m2 m3
m3
Figure H.3: Translation of sequence diagram into activity diagram for case c
Case d: In this case, sequence diagram has three object O1, O2, and O3 and three messages m1, m2, and m3 as shown in Figure H.4. All three message m1 is called on object O1. Inside message m1 message m2 is called on object O2; and inside message m2 message m3 on object O3. The corresponding activity diagram is nested message calls as shown in Figure H.4. In this design either object O1 or message m1 should have instance of object O2 to call message m2. Similarly, either object O2 or message m2 should have instance of object O3 to call message m3. Messages m2 and m3 are public methods in objects O2 and O3, respectively. The test scenario generated from this design is Sd1 = S → m1 → m2 → m3 → E.
219
H. More Examples for Translation of Sequence Diagram to Activity Diagram
O1:Object1
O3:Object3
O2:Object2
O1
m1
O2
m2 m3
O3
m1 m2
m3
Figure H.4: Translation of sequence diagram into activity diagram for case d
In summary, in all the above different designs test scenario generated from them is same except for case b; the scenario is S = S → m1 → m2 → m3 → E. In case b, the visibility of the methods will give two different test scenarios as given.
220
Publications out of This Work 1. M. Shirole and R. Kumar, “A Hybrid Genetic Algorithm Based Test Case Generation Using Sequence Diagrams”, In Proceedings of the Contemporary Computing, volume 94 of Communications in Computer and Information Science, pages 53–63, 2010. 2. M. Shirole, A. Suthar, and R. Kumar, “Generation of Improved Test Cases from UML State Diagram using Genetic Algorithm”, In Proceedings of the 4th India Software Engineering Conference, ISEC ’11, pages 125–134, 2011. 3. M. Shirole, K. Mounika, and R. Kumar, “Transition Sequence Exploration of UML Activity Diagram using Evolutionary Algorithm”, In Proceedings of the 5th India Software Engineering Conference, ISEC ’12, pages 97–100, 2012. 4. M. Shirole and R. Kumar, Testing for Concurrency in UML Diagrams, In SIGSOFT Softw. Eng. Notes, 37(5):1–8, Sept. 2012. 5. M. Shirole and R. Kumar, UML Behavioral Model Based Test Case Generation: A Survey, In SIGSOFT Softw. Eng. Notes, pages 1-13, July 2013. 6. M. Shirole and R. Kumar, Test scenario selection for concurrency testing from UML models, In Proceedings of the Eighth International Conference on Contemporary Computing (IC3), pages 531-536 , Aug 2015.
Curriculum Vitae
Mahesh Shirole received a B.E. degree in Computer Science and Engineering from Walchand College of Engineering, Sangali in 1996. Further he received an M.E. degree in Computer Science from Veermata Jijabai Technological Institute, Mumbai in 2004. From August 1998 till April 2000, he worked in Bharati Vidyapeeths College of Engineering, Mumbai, as a Lecturer. Since April 2000, he has been working as a faculty member in Veermata Jijabai Technological Institute, Mumbai. In July 2010, he has joined as a research scholar under Quality Improvement Program (QIP) in the department of Computer Science and Engineering in Indian Institute of Technology, Kharagpur. During the period of Ph.D program (2010-2014), at Indian Institute of Technology, Kharagpur, he has been engaged with the Ph.D work on Concurrency Test Scenario Generation using UML Transition Sequences. He has also been associated in teaching and research in the broad field of software engineering. His research interests include software testing, object oriented technology, evolutionary algorithm, image processing, and networking.
References Abdurazik, A. and Offutt, J. (2000). Using UML collaboration diagrams for static checking and test generation. In Proceedings of the 3rd International Conference on the Unified Modeling Language: Advancing the Standard, pages 383 – 395. SpringerVerlag. 2.2.1, 2.2.2, 3.1.3, 3.3.1, 3.3.1, 3.3.2, 8.3.3 Abdurazik, A. and Offutt, J. (2004). A Controlled Experimental Evaluation of Test Cases Generated from UML Diagrams. Technical report. 2.4 Ali, S., Briand, L., Hemmati, H., and Panesar-Walawege, R. (2010). A systematic review of the application and empirical investigation of search-based test case generation. IEEE Transactions on Software Engineering, 36(6):742 –762. 2.4, 7 Ammann, P. and Offutt, J. (2008). Introduction to Software Testing. Cambridge University Press, New York, NY, USA, 1 edition. 3, 3.1.3, 3.2.1, 3.2.1 Ammann, P., Offutt, J., and Xu, W. (2008). Coverage Criteria for State Based Specifications. In Hierons, R., Bowen, J., and Harman, M., editors, Formal Methods and Testing, volume 4949 of Lecture Notes in Computer Science, pages 118–156. Springer Berlin / Heidelberg. 3.1.3 Andrews, A., France, R., Ghosh, S., and Craig, G. (2003). Test adequacy criteria for uml design models. Journal of Software Testing, Verification and Reliability, 13:95–127. 3.3.2 Andrews, J., Briand, L., Labiche, Y., and Namin, A. (2006). Using mutation analysis for assessing and comparing testing coverage criteria. IEEE Transactions on Software Engineering, 32(8):608–624. 8.3.4 Ball, T., Burckhardt, S., de Halleux, P., Musuvathi, M., and Qadeer, S. (2011). 223
References Predictable and progressive testing of multithreaded code. Software, IEEE, 28(3):75 –83. 4 Baresel, A., Sthamer, H., and Schmidt, M. (2002). Fitness Function Design To Improve Evolutionary Structural Testing. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO ’02, pages 1329–1336, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. 7.1 Beizer, B. (1990). Software testing techniques (2nd ed.). Van Nostrand Reinhold Co., New York, NY, USA. 1.1, 3, 3.1.3, 3.2.1, 5.2.2 Bradbury, J. S., Cordy, J. R., and Dingel, J. (2006). Mutation Operators for Concurrent Java (J2SE 5.0). In Proceedings of the Second Workshop on Mutation Analysis, MUTATION ’06, page 11, Washington, DC, USA. IEEE Computer Society. (document), 8.3.4, 8.6 Briand, L., Di Penta, M., and Labiche, Y. (2004). Assessing and Improving StateBased Class Testing: A Series of Experiments. IEEE Transactions on Software Engineering, 30(11):770–783. 2.4 Briand, L. C., Labiche, Y., and Cui, J. (2005). Automated support for deriving test requirements from UML statecharts. Software and System Modeling, 4(4):399–423. 2.1, 2.2.3 Bron, A., Farchi, E., Magid, Y., Nir, Y., and Ur, S. (2005). Applications of Synchronization Coverage. In Proceedings of the tenth ACM SIGPLAN symposium on Principles and practice of parallel programming, pages 206–212, New York, NY, USA. ACM. 1.3, 2.1, 3.1.3, 3.3.2.2 Buchs, D., Pedro, L., and Lucio, L. (2006). Formal Test Generation from UML Models. In Research Results of the DICS Program, volume 4028 of Lecture Notes in Computer Science, pages 145–171. Springer. 2.2.2 Campbell, C., Veanes, M., Huo, J., and Petrenko, A. (2005). Multiplexing of partially ordered events. In TestCom, Lecture Notes in Computer Science, pages 97–110. Springer. 8.1 Cartaxo, E., Neto, F., and Machado, P. (2007). Test Case Generation by means of UML Sequence Diagrams and Labeled Transition Systems. In Proceedings of 224
References the IEEE International Conference on Systems, Man and Cybernetics, ISIC, pages 1292–1297. 2.2 Carver, R. and Lei, J. (2010). A Stateful Approach to Testing Monitors in Multithreaded Programs. In Proceedings of the IEEE 12th International Symposium on High-Assurance Systems Engineering (HASE), 2010, pages 54 –63. 2.1.2 Carver, R. and Lei, Y. (2004). A General Model for Reachability Testing of Concurrent Programs. In Formal Methods and Software Engineering, volume 3308 of Lecture Notes in Computer Science, pages 76–98. Springer Berlin / Heidelberg. 2.1.2, 4.2.2 Chandler, R., Lam, C. P., and Li, H. (2005). AD2US: An Automated Approach to Generating Usage Scenarios from UML Activity Diagrams. In Proceedings of the 12th Asia-Pacific Software Engineering Conference, pages 9–16. 1.1, 1.2, 2.2.1, 6, 6.1, 6.3, 7, 8.3.4 Chen, M., Mishra, P., and Kalita, D. (2008). Coverage-driven Automatic Test Generation for UML Activity Diagrams. In Proceedings of the 18th ACM Great Lakes symposium on VLSI, pages 139–142. 2.2.2 Chen, M., Mishra, P., and Kalita, D. (2010). Efficient test case generation for validation of UML activity diagrams. Design Autom. for Emb. Sys., 14(2):105–130. 3.1.3 Committee, I. C. S. S. C. (1990). IEEE Standard Computer Dictionary: A Compilation of IEEE Standard Computer Glossaries, 610. ANSI / IEEE Std. IEEE. 3.1.3 Corbett, J., Dwyer, M., Hatcliff, J., Laubach, S., Pasareanu, C., Robby, and Zheng, H. (2000). Bandera: extracting finite-state models from java source code. In Proceedings of the 2000 International Conference on Software Engineering, 2000, pages 439 –448. 2.1.1 Dinh-Trong, T., Ghosh, S., and France, R. (2006). A Systematic Approach to Generate Inputs to Test UML Design Models. In Proceedings of the 17th International Symposium on Software Reliability Engineering, ISSRE ’06, pages 95–104. 2.2, 2.2.3, 3.1.3, 3.3.1, 3.3.1 225
References Dinning, A. and Schonberg, E. (1991). Detecting access anomalies in programs with critical sections. SIGPLAN Not., 26(12):85–96. 1 Doungsa-ard, C., Dahal, K., Hossain, A., and Suwannasart, T. (2007). Test Data Generation from UML State Machine Diagrams using GAs. In Proceedings of the International Conference on Software Engineering Advances, ICSEA 2007, page 47. 2.3, 2.2.4, 7.1 Edelstein, O., Farchi, E., Nir, Y., Ratsaby, G., and Ur, S. (2002). Multithreaded java program test generation. IBM Systems Journal, 41(1):111–125. 1, 2.1, 2.1.3, 6 Eshuis, R. (2006). Symbolic Model Checking of UML Activity Diagrams. ACM Trans. Softw. Eng. Methodol., 15:1–38. 1 Eshuis, R. and Wieringa, R. (2004). Tool Support for Verifying UML Activity Diagrams. IEEE Transactions on Software Engineering, 30(7):437 – 447. 2.2.2 Fan, X., Shu, J., Liu, L. L., and Liang, Q. J. (2009). Test Case Generation from UML Subactivity and Activity Diagram. In Proceedings of the Second International Symposium on Electronic Commerce and Security, 2009. ISECS ’09, volume 2, pages 244 –248. 2.1, 2.2.3 Farchi, E., Nir, Y., and Ur, S. (2003). Concurrent bug patterns and how to test them. In Proceedings of the 17th International Symposium on Parallel and Distributed Processing, IPDPS ’03, pages 286–292, Washington, DC, USA. IEEE Computer Society. 2.1.3, 8.3.4 Farooq, U., Lam, C. P., and Li, H. (2008). Towards Automated Test Sequence Generation. In Proceedings of the 19th Australian Conference on Software Engineering, pages 441–450. IEEE Computer Society. 2.3, 2.2.4, 7 Farzan, A. and Madhusudan, P. (2006). Causal Atomicity. In Ball, T. and Jones, R., editors, Computer Aided Verification, volume 4144 of Lecture Notes in Computer Science, pages 315–328. Springer Berlin Heidelberg. 1 Ferreira, R., Faria, J., and Paiva, A. (2010). Test Coverage Analysis of UML State Machines. In Proceedings of the Third International Conference on Software Testing, Verification, and Validation Workshops (ICSTW), 2010, pages 284 –289. 3, 3.1.3 226
References Flake, S. and M¨ uller, W. (2003). Formal semantics of static and temporal stateoriented OCL constraints. Software and System Modeling, 2(3):164–186. 2.2.2 Fowler, M. (2004). UML Distilled: A Brief Guide to the Standard Object Modeling Language. Addison-Wesley object technology series. Addison-Wesley. 3.1.1, 3.1.2, 5, 5.2 Fraikin, F. and Leonhardt, T. (2002). SeDiTeC-Testing Based on Sequence Diagrams. In Proceedings of the 17th IEEE International Conference on Automated Software Engineering, ASE 2002, pages 261–266. 2.2.1 Gagnon, P., Mokhati, F., and Badri, M. (2008). Applying model checking to concurrent uml models. Journal of Object Technology, 7(1):59–84. 2.3.3 Gnesi, S., Latella, D., and Massink, M. (2004). Formal Test-case Generation for UML Statecharts. In Proceedings of the Ninth IEEE International Conference onEngineering Complex Computer Systems, 2004, pages 75 – 84. 2.2.2 Godefroid, P. (1997). Model Checking for Programming Languages using VeriSoft. In Proceedings of the 24th ACM SIGPLAN-SIGACT symposium on Principles of programming languages, POPL ’97, pages 174–186. ACM. 1, 2.1.1 Goldberg, D. E. (1989). Genetic Algorithms in Search, Optimization and Machine Learning. Pearson Education, Inc., 1st edition. 7.1, 7.1 Goodenough, J. B. and Gerhart, S. L. (1975). Toward a theory of test data selection. IEEE Transactions on Software Engineering, SE-1(2):156–173. 3.1.3 Hammer, C., Dolby, J., Vaziri, M., and Tip, F. (2008). Dynamic Detection of AtomicSet-Serializability Violations. In Proceedings of the 30th international conference on Software engineering, ICSE ’08, pages 231–240, New York, NY, USA. ACM. 1, 4 Hanrahan, G. (2011). Artificial Neural Networks in Biological and Environmental Analysis. Analytical Chemistry. CRC Press. 7.1 Harman, M. (2007). The Current State and Future of Search Based Software Engineering. In 2007 Future of Software Engineering, FOSE ’07, pages 342–357, Washington, DC, USA. IEEE Computer Society. 2.3.2 227
References Harrold, M. J. (2000). Testing: a roadmap. In Proceedings of the Conference on The Future of Software Engineering, ICSE ’00, pages 61–72, New York, NY, USA. ACM. 3 Hartmann, J., Vieira, M., Foster, H., and Ruder, A. (2005). A UML-based approach to system testing. Innovations in Systems and Software Engineering, 1:12–24. 1 Havelund, K. and Pressburger, T. (2000). Model checking JAVA programs using JAVA PathFinder. International Journal on Software Tools for Technology Transfer, 2:366–381. 1, 2.1.1, 2.1.3 Hwang, G.-H., Tai, K.-C., and Huang, T.-L. (1994). Reachability testing: an approach to testing concurrent software. In Proceedings of the First Asia-Pacific Software Engineering Conference, 1994, pages 246 –255. 2.1.2 Ipate, F. and Lefticaru, R. (2008). Genetic Model based Testing: a Framework and a Case Study. In Romanian Journal of Information Science and Technology, volume 11, pages 209–227. 7.1, 7.1 Jacobson, I. (1992). Object-oriented software engineering: a use case driven approach. ACM Press Series. ACM Press. 5.1 JGAP. Java Genetic Algorithm Package. http://jgap.sourceforge.net/. 7.4 Jones, T. (1995). Evolutionary Algorithms, Fitness Landscapes and Search. PhD thesis, University of New Mexico. 7.1 Jorgensen, P. C. (2008). Software Testing: A Craftsman’s Approach, Third Edition. AUERBACH, 3 edition. 3.1.2 Joshi, P., Naik, M., Park, C.-S., and Sen, K. (2009a). CalFuzzer: An Extensible Active Testing Framework for Concurrent Programs. In Bouajjani, A. and Maler, O., editors, Computer Aided Verification, volume 5643 of Lecture Notes in Computer Science, pages 675–681. Springer Berlin Heidelberg. 2.1.3 Joshi, P., Park, C.-S., Sen, K., and Naik, M. (2009b). A Randomized Dynamic Program Analysis Technique for Detecting Real Deadlocks. In Proceedings of the 2009 ACM SIGPLAN conference on Programming language design and implementation, PLDI ’09, pages 110–120, New York, NY, USA. ACM. 2.1.3 228
References Kansomkeat, S. and Rivepiboon, W. (2003). Automated-Generating Test Case Using UML Statechart Diagrams. In Proceedings of the Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists on Enablement through technology, SAICSIT ’03, pages 296–300. 2.2 Kansomkeat, S., Thiket, P., and Offutt, J. (2010). Generating Test Cases from UML Activity Diagrams using the Condition-Classification Tree Method. In Proceedings of the 2nd International Conference on Software Technology and Engineering (ICSTE), 2010, pages 62 –66. 2.1, 2.2.3 Khandai, M., Acharya, A., and Mohapatra, D. (2011). A Novel Approach of Test Case Generation for Concurrent Systems Using UML Sequence Diagram. In Proceedings of the 3rd International Conference on Electronics Computer Technology (ICECT), 2011, volume 1, pages 157–161. (document), 1.1, 2.3.1, 8.1, 8.3, 8.3.3, 8.5, 8.3.4, 8.3.4 Kim, H., Cho, D., and Moon, S. (2011). Maximizing synchronization coverage via controlling thread schedule. In Proceedings of the IEEE Consumer Communications and Networking Conference (CCNC), 2011, pages 1186 –1191. 3.1.3, 3.3.2.2 Kim, H., Kang, S., Baik, J., and Ko, I. (2007). Test Cases Generation from UML Activity Diagrams. In Proceedings of the Eighth ACIS International Conference on Software Engineering, Artificial Intelligence, Networking, and Parallel/Distributed Computing, pages 556–561. 6, 7, 8.3.4 Kim, S.-K., Wildman, L., and Duke, R. (2005). A UML Approach to the Generation of Test Sequences for Java-Based Concurrent Systems. In Proceedings of the Australian Conference on Software Engineering, ASWEC ’05, pages 100–109. 2.3.3 Kim, Y., Hong, H., Bae, D., and Cha, S. (1999). Test Cases Generation from UML State Diagrams. IEE Proceedings -Software, 146(4):187 –192. 1 Koppol, P. V., Carver, R. H., and Tai, K.-C. (2002). Incremental integration testing of concurrent programs. IEEE Trans. Software Eng., 28(6):607–623. 2.1.2 Kosmatov, N., Legeard, B., Peureux, F., and Utting, M. (2004). Boundary Coverage Criteria for Test Generation from Formal Models. In Proceedings of the 15th International Symposium on Software Reliability Engineering, 2004. ISSRE 2004, pages 139 – 150. 3.1.3 229
References Kumar, R. and Banerjee, N. (2011). Multiobjective network topology design. Appl. Soft Comput., 11(8):5120–5128. 7.3 Kumar, R. and Rockett, P. (1998). Multiobjective genetic algorithm partitioning for hierarchical learning of high-dimensional pattern spaces: a learning-followsdecomposition strategy. IEEE Transactions on Neural Networks, 9(5):822–830. 7.3 Kundu, D. and Samanta, D. (2009). A Novel Approach to Generate Test Cases from UML Activity Diagrams. Journal of Object Technology, 8(3):65–83. 1.1, 2.2, 2.2.3, 2.3.1, 3.2.2.2, 4.3, 5.2.2, 6.1, 6.3, 7, 7.4 Kundu, S., Ganai, M. K., and Wang, C. (2010). Contessa: Concurrency Testing Augmented with Symbolic Analysis. In CAV, LNCS, pages 127–131. Springer. 6 Lamport, L. (1978). Time, clocks, and the ordering of events in a distributed system. Communications of the ACM, 21(7):558–565. 6.2 Latella, D. and Massink, M. (2001). A formal testig framework for UML statechart diagrams behaviours: from theory to automatic verification. In Proceedings of the Sixth IEEE International Symposium on High Assurance Systems Engineering, 2001, pages 11 –22. 2.2.2 Lefticaru, R. and Ipate, F. (2008). Functional Search-based Testing from State Machines. In Proceedings of the First International Conference on Software Testing, Verfication and Validation (ICST 2008), pages 525 –528. 2.3, 2.2.4 Lei, B., Wang, L., and Li, X. (2008). UML Activity Diagram Based Testing of Java Concurrent Programs for Data Race and Inconsistency. In Proceedings of the International Conference on Software Testing, Verification, and Validation, pages 200–209. IEEE Computer Society. (document), 2.3.1, 2.1, 4.1, 4.2, 4.2.1, 4.3, 4.3, 4.3 Lei, Y. and Carver, R. (2005). A new algorithm for reachability testing of concurrent programs. In Proceedings of the 16th IEEE International Symposium on Software Reliability Engineering, 2005. ISSRE 2005, pages 10 pp. –355. 2.1.2 Lei, Y. and Carver, R. (2006). Reachability testing of concurrent programs. IEEE Transactions on Software Engineering, 32(6):382 –403. 2.1.2, 4 230
References Lei, Y. and Lin, N. (2008). Semiautomatic Test Case Generation Based on Sequence Diagrams. International Computer Symposium, ICS2008, pages 349–355. 2.2, 8.1 Li, B., Li, Z., Qing, L., and Chen, Y. (2007). Test Case Automate Generation from UML Sequence Diagram and OCL Expression. In Proceedings of the International Conference on Computational Intelligence and Security, 2007, pages 1048 –1052. 2.1, 2.2.3 Li, H. and Lam, C. (2005a). An Ant Colony Optimization Approach to Test Sequence Generation for State-based Software Testing. In Fifth International Conference on Quality Software, 2005. (QSIC 2005), pages 255 – 262. 2.3, 2.2.4, 7 Li, H. and Lam, C. (2005b). Using Anti-Ant-like Agents to Generate Test Threads from the UML Diagrams. In Testing of Communicating Systems, volume 3502 of Lecture Notes in Computer Science, pages 69–80. Springer Berlin / Heidelberg. 2.3, 2.2.4 Linzhang, W., Jiesong, Y., Xiaofeng, Y., Jun, H., Xuandong, L., and Guoliang, Z. (2004). Generating Test Cases from UML Activity Diagram based on Gray-Box Method. In Proceedings of the 11th Asia-Pacific Software Engineering Conference, pages 284–291. 2.2.1, 3.1.3, 3.2.1, 3.3, 3.4, 3.2.2 Lu, S., Jiang, W., and Zhou, Y. (2007). A study of interleaving coverage criteria. In Proceedings of the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering, pages 533–536. ACM. 1.3, 3.1.3, 3.2.2.2, 3.3.2.1, 3.3.2.3 Lu, S., Park, S., Seo, E., and Zhou, Y. (2008). Learning from Mistakes: A Comprehensive Study on Real World Concurrency Bug Characteristics. SIGOPS Oper. Syst. Rev., 42(2):329–339. 1, 4 Lu, S., Park, S., and Zhou, Y. (2012). Finding Atomicity-Violation Bugs through Unserializable Interleaving Testing. IEEE Transactions on Software Engineering, 38(4):844–860. 1, 6 Lund, M. S. and Stølen, K. (2006). Deriving Tests from UML 2.0 Sequence Diagrams with neg and assert. In Proceedings of the 2006 International Workshop on Automation of Software Test, AST ’06, pages 22–28. ACM. 2.2.2 231
References Malik, Q., Truscan, D., and Lilius, J. (2010). Using UML Models and Formal Verification in Model-Based Testing. In Proceedings of the 17th IEEE International Conference and Workshops on Engineering of Computer Based Systems (ECBS), 2010, pages 50 –56. 2.2.2 Manna, Z. and Pnueli, A. (1992). The temporal logic of reactive and concurrent systems. Springer-Verlag New York, Inc., New York, NY, USA. 2.1.1 Massink, M., Latella, D., and Gnesi, S. (2006). On testing UML statecharts. Journal of Logic and Algebraic Programming, 69(1-2):1–74. 2.2.2 McCabe, T. (1976). A complexity measure. IEEE Transactions on Software Engineering, SE-2(4):308 – 320. 3.2.1 McQuillan, J. A. and Power, J. F. (2005). A Survey of UML-Based Coverage Criteria for Software Testing. Technical report, Department of Computer Science National University of Ireland, Maynooth, Ireland. 3.1.3 Micskei, Z. and Waeselynck, H. (2011). The many meanings of UML 2 Sequence Diagrams: a survey. Journal of Software and Systems Modeling, 10:489–514. 3.1, 3.1.1 Mingsong, C., Xiaokang, Q., and Xuandong, L. (2006). Automatic Test Case Generation for UML Activity Diagrams. In Proceedings of the 2006 International Workshop on Automation of Software Test, AST ’06, pages 2–8. ACM. 3.1.3, 3.2.1, 3.5, 3.6, 3.2.2.2, 5.2.2, 8.3.4 Mouchawrab, S., Briand, L., Labiche, Y., and Di Penta, M. (2011). Assessing, Comparing, and Combining State Machine-Based Testing and Structural Testing: A Series of Experiments. IEEE Transactions on Software Engineering, 37(2):161 – 187. 2.4 Musuvathi, M. and Qadeer, S. (2007). Iterative Context Bounding for Systematic Testing of Multithreaded Programs. SIGPLAN Not., 42(6):446–455. 2.1, 2.1.1 Musuvathi, M. and Qadeer, S. (2008). Fair stateless model checking. SIGPLAN Not., 43(6):362–371. 2.1, 2.1.1 Musuvathi, M., Qadeer, S., Ball, T., Basler, G., Nainar, P. A., and Neamtiu, I. (2008). Finding and reproducing heisenbugs in concurrent programs. In Proceedings 232
References of the 8th USENIX conference on Operating systems design and implementation, OSDI’08, pages 267–280, Berkeley, CA, USA. USENIX Association. 4 Naik, M., Park, C.-S., Sen, K., and Gay, D. (2009). Effective static deadlock detection. In Proceedings of the 31st International Conference on Software Engineering, ICSE ’09, pages 386–396, Washington, DC, USA. IEEE Computer Society. 1 Nayak, A. and Samanta, D. (2010). Automatic Test Data Synthesis using UML Sequence Diagrams. Journal of Object Technology, 9(2):115–144. (document), 1.1, 1.2, 2.2, 2.2.3, 2.3.1, 6.1, 8.1, 8.3, 8.3.3, 8.5, 8.3.4, 8.3.4 Nayak, A. and Samanta, D. (2011). Synthesis of test scenarios using UML activity diagrams. Software and System Modeling, 10(1):63–89. 2.2, 2.2.3 Offutt, A. J. and Untch, R. H. (2001). Mutation testing for the new century. chapter Mutation 2000: uniting the orthogonal, pages 34–44. Kluwer Academic Publishers, Norwell, MA, USA. 8.3.4 OMG (2011). UML superstructure v2.4.1. http://www.omg.org/spec/UML/2.4/Superstructure. 3.1.1, 8, 8.1 Park, C.-S. and Sen, K. (2008). Randomized Active Atomicity Violation Detection in Concurrent Programs. In Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering, SIGSOFT ’08/FSE-16, pages 135–145, New York, NY, USA. ACM. 2.1.3 Pilone, D. and Pitman, N. (2005). UML 2.0 in a Nutshell (In a Nutshell (O’Reilly)). O’Reilly Media, Inc. 3.1.2, 8.3 Pilskalns, O., Andrews, A., Ghosh, S., and France, R. (2003). Rigorous Testing by Merging Structural and Behavioral UML Representations. In Proceedings of the 6th International Conference on the Unified Modeling Language, pages 234–248. 2.2, 2.2.3, 3.3.1, 8.3.3 Reuys, A., Reis, S., Kamsties, E., and Pohl, K. (2003). Derivation of Domain Test Scenarios from Activity Diagrams. In Proceedings of the International Workshop on Product Line Engineering the Early Steps: Planning, Modeling, Managing (PLEES03). 1, 3.1.3 233
References Rountev, A., Kagan, S., and Sawin, J. (2005). Coverage Criteria for Testing of Object Interactions in Sequence Diagrams. In Fundamental Approaches to Software Engineering, volume 3442 of Lecture Notes in Computer Science, pages 289–304. Springer. 3, 3.1.3, 3.3.1, 3.3.2 Safe, M., Carballido, J., Ponzoni, I., and Brignole, N. (2004). On Stopping Criteria for Genetic Algorithms. In Advances in Artificial Intelligence SBIA 2004, volume 3171 of Lecture Notes in Computer Science, pages 405–413. Springer Berlin Heidelberg. 7.1 Saha, S., Kumar, R., and Baboo, G. (2013). Characterization of graph properties for improved Pareto fronts using heuristics and EA for bi-objective graph coloring problem. Appl. Soft Comput., 13(5):2812–2822. 7.3 Sarma, M., Kundu, D., and Mall, R. (2007). Automatic Test Case Generation from UML Sequence Diagram. In Proceedings of the International Conference on Advanced Computing and Communications, 2007. ADCOM 2007, pages 60 –67. 2.2 Savage, S., Burrows, M., Nelson, G., Sobalvarro, P., and Anderson, T. (1997). Eraser: a dynamic data race detector for multithreaded programs. ACM Trans. Comput. Syst., 15:391–411. 1, 4.2 Seifert, D., Helke, S., and Santen, T. (2003). Test Case Generation for UML Statecharts. In Perspectives of System Informatics, volume 2890 of Lecture Notes in Computer Science, pages 93–109. Springer Berlin / Heidelberg. 2.2.2 Sen, K. (2008). Race directed random testing of concurrent programs. SIGPLAN Not., 43(6):11–21. 1, 2.1, 2.1.3 Seo, H.-S., Chung, I. S., and Kwon, Y. R. (2006). Generating Test Sequences from Statecharts for Concurrent Program Testing. IEICE - Trans. Inf. Syst., E89D:1459–1469. 2.3.3 Shousha, M., Briand, L., and Labiche, Y. (2008). A UML/SPT Model Analysis Methodology for Concurrent Systems Based on Genetic Algorithms. In Proceedings of the 11th International Conference on Model Driven Engineering Languages and Systems, MoDELS ’08, pages 475–489. 2.3.2, 2.4 234
References Shousha, M., Briand, L., and Labiche, Y. (2011). UML/MARTE Model Analysis Method for Uncovering Scenarios Leading to Starvation and Deadlocks in Concurrent Systems. IEEE Transactions on Software Engineering. 7 Shousha, M., Briand, L., and Labiche, Y. (2012). A UML/MARTE Model Analysis Method for Uncovering Scenarios Leading to Starvation and Deadlocks in Concurrent Systems. IEEE Transactions on Software Engineering, 38(2):354–374. 1, 2.3.2, 2.4, 8.3.4 Shousha, M., Briand, L. C., and Labiche, Y. (2009). A UML/MARTE Model Analysis Method for Detection of Data Races in Concurrent Systems. In Proceedings of the 12th International Conference on Model Driven Engineering Languages and Systems, MODELS ’09, pages 47–61, Berlin, Heidelberg. Springer-Verlag. 2.3.2, 2.4, 8 Sofokleous, A. A. and Andreou, A. S. (2008). Automatic, evolutionary test data generation for dynamic software testing. Journal of Systems and Software, 81(11):1883– 1898. 2.4 Souza, S. R. S., Brito, M. A. S., Silva, R. A., Souza, P. S. L., and Zaluska, E. (2011). Research in Concurrent Software Testing: A Systematic Review. In Proceedings of the Workshop on Parallel and Distributed Systems: Testing, Analysis, and Debugging, PADTAD ’11, pages 1–5. 1, 2.1.2, 4 Sthamer, H. (1995). The Automatic Generation of Software Test Data Using Genetic Algorithms. PhD thesis, University of Glamorgan. 7.1 Stoller, S. D. (2002). Testing concurrent java programs using randomized scheduling. Electronic Notes in Theoretical Computer Science, 70(4):142–157. 1, 2.1, 2.1.3 Sun, C. (2008). A Transformation-Based Approach to Generating Scenario-Oriented Test Cases from UML Activity Diagrams for Concurrent Applications. In Proceedings of the 32nd Annual IEEE International Conference of Computer Software and Applications, COMPSAC ’08, pages 160 –167. 2.3.1 Sun, C., Zhang, B., and Li, J. (2009). TSGen: A UML Activity Diagram-Based Test Scenario Generation Tool. In Proceedings of the 2009 International Conference on Computational Science and Engineering - Volume 02, CSE ’09, pages 853–858. 6, 7 235
References S¨ uß, M. and Leopold, C. (2008). Common mistakes in openmp and how to avoid them: A collection of best practices. In Proceedings of the 2005 and 2006 International Conference on OpenMP Shared Memory Parallel Programming, IWOMP’05/IWOMP’06, pages 312–323, Berlin, Heidelberg. Springer-Verlag. 1, 6 Tretmans, J. (2008). Model Based Testing with Labelled Transition Systems. In Formal Methods and Testing, volume 4949 of Lecture Notes in Computer Science, pages 1–38. Springer Berlin / Heidelberg. 10.1007/978-3-540-78917-81. 3.1.3 Utting, M. and Legeard, B. (2006). Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. 3.1.3 Vaziri, M., Tip, F., and Dolby, J. (2006). Associating synchronization constraints with data in an object-oriented language. SIGPLAN Notes., 41(1):334–345. 1, 3.3.2.2, 8 Weißleder, S. (2010). Test Models and Coverage Criteria for Automatic Model-Based Test Generation with UML State Machines. Doctoral thesis, Humboldt-University Berlin, Germany. 3, 3.1.3 Xu, D., Li, H., and Lam, C. P. (2005). Using Adaptive Agents to Automatically Generate Test Scenarios from the UML Activity Diagrams. In Proceedings of the 12th Asia-Pacific Software Engineering Conference, pages 385–392. IEEE Computer Society. 2.3, 2.2.4, 5, 5.2, 7, 7.1, 7.4 Yamada, Y., Iwasaki, H., and Ugawa, T. (2011). SAW: Java Synchronization Selection from Lock or Software Transactional Memory. In Proceedings of the IEEE 17th International Conference on Parallel and Distributed Systems (ICPADS), pages 104–111. 1 Yang, C.-S. D., Souter, A. L., and Pollock, L. L. (1998). All-du-path coverage for parallel programs. SIGSOFT Softw. Eng. Notes, 23(2):153–162. 3.1.3 Yang, L. and Stacey, D. A. (2001). Solving the traveling salesman problem using the enhanced genetic algorithm. In Advances in Artificial Intelligence, pages 307–316. Springer. 7.1 Zhou, H., Huang, Z., and Zhu, Y. (2008). Polymorphism Sequence Diagrams Test Data Automatic Generation Based on OCL. In Proceedings of the 9th International 236
References Conference for Young Computer Scientists, 2008, ICYCS 2008, pages 1235 –1240. 2.2 Zhu, H., Hall, P. A. V., and May, J. H. R. (1997). Software unit test coverage and adequacy. ACM Comput. Surv., 29(4):366–427. 3.2.1
237
