An Automated Approach to System Testing based on

An Automated Approach to System Testing based on Scenarios and Operations Contracts Najla Raza, Aamer Nadeem, Center for Software Dependability, Mohammad Ali Jinnah University, Islamabad, Pakistan [email protected], [email protected] Abstract Scenarios are an important mechanism for requirements specification and can be used to generate test paths at the requirements level. UML 2.0 provides a rich set of models for specifying scenarios, however, little work has focused on using UML 2.0 models for system testing. We propose a novel testing approach that uses UML 2.0 models for automated generation of test paths. This is done by enhancing the information with contracts and then formalizing it into a transition system. Test paths are then generated by path traversal of the transition system. The proposed approach is fully automatable and supported by a prototype tool, thus providing a platform for reducing the gap between high level scenarios and system testing.

Keywords Scenarios, Testing, UML, Contracts.

1. Introduction Scenarios have been widely used in elicitation and specification of requirements [14][15][16]. The idea of using scenarios in system level testing is not new [4][8][12][9], however lack of automated approaches and tool support is a major hurdle in employing these approaches for practical purpose. Formal methods have been used for automated testing [6][7], but, the cost of use of formal methods is high and is only justified for safety critical systems. The latest version of UML [1] provides a rich set of models to represent scenarios. One diagram presented by UML 2.0 is Interaction Overview Diagram (IOD) which is a specialized form of activity diagram used to show control flow. The nodes in an IOD are either interaction diagrams (Sequence Diagrams, Collaboration Diagrams, Timing

Muhammad Zohaib Z. Iqbal Department of Computer Science International Islamic University, Islamabad, Pakistan [email protected]

Diagrams, etc.) or Interaction Occurrences that show an operation invocation. Scenarios corresponding to a use case are easily captured by an IOD. The idea of an IOD is similar to HMSCs (Hierarchical Message Sequence Charts) [19]. HMSCs provide a graphical structure to define the composition of Message Sequence Charts (MSC). MSCs can be defined as a non object-oriented variant for sequence diagrams. One difference between and IOD and an HMSC is that we can refer all interactions diagrams in an IOD. To our knowledge, there is no approach that makes use of an IOD for system testing, however, testing using HMSCs is not a new concept and has been attempted before in [20][21][22]. In this paper, we use the Interaction Overview Diagram, defined in UML 2.0, for automated generation of test-paths. Scenarios are represented through the control flow in the IOD. By using scenarios in system testing, we are able to start testing at a very early stage of software development. The IOD that we have used in the proposed approach does not require detailed design information. It can easily be generated at early phases of system analysis. The information gathered from the interaction overview diagram is enhanced by operation contracts and then formalized into a transition system that we call the Contracts Transition System. It should be noted that the Contracts Transition System (CTS) models all scenarios represented by the IOD. By traversing the paths in the transition system a set of test paths are generated for the application under test. As highlighted in [23], scenarios depict the most common system behavior and an exhaustive specification of possible traces (negative, positive and unspecified system behavior) might be missing. Through our transition system we are able to identify such traces that may occur in the system, but have not been explored yet. These traces are candidates for either proscribed

behavior or possible valid behavior for a software system. Therefore, such a specification may have two fold benefits. Firstly, it would help in elaboration of requirements by identifying possible system behavior that the user might have missed. Secondly, it would help to identify exceptional system behavior, thus leading to a possible platform for generation of negative test cases. Therefore, our contribution is to provide an automated approach for testing software system that not only helps in starting testing at early stages of software development, but also provides a mechanism to elaborate and refine the specifications. The rest of the paper is organized as follows: in section 2 we have discussed related research in the area of scenario based testing. Section 3 gives an overview of the proposed approach. In section 4 the generation of CTS and test paths is explained with the help of an example and results are evaluated. We conclude in section 4 and give a brief detail of future work.

2. Related work Many approaches that deal with scenario based testing have been reported in literature. Some of them also use contracts. The major contribution for testing based on use cases enhanced with contracts can be found in [5]. The idea of a transition system presented in [5] is similar to our proposed approach. However, the transition system in [5] is generated on system level and is based on the pre and post conditions of use cases. In [7] specifications are given in the form of formal specifications. An FSA is generated whose transitions represent the sub operations that are generated by partition analysis of operations. Two sets of constraints, the before and after state are identified for each sub-operation and the states of the FSA are identified by partition analysis of system states. Our approach differs by the fact that we are dealing with specifications given in terms of UML model and the transition system in our approach maps to the scenarios of the IOD. The transitions represent operations that are taken directly from the sequence diagrams and IOD references. Apart from safety critical systems, use of requirement based testing techniques based on formal methods [6] and [7] is not justifiable in software testing with respect to the effort required to build them. It should be noted here that in our approach operation contract is the only document written in OCL and this document is provided as input to the proposed approach. The proposed approach does not requires any further formalization. In [10], a detailed approach to create, analyze and validate scenarios is presented. They formalize

scenarios into state charts and test cases are generated by path traversals. Furthermore, a comprehensive notation for modeling scenario dependencies has been proposed, however the approach is manual and state chart generation process is declared to be a creative activity and is done solely by the experience of the tester. An approach for system testing using embedded meta-data is presented in [3], however the main focus of the approach is to track the impact of changing requirements on software testing. Other approaches dealing with scenario/use case based testing are [4], [8], [12] and [9]. [4] is focused towards distributed systems and [8], [9] and [12] require artifacts from design phase of software development whereas our approach only requires analysis phase artifact thus supporting early start of testing activities. Liu et al [24] present an approach for specification based inspection using pre and post conditions of operations to inspect the implementation of each functional scenario in the program code based on SOFL [25]. SOFL language however is not purely formal, but introduces a high level of formality for specification. The approach presented in [11] uses use case diagrams, activity diagrams, and sequence diagrams for generation of system level test cases. Use case dependencies are modeled by activity diagrams. Concrete test cases are obtained in the form of regular expressions written in OCL.

3. The Proposed Approach We propose a scenario based approach for test path generation. The scenarios expressed as UML Interaction Overview Diagram (IOD) [1] are taken as input. An IOD represents all scenarios (i.e., success scenario and all alternative scenarios) of a use case. A transition system, called CTS (Contracts Transition System) is built by enhancing the operations in the IOD with contracts. The contracts, written in OCL, specify the pre and post conditions for each operation. The states in the CTS are the contracts and the transitions represent the operations. Test paths can be generated from the CTS by path traversals from the initial node to a final node. The test set can be improved by selecting an appropriate coverage criterion, such as state coverage, transition coverage, or transition pair coverage. Following is a brief outline of the test generation procedure: • Identify operations from the IOD. The operations are taken from the sequence

diagrams or other operations expressed as interaction occurrences in the IOD. • Identify states for the Contracts Transition System (CTS) using the contracts of operations identified above. • Prepare CTS matrix • Build a CTS for each use case using the mapping rules and CTS matrix • Generate test paths from the CTS by applying a coverage criterion In the following section, we give a detailed explanation of generation of transition system with the help of an example.

3.1 CTS Generation In order to elaborate the proposed approach we will use the GenerateRollNumberSlip use case of an online examination system example. On request of a student, the Registrar can generate roll number slip after verifying the dues status and attendance status of the student. Figure 1 gives the IOD for GenerateRollNumberSlip use case. The IOD directly maps the use case steps and can be built without any detailed design information. Our tool takes the XMI file for the IOD generated using Visual Paradigm Suite

for UML [2] and operation contracts as input. The list of operations is obtained automatically from the XMI File. These operations either appear as interaction use which represents the referred as O1 issueRollNumSlip(regNo) operation or in the sequence diagrams which are represented as inline interactions. The user then provides the operation contracts written in OCL for these operations. For identification of states, the tool parses the OCL contracts file. Figure 2 shows an excerpt from the OCL file for the operations identified from the IOD. A state is created against each precondition and each post condition of operations. The operations having no pre conditions do not require any specific conditions for their occurrence and therefore their pre conditions are taken to be true. For this, a separate state S1 is created which has the value “true” and this is taken as the initial state of the CTS. Therefore, for issueRollNumSlip(regNo) the pre condition is S1: true and postconditon is S2: (self.regNo = regNo, session = true). We need to have special rules to resolve the logical if-then-else conditions in the contracts. This is because if these conditions result in true then they will represent a different system state and their false value will represent a different system state. We also need to resolve the re-occurring statements within states.

O1

Figure 1. Interaction overview diagram for use case generate roll number slip

represent operations. The transition function τ is defined as S x O Æ S. Table 1. CTS matrix

Operations O1 O2 O3 O4 O5 Figure 2. Excerpt from OCL File In order to resolve these two issues we have proposed the following rules for definition of states for our transition system: 1. The logical if-then-else conditions are resolved by combining their testing condition with the result. In this case, two different sub states are created and the post condition will be the composition of the two sub states and any other applicable conditions. Following this rule, post condition for O3:verifyClearance(regNo) will compose of two states; S6 (self.feePaid = feeDue, Student.duesStatus = true, result = Student.duesStatus) and S7 (self.feePaid feeDue, Student.duesStatus = false, result = Student.duesStatus) 2. If statements in a contract of an operation appear in the contract of another operation (and they do not form part of the if-then-else conditions) then split the identical statements into sub states. The statement self.regNo = regNo occurs in the post condition of operation O1 and pre condition of operation O5:updateStudentRecord(regNo,rns) and does not form part of any logical if-then-else condition. Therefore, we will split the state S2 into two states; S2.1 (self.regNo = regNo, S2.2 (session = true), and pre condition of O5 will be given as the logical AND of S2.1 and S8 (self.attendanceVerified = true) The next step is to create the CTS matrix which defines the relationship between operations and different states. For example, as illustrated above, the pre condition of operation O1:issueRollNumSlip(regNo) is S1 and post condition is the logical AND of S2.1 and S2.2. For the sake of simplicity, “S2.1 AND S2.2” is represented by the composite state A. The complete CTS Matrix for our example is shown in Table 1. In CTS we only represent the composite states. The CTS can be expressed by the following 5-tuple function: CTS = (S, O, Si, Sf, τ,) Here S is the set of states, O represents the set of operations, Si is the initial node and Sf is the final node and τ is the transition function. The nodes in our transition system represent states and the transitions

Pre

Post

Composite States A B C D

S1 S2.1 AND S2.2 S1 S4 OR S5 S4 S6 OR S7 S6 S8 OR S9 S8 AND S10 S2.1 O6 S9 S8 For this 5-tuple function S = {S1, A, B, C, D, S10}, O = {O1, O2, O3, O4, O5, O6}, Si = S1, and Sf = the terminating node in Figure 3. We also need to consider the following points in generation of CTS. 1. The operations with pre condition equal to “true” do not require any specific conditions for their invocation and can be invoked from all other states in the CTS, e.g., operations O1 and O2. Their invocation is represented by adding one transition initiating from each state and terminating at the post condition states A and B for operations O1 and O2 respectively. 2. Conditional flows leading to terminating node can be identified from the IOD and are added in the CTS. In our example, the transition from states C, D and S10 lead to the terminating node. 3. Additional transitions are added for all conditional flows leading to alternative scenarios and their guard conditions are attached with them. 4. Iterations such as the iteration in the sequence diagram for verify attendance where each course is taken by the student in that semester is checked for 75% attendance. The transition system is annotated with an asterisk sign “*” for such iterations. 5. Join node is formed in the CTS if an operation has the AND of more than one states in its precondition, for example O5. The procedure given in [13] is followed to eliminate the join nodes from the CTS. The group of the join nodes is replaced by several paths in a sequence in the CTS and adding constraint to the last edge. However, the user needs To remove the join node the condition “A must be visited in the path” is attached with operation O5. In the CTS some new flows are also identified which were not present in the original scenario. These flows can help the user to further refine the requirements. Such scenarios can be potential candidates for unspecified behavior or unwanted behavior. In any case, the user would be able to explicitly mark such behavior thus filling out gaps and new possible test paths are identified. For example the transition AÆA is directly shown in the matrix, however it is deduced from point 1 explained above. Now a path starting from S1ÆAÆA would mean

that O1:issueRollNumSlip(regNo) is exercised twice. This initially seems a bit illogical, however, on close observation the user may identify that this scenario is likely in the event that after the instantiation of the first issueRollNumberSlip(regNo) operation, the user may hit cancel. After the occurrence of this event, a second issueRollNumberSlip(regNo) is likely. This would therefore result in refinement of requirements. Similarly, if a transition is identified as an invalid behavior then such a transition is removed from the CTS using the Transition Editor of our tool.

transition coverage provide the complete scenario coverage of IOD.

3.2. Test path generation from CTS Test paths are generated from the CTS by traversing paths in the CTS using the following coverage criteria: Transition coverage: A set of test paths satisfies the transition coverage criterion if and only if each transition in the CTS is covered by at least one test path. State coverage: A set of test paths satisfies the all states coverage criterion if and only if each state in the CTS is covered by at least one test path. Transition pair coverage [18]: A set of test paths satisfies the transition pair coverage criterion if and only if each possible pair of transition in the CTS is covered by at least one test path. The results show that the test paths generated from the state coverage criterion are covered by the transition coverage criterion. The transition pair coverage criterion subsumes both the transition and state coverage criteria. The number of test paths generated for state, transition and transition pair coverage are 1, 13 and 26 respectively. It is reflected that the transition pair coverage criterion is the most expensive in terms of number of test paths. The all transitions coverage criterion seems reasonable in terms of number of test paths but as concluded by Briand et al in [17] and Ali et al [26], the all transition coverage criterion is not suitable in all situations for fault detection. Briand et al further concludes that all transition pair guarantees maximum fault coverage but is expensive to be used in most practical cases, as is shown by our results as well. In terms of functionality coverage, the state coverage criterion ensures that all states are covered in the Contracts Transition System. These states represent the pre and post conditions of the operations. Therefore, covering all states ensures that all operations belonging to the success scenario are captured. The Transition Coverage requires that each transition must be covered through at least one test path. The conditions leading to alternative flows in the Interaction Overview Diagram are mapped on the appropriate transitions of the CTS, therefore, the Transition Coverage criteria guarantees coverage of all alternative flows in the IOD. Therefore, the test paths generated through our approach based on

Figure 3. Contracts Transition System

4. Conclusion and future work This paper presents an automated approach for scenario based generation of test paths. The focus is to automate the process of generation of test paths using the richer set of models given by the UML 2.0 specifications for object oriented software. Our approach helps in identifying undefined system behavior and negative system scenarios. The approach builds a Contracts Transition System by using information from the Interaction Overview Diagram and Contracts. The CTS is later used to generate test paths by applying suitable coverage criterion. The approach has been implemented using a prototype tool. The future work will focus on applying the proposed testing approach on larger case studies to evaluate its effectiveness. Also, an extension of the proposed approach could be to model the dependencies among different use cases at the system level.

References [1] UML 2.0 Profile. http://www.uml.org/ [2] http://www.visual-paradigm.com/product/. [3] M. Strembeck, and U. Zdun, “Scenario-based Component Testing Using Embedded Metadata,” in: Proc. of the Workshop on Testing of Component-based Systems (TECOS), Lecture Notes in Informatics (LNI), Erfurt, Germany, September, 2004, pp. 31-45.

[4] W. T. Tsai, A. Saimi, L. Yu, R. Paul, “Scenario-based Object-Oriented Test Frameworks,” Proc. of Third Intl. Conference on Quality Software (QSIC03), 2003, pp. 410-417. [5] C. Nebut, F. Fleurey, Y. L. Traon, J. M. Jezequel, “Automatic Test Generation: A Use Case Driven Approach,” IEEE Transactions on Software Engineering, vol. 32, issue 3, March, 2006, pp. 140-155. [6] L. Tahat, B. Vaysburg, B. Koreland, and A. Bader, “Requirement Based Automated Black-Box Test Generation,” 25th Annual Intl. Conference (COMPSAC), 2001, p. 489. [7] J. Dick and A. Faivre, “Automating the Generation and Sequencing of Test Cases from Modelbased Specifications,” Proc. International Symposium Formal Methods, Europe, 1993, pp. 268-284. [8] F. Basanieri, A. Bertolino, E. Marchetti, "The Cow_Suite Approach to Planning and Deriving Test Suites in UML Projects", Proc. Fifth International Conference on the Unified Modeling Language - the Language and its applications UML 2002, LNCS 2460, Dresden, Germany, September 30 - October 4, 2002, p. 383-397. [9] F. Basanieri, A. Bertolino, "A practical approach to UMLbased derivation of integration tests", Proc. Software Quality Week Europe QWE2000, 20-24 November, 2000, paper 3T. [10] J. Ryser and M. Glinz, “Using Dependency Charts to Improve Scenario-Based Testing,” Proc. 17th Int’l Conf. Testing Computer Software, June 2000. [11] L. Briand and Y. Labiche, “A UML-Based Approach to System Testing,” J. Software and Systems Modeling, 2002, pp. 10-42. [12] P. Frohlick, J. Link, “Automated Test Case Generation from Dynamic Models,” Proc. Of the 14th European Conference on Object Oriented Programming, 2000, pp. 472-492. [13] X. BAI, C. P. Lam, and H. Li, “An Approach to Generate the Thin-threads from the UML Diagrams,” Proc. Of the 28th annual Intl. Conference (COMPSAC’04), Hong Kong, September 28 – 30, 2004, pp. 546-552. [14] Leite et al, “Enhancing a requirements baseline with scenarios”, Proc. Of the 3rd IEEE International Symposium on Requirements Engineering, 1997, pp. 44-53. [15] A. G. Sutcliffe, N.A.M. Maiden, S. Minocha, and D. Manuel, “Supporting scenario-based requirements engineering”, IEEE Transactions on Software Engineering, Dec, 1998, pp. 1072-1088. [16] J. M. Carroll et al, “Requirements development in scenariobased design”, IEEE Transactions on Software Engineering, Dec, 1998, pp. 1156-1170. [17] L. C. Briand, Y. Labiche, Y. Wang, “Using Simulation to Empirically Investigate Test Coverage Criteria Based on

Statechart”, 26th International Conference Engineering (ICSE’04), pp. 86-95.

on

Software

[18] A. J. Offutt, Y. Xiong and S. Liu, “Criteria for Generating Specification-Based Tests,” Proc. 5th International Conference on Engineering of Complex Computer Systems (ICECCS), Las Vegas, NV, October, 1999, pp. 119-129. [19] E. Rudolph, J. Grabowski, and P. Graubmann, “Tutorial on Message Sequence Charts (MSC '96)”. In: Tutorials of the First Joint International Conference on Formal Description Techniques for Distributed Systems and Communication Protocols, and Protocol Specification, Testing, and Verification (FORTE/PSTV'96), Kaiserslautern, Germany, October 1996. [20] P. Graubmann, and E. Rudolph, “HyperMSCs and Sequence Diagrams for Use Case Modelling and Testing”, "UML" 2000 - The Unified Modeling Language. Advancing the Standard: Proc. Of Third International Conference, York, UK, October 2000, pp. 32-46. [21] S. Blaustein, V. Braberman, and F. Oliveto, “Scenariobased Validation and Verification for Real-Time Software: On Run Conformance and Coverage for MSC-Graphs”, 2nd International Workshop on Scenarios and State Machines: Models, Algorithms and Tools (SCESM’03), Portland, Oregon, USA, May 2003. [22] E. Rudolph, I. Schieferdecker, J. Grabowski: Development of a Message Sequence Chart/ UML Test Format. Proceedings of FBT-2000, Lübeck, Germany. Shaker-Verlag, 2000. [23] S. Uchitel, J. Kramer, and J. Magee, “Behaviour Model Elaboration using Partial Labelled Transition Systems”, fourth joint meeting of the European Software Engineering Conference and ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE), Helsinki, September 2003. [24] S. Liu, F. Nagoya, Y. Chen, M. Goya and J. McDermid, "An Automated Approach to Specification-Based Program inspection", In the 7th Intl. Conference on Formal Engineering Methods (ICFEM 2005), Manchester, UK, LNCS 3785 Springer-Verlag, 1-4 November 2005, pp. 421-434. [25] S. Liu, "Formal Engineering for Industrial Software Development using the SOFL Method", Springer-Verlag, March 2004, 428 pages, ISBN 3-540-20602-7. [26] S. Ali, L. C. Briand, M. J. Rehman, H. Asghar, M. Z. Z. Iqbal, A. Nadeem, “A State-based approach to Integration Testing based on UML Models”, Information and Software Technology, Vol. In Press, (2007), doi:10.1016/j.infsof.2006.11.002