Towards Traceability of Model-based Testing Artifacts Hadar Ziv
Debra J. Richardson
Donald Bren School of Information and Computer Sciences University of California, Irvine Irvine, CA 92697-3425 USA
Leila Naslavsky
Donald Bren School of Information and Computer Sciences University of California, Irvine Irvine, CA 92697-3425 USA
Donald Bren School of Information and Computer Sciences University of California, Irvine Irvine, CA 92697-3425 USA
[email protected]
[email protected]
[email protected]
ABSTRACT
widespread adoption is an outcome from the need to handle the increasing software systems’ complexity. Dissemination of the UML metamodel specification and the increasing availability of tools that support code generation from models are additional factors contributing to MDD adoption. The consequences are twofold: (1) code is no longer the major software artifact, (2) code is no longer the single source for automated test case selection. Model-based testing now complements code-based testing.
Practitioners regard software testing as the central means for ensuring that a system behaves as expected. Due to the recent widespread adoption of model-driven development (MDD), code is no longer the single source for selecting test cases. Testing against original expectations can be done with modelbased testing that adopts high-level models as the basis for test generation. In addition to test generation, challenges to modelbased testing include creation and maintenance of traceability information among test-related artifacts. Traceability is required to support activities such as result evaluation, regression testing and coverage analysis. MDD and model transformation solutions address the traceability problem by creating relationships among transformed artifacts throughout the transformation process. This paper proposes an approach that leverages model transformation traceability techniques to create fine-grained relationships among model-based testing artifacts. Relationships are created during the test generation process. Their fine granularity enables the support for result evaluation, coverage analysis and regression testing.
Traditional testing activities comprise: test case generation, teste execution, result evaluation, coverage analysis, and regression testing. Most model-based testing automated approaches often concentrate on the test generation and execution activities, while neglecting the others. As discussed in [16] , model-based testing requires the ability to relate the “abstract values of the specification to the concrete values of the implementation”. The role played by relationships among artifacts to support automation of testing activities had long been recognized [32, 16]. Relationships can be established with different purposes: relationships that map concepts in the model to concepts in the implementation support test scripts generation or abstract test scripts execution; relationships from models to test cases and from test cases to test results support coverage measurement and selective regression testing. Creation and maintenance of explicit relationships among test-related artifacts is therefore, a main challenge to the automated support of such activities.
Categories and Subject Descriptors D.2.5 [Testing and Debugging]: Testing tools (e.g., data generators, coverage testing)
General Terms
The artifacts at their end points define a relationship’s level of granularity. When relationships are used to support identification of impacted test cases (or impacted test models), their granularity influences the selectivity precision. The more fine-grained these endpoints are, the more precise the selection could be (e.g. code-based regression testing). There is however, a trade-off between the precision obtained and the cost to externalize and maintain such relationships [9]. In this paper, we are leveraging one solution that allows externalization of relationships with different levels of granularity [18]. This solution is adopted because it supports the investigation of such trade-offs.
Measurement, Documentation, Design.
Keywords Model-driven development, model-based testing, traceability.
1. INTRODUCTION In traditional software development processes, the source code was the only artifact to be used for testing purposes. Given the level of abstraction and formalism of source code, many solutions are currently available in the industry [6] and in the academia [38, 15] that automate some testing activities and reduce testers efforts. Code-based testing uses the implementation to derive test cases. Thus, it can be used to test the system, but it cannot be used to test the original expectations about the system. Adoption of high-level models as the basis for test derivation can test systems against original expectations. It also leverages the effort put into creating models by supporting testing.
Our approach uses model transformation traceability techniques [26] to create relationships among model-based testing artifacts. Relationships are created during the test generation process. Tool support is based on the Eclipse IDE [3] to manipulate models. We adapt a model-based control flow model [33] and use it to support test case data generation from sequence diagrams in UML 2.0 models. Additionally, we adapt a test hierarchy model [35] and use it to describe a hierarchy of test data partitions. A fourth (traceability) metamodel is used to
Model-driven development (MDD) adds a new level of abstraction to software development artifacts. Its recent Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. AMOST'07, July 9-12, 2007, London, UK Copyright 2007 ACM 978-1-59593-850-3/07/0007...$5.00
105
support creation and persistence of relationships among these models [17].
underlie our approach. We also describe the creation of finegrained relationships among models to support testing activities. This section demonstrates and motivates our approach with an example application of the infrastructure and workflow defined by our models and their relationships. We plan to implement this application in the near future. Figure 1 depicts views of the models related by our approach: a sequence diagram from an UML 2.0 model (uml), a model-based control flow graph (mbcfg), and a test generation hierarchy (tgh). Later, sections 4 and 5 describe the models and their creation in more detail. Figures 11, and 13 depict in detail the mbcfg, and the tgh models from figure 1, whereas figure 10 depicts another version of the uml model, before addition of message logSessionAccount.
The rest of this paper discusses the motivation for the approach (section 2); presents some background and related work on the subjects of model transformation, model-based test generation, and traceability applied to automated testing approaches (section 3); presents the overview of our approach (section 4); presents an example (section 5); concludes (section 6), and explains our future work.
2. MOTIVATION In this paper we describe several related models, metamodels, model transformations, and a test generation process that
Figure 1 – Views of related models. and semantics of the relationships available. There is a trade-off between the precision obtained and the cost to externalize and maintain such relationships. The infrastructure created with our approach will be used the to investigate this trade-off.
The proposed infrastructure underlying our approach enables automated support for tasks other than test generation (e.g. regression testing). The availability of fine-grained relationships can be used to determine the impact a change to a model have on a test suite. Depending on the relationships’ granularity, it is possible to conclude that a change to a message in a sequence diagram impacts a sub-set of test cases, while other test cases are not impacted. Therefore, re-run of fewer test cases is required to test if the change inserted new faults to the system. Different selectivity precision is achieved depending on the granularity
For instance, figure 1 shows three models with coarse-grained and fine-grained relationships (represented dashed two-way arrows) among them. Relationships represent one or more underlying traceability models. The dashed arrows show that message setCurrentAccount (in the uml model) is related to node setCurrentAccount (in the mbcfg model), which is related to the node N12 (in the tgh model). Suppose the uml model in
106
figure 1 originally did not have message logSessionAccount (see figure 10), so node N12’ in mbcfg and tgh models did not originally exist, nor did edges N12-N12’ and N12’-N13 in the mbcfg model. The addition of message logSessionAccount resulted in: the addition of node N12’ to mbcfg model; the addition of edges N12-N12’ and N12’-N13, and the removal of edge from N12-N13, all depicted in figure 1. This change, in turn, impacts some paths on the test generation hierarchy (Path5 and Path6). Therefore, it should impact test cases under those partitions. The other 32 test cases generated from the same sequence diagram (TGH Root startSession in the tgh model), should not be impacted. Acquiring this information is possible because the depicted relationships connect fine-grained entities (messages) to test case partitions, which creation was influenced by these entities.
fine-grained entities (e.g. messages) in the original model (uml model) is measured with the support of the explicit fine-grained relationships.
3. BACKGROUND AND RELATED WORK 3.1 Traceability on Testing Approaches Support for software traceability is achieved with managing relationships among software artifacts. Relationship management aims at supporting (automatically, semiautomatically or manually) creation, persistence, maintenance, and destruction of meaningful relationships across software artifacts [8, 34]. It also aims at describing and classifying those relationships [8, 28, 34]. Establishing and maintaining relationships among software artifacts is important because relationships can be used in a number of different software engineering activities such as software change impact analysis and software validation, verification and testing [34]. A commonality between model-based testing and traceability is the need to manage relationships among artifacts.
If fine-grained relationships were not made explicit and persistent, they would be embedded in the test generation algorithm. As a result, identification of a precise subset of test cases to be regression tested would be a more labor-intensive task. Additionally, given the same change, coarse-grained relationships will identify 37 test cases to be regression tested (all test cases under TGH Root startSession could be impacted).
Regarding traceability of testing artifacts, solutions available typically use a traceability matrix to support relationship management among requirements and test cases. One example of a semi-automatic solution [13] creates a traceability matrix from requirements to test cases during the test generation process. It expects formal models (written with B specification language) to be annotated with requirements identifiers. When the test cases are generated from the models, the identifiers are used to create the traceability matrix relating requirement identifiers to test cases identifiers. Our approach also proposes creation of relationships among testing artifacts during the test generation process. However, it does not relate requirements to test cases, but entities that compose the models to a test hierarchy, and it does not require previous model annotation. Moreover, the level of granularity of relationships in traceability matrixes is high and fixed, while the traceability approach we adopt allows this level of granularity to vary. This could be an important factor when the aim is supporting activities other than generation (e.g. model-based regression testing).
Another promising application for the infrastructure established with our approach is behavioral result evaluation of test cases. In fact, one advantage of model-based testing is using the models as test oracles. Test oracles consist of oracle procedure and oracle information. The procedure uses the information to verify that the execution behavior complies with the expected [31]. An oracle can therefore check that a final result obtained was the expected. It can also compare that the steps taken to reach that result are the same steps described in the model. This is possible because the test cases belong to a hierarchy that defines a path as its parent, so the paths are used as behavioral oracles. If we assume mapping relationships from concepts in the model to concepts in the implementation is available with the uml model, the test cases in the test generation hierarchy can be executed against the implementation. Thus, if results obtained match expected results, but obtained sequence of message does not match the path, the implementation does not comply with the model. For instance, assuming the expected result for executing test cases under partition Path6 is: session is created, and account with number accNum is set (figure 1). Suppose the execution of one test case against some implementation results in the following sequence of message calls: insertCard, sessionCreateMessage, getAccount, setCurrentAccount. Then, the result obtained matches the expected result (session is created with sessionCreateMessage, account is set setCurrentAccount). However, evaluating Path6, and using the fine-grained relationships from its nodes to their related messages (in the uml model), we identify that a call to validatePin (N9 in the tgh model) is missing.
Concerning relationships, automated model-based testing approaches exploit implicit and/or explicit relationships. Implicit relationships are embedded in the tool’s algorithms and models, whereas explicit relationships are either automatically created and made explicit by the tool, or created by the users. Some approaches use implicit relationships to support test generation, execution and evaluation [21, 30, 37]; while others use implicit relationships to support regression testing [14]. Additional approaches use explicit relationships to support test generation [10], test execution and evaluation [23, 24], or coverage analysis [24]. Relationships among UML models used by COWtest pluS UIT Environment (COW_SUITE) [10] are made explicit when the model is loaded into the tool. They are created as edges in hierarchical trees that represent the UML model. The Automated Generation and Execution of test suites for DIstributed component-based Software (AGEDIS) tool [24] uses explicit relationships created by the user to execute and evaluate the test scripts. In this case, relationships map abstract stimuli to method invocations, and abstract observations to value checking.
Finally, since models are used as basis for test generation, model-based coverage measurement is another activity that should be supported by model-based testing approaches. It is also another promising application for the infrastructure proposed in this paper. In figure 1, test cases under the TGH Root startSession were created to trigger different paths from the startSession sequence diagram (uml model, figure 1). When these test cases are executed, coverage obtained with regards to
107
AGEDIS also externalizes relationships between abstract test suites and test trace results during test execution. It supports visualization of test traces and the abstract test suite that generated them, which supports manual coverage analysis.
expected results. This specification is written in Java, which is the language used by the JUnit framework. The approach assumes existence of direct mapping relationships between concepts (e.g. lifelines and messages) in the sequence diagram and concepts from the implementation (e.g. objects and methods).
Analogously to AGEDIS, the Abstract state machine Language (AsmL) [23] also uses explicit relationships created by the user to execute and evaluate the abstract test scripts. In this case, they support the parallel execution of the model and its implementation. Uses of relationships in AsmL are threefold (1) map methods in the models to methods in the implementation according to their signatures, (2) relate global constants in the model and implementation, and (3) compare states of the model and states of the implementation.
COWtest pluS UIT Environment (COW_SUITE) [10] is a tool that provides systematic and automated support for test planning and test case generation, based on UML diagrams. It is largely based on category partition. It supports test case plan generation directly from UML diagrams (use case, sequence, and class diagrams). It’s test case generation process analysis sequence diagram and interacts with the user to gather choices values, and constraints for the case of contradictory choice combinations.
Similarly to these approaches, we are proposing the use of explicit relationships among testing artifacts. Differently from them, we are externalizing fine-granular relationships from model to a test case generation hierarchy, which, as discussed in section 2 enables the support for more precise regression testing, model-based coverage analysis and result evaluation.
In [19], a graph that integrates class and sequence diagrams information is used as basis for test case generation. This approach adopts a symbolic execution approach to derive test input constraints from the presented graph and solves these constraints with the Alloy constraint solver. While SCENTOR and SeDiTeC are concerned with generating test scripts, COW_SUITE is concerned with managing the test cases and prioritizing them. Emphasis is not given to creating test data input. Regarding test generation, our approach is close to [19], since it also proposes the use of constraint solvers to discover test data input satisfying a composition of constraints obtained from a UML model.
3.2 Model Driven Development Model-driven development (MDD) describes software development approaches that consider software models as the primary artifact. Models conform to metamodels. In this context, metamodels state how instance models conforming to the metamodel should be built. The software engineering field recently witnessed a large increase in MDD adoption. MDD tools support code generation from models, which is a kind of model transformation that converts a model at a higher level of abstraction into source code. Indeed, model transformation is a core activity in MDD. The OMG proposed a specification (Query/Views/Transformations – QVT) that standardized the ways source models are transformed into target models [7]. There are many implementations partially or totally compliant with this standard (e.g. M2M, Borland Together, ATL, Tefkat).
4. APPROACH OVERVIEW We build upon the techniques described in the background section to achieve traceability of model-based testing artifacts. Figures 2 and 3 show an overview of the approach. It ultimately generates a test case hierarchy [35] from sequence and class diagrams belonging to a UML 2.0 model. First, step 1 in figure 2 creates a UML 2.0 model. Then, step2 transforms each sequence diagram into a model-based control flow graph model. During this transformation, the traceability information is made explicit into a separate model (step 2).
Our approach adopts the Atlas Transformation Language (ATL) implementation, which is a model transformation language compliant to the QVT standard [27]. Its implementation is available as an Eclipse plug-in [1]. To establish relationships among models, our approach adopts the traceability metamodels available with the Atlas Model Weaver (AMW). Its implementation builds upon ATL and is also available as an Eclipse plug-in [2].
3.3 Model-based Testing Approaches UML sequence diagrams are scenarios at the design level of abstraction [29], in this section we discuss some related scenario-based testing approaches. Sequence Diagram Test Center (SeDiTeC) [21] supports the execution of sequence diagrams, so it uses the sequence diagrams to test the integration among objects. The tester is in charge of describing concrete parameters and expected results of the sequence diagrams. Result evaluation is based on source code instrumentation to log obtained method calls, concrete parameters and obtained results. SCENTOR [37] supports generation scripts through extending JUnit framework classes. Similarly to SeDiTeC, the tester is in charge of specifying the methods’ concrete parameter values and
Figure 2 - Approach overview: steps 1 and 2.
108
Then, test generation takes place (step 3 in figure 3); it consists of using both the UML 2.0 model and the model-based control flow graphs to generate a test case hierarchy. Again, during the test data generation, the traceability information is made explicit and persistent (step 3).
4.1.2 Model-Based Control Flow Graph Metamodel Sequence diagrams model the expected flow of messages between the objects of a system, given a particular input. As a result, it is possible to describe different message sequence calls in a single sequence diagram. A well known strategy for test case generation from sequence diagrams [12] consists of deriving a control flow model from the sequence diagram, developing a set of paths that achieves particular coverage criteria based on the control flow model (e.g. branch coverage, or path coverage), and identifying inputs that cause the paths in the set to be taken. Thus, each path derived from the sequence diagram partitions the valid input set into test case data inputs. Control flow graphs derived from models are named modelbased control flow models. In the literature, there are some proposed models [22, 33]. Garousi et al. [22] discusses a comprehensive set of such models. They propose an extension of UML 2.0 activity diagram metamodel, named Concurrent Control Flow Graph (CCFG), to support control flow analysis of sequence diagrams. An important point of this work is the support for concurrency. The main objective of our approach is to explore the relationships among testing artifacts. The model-based control flow graph is used to support path derivation from the sequence diagrams, and to establish the relationships from the sequence diagrams to the test generation hierarchy. We are not currently addressing test generation from concurrent models. Therefore, we are using a simpler model-based control flow metamodel, which is the inter-procedural restricted control flow graph (IRCFG). It “represents in a compact manner the set of message sequences in a sequence diagram” [33]. That work, however, does not define a metamodel for the control flow graph and does not support the use of loop definitions. We created an EMF metamodel that is a variation of IRCFG, depicted in figure 4. It supports description of conditioned edges, loops (LoopNode), and distinguishes among regular messages (MessageNode) and messages that create objects (MessageCreateNode). One modelbased control flow graph is an instance of IRCFG. Although some constraints are not shown in the figure, an IRCFG should have one node representing the start of the graph (StartNode), one node representing the end of the graph (EndNode), zero or more Nodes, and zero or more Edges.
Figure 3 - Approach overview: step 3. The following subsections describe, in more detail, the steps and main components of our approach.
4.1 Test-related Artifacts 4.1.1 Sequence Diagrams in UML 2.0 Metamodel In software engineering, scenarios are used to express system requirements, components and objects interactions, and test scripts [20, 25, 36]. They play an important role in software testing because concrete test scripts are code level scenarios annotated with further information to support comparison between obtained and expected results. Scenarios are used to describe expectations about the system. Depending on the level of the abstraction in which they are described, they can be used and understood by the majority of stakeholders (users, customers, developers, testers, etc.). UML sequence diagrams are scenarios at the design level of abstraction. They are used to model the flow of messages between objects within a system in a visual manner, focusing on the order in which messages are sent. They are commonly used for describing main usage scenarios, the logic of methods or services. In fact, model-based testing approaches consider sequence diagrams as use case realizations [14, 30]. Therefore, testing approaches based on scenarios address the concerns of using of expectations to test the system. For this reason, we adopt sequence diagrams as the basis for test generation. OMG standardizes the UML 2.0 metamodel. In our approach, we use one plug-in implementation of this metamodel that is based on the Eclipse Modeling Framework metamodel (Ecore) [4]. It supports creation of UML 2.0 models and transformation from model to code.
Figure 4 - Model-Based Control Flow Graph (MBCFG) metamodel. Figure 6 depicts an instance of a model-based control flow model derived from the sequence diagram on figure 5. The first node in the graph (switchOnInteraction StartNode) represents the start of the whole sequence diagram in figure 5. It is followed by the switchOn MessageNode, representing the
109
message call switchOn of the sequence diagram. According to the metamodel, MessageNode can be associated to another IRCFG, so switchOn MessageNode has an associated IRCFG, which starts with the StartswitchOn StartNode. This node is followed by (1) showMessage MessageNode, which is followed by the EndswitchOn1 EndNode, and then by the switchOnInteraction EndNode that terminates the sequence diagram, or (2) setInitialCash MessageNode, then the openConnection MessageNode, then the EndswitchOn2, also followed by switchOnInteraction EndNode.
diagrams, we expect the first strategy applied to be path coverage on the model-based control flow graph derived from each sequence diagram. That way, straightforward relationships are established between the messages in a sequence diagram and the nodes in the derived paths. Figure 7 depicts the main components of the test generation hierarchy metamodel.
Figure 7 - Test Generation Hierarchy (TGH) metamodel.
4.1.4 Traceability Metamodel MDD and model transformation solutions address the traceability problem by creating relationships among transformed artifacts as part of the transformation process [26].
Figure 5 - Sequence diagram.
A typical model transformation program receives as input one or more (source) models and transforms them into one or more (target) models. The ATLAS group developed a tool-supported approach for model transformation, namely Atlas Transformation Language (ATL). They built upon this approach and also developed the Atlas Model Weaver (AMW) [2, 18], which is a model weaving solution. As part of the weaving solution, they propose a core weaving metamodel and an extension of this metamodel that supports traceability. Our approach currently uses their traceability metamodel “as is” to externalize the relationships among the test-artifact models (sequence diagram models, model-based control flow graph models and test hierarchy models) (figure 1). Figure 8 depicts a simple traceability model; we refer the interested reader to [17, 18].
Figure 6 - Model-based control flow graph.
4.1.3 Test Generation Hierarchy Metamodel Stocks and Carrington proposed a testing template framework (TTF) that unifies test derivation, oracles and reification [35]. The framework is defined using Z, which is also used as the test description language. The essential idea in their work is the test template (TT). The test template is a formal statement that describes partitions of the data space. Test templates are organized into a hierarchy where coarser templates (larger partitions) are divided into finer templates (smaller partitions) using testing strategies. Each operation to be tested has and equivalent test template hierarchy. The root of the hierarchy describes the valid input space for that operation. It is subdivided into a set of child templates (partitions). After the desired strategies are applied, the hierarchy is considered complete. Later, the leaves are reified and used as a concrete test data. As noted in [35] this kind of structure establishes dependency relationships between specifications and the test case data, which can improve regression testing. We apply the idea of a test template hierarchy (TTH) to generate test case data based on sequence diagrams. We use a metamodel named test generation hierarchy (TGH.ecore in figure 3) that also supports parent/child relationships among the valid input space and its partitions. In our metamodel, each root represents the valid input space for a sequence diagram from the original UML 2.0 model.
Figure 8 - A simple traceability model.
4.1.5 Approach Automation Automation of the approach described in this paper is in progress. In this section we describe our efforts to date and those underway, following the three main steps of the approach as
Since the test cases generated are based on the sequence
110
described earlier. Additional implementation plans are described later in Future Work. We used the Eclipse Modeling Framework (EMF) to create the model-based control flow graph and test hierarchy metamodels. We also used EMF to generate editors for these metamodels.
diagrams (e.g. [X>0] in alt figure 5), constraints on parameters of the sequence diagrams, constraints on message parameters, pre and post conditions in messages, and so forth. A solution that satisfies the conjunction of these constraints is one test case input data. To automate this step, we will use a constraint solver to generate the test case data values. Applying constraint solvers to OCL constraints had been used [11] for test generation based on UML class diagrams. Dinh-Trong et al. [19] also uses UML Sequence diagrams and the Alloy constraint solver to find one solution to the constraint.
The first step of the approach (model creation) creates the UML 2.0 model (with class and sequence diagrams) with a CASE tool capable of exporting a model that conforms to the EMF metamodel (Ecore) implementation of the UML 2.0 specification. Ideally, an integrated tool with graphical interface would exist for creating these models directly with Eclipse. However, the provision of an integrated tool is not the main concern of our approach. We expect a graphical solution for creating UML 2.0 models based on the Ecore metamodel to be available shortly.
5. EXAMPLE To experiment with this idea, we created the UML model of an ATM example with its class diagram and sequence diagrams. In this section, we show how the approach works step by step.
Implementation of the second step (model transformation) is based on the “ATL to Tracer” model transformation [17], which relies on the notion that an ATL transformation is itself a model that conforms to the ATL metamodel. An ATL transformation can therefore, be transformed into another model. The “ATL to Tracer” transformation receives as input an ATL transformation model (Source) and transforms it into another ATL transformation model (Target).
Figure 10 depicts the sequence diagram “start session”. Table 1 shows additional constraints related to the sequence diagram. C1 and C2 are constraints on the parameters of the sequence diagram, namely accNum, which is the account number, and pinNum, which is the pin Number. Depending on the values of these two arguments, different paths in the sequence diagram are triggered. For instance, if either accNum or pinNum are equal to zero, the first break (first brk in figure 10) is enabled and an error message is shown. Otherwise, if both are different from zero, and both are valid, the first operand of the alternation (alt in figure 10) is enabled. Then, a session is created, the account object is retrieved from the bank, and the current account for the created session is set.
We are implementing an ATL transformation that receives as input a UML 2.0 model and transforms it into a model-based control flow graph model. To support traceability, we use “ ATL to Tracer” to transform our transformation into a transformation capable of also creating a traceability model. Figure 9 depicts this idea. It shows an ATL transformation (Source) able to transform UML 2.0 models into model-based control flow graph models. This transformation (Source) is used as input to another transformation (Transformation), which creates a third transformation (Target). The Target transformation creates a model-based control flow graphs from the UML 2.0 models, along with a second model that describes relationships among entities from the UML 2.0 and the model-based control flow graph models.
Figure 9 – Leveraging “ATL to Tracer” transformation to add traceability capability to our transformation. The implementation of the third step (create the test case generation hierarchy, while creating the respective traceability model) is currently manual. Given its complexity, it is not being implemented as a model transformation. It will be a separate module that generates the test case hierarchy and the traceability model. Test case data generation happens as follows. A path from the root of the test generation hierarchy to a leaf defines a set of constraints on the input values of the sequence diagram. These constraints are guards (boolean expression often written in OCL) described in the interaction operands of the sequence
Figure 10 - Sequence diagram "start session". Model creation was described as step 1 of the approach (figure 2). The following step is transforming the sequence diagrams from the UML model into model-based control flow graphs, and creating the traceability information (step 2 in figure 2). The graphical representation of the transformed model is depicted in
111
figure 11, while one part of the traceability information is depicted in figure 12. Table 1 - Additional constraints related to elements of "start session".
Figure 12 - A small part of the traceability model. The next and last step in the approach (step 3 in figure 3) is creation of the test generation hierarchy, and the traceability information. We use information from “star session” sequence diagram (figure 10, from now on called ssSD), from the additional constraints, from the “start session” model-based control flow graph (figure 11, from now on called ssMBCFG), and from the traceability model.
Figure 11 - model-based control flow graph for "start session". Figure 13 - Test generation hierarchy example. In figure 12, the traceability model is named TraceUML2MBCFG. It relates models UML2ATMModel and MBCFGATMModel. It has one trace link named startSessionTraceLink that relates the StartSessionSD in the UML2ATMModel to the StartSessionMBCFG in the MBCDFATMModel. startSessionTraceLink has many children; figure 12 shows insertCardTraceLink, which relates the InsertCard message in the StartSessionSD to the InsertCard node in the StartSessionMBCFG.
First, we apply the path selection strategy to ssMBCFG, and derive the possible paths (e.g. N1-N2-N3-N9-N10-N11-N12N21). Using the traceability model and the ssSD, we know that this path is enabled provided that some guards and constraints are satisfied. First, the accNum and the pinNum must be different from zero (guard for first brk in ssSD). Second, the accNum must be the number of a valid account (guard for second brk in ssDD). Third, the pinNum must be the valid pin for the account (guard for loop and for alt in ssSD). We then
112
analyze the domains of accNum and pinNum. C3 in table 1 says that validateAccount returns true when the accNum belongs to a set of valid accounts. C4 in table 1 says that validatePin(accNum, pinNum) returns true if there are account and pin in the set of predefined valid accounts, where account and pin match the accNum and pinNum received as argument. Therefore, what is needed as input to test this path is a combination of valid account and matching pin. This conjunction of constraints is simple enough that a solution can be randomly retrieved directly from the set of valid accounts, which is either defined in the model, or in a separate database. In case of a more complex conjunction, we could transform it into a valid input to a constraint solver [5] that would create one ore more solutions. This feature is not implemented yet. Figure 13 depicts the test generation hierarchy created with this process.
the models such as the screenshot presented in figure 1 is part of our future work. The traceability metamodel by [2] is adopted “as is” in our approach. In their empirical study about the use of traceability, Balasubramaniam and Matthias [9] observe that “the semantics or the meaning attributed to linkage is guided by the reasoning that the user will be performing with the linkage”, so one can define different relationships, or different attributes to the same relationships to among entities support specific activities. Therefore, we will investigate how the traceability metamodel should be extended to better support the discussed testing activities. Although we presented an ATM example, our future work includes evaluation of the approach’s scalability with more complex examples. As part of the evaluation, we will evaluate the trade-offs between gains obtained with respect to regression testing, coverage analysis, result evaluation and the cost to externalize and maintain the relationships.
6. CONCLUSIONS In this paper, we presented an approach that leverages currently available model transformation traceability techniques to create fine-grained relationships among model-based testing artifacts. Relationships are created among sequence diagrams in UML models, their model-based control flow graph, and a test generation hierarchy that describes parent/child relationships among the sequence diagrams’ valid input space and its partitions. The relationships between the UML model and the model-based control flow graphs are made explicit and persistent during the transformation from one model into the other, whereas the relationships between the model-based control flow graph and the test generation hierarchy are made explicit and persistent during the generation of the test data input (creation of the test generation hierarchy from the model-based control flow graph). The relationships’ fine granularity enables the support for model-based tests result evaluation, coverage analysis and regression testing.
8. ACKNOWLEDGMENTS The authors acknowledge a 2005 and 2006 IBM faculty award, a research gift from Siemens Corporate Research. The authors would like to thank Henry Muccini at the Dipartimento di Informatica University of L’Aquila, Italy for his valuable insights.
9. REFERENCES [1] [2] [3] [4]
Our approach to test generation assumes the existence of sequence diagrams in the UML model. It creates a model-based control flow graph from each sequence diagram, and applies path coverage testing strategy followed by domain analysis on the parameters of the sequence diagram. To solve the conjunction of guards and constraints that define the different paths in the model-based control flow graph, the conjunction is converted into a valid input to a constraint solver that creates one or many test case data satisfying the constraints. Our future work places include automation this process.
[5] [6] [7] [8]
The main contribution of this work is an infrastructure and workflow for model-based testing that applies model transformation and test generation techniques to create modelbased control flow graphs, test generation hierarchy models, and traceability models.
[9] [10]
7. FUTURE WORK As explained in section 4.1.5 the proposed infrastructure is under implementation. In the immediate future, we are finalizing automation of steps 2 and 3 (transformation from UML 2.0 model to model-based control flow graph, and creation of test generation hierarchy).
[11]
[12]
Given the availability of open-source visualization tools available for the models we are using [4], better visualizations of
113
ADT: ATL Development Tools, http://www.eclipse.org/m2m/atl/. Atlas Model Weaver (AMW), http://www.eclipse.org/gmt/amw/. Eclipse - an open development platform. Eclipse Modeling Framework Project (EMF), http://www.eclipse.org/modeling/emf/?project=emf. JCL Java Constraint Library, http://liawww.epfl.ch/JCL/. Parasoft, http://www.parasoft.com. MOF™ Query / Views / Transformations, OMG, 2005. I. Alexander, Towards automatic traceability in the industrial practice, 1st international workshop on traceability in emerging forms of software engineering, Edinburgh, UK, 2002. R. Balasubramaniam, Matthias,J., Toward Reference Models for Requirements Traceability, IEEE Press, 2001. F. Basanieri, Bertolino, A., Marchetti, E., The Cow_Suite Approach to Planning and Deriving Test Suites in UML Projects, Proceedings of the 5th International Conference on The Unified Modeling Language, Springer-Verlag, 2002, pp. 383-397. M. Benattou, Bruel, J. M., Hameurlain, N., Generating Test Data from OCL Specication, ECOOP 2002 Workshop on integration and transformation of UML models (WITUML'2002), Málaga, Spain, 2002. R. V. Binder, Round-trip Scenario Test, Testing Object-Oriented Systems - Models, Patterns, and Tools, Addison-Wesley, 2000, pp. 579-591.
[13]
[14]
[15] [16] [17]
[18]
[19]
[20] [21]
[22]
[23]
[24]
[25]
[26]
[27]
F. Bouquet, Jaffuel, E., Legeard, B., Peureux, F., Utting, M., Requirements Traceability in Automated Test Generation - Application to Smart Card Software Validation, ICSE Int. Workshop on Advances in Model-Based Software Testing (A-MOST'05), ACM Press, St. Louis, USA, 2005. L. C. Briand, Labiche, Y., A UML-Based Approach to System Testing, 4th International Conference on the Unified Modeling Language (UML), Toronto, Canada, 2001, pp. 194-208. C. Csallner, Smaragdakis, Y., JCrasher: An automatic robustness tester for Java, 2004, pp. 1025-1050. J. Dick, Faivre, A., Automating the Generation and Sequencing of Test Cases from Model-Based Specifications, Springer-Verlag, 1993. M. Didonet Del Fabro, Generating an ATL Execution Trace as a Traceability Weaving Model (ATL to Tracer),http://www.eclipse.org/gmt/amw/examples/#A TL2WTracer. M. Didonet Del Fabro, Bézivin, J., Valduriez, P., Weaving Models with the Eclipse AMW plugin, Eclipse Modeling Symposium, Eclipse Summit Europe 2006, Esslingen, Germany, 2006. T. T. Dinh-Trong, Ghosh, S., France, R. B., A Systematic Approach to Generate Inputs to Test UML Design Models, 17th International Symposium on Software Reliability Engineering (ISSRE'06), 2006, pp. 95-104. A. Egyed, A scenario-driven approach to trace dependency analysis, IEEE Transactions on Software Engineering, 2003. F. Fraikin, Leonhardt, T., SeDiTeC — Testing Based on Sequence Diagrams, 17th IEEE International Conference on Automated Software Engineering, 2002, pp. 261 - 266. V. Garousi, Briand, L. C., Labiche Y., Control Flow Analysis of UML 2.0 Sequence Diagrams, European Conference on Model Driven ArchitectureFoundations and Applications (ECMDA-FA), 2005. W. Grieskamp, Nachmanson, L., Tillmann, N., Veanes, M., Test Case Generation from AsmL Specifications - Tool Overview, 10th International Workshop on Abstract State Machines, Taormina, Italy, 2003. A. Hartman, Nagin, K., The AGEDIS tools for model based testing, 2004 ACM SIGSOFT international symposium on Software testing and analysis, ACM Press, Boston, Massachusetts, USA, 2004, pp. 129132. W. Jeremiah, Frank, M., Using UML to Partially Automate Generation of Scenario-Based Test Drivers, 7th International Conference on Object Oriented Information Systems, 2001, pp. 303-306. F. Jouault, Loosely Coupled Traceability for ATL, European Conference on Model Driven Architecture Workshop on Traceability, Nuremberg, Germany, 2005. F. Jouault, Kurtev, I., On the Architectural Alignment of ATL and QVT, ACM Symposium on Applied Computing (SAC 06), ACM Press, Dijon, France, 2006, pp. 1188--1195.
[28] [29]
[30]
[31] [32]
[33]
[34]
[35] [36]
[37]
[38]
114
M. Lindval, Sandahl, K., Practical Implications of Traceability, Software Practice and Experience, 1996, pp. 1161-1180. L. Naslavsky, Alspaugh, T. A., Richardson, D. J., Ziv, H., Using Scenarios to Support Traceability, International Workshop on Traceability in Emerging Forms of Software Engineering, ACM Press, Long Beach, 2005, pp. 25-30. C. Nebut, Fleurey, F., Traon, Y. L., Jézéquel, J., Automatic Test Generation: A Use Case Driven Approach, IEEE Transactions on Software Engineering, 32 (2006), pp. 140-155. D. J. Richardson, TAOS: Testing with Analysis and Oracle Support, ACM Press, Seattle, Washington, United States, 1994. D. J. Richardson, Aha, S. L., O'Malley, T. O., Specification-based test oracles for reactive systems, Proceedings of the 14th international conference on Software engineering, ACM Press, Melbourne, Australia, 1992, pp. 105-118. A. Rountev, Kagan, S., Sawin, J., Coverage Criteria for Testing of Object Interactions in Sequence Diagrams, Fundamental Approaches to Software Engineering, Springer Berlin / Heidelberg, 2005, pp. 289-304. G. Spanoudakis, Zisman, A., Software Traceability: A Roadmap, Advances in Software Engineering and Knowledge Engineering, World Scientific Publishing, 2005. P. Stocks, Carrington, D., A Framework for Specification-Based Testing, IEEE Transactions on Software Engineering, 1996, pp. 777-793. S. Uchitel, Chatley, R., Kramer, J., Magee, J., System Architecture: the Context for Scenario-based Model Synthesis, 12th ACM SIGSOFT twelfth international symposium on Foundations of software engineering, 2004, pp. 33-42. J. Wittevrongel, Maurer, F., SCENTOR: ScenarioBased Testing of E-Business Applications, Tenth IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises, 2001, pp. 41 - 46. T. Xie, Marinov, D., Notkin, D., Rostra: A Framework for Detecting Redundant Object-Oriented Unit Tests, 19th IEEE International Conference on Automated Software Engineering (ASE'04), 2004, pp. 196-205.
