Using Model Transformation to Support Model-Based Test Coverage Measurement Leila Naslavsky
Hadar Ziv
Debra J. Richardson
Donald Bren School of Information and Computer Sciences University of California, Irvine Irvine, CA 92697-3425 USA
Donald Bren School of Information and Computer Sciences University of California, Irvine Irvine, CA 92697-3425 USA
Donald Bren School of Information and Computer Sciences University of California, Irvine Irvine, CA 92697-3425 USA
[email protected]
[email protected]
[email protected]
diagram). Moreover, some tools also automate code generation from behavioral models (e.g. UML state machines). Irrespective of the code generation process, model and code evolve in parallel while maintenance of their consistency is not fully automated (the round-trip engineering problem).
ABSTRACT Adoption of model-driven development is leading to increased use of models jointly with source code for software testing, by means of Model-Based Testing (MBT). MBT uses models to derive concrete test cases to test code. With MBT, test adequacy criteria are described in relation to the models. They are used to evaluate reliability of derived test cases and as predictor for determining when to stop testing. Hence, when concrete test cases are executed, it is important to measure coverage achieved with regards to the model, rather than only to the code. This places new challenges for testing activities, which include creation and maintenance of relationships between model and code elements. To deal with such challenge, we propose an approach that leverages model-to-text transformation traceability techniques to create relationships required to measure model coverage achieved with test cases executions. We illustrate the approach by applying it to a small ATM example.
The increased use of models creates opportunities for software testing. In particular, it creates the opportunity to drive the testing process at a higher level of abstraction and to demonstrate compliance of models with source code, by means of ModelBased Testing (MBT). Although most automated approaches to MBT concentrate on test case generation, execution and evaluation, other activities such as measuring model coverage achieved when executing test cases are important [16]. Test adequacy criteria have been used as predictor for determining when to stop testing a program [11]. They rely on the idea that test suite reliability can be evaluated according to the extent to which parts of the program are exercised by the test cases [13]. Thus, it is important to measure test suite adequacy to quantify the level of confidence on the testing performed. For instance, when concrete test cases are executed, a traditional way to measure their adequacy for testing the program is learning which parts of the program where executed. This can be measured by instrumenting the code and observing which statements/edges were exercised by each test case.
Categories and Subject Descriptors D.2.5 [Testing and Debugging]: Testing tools (e.g., data generators, coverage testing)
General Terms Measurement, Documentation, Design.
Since MBT uses models to derive test cases, derived test suite adequacy should be determined with respect to the model, rather than the code. The need to determine test suite adequacy with respect to models becomes more evident when test cases are generated based on one type model (e.g. state machines) and test suite adequacy is measured with respect to other type of model (e.g. sequence diagrams). To achieve these objectives, when a concrete test case is exercised against the code, there should be support for identifying which model elements were covered. This poses new challenges for MBT activities, including creation and maintenance of relationships between model and code entities (a.k.a. mapping or traceability between model and code).
Keywords Model-driven development, model-based testing, traceability.
1. INTRODUCTION Adoption of model-driven software development is leading to increased use of models as primary software artifacts. This approach to software development uses structural and behavioral models as guidance for code generation, which can be manual or automatic. Tool-supported approaches to code generation often generate code skeleton from structural models (e.g. UML class
In model-driven development, creation of relationships between model and code has been addressed by model-to-text transformation techniques (e.g. MofScript [5]). These techniques create such relationships during transformations, according to the transformations rules. For instance, with a MofScript transformation, rules can describe how a class diagram is transformed into Java class source code. During this process,
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. AST’08, May 11, 2008, Leipzig, Germany. Copyright 2008 ACM 978-1-60558-030-2/08/05...$5.00.
1
support coverage measurement do so by creating code-based coverage reports [14]. Thus, they do not address model-based coverage measurement. In this paper, we do not discuss these approaches in detail; rather, we refer the interested reader to [19].
relationships can be created between methods described in the class diagram and method signatures in the source code. Indeed, the idea of creating relationships among source and target artifacts according to transformations rules is how model transformation languages support traceability (e.g. ATL weaver [15], MofScript [20]).
Ghosh et al. proposed a design testing method that incorporates test assessment of test adequacy criteria for UML design models [12]. Their work aims at generating test cases to test design and measuring adequacy in terms of covered model elements. Our work is similar since it also measures covered model elements. However, their work measures model coverage when exercising the models, whereas our proposed approach does so when exercising code.
In this paper, we propose using relationships created with modelto-text transformations in the context of MBT to support modelbased test coverage measurement. We assume existence of concrete test cases (executable test scripts). Model-to-text transformations are used to generate code skeleton and explicit fine-grained relationships between model and code elements. These relationships support measurement of model coverage achieved with execution of concrete test cases. We illustrate usefulness of the approach with a small example.
Sequence Diagram Test Center (SeDiTeC) supports the execution of sequence diagrams, so it uses the sequence diagrams to test the integration among objects [10]. The tester is in charge of describing concrete parameters and expected results of the sequence diagrams. SeDiTeC uses code instrumentation to evaluate results of test execution. It logs obtained method calls, concrete parameter and results. Differently from SeDiTeC, the proposed approach is based on a model-to-text transformation technique that makes model-code relationships persistent and explicit (in the form of a traceability model), and explores these relationships in the support of coverage measurement.
2. BACKGROUND Model-driven development (MDD) describes software development approaches in which software models are primary artifact. The software engineering field recently witnessed an increase in MDD adoption. Core MDD techniques are model-tomodel and model-to-text transformations. MDD and model transformation solutions address the traceability problem by creating relationships among transformed artifacts during the transformation process (model-to-model [15, 17], and model-totext [20]). We base our work in the second (model-to-text) solution.
4. APPROACH OVERVIEW Previously, we used model-to-model transformation to support test case generation from models (UML class and sequence diagrams) [18]. In this work, we assume existence of a test suite, composed of concrete test cases previously generated from models.
The OMG proposed a specification (Query/Views/ Transformations – QVT) that standardized the ways source models are transformed into target models (model-to-model transformations) [6]. There are many implementations partially or totally compliant with this standard (e.g. M2M, Borland Together, ATL, Tefkat). MDD tools support code generation from models, which is a kind of model transformation that converts a model at a higher level of abstraction into source code. OMG also proposed a specification for standardized model-to-text transformations and also specifies a traceability solution that is implemented by MofScript. Our approach adopts such implementation. Model-based testing requires the ability to relate the “abstract values of the specification to the concrete values of the implementation” [8]. In fact, MDD supports this ability: models are transformed into code and relationships are made explicit and persistent (traceability) based on transformation rules. One testing activity that benefits from such relationships is coverage measurement. In this paper we address model-based coverage measurement by leveraging traceability created with model-totext transformations. To the best of our knowledge, this had not been considered by other MBT automated approaches.
3. RELATED WORK
Figure 1 - approach overview.
Traditional testing activities comprise: test case generation, test execution, result evaluation, coverage measurement, and regression testing. Most model-based testing automated approaches often concentrate on the test generation, execution, and result evaluation activities, while support to other activities such as coverage measurement is scarce [16, 22]. Approaches that
Figure 1 depicts an overview of the proposed approach. We assume that (a) UML model (with class and sequence diagrams) was previously created, and (b) the target programming language for code generation is Java. In the first step, the class diagram model is used as basis for code skeleton generation with a model-
2
unique identifiers (xmi:id). Java Annotations are user-defined metatags used in java code [3]. Therefore, relationships (code) endpoints are independent of position in the code file. This way, when code is modified (e.g. addition of method body), relationships’ (code) endpoints do not need to be updated. In addition, annotations can be preprocessed to instrument code automatically to trace method calls. This step is required to measure coverage achieved.
to-text transformation technique. During transformation, finegrained relationships are created between model and code. These relationships are persisted in a traceability model. A developer should implement the methods’ behavior (body). The methods should be instrumented to emit method-call notifications. In the second step, concrete test cases are executed. During test case execution, the traceability model supports identification of exercised parts of the behavioral model (step 3 in Figure 1).
Figure 2 depicts an example of relationships between methods in a class diagram and methods in the respective generated code skeleton. Each annotation has an element id with same value as the originating model element identifier. For instance, method isAmountAvailable in the UML model generated the method isAmountAvailable in Account.java. The Java method has a Java Annotation with same identifier as isAmountAvailable in the UML model.
5. APPROACH AUTOMATION DETAILS In this section, we explain automation details, and describe our efforts to date.
5.1 Generating Code Skeleton Model-driven software development explores model-to-text transformations for code generation. Our proposed approach adopts MofScript [5], which is an Eclipse-based open-source model-to-text transformation tool. MofScript uses the Eclipse Metamodel Framework (EMF) [2] as meta-model input. It has an adaptation to UML 2.0 metamodel (also EMF-based). MofScript supports generation of code skeleton from the structural model (UML class diagram). In this case, the model should conform to UML 2.0 metamodel (also based EMF-based). We adapted a publicly available MofScript code generation example.
For pages other than the first page, start at the top of the page, and continue in double-column format. The two columns on the last page should be as close to equal length as possible.
Code skeleton generation (model-to-text transformation, or step 1 in figure 1) is done from class diagram to Java code. Currently, our approach assumes a one-to-one relationship between classes in the class diagram and Java classes to be implemented. One method signature is created for each method in the class diagram. If higher level of abstraction were used to describe the models, this assumption would not hold and a mapping from high level model entities to lower level code entities would be required.
5.2 Creating Relationships Relationships between model and code elements can be made explicit and persistent at different levels of abstraction. Coarsegrained relationships can be created from classes in class diagrams to classes’ source code file; or fine-grained relationships can be created from methods in class diagrams to their implementation the source code file. In fact, endpoint elements define the granularity of a relationship.
Figure 2 – Relationships between model and code with annotations. Due to space constraints, the traceability model in Figure 2 does not show all relationships in detail. We refer the interested reader to [9, 14, 17] for an explanation of the adopted metamodel.
Regarding the level of abstraction, relationships created with MofScript are at the statement/block level. Position of statements and blocks in the generated code file identify the (code) endpoints of such relationships. Thus, when developers modify generated files, positions of subsequent blocks must be updated to maintain consistency of relationships. Although this step can be automated, maintaining consistency among model, traceability model, and code might be costly. To support our approach to model-based coverage measurement, granularity of relationships need not be finer than method-call level. Thus, creation of such fine-grained (statement level) relationships serves to no purpose.
5.3 Code Instrumentation Gathering information during concrete test case execution is an integral part of the proposed approach. Different techniques could be adopted: (a) directly instrument the source code to emit methodcall notifications, (b) use aspect oriented by creating aspects with monitoring concerns to be woven into the code [1]; or even (c) instrument the byte code or monitor the java virtual machine (Java Platform Debugging Architecture) [4]. Since we have the code to can be instrumented, we could use either aspect oriented (with AspectJ) or we could directly instrument the code.
Although we use MofScript “as-is” for code generation, we propose modifying its support for traceability. We create relationships at the method-level of abstraction, which we regard as fine-grained. Instead of using statement/block positions as the relationships’ endpoints, we use Java Annotations with modeling elements’
3
consider the situation where the user is trying to withdrawal money, but the ATM is not at a ‘SERVING_CUSTOMER_STATE’ state, so that path is not traversed (we assume that if a user is attempting to withdraw, that user was previously authenticated, and ATM is consequently serving that user).
We propose adopting of AspectJ [1] to create aspects that monitor the Java methods. This decision supports instrumentation functionality with our direct code modification, which is a wellknown advantage of aspect-oriented programming languages. Per our solution, aspects woven into the code emit method-call notifications with the traceability id obtained from the Java Annotation. When concrete test cases are executed, the approach gathers the method-call notifications obtained with the weaving code. Then, it uses the traceability model to locate the exercised parts of the model (coverage achieved).
Table 1 – part of test suite for ATM model.
Figure 3 depicts the process. MethodCovered.java describes an aspect that will be woven into all methods with annotation of type TraceabilityAnnotation (step 1). Then, test cases are executed (step 2). When methods are called, the woven aspect should store in the previously created traceability model information about execution trace and methods called. Then, the traceability model should be used to locate and mark elements from the behavioral model that where covered with the test case (step 3).
In this example, we consider a test suite to be reliable when it satisfies test adequacy criterion that covers all message paths (covers all paths in the sequence diagram) [12]. Thus, the information obtained with the proposed model-based coverage measurement approach allows the tester to conclude the test suite used does not satisfy the desired test adequacy criteria. In another situation, suppose test case generation strategy adopted aims at covering all paths of the sequence diagram. Indeed, a well known strategy (round-trip scenario) for test case generation from sequence diagrams consists of deriving a control flow model from the sequence diagram, developing a set of paths that achieves particular coverage criteria based on the control flow model (e.g. branch coverage, or path coverage), and identifying inputs that cause the paths in the set to be taken [7]. Each path derived from the sequence diagram partitions the valid input set into test case data inputs. Table 2 - part of test suite for ATM model.
Figure 3 - Instrument code and measure coverage.
6. THE ATM EXAMPLE This section illustrates the proposed approach and its usefulness with an example. We assume that (1) ATM UML class diagram was created; (2) code skeleton with traceability was generated (sections 5.1, and 5.2); and (3) code was instrumented (section 5.3). We will use concrete test cases obtained from a sequence diagram that describes the ‘withdrawal’ functionality of an ATM model. Suppose concrete test cases are created to test functionality of the sequence diagram. The test case generation strategy adopted aims at data coverage. It is a useful criterion for choosing representative inputs when the domain is too large. Note the example sequence diagram uses two input arguments (type of transaction, and amount to withdrawal). The adopted strategy results in test cases with different values for the amount to withdraw. Test cases shown in table 1 were generated. These test cases are used to exercise the instrumented ATM Java code. The traceability model previously created during code generation (model-to-text transformation), allows elements in the model to be marked as covered (thicker arrows in figure 4). The obtained model-based coverage measurement supports observation that some paths in the model were not traversed with those test cases. For example, the test case generation strategy did not
Test cases in table 2 were generated using the aforementioned strategy. The same activities as previously described were performed. In this case, the tester would be able to notice all message paths in the model (sequence diagram) were covered with test cases in table 2.
4
Figure 4 - an example sequence diagram with coverage measurement information (test suite table 1). Coverage measurement measures test suite adequacy by quantifying the level of confidence on the testing performed. Indeed, test suite reliability can be evaluated according to the extent to which parts of the program are exercised with the test cases. We brought this argument to a higher level of abstraction to measure test suite reliability when test suite is derived from models.
7. DISCUSSION Although the example used in this section is simple, it illustrates usefulness of our approach to model-based coverage measurement. It exemplifies the approach is useful to identify elements in the model not covered with the test cases. It also exemplifies the approach is useful to quantify the level of confidence on the testing performed by measuring if test suite achieved desired adequacy criteria with regards to the model.
Traditionally, when concrete test cases are executed, a way to measure their adequacy for testing the program is learning which parts of the program where exercised. This can be measured with instrumenting the code and observing which statements/edges were exercised with each test case. Our approach supports the same idea by instrumenting the code and observing which methods/edges of the model were exercised with each test case.
Clearly, the first set of test cases on table 1 was not meant to exercise all paths, and this explains the failure on satisfying test coverage adequacy criteria. Nevertheless, this is a valid illustration given the fact that the criteria used to select test cases is a prescription of the way test cases should be generated. The results of test cases execution are measured to check how well they satisfy the test adequacy criteria [21]. Indeed, MBT tools are not always able to generate test cases fully satisfying the selection criteria. Hence, model-based coverage measurement is used to identify non-covered parts in the model, which should be considered an integral part of Model-Based Testing automated solutions.
This automated approach does not require developers to provide the mapping manually, rather, it takes advantage of the automation created by model-to-text transformation techniques. We are currently working on further automating the proposed approach. In the future, we plan to integrate our model-based test case generation/selective regression testing approach and traceability infrastructure with the proposed approach [18]. This would allow us to create an integrated model-based testing automated approach that not only supports test case generation, but also selective regression testing and coverage measurement.
8. CONCLUSION This paper contributes an application for model-to-text transformations in the context of model-based testing (MBT). It assumes model-to-text transformation is used to support code generation. Then, it uses the traceability obtained from model-totext transformation to create a fine-grained relationships infrastructure that provides the necessary support for measuring coverage achieved by test case execution with regards to parts of the models exercised.
9. REFERENCES [1] AspectJ, http://www.eclipse.org/aspectj/. [2] Eclipse Modeling Framework Project (EMF), http://www.eclipse.org/modeling/emf/?project=emf.
5
[3] Java Annotations, http://java.sun.com/docs/books/tutorial/java/javaOO/annotat ions.html.
[14] Hartman, A., Nagin, K., The AGEDIS tools for model based testing, 2004 ACM SIGSOFT international symposium on Software testing and analysis, 2004.
[4] Java Platform Debugger Interface, http://java.sun.com/j2se/1.4.2/docs/guide/jpda/index.html.
[15] Jouault, F., Loosely Coupled Traceability for ATL, European Conference on Model Driven Architecture Workshop on Traceability, 2005.
[5] MofScript, http://www.eclipse.org/gmt/mofscript/.
[16] Naslavsky, L., Muccini, H., Richardson, D. J., Scenariobased and State Machine-based Testing: An evaluation of automated approaches, 1st International Workshop on Model-Based Testing and Object-Oriented Systems (MTOOS-2006), 2006.
[6] MOF™ Query / Views / Transformations, http://www.omg.org/cgi-bin/doc?ptc/2005-11-01 [7] Binder, R. V., Round-trip Scenario Test, Testing ObjectOriented Systems - Models, Patterns, and Tools, AddisonWesley, 2000, pp. 579-591.
[17] Naslavsky, L., Ziv H., Richardson D. J., Towards Traceability of Model-based Testing Artifacts, Third Workshop on Advances in Model Based Testing (A-MOST 2007), 2007.
[8] Dick, J., Faivre, A., Automating the Generation and Sequencing of Test Cases from Model-Based Specifications, First International Symposium of Formal Methods Europe on Industrial-Strength Formal Methods, 1993.
[18] Naslavsky, L., Ziv, H., Richardson D. J., Towards Leveraging Model Transformation To Support Model-based Testing, International Conference on Automated Software Engineering (ASE'07), 2007.
[9] Didonet Del Fabro, M., Bézivin, J., Valduriez, P., Weaving Models with the Eclipse AMW plugin, Eclipse Modeling Symposium, Eclipse Summit Europe 2006, 2006. [10] Fraikin, F., Leonhardt, T., SeDiTeC — Testing Based on Sequence Diagrams, 17th IEEE International Conference on Automated Software Engineering, 2002.
[19] Naslavsky, L., Ziv, H., Richardson, D. J., Scenario-based and State Machine-based Testing: An Evaluation of Automated Approaches, ISR, University of California, Irvine, 2006.
[11] Frankl, P. G., Weyuker, E. J., An applicable family of data flow testing criteria, IEEE Transactions of Software Engineering, 14 (1998), pp. 1483 - 1498.
[20] Olsen, G. K., Oldevik, J., Scenarios of Traceability in Model to Text Transformations, Model Driven ArchitectureFoundations and Applications, Springer Berlin / Heidelberg, 2007, pp. 144-156.
[12] Ghosh, S., France, R., Braganza, C., Kawane, N., Test Adequacy Assessment for UML Design Model Testing, International Symposium on Software Reliability Engineering, 2003.
[21] Utting, M., Legeard, B., Selecting Your Tests, Practical Model-Based Testing, Morgan Kaufmann, 2007.
[13] Goodenough, J.B, Gerhart, S. L., Toward a theory of test data selection, International Conference on Reliable Software, 1975.
[22] Utting, M., Pretschner, A., Legeard, B., A Taxonomy of Model-Based Testing, Department of Computer Science, The University of Waikato, New Zealand, 2006.
6
