Debugging Model-Transformation Failures Using Dynamic Tainting ...

Debugging Model-Transformation Failures Using Dynamic Tainting Pankaj Dhoolia, Senthil Mani, Vibha Singhal Sinha, and Saurabh Sinha IBM Research – India {pdhoolia,sentmani,vibha.sinha,saurabhsinha}@in.ibm.com

Abstract. Model-to-text (M2T) transforms are a class of software applications that translate a structured input into text output. The input models to such transforms are complex, and faults in the models that cause an M2T transform to generate an incorrect or incomplete output can be hard to debug. We present an approach based on dynamic tainting to assist transform users in debugging input models. The approach instruments the transform code to associate taint marks with the input-model elements, and propagate the marks to the output text. The taint marks identify the input-model elements that either contribute to an output string, or cause potentially incorrect paths to be executed through the transform, which results in an incorrect or a missing string in the output. We implemented our approach for XSL-based transforms and conducted empirical studies. Our results illustrate that the approach can significantly reduce the fault search space and, in many cases, precisely identify the input-model faults. The main benefit of our approach is that it automates, with a high degree of accuracy, a debugging task that can be tedious to perform manually. Keywords: Model-driven engineering, input-model faults, fault localization, dynamic tainting.

1

Introduction

Model-Driven Engineering (MDE) [20] is the paradigm of software development that uses formal models, at different abstraction levels, to represent the system under development, and uses automated transforms to convert one model to another model or text.1 A model is typically represented using a structured format (e.g., XML or UML). A significant class of model transforms, called model-totext (M2T) transforms, generate text output (e.g., code, configuration files, or HTML/JSP files) from an input model. The input models to the transforms are often large and complex. Therefore, the models can contain faults, such as a missing element or an incorrect value of an attribute, that cause a transformation to fail: the transformation either generates no output (i.e., it terminates with an exception) or generates an incorrect output. 1

We follow the terminology introduced by Baxter [4]. A transform is a function, or a program, that maps one model to another model or text. A transformation is the application, or the execution, of a transform on a model instance.

T. D’Hondt (Ed.): ECOOP 2010, LNCS 6183, pp. 26–51, 2010. c Springer-Verlag Berlin Heidelberg 2010 

Debugging Model-Transformation Failures Using Dynamic Tainting

27

Fig. 1. Example of an input-model fault that causes an incorrect output

The structure of a model is defined by a metamodel. In many cases, a metamodel also specifies the semantic constraints that a model must satisfy. For example, to be a valid instance, a UML model may have to satisfy OCL constraints. A model can contain faults that violate such syntactic and semantic well-formedness properties. Such faults can be detected easily using automated validators that check whether a model conforms to the metamodel constraints. However, a large class of faults may violate no constraints and yet cause a transformation to fail; such faults cannot be detected using model validators. To illustrate, consider the model and output fragments shown in Figure 1. The left side of the figure shows a correct input model to a transform that generates a configuration file consisting of name-value pairs. (We present the transform code in Section 2.) The input model on the right contains a fault: the isGen attribute of the second property has an incorrect value. This fault causes a wrong transform path to be executed and, consequently, the incorrect substring “NIL” to be generated in the output. However, the value of isGen is not constrained to be “nameValue” and a different value is, in fact, valid in cases where the user expects “NIL” to be generated. Thus, the interpretation of whether the isGen value represents a fault depends on what the user expects in the output. In this case, the value is a fault, but no automated validator can detect it. In a large and complex model, consisting of thousands of elements and attributes, locating such subtle faults can be difficult and time-consuming. Although a transformation failure can be caused by faults in the transform, the goal of our work is to develop techniques for investigating failures caused by input-model faults. In MDE, it is a common practice for transform users to use transforms that are not written by them (e.g., many tools provide standard built-in transforms). Thus, a user’s knowledge of the transform is limited to the information available from documentation and example models. Even if the code is available, the end-users often lack the technical expertise to debug the problem by examining the code. Thus, when a transformation fails, the pertinent task for transform users is to understand the input space, how it maps to the output, and identify faults in the input; investigating the transform code is irrelevant, and, in the absence of access to the transform implementation, impossible.

28

P. Dhoolia et al.

Most of the existing research in fault localization focuses on identifying faults in the program. The goal of these approaches is to narrow down the search space of program statements that should be examined to locate the fault; they use different techniques, such as program slicing (e.g., [2,8]) or spectra comparisons for passing and failing executions (e.g., [13,19]). These approaches are not applicable to localizing input-model faults. Some researchers (e.g., [6,23]) have investigated ways to extend the statement-centric view of debugging to consider also the subset of the input that is relevant for investigating a failure. For example, given an input i that causes a failure, delta debugging [23] identifies the minimal subset of i that would also cause the failure. Similarly, the penumbra tool [6] identifies the subset of i that is relevant for investigating the failure. These approaches could conceivably be used for debugging input models because the failure-relevant subset of the input model is likely to contain the fault. However, because these techniques are not targeted toward detecting input-model faults, in practice, they may perform poorly when applied to model debugging. For the example in Figure 1, delta debugging would fail to identify a minimal failure-inducing input, whereas penumbra could identify the entire input model as being failure-relevant. Model-tracing techniques [7] create links between input-model and outputmodel entities, which can be useful for supporting fault localization in cases where an incorrect value of an input-model entity flows to the output through value propagation. However, for faults such as the one illustrated in Figure 1, tracing techniques can provide no assistance in localizing the faults. Similarly, if the fault is a missing entity in the input or the manifested failure is a missing substring in the output, tracing techniques cannot assist with fault localization. In this paper, we present an approach for assisting transform users in locating faults in input models that cause a model-to-text transformation to fail. The goal of the approach is to narrow down the fault search space in a failureinducing input model. The approach uses dynamic tainting [5] (or informationflow analysis [16]) to track the flow of data from input-model entities to the output string of a model-to-text transform. Given the input model I for a failing execution of a transform program P , the approach instruments P to associate taint marks with the elements of I and propagate the marks to the output string. The execution of the instrumented program generates a taint log, in which substrings of the output string have taint marks associated with them. The taint marks associated with a substring indicate the elements of I that influenced the generation of the substring. To locate the faults in I, the user first identifies the point in the output string at which a substring is missing or an incorrect substring is generated. Next, using the taint marks, the user can navigate back to entities of I, which constitute the search space for the fault. A key feature of our approach is that, in addition to identifying input-model entities from which data flows to the output, the taint marks also identify the entities that determine whether an alternative substring could have been generated at a particular point in the output string, had the failing execution traversed a different path through the transform. We call such taint marks “control-taint

Debugging Model-Transformation Failures Using Dynamic Tainting

29

marks” and distinguish them from “data-taint marks.” Unlike data-taint marks, which are propagated at assignment statements and statements that construct the output string, a control-taint mark is propagated to the output string at conditional statements. The propagation of control taints lets our approach identify faults that cause an incorrect path to be taken through the transform, which results in a missing or an incorrect substring in the output. The control taints are similar to the notion of control-flow-based tainting [5] and, as our empirical results indicate, are essential for the application of dynamic tainting to debugging model-transformation failures. We also introduce the notion of “loop-taint marks,” which, intuitively, scope out the execution of a loop. These taints help in locating faults that cause an incorrect number of loop iterations. We implemented the approach for XSL-based model transforms that generate configuration files, Java programs, and XML files as output. Using the implementation, we conducted empirical studies to investigate the effectiveness of the technique in reducing the fault search space. For our subjects and failureinducing inputs, the approach significantly reduced the fault search space: for 51% of our faulty input models, the fault was identified precisely; for remaining 49% of the faults, the fault-space reduction was greater than 94%. These results indicate that the approach can be highly effective in assisting transform users in locating input-model faults. The main benefit of our approach is that it can automate, with a high degree of accuracy, a debugging task that can be tedious and time-consuming to perform manually. It is especially useful for localizing faults that cause an incorrect path to be executed or an incorrect number of iterations of a loop. Although we present the approach in the context of model-to-text transforms, it is applicable more generally in cases where the programs take large structured inputs and generate structured output, and where the goal of investigating a failure is to locate faults in the inputs. The contributions of the paper include – A novel dynamic-tainting-based approach for localizing input-model faults that cause model-transformation failures – The description of an implementation of the approach for XSL-based modelto-text transforms – The results of empirical evaluation, conducted using real subjects, which illustrate the effectiveness of the approach

2

Background and Illustration of the Problem

In this section, we briefly discuss model-to-text transforms and illustrate the problem using a sample transform. Model-to-text transforms are a special class of software applications that transform a complex input model into text-based files. Examples of such transforms include UML-to-Java code generators and XML-to-HTML format converters. A model-to-text transform can be coded using a general-purpose programming language, such as Java. Such a transform reads content from input files, performs the

30

P. Dhoolia et al.

Fig. 2. Illustration of input model faults, fault propagation through the transform, and resulting failures

transformation logic, and writes the output to a file as a text string. Alternatively, a transform can be implemented using specialized templating languages, such as XSLT (Extensible Stylesheet Language Transformation) and JET (Java Emitter Templates),2 that let developers code the transform logic in the form of a template. The associated frameworks—Xalan3 for XSLT and the Eclipse Modeling Framework (EMF)4 for JET—provide the functionality to read the input into a structured format and write the output to a text file. A model is a collection of elements (that have attributes) and relations between the elements. (We use the term entity to refer to either an element or an attribute.) A model is based on a well-defined notation that governs the schema and the syntax of how the model is represented as a physical file, and how the file can be read in a structured way. XML and UML are examples of commonly used notations to define a model. Figure 1 shows an example of a model defined using XML. The model contains instances of property elements. Each property has an attribute isGen and contains elements foo and bar. Figure 2 presents an intuitive illustration of the propagation of input-model faults through a transform, and the manifested failures. A fault can be a missing entity or an incorrect value of an entity. A missing entity can cause a wrong path to be traversed through the transform. An incorrect entity value can cause either a wrong path or the propagation of the incorrect value along a correct path. An incorrect path through the transform manifests as either a missing substring or an incorrect substring in the output. The propagation of an incorrect value through the transform results in an incorrect string or a missing string (in cases where the incorrect value is an empty string). To illustrate these scenarios using a concrete example, consider Figure 3, which elaborates upon the example from Figure 1. Figure 3(a) shows a sample transform, written using XSL, that generates name-value pairs from the model. Part (b) of the figure shows the transformation logic in the form of procedural psuedo-code that could be implemented using a general-purpose programming language. The transform iterates over each property element in the input model and, based on the value of isGen, writes name-value pairs to the output file. Part (c) of Figure 3 shows three faulty models and the generated incorrect outputs. The solid boxes highlight the faults, whereas the dashed boxes highlight the incorrect parts of the output. In the first model, element bar for the second 2 3 4

http://wiki.eclipse.org/M2T-JET http://xml.apache.org/xalan-j http://www.eclipse.org/modeling/emf

Debugging Model-Transformation Failures Using Dynamic Tainting

31

Fig. 3. Illustrative examples: (a) an XSL transform that generates name-value pairs; (b) the corresponding psuedo-code; (c) three faulty input models and incorrect outputs

property is empty. This causes a missing substring in the output: the second name-value pair has a missing value. During the failing execution, in the first iteration of the loop in line 1, the condition in line 2 evaluates true and the string name1=value1 is written to the output. In the second iteration of the loop, the condition evaluates true, but because element bar is empty in the input model, an empty string is written to the output at line 5. Thus, a missing value of an element in the input model causes an empty string to be propagated along a correct path, resulting in a missing substring in the output; this corresponds to path 2 → 4 → 5 in Figure 2. In the second faulty model, attribute isGen of the second property has an incorrect value, which causes an incorrect path to be taken: in the second iteration of the loop, the else-if branch is taken instead of the if branch. This results in an incorrect string in the output—NIL instead of name2=value2. This case corresponds to path 2 → 3 → 6 in Figure 2. In the third faulty model, the second property is missing attribute isGen. This causes an incorrect path to be taken through the transform: in the second iteration of the loop, both the if and the else-if branches evaluate false. The

32

P. Dhoolia et al.

Fig. 4. Overview of the approach

resulting output has a missing substring. This case corresponds to path 1 → 3 → 5 in Figure 2. In a large model that contains thousands of elements and attributes, locating such subtle faults can be very difficult. Next, we present our approach that guides the user in locating such input-model faults.

3

Dynamic Taint Analysis for Debugging

Figure 4 presents an overview of our approach. Given a transform program P and a failure-inducing input model I, the approach requires the user to identify error markers, which indicate the points in the output string at which a substring is missing or an incorrect substring is generated. Next, the approach instruments P to add probes—these probes associate taint marks with the elements of I and propagate the taint marks to track the flow of data from the elements of I to the output string. The execution of the instrumented transform on I generates a taint log, in which taint marks are associated with substrings of the output. Finally, we analyze the taint log and, using the information about the error markers, identify the fault space in I. In the rest of this section, we discuss the three main steps of the approach: (1) identification of error markers, (2) association and propagation of taint marks, and (3) analysis of taint logs. 3.1

Identification of Error Markers

The starting point for failure investigation is a relevant context [6], which provides information about where the failure occurs. In conventional fault localization, the relevant context is typically a program statement and the data that is

Debugging Model-Transformation Failures Using Dynamic Tainting

33

observed to be incorrect at that statement. In contrast, the relevant context in our approach is a location in the output string at which a missing substring or an incorrect substring (i.e., the failure) is observed. For a model-to-text transform, such a relevant context is appropriate because a transform typically builds the output text in a string buffer b that is printed out to a file at the end of the transformation. If the fault localization were to start at the output statement and the string buffer b as the relevant variable—for example, as the penumbra tool would do—the entire input model would be identified as the fault space. In our approach, the relevant context for fault localization is an error marker. An error marker is an index into the output string at which a substring is missing or an incorrect substring is generated. In most cases, the user would examine the output text and manually identify the error marker. However, for certain types of output texts, the error-marker identification can be partially automated. For example, if the output is a Java program, compilation errors can be identified automatically using a compiler; these errors can be used to specify the error marker. Similarly, for an XML output, error markers can be identified using a well-formedness checker. Identification of error markers can be complex. In some cases, a failure may not be observable by examining the output string: the failure may manifest only where the output is used or accessed in certain ways. In other cases, a failure may not be identifiable as a fixed index into the output string. In our current approach, we assume that the failure can be observed by examining the output string and that the error marker can be specified as a fixed index. Our focus in this paper is on the core analysis for fault localization; we leave it to future work to address the complexity in the identification of error markers. 3.2

Taint Association and Propagation

The approach associates taint marks with the input model. Taint marks can be associated at different levels of granularity of the input-model entities, which involve a cost-accuracy tradeoff. A finer-grained taint association can improve the accuracy of fault localization, but at the higher cost of propagating more taint marks. In our approach, we associate a unique taint mark with each model entity, from the root element down to each leaf entity in the tree structure of the input model. The top part of Figure 5 illustrates the taint associations for the three faulty input models of Figure 3. Each model element and attribute is initialized with a unique taint mark ti . Thus, the first two models have nine taint marks, whereas the third model has eight taint marks because the isGen attribute is missing in that model. During the execution of the instrumented transform, these taint marks are propagated to the output string through variable assignments, library function calls, and statements that construct the output string. Data-Taint, Control-Taint, and Loop-Taint Marks. A key feature of our approach is that, in addition to propagating taint marks at assignment and

34

P. Dhoolia et al.

Fig. 5. Taint associations with the three faulty input models and the output texts of the example from Figure 3

string-manipulation statements, our approach propagates taint marks at conditional statements.5 We classify such taint marks as control-taint marks, and distinguish them from data-taint marks, which are propagated at non-conditional statements. In addition, we propagate taints marks at looping constructs to scope out, in the output string, the beginning and end of each loop; we call these taint marks loop-taint marks. Intuitively, a control-taint mark identifies the input-model elements that affect the outcome of a condition in a failing execution E. Such taint marks assist with identifying the faults that cause an incorrect path to be taken through the transform code in E. At a conditional statement c, the taint marks {t} associated with the variables used at c are propagated to the output string and classified as control-taint marks. In the output string, the taints in {t} identify locations at which an alternative substring would have been generated had c evaluated differently (e.g., “true” instead of “false”) during the execution. (The controltaint marks are based on the notion of control-flow-based tainting [5]. We discuss related work further in Section 6.) A loop taint is a further categorization of control taints; it bounds the scope of a loop. Loop taints are useful for locating faults that cause an incorrect number of iterations of a loop. In cases where an instance of an iterating input-model element is missing and the user of the transform is able only to point vaguely to a range as an error marker, the loop bounds allow the analysis to identify the input-model element that represents the collection with a missing element. Figure 5 presents an intuitive illustration of the taint log that is generated by the execution of the instrumented transform. In the taint log, substrings (other 5

We use the term “conditional” to refer to the different language constructs that provide for conditional execution of statements, such as if statements, looping constructs, and switch statements.

Debugging Model-Transformation Failures Using Dynamic Tainting

35

than string literals) of the output string have taint marks associated with them, and each taint mark is classified as a data taint, a control taint, or a loop taint. Consider the taint log for the first faulty model, shown on the left in the figure. Data taint t4,d is associated with substring name1, which indicates that the name1 is constructed from the input-model element that was initialized with taint t4 (element foo of the first property). A data taint may be associated with an empty substring, as illustrated by t9,d . This indicates that element bar of the second property, which was initialized with t9 , is empty. A control taint has a scope that is bound by a start location and an end location in the output string. The scope of control taint t3,c indicates that name1=value1 was generated under the conditional c at which t3 was propagated to the output string; and, therefore, that the substring would not have been generated had c evaluated differently. In the psuedo-code shown in 3, c corresponds to the conditional in line 2. Also, attribute isGen of the first property was initialized with t3 ; thus, that attribute determined that name1=value1 was generated. A different value for that attribute could have caused conditional 2 to evaluate differently and, consequently, the generation of an alternative substring. A control taint may have an empty scope: this occurs when no output string is generated along the “taken branch” from a conditional. In the taint log for the third faulty model, control taint t6,c has an empty scope. This happens because in the second iteration of the loop in 3, the conditionals 2 and 7 evaluated false, and along the taken branch, no string was generated. Loop-taint mark t1,L scopes out the loop iterations; a control taint is generated for each iteration of the loop. To summarize, data taints are propagated at each assignment statement and each statement that manipulates or constructs the output string. At a conditional statement s that uses model entity e, the data taints associated with e are propagated, as control taints, to bound the output substring generated within the scope of s. Similarly, at a loop header L that uses entity e, the data taints associated with e are propagated, as loop taints, to bound the output string generated within the body of L. Computation of Control- and Loop-Taint Propagation Points. Controltaints have a scope, defined by a start index and an end index, in the output string. To propagate the start and end control-taints to the output string, our approach identifies the program points at which conditionals occur and the join points for those conditionals. For each conditional c, the approach propagates the taint marks associated with the variables used at c to the output string, and classifies the taint marks as control-taints. Similarly, it propagates the corresponding end control-taints before the join point of c. To explain the computation of control-taint propagation points, we provide some definitions. A control-flow graph (CFG) contains nodes that represent statements, and edges that represent potential flow of control among the statements; a CFG has a unique entry node, which has no predecessors, and a unique exit node, which has no successors. A node v in the CFG postdominates a node u if and only if each path from u to the exit node contains v. v is the

36

P. Dhoolia et al.

Fig. 6. Examples to illustrate the propagation of control taints: (a) CFG of the sample transform (Figure 3); (b) nonstructured if statement; (c) loop with break statement

immediate postdominator of node u if and only if there exists no node w such that w postdominates u and v postdominates w. A node u in the CFG dominates a node v if and only if each path from the entry node to v contains u. An edge (u, v) in the CFG is a back edge if and only if v dominates u. A node v is control dependent on node u if and only if v postdominates a successor of u, but does not postdominate u. A control-dependence graph contains nodes that represent statements and edges that represent control dependences: the graph contains an edge (u, v) if v is control dependent on u. A hammock graph H is a subgraph of / H such CFG G with a unique entry node he ∈ H and a unique exit node hx ∈ that: (1) all edges from (G − H) to H go to he , and (2) all edges from H to (G − H) go to hx [11]. Figure 6 illustrates the identification of control-taint propagation points. Part (a) of the figure shows the CFG for the sample transform of Figure 3; each hammock in the CFG is highlighted with a dashed bounding box. For if statement 2, a start control-taint, t3,c(start) , is propagated before the execution of the statement. The join point of statement 2 is statement 10, which is the immediate postdominator of statement 2. Therefore, a corresponding end control-taint, t3,c(end) , is propagated before node 10, along each incoming edge. Similarly, start control-taint t4,c(start) is propagated before the nested if statement. The immediate postdominator of this statement is also node 10. However, end control-taint t4,c(end) is propagated along incoming edges (7, 10) and (9, 10) only—and not along inedge (6, 10) because the start taint is not reached in the path to node 10 along that edge. If t4,c(end) were to be propagated along edge (6, 10), the path (entry, 1, 2, 3, 4, 5, 6, 10) would have no matching start taint for t4,c(end) . Our approach requires that, along each path in the CFG, the propagation of start and end control-taint marks must be properly matched: each start control-taint must have a corresponding end control-taint and each end control-taint must be preceded by a corresponding start control-taint.

Debugging Model-Transformation Failures Using Dynamic Tainting

37

For loop header 1, start loop-taint t1,L(start) and start control-taint t2,c(start) are propagated before the loop header; corresponding end taints (t1,L(end) and t2,c(end) ) are propagated before node 11, the immediate postdominator of node 1. In addition, control taints are also propagated along the back edge, which ensures that each iteration of the loop generates a new control-taint scope. Part (b) Figure 6 illustrates a nonstructured if statement: the nested if statement is nonstructured because its else block has an incoming jump from outside the block (through edge (2, 4)). For such if statements, start and end taint propagation can result in the taints not being properly matched along some path in the CFG. If t2,c(start) and t2,c(end) were propagated as shown in the figure, path (entry, 2, 4, 7) contains an unmatched end taint: t2,c(end) . To avoid such cases and ensure that may taints are properly matched along all paths, our approach performs taint propagation for only those conditionals that form a hammock graph. A hammock graph H has the property that no path enters H at a node other than he and no path exits H at a node other than hx . Therefore, propagating a start control-taint before he and an end controltaint before after each predecessor of hx guarantees that the control taints are properly matched through H. In the CFG shown in Figure 6(b), because the nested if statement does not form a hammock, no control-taint propagation is performed (shown as the crossed-out control-taints). Part (c) of Figure 6 shows a loop L with a break statement: node 3 represents a break statement that transfers control outside L. In this case, as illustrated, end control-taints need to be propagated along the edge that breaks out of the loop. Moreover, conditional statements within L that directly or indirectly control a break statement do not induce hammocks: e.g., if statement 2 does not form a hammock. For such statements, control taints need to be propagated appropriately, as illustrated in Figure 6(c). Because of space constraints, we omit the details of the algorithm for propagating the control taints. Similar to nonstructured if statements, a loop may be nonreducible: control may jump into the body of the loop from outside of the loop without going through the loop header [3]. Our analysis performs no control-taint propagation for such loops because matched control-taints cannot be created along all paths through the loop. 3.3

Analysis of Taint Logs for Fault Localization

The execution of the instrumented transform generates a taint log, in which substrings of the output string have taint marks associated with them. The third step of our approach analyzes the taint log to identify the fault space in the input model. Overall, the log analysis performs a backward traversal of the annotated output string, and iteratively expands the fault space, until the fault is located. To start the analysis, the user specifies an error marker and whether the error is an incorrect substring or a missing substring. The bottom part of Figure 5 shows the taint logs, the error markers, and the computed fault spaces for the three failure-inducing models of the sample

38

P. Dhoolia et al.

transform (Figure 3). The first and the third faulty models cause missing strings in the output, whereas the second faulty model causes an incorrect substring in the output. Missing Substrings. A failing transformation that results in a missing substring could be caused by the incorrect empty value of an element or attribute. The first faulty model in Figure 5 illustrates this. Alternatively, a missing substring could be caused by a wrong path through the transformation: i.e., a conditional along the traversed path could have evaluated incorrectly, which caused the substring to not be generated along the taken-path. The third faulty model in Figure 5 illustrates this. To compute the fault space for missing substrings, the log analysis identifies empty data taints and empty control taints, if any, that occur at the error marker, and forms the first approximation of the fault space, which consists of the inputmodel entities that were initialized with these taints. If the initial fault space does not contain the fault, the analysis identifies the enclosing control taints, starting with the innermost scope and proceeding outward, to expand the initial fault space iteratively, until the fault is located. For the first faulty model in Figure 5, the analysis identifies empty data taint t9,d and sets the initial fault space to contain element bar of the second property. Because the fault space contains the fault, the analysis terminates. Similarly, for the third faulty model, the analysis identifies empty control taint t6,c and sets the initial fault space to the second property element, which contains the fault. Thus, in both the cases, the analysis precisely identifies the fault in the first approximation of the fault space. Incorrect Substrings. An incorrect substring could be generated from the incorrect value of an input-model entity; alternatively, the incorrect string could be generated along a wrong path traversed through the transform. To compute the fault space for incorrect substrings, the log analysis identifies the data taint associated with the substring at the error marker. For the second faulty model in Figure 5, the analysis looks for data taints. Because no data taints are associated with the output string at the error marker, the analysis considers the enclosing control taint, t7,c , and adds the input-model element initialized with t7 to the fault space. This fault space contains the second property element; thus, the analysis identifies the fault. Summary. To summarize, for a missing substring, the log analysis starts at an empty data taint or an empty control taint, and computes the initial fault space. For an incorrect substring, the analysis starts at a non-empty data taint to compute the initial fault space. Next, for either case, the analysis traverses backward to identify enclosing control taints—in reverse order of scope nesting— and incrementally expands the fault space. The successive inclusion of control taints lets the user investigate whether a fault causes an incorrect branch to be taken at a conditional, which results in an incorrect string or a missing string at the error marker.

Debugging Model-Transformation Failures Using Dynamic Tainting

39

Fig. 7. Architecture of the implementation for XSL-based transforms

4

An Implementation of the Technique

We implemented the technique for XSL-based transforms. Figure 7 presents the architecture and flow of the implementation. The top part of the figure shows the process steps and the artifacts that are generated or transformed by each step. The middle part of the figure shows the components that we have implemented. These include: (1) a taint API that contains taint-initialization and taint-propagation methods; (2) an instrumentation component that adds probes to invoke control-tainting and loop-tainting methods; (3) an aspect-weaver component that weaves in aspects to the instrumented bytecode to invoke taintinitialization and data-tainting methods; and (4) an indexer component that sanitizes and indexes the raw taint log to make it appropriate for querying. The bottom part of the figure shows the external software that we use out-of-the-box. Note that the addition of probes that invoke tainting methods is split into two steps. In the first step, we use bytecode instrumentation to add calls to control- and loop-tainting methods. In the second step, we use aspects to add calls to data-tainting methods. For XSL-based transforms, data propagation occurs through calls to the Xalan library. Aspects provide an easy way to add instrumentation code around method calls, thereby removing the need to instrument the actual library code. Therefore, we selected aspects for data-taint propagation. However, AspectJ does not provide any join-points for conditionals; therefore, we performed direct bytecode instrumentation to propagate control and loop taints. Next, we discuss the five steps of the process. Transform Compilation. Because our analysis infrastructure is Java-based, we first compile the XSL transform into Java bytecode. We use Apache XSL transform compiler (xsltc)6 for this purpose. The xsltc compiler generates 6

http://xml.apache.org/xalan-j/xsltc

40

P. Dhoolia et al.

an equivalent bytecode program (called translet) for the XSL. This transform program can be executed using the xsltc runtime API. Bytecode Instrumentation. The instrumentation component of our implementation adds probes to the translet bytecode to propagate control and loop taints. The component consists of a taint-location analyzer and a bytecode instrumenter. The taint-location analyzer is developed using the wala analysis infrastructure.7 It uses wala to perform control-flow analysis and dominance/postdominance analysis. Using these, it identifies loops and loop back-edges8 and, for each conditional c, checks whether c is the entry node of a hammock graph. The analyzer identifies all taint-propagation locations according to the algorithm presented in Section 3.2. Each taint location is specified using a bytecode offset and information about what instrumentation action to perform at that offset. The instrumenter processes the taint locations, and uses bcel9 to add bytecode instructions and modify existing instructions. The instrumenter performs three types of actions: (1) add calls to the tainting methods; (2) redirect existing branch and goto instructions, and (3) add new goto instructions. The following code fragments illustrate these actions. 0: iload_1 1: istore_3 2: iload_2 3: ifne 9 6: iinc 3 1 9: return

0: 1: 2: 3: 6: 9: 12: 15: 16: 19: 22: 25:

iload_1 istore_3 iload_2 invokestatic markStartControlTaint()V // added invoke ifne 16 // redirected branch iinc 3 1 goto 22 // added goto return invokestatic markEndControlTaint()V // new jump target goto 15 invokestatic markEndControlTaint()V // new jump target goto 15

The fragment on the left shows the original bytecode (P ) that encodes an if-then statement; the fragment on the right shows the instrumented bytecode

(P  ), in which calls to tainting methods (from our taint API) have been added. In P  , at offset 3, a call to tainting method markStartControlTaint() has been added. In P , the if statement at offset 3 transfers control to offset 9, which is the end of the if-then block. In P  , the branch has been redirected to first invoke (at offset 16) the end control-taint method markEndControlTaint(), and then jump to the original target (offset 9 in P , offset 15 in P  ) of the branch. At the end of the then branch (offset 6 in P , offset 9 in P  ), a goto instruction has been added to ensure that the end control-taint method is called before control flows out of the then block. Aspect Weaving. The aspect-weaver component defines abstract aspects for taint initialization and data-taint propagation. We implement these abstract 7 8 9

http://wala.sourceforge.net Because the analysis is performed on bytecode, which encode loops using if and goto instructions, loop detection is based on the identification of back-edges. http://jakarta.apache.org/bcel

Debugging Model-Transformation Failures Using Dynamic Tainting

41

aspects by providing a set of specific point-cut definitions and corresponding advices. The advices invoke tainting methods from the taint API. The taintinitialization aspect, woven to the XML parser, assigns a unique taint mark to each element, and for each element, to each of its attributes and content. The point-cuts and advices of the data-taint-propagation aspect, are implemented based on an understanding of the general profile of transform programs generated by the xsltc compiler. Translet Execution. Next, we execute the fully instrumented translet (instrumented for taint initialization, data-taint propagation, and control-taint propagation) on the faulty input. We use the xsltc command-line API. The execution of the instrumented translet produces an annotated taint log. For a data-taint tag, the taint information contains either a taint mark, or an association to an intermediate variable created and used in the XSL transform. The taint information for a variable tag may itself contain either taint marks, or associations to other intermediate variables. A control-taint tag may contain a taint mark or an association to an intermediate variable, and/or the conditions. The condition tag may contain a taint mark or variable associations for both the left-hand and right-hand expressions of the conditional statement, along with the conditional operand. For loop constructs, the annotations contain just the loop tag. Taint Log Indexing. Finally, the indexer component sanitizes, analyzes, and indexes the taint-marks associations with the output substrings. It performs the two steps. First, we sanitize the taint the log to process it as an XML document. However, the actual output of the transform may either itself be an XML (leading to a possible interleaving of its tags with ours) or it may contain special characters (e.g., the greater-than comparison operator in an output Java program). Either of these cases can make the taint log an invalid xml. To avoid this, we sanitize the taint log by encapsulating all the actual output chunks between our tags as CDATA sections. In XML, a CDATA section is a section of element content that is marked for the parser to interpret as only character data, not markup. Second, the indexer analyzes and indexes the sanitized taint log. It uses JDOM10 and XML processing to traverse the sanitized taint log as an XML document. It processes the special CDATA sections, created during the sanitizing pass, sequentially in the order of their occurrence. It associates the parent taint element tags with the ranges of the output segments bounded within the CDATA sections. For the CDATA ranges associated with intermediate variables, the indexer keeps a temporary mapping of variables with taint marks, which it uses for resolving tainted ranges associated with the use of those variables. Further, based on the containment hierarchy of taint tags, a list of taint marks representing an iterative expansion of the fault space is indexed for relevant ranges in the output. Finally, the indexer provides an API on the taint index that supports queries for taint marks (or probable taint marks) associated with 10

http://www.jdom.org

42

P. Dhoolia et al. Table 1. Subjects used in the empirical studies XSLT Constructs used in the Subjects Subject Templates Loop Condition Data Variables ClsGen1 4 8 14 147 27 ClsGen2 19 16 94 519 103 IntGen1 2 2 5 16 9 IntGen2 6 3 40 101 15 PluginGen 7 9 48 107 52 PropGen 3 0 1 2 0 Total 41 28 202 892 206

Faulty Input Models 359 98 111 121 129 95 913

a position (or a range) in the output, with additional information about whether the output is missing or incorrect. An Apache Ant build script, which takes the XSL transform program and the input model as inputs, completely automates the entire process and enables a one-click execution of the process.

5

Empirical Evaluation

We conducted empirical studies to evaluate our technique. The goals of the studies were to investigate (1) the effectiveness of the technique in reducing the fault space in failure-inducing input models, (2) the importance of propagating control taints to localize faults, and (3) the cost of the technique. In this section, first, we describe the experimental setup; following that, we present the results of the three studies. 5.1

Experimental Setup

We used six XSL transform programs as our experimental subjects. Table 1 lists the subjects. These transforms are real programs that have been developed as part of research projects in IBM. Each transform takes as input a domainspecific Ecore EMF model, and generates different types of text output: ClsGen1 and ClsGen2 generate Java classes; IntGen1 and IntGen2 each generate Java interfaces; PluginGen generates an XML configuration file; and PropGen generates a properties file consisting of name-value pairs. Table 1 also lists different XSLT constructs used in the subjects, which indicates the complexity of the subjects. Columns 2–6 list, respectively, the counts of template (Column 2), for-each (Column 3), if, when, otherwise (Column 4), value-of (Column 5), and variable, param (Column 6). An XSLT template corresponds to a function in a procedural language context, variable represents a variable declaration, and the value-of construct corresponds to either a variable assignment or a write into the output string. As the data indicate, ClsGen2 is the most complex subject, whereas PropGen is the least complex. To generate failure-inducing input models, we used the technique of generating test inputs by applying mutation operators to existing test inputs (e.g., [9,18]). More specifically, we defined appropriate mutation operators for the input

Debugging Model-Transformation Failures Using Dynamic Tainting

43

models of our subjects, and generated faulty models (i.e., mutants) by applying the operators to a valid input model. Specifically, we used four mutation operators: (1) DeleteElement deletes a model element, (2) DeleteAttribute deletes a model attribute; (3) EnumerateAttribute modifies the value of an attribute based on a predefined enumeration of values for the attribute; and (4) EmptyAttribute sets an attribute to empty. Each faulty model was generated by applying one mutation operator; therefore, each faulty model had one fault only. We used two valid input models; the sizes of these models, measured as the total number of elements and attributes, are approximately 38,100 and 40,500. For each input, we generated a different set of mutants. Together, the two sets of mutants formed the total pool of mutants If , which we used to construct faulty input models I(f,P ) ⊆ If for each subject P . For each P and pair (i, if ), where i is a valid input and if is a mutant generated from i, we executed P on i and if and compared the outputs. If the outputs differed, we added if to I(f,P ) ; otherwise, we ignored if because it was an equivalent mutant with respect to P . The last column of Table 1 lists the size of I(f,P ) (the number of faulty input models) for each subject. Over the six subjects, there were a total of 913 faulty models. Of these, 30% of the faults were created by the application of the DeleteElement mutation operator; another 30% were created by the application of the DeleteAttribute operator; EnumerateAttribute and EmptyAttribute contributed to 14% and 26% of the faults, respectively. To automate the step of computing the error markers, we implemented a differencing component, filediff. filediff compares the output generated by a program P on a valid input i with the output generated by P on a faulty input if (derived from i) to identify the first point (line and column numbers) at which the two outputs differ. This point is specified as the error marker for the failing execution of P on if . 5.2

Study 1: Effectiveness of the Technique

Goals and method. The goal of the first study was to evaluate the effectiveness of our technique in reducing the fault search space. We define fault space reduction as: FSreduction =

FStotal − FSanalysis × 100 FStotal

FStotal is the total fault space (i.e., the size of the faulty input model); FSanalysis is the number of elements/attributes computed by the analysis (i.e., the subset of the faulty model that must be examined to identify the fault).11 The maximum reduction occurs when FSanalysis = 1, which depends on the type of fault or mutation operator. For the EnumerateAttribute and EmptyAttribute operators, FSanalysis = 1 if the analysis computes only the modified attribute. For the DeleteElement and DeleteAttribute operators, FSanalysis = 1 if the analysis computes only the parent element of the deleted element/attribute. The entire execution and data collection was automated using an Ant build script (as discussed in Section 4). We compiled each XSL subject program to 11

FSanalysis represents the complete fault space computed by the analysis, after performing all iterative expansions.

44

P. Dhoolia et al.

Fig. 8. Fault-space reduction attained by our approach. The chart on the left shows the percentage of faults for which different ranges of reduction were attained. The table of the right shows the absolute sizes of the fault spaces.

get the corresponding Java translet. We executed translet on the valid input i to compute the valid output Ovalid . Next, we instrumented the translet, weaved in aspects, and executed the modified translet on each input if ∈ I(f,P ) to compute the incorrect output Oi,incorrect . We used filediff compute the error marker for Oi,incorrect , and then queried the taint indexer to compute FSanalysis . Finally, we determined whether the fault in if occurred in FSanalysis (which was the case for all the faults) and computed FSreduction . Results and analysis. Figure 8 presents data to illustrate the fault-space reduction achieved by our technique: the segmented-bar chart on the left shows the percentage fault reduction, whereas the table on the right shows the absolute sizes of the fault spaces. For each subject, the chart contains a segmented bar, in which the segments represents different ranges of FSreduction . The vertical axis represents 100% of the faulty models for a subject; the number at the top of a bar is the number of faulty models for that subject. The data demonstrate that our approach can be highly effective in reducing the fault space. For example, for ClsGen1, the maximum reduction was attained for 56% of the faults, a reduction of 98% or better was attained for another 26% of the faults; for the remaining faults, a reduction of at least 94% was achieved. For four of the subjects, the approach attained the maximum reduction for at least 50% of the faulty inputs in those subjects. Over all subjects, the technique accomplished the maximum reduction for 468 (51%) of the 913 faulty models, and at least a 98% reduction for another 378 (41%) of the faults. The table on the right in Figure 8 presents a different view of the faultreduction data: it shows the number of faults for which the fault-space sizes were in the different ranges. For example, consider ClsGen1. For 222 of the 359 faulty inputs, FSanalysis = 1; for 16 of the faults, FSanalysis is between 2 and 100;

Debugging Model-Transformation Failures Using Dynamic Tainting

45

Fig. 9. Data to illustrate the significance of using control taints for fault localization

for another 41 faults, FSanalysis is between 101 and 500; and for the remaining 80 faults, FSanalysis contains more than 500 entities. The data show that the sizes of the fault spaces can be quite large. However, this should be interpreted keeping in view the sizes of our input models, which are very large: the faulty models derived from one valid model contain about 38,100 elements and attributes, whereas the faulty models derived from the other input model contain about 40,500 elements and attributes. Thus, although a fault space of 500 elements/attributes appears quite large, it is substantially smaller than the complete fault space, which is what the user might have to examine to identify the fault, if no fault-space reduction were performed. 5.3

Significance of Control-Taint Tags

Goals and Method. One of the key features of our approach is the propagation of control taints. The goal of the second study was to evaluate the importance of using control taints. In general, propagation of control taints is necessary to enable the approach to compute the fault space (FSanalysis ), conservatively— i.e., to ensure that FSanalysis contains the fault. Stated differently, exclusion of control taints can cause FSanalysis to not contain the fault. Thus, to measure the significance of control taints, we collected data about the percentage of faulty models for which FSanalysis would not contain the fault if control taints were not propagated. Results and Analysis. Figure 9 presents the data to illustrate the importance of control taints. The figure contains one segmented bar per subject. The segments indicate the percentage of faulty models for which control taints were necessary for locating the fault, and the remaining faulty models for which data taints alone were sufficient for locating the fault. As the data illustrate, for a substantial percentage of the faulty inputs, control taints were necessary for locating the faults. For example, for ClasGen1, control taints were required for locating the faults for 80% of the faulty inputs; data taints were sufficient for only 20% of the faults. For IntGen1, none of the faults could be localized using data

46

P. Dhoolia et al.

Table 2. Cost of the approach in terms of the execution and instrumentation overheads Average Execution Time (s) Subject Original Instrumented × Increase Original ClsGen1 3.62 28.5 8 3722 ClsGen2 3.2 91.88 28 13270 IntGen1 1.75 28.73 16 938 IntGen2 2.02 18.04 9 3264 PluginGen 1.71 12.71 7 5888 PropGen 0.22 1.03 5 376

Translet Instructions With control With control taints and data taints 3911 15783 14120 58326 999 2644 3521 14129 6359 12013 404 698

taints alone—all the faults required control-taint propagation. Data taints were most effective for PropGen, for which 41% of the faults could be localized without control taints. Over all subjects, 83% of the faults required the propagation of control taints, and only 17% could be localized using data taints alone. The data indicate that for debugging model transforms, propagation of control taints is essential. A technique that does not perform control-taint propagation can very often fail to compute the fault space conservatively, and may be of limited value. 5.4

Cost of Analysis

Goals and Method. In the final study, we evaluated the cost of the technique in terms of the execution overhead incurred because of the instrumentation for propagating data and control taints. (The instrumented code refers to the translet code after bcel instrumentation and aspect weaving.) We measured the average execution time over all the faulty input models on the original translet and the instrumented translet. We also measured the instrumentation overhead in terms of the sizes of the original and the instrumented translets. Results and Analysis. Table 2 presents data to illustrate the cost of our technique. It shows the average execution times (in seconds), and the increase in the sizes of the translets (measured as bytecode instructions) caused by instrumentation. As might be expected, code instrumentation incurs a heavy cost: the average increase in execution time varies from five times to 28 times. In absolute values, the execution times are reasonable for our subjects and acceptable for on-line execution of the instrumented transform within an interactive debugging environment. For longer-running transforms, this may not be practical. However, with appropriately designed usage scenarios, the execution overhead would not be a mitigating factor that limits the use of our approach. In practice, the instrumented code can be executed off-line or as a background process to generate the taint logs, which can then be queried in an interactive debugging environment. Columns 5–7 of the table show the increase in the number of bytecode instructions in the translet after the addition of probes for propagating control and data taints. As the data illustrate, the addition of probes for control-taint propagation causes a moderate increase in the number of bytecode instructions:

Debugging Model-Transformation Failures Using Dynamic Tainting

47

on average, less than 4% increase. However, the weaving of aspects for datataint propagation causes a significant increase in the number of instructions. For example, the number of instructions increases about four times for ClsGen1, ClsGen2, and IntGen2. 5.5

Threats to Validity

Our evaluation shows that, for the subjects and faulty inputs that we studied, our approach for fault localization can be very effective in reducing the fault search space in the input models. Moreover, our results indicate that the propagation of control taints is essential for conservative fault localization. Unlike the results of Clause and Orso [6] which did not indicate any significant benefit accruing from control-flow-based taint propagation, our results suggest that, in the domain of model transforms, control taints have a significant role to play; therefore, an approach that does not perform such propagation would be little value. Our results are preliminary and there are several threats to the validity of our observations. Threats to external validity arise when the results of the experiment cannot be generalized to other situations. Our evaluation is limited to six subjects, and thus, we cannot conclude whether our results might hold for general transforms. However, our transforms are varied in that they generate different types of outputs (code, properties, and configuration files), which gives us confidence that the results might apply to different types of model-to-text transforms. Another threat to validity is the representativeness of the faulty input models. We used mutation analysis to generate the faulty models, which may not represent the types of faults that occur frequently in practice; moreover, each faulty model contained only one fault. However, based on our experience with developing transforms, we designed our mutation operators to capture the commonly occurring faults. This threat is further mitigated by the fact that different types of faults were well-represented in our set of faulty models. Threats to internal validity arise when factors affect the dependent variables without the researchers’ knowledge. In our case, our implementation and data collection could have flaws that would affect the accuracy of the results we obtained. To alleviate this threat, we manually verified a random sampling of the results. Construct threats to validity arise when the measures do not adequately capture the concepts they are intended to measure. Our FSreduction measure is defined such that, for a deleted element, maximum reduction occurs if the parent element of the deleted element is identified by the analysis. This is somewhat coarse-grained because the user might have to examine several elements in the parent element to identify the missing element. However, we think that such a choice is a reasonable one. In theory, the most precise solution would point out exactly the missing element; in practice, this may not be possible in every case. We leave it to future work to investigate ways to improve the accuracy of our approach for faults that involve deletions of elements.

48

6

P. Dhoolia et al.

Related Work

There is much research in the area of fault localization. However, most of the existing research focuses on identifying faults in the program. The goal of these techniques is to narrow down the search space of program statements that should be examined to locate the fault. The most widely studied among these are based on dynamic program slicing [15]. A dynamic slice, computed with respect to an input i, a set of variables V , and a statement occurrence s in the execution trace induced by i, identifies the statements that affect the value of the variables in V on execution against i. In addition to investigating the use of dynamic slicing, in its basic form, for debugging [1], researchers have presented many extensions to, and variants, of dynamic slicing (e.g., [2,8]). Slicing-based techniques do not trace the incorrect output values, observed in a failing execution, back to program inputs; therefore, they are inappropriate for model debugging. Dynamic tainting [5], which tracks the flow of data from inputs to outputs during an execution, naturally fits the requirements of model debugging. Dynamic tainting has been explored extensively for identifying security vulnerabilities (e.g., [12,17]). More recently, Clause and Orso [6] investigated the use of dynamic tainting for identifying a failure-relevant subset of a failure-inducing input. Their work is the one that is most closely related to ours. Given information about the program statement and data involved in a failure, their technique performs taint propagation over data and control flow to identify the subset of input that is relevant for investigating the failure. We apply dynamic-tainting-based debugging to a new domain—i.e., model transformation—and our goal is to identify faults in the transform inputs. To apply dynamic tainting effectively in this domain, we made several adaptations and extensions to Clause and Orso’s technique. In our approach, the relevant context is a location in the output string of a modelto-text transform, which differs from the starting point of failure investigation of existing techniques. Our control taints are an adaptation of “taint propagation over control flow,” discussed by Clause and Orso (and also discussed as “implicit information flow” by Masri and colleagues [16]). The distinguishing aspect of our use of control taints is that we formulate control taints as first-class entities, and we track the nesting scope of control taints (including empty scopes). This lets our approach perform an incremental expansion of the fault space and identify faults that cause the execution of an incorrect path through the transform. The empirical results presented by Clause and Orso did not illustrate any substantial gain from propagating taints over control flow. In contrast, our results indicate that for model-to-text transforms, tracking control taints is absolutely essential. Model-tracing techniques [7] infer links between input-model and outputmodel entities. Recent work in the areas of traceability metamodels [10], modelcomparison languages [14], and higher-order transformations [22] can be applied to analyze model transforms and trace backward from error markers to inputmodel entities. However, tracing techniques can support fault localization only in cases where an incorrect value of an input-model entity flows to the output through value propagation; thus, traceability links provide a capability similar to

Debugging Model-Transformation Failures Using Dynamic Tainting

49

what the data taints provide in our approach. However, in cases where the fault is a missing model entity or the failure is a missing output substring, tracing techniques cannot be applied because no links are created. Moreover, for faults that cause the execution of an incorrect path, traceability links cannot guide the user to the relevant input-model entities. Delta debugging [23] is a technique that identifies a minimal subset of a failureinducing input i that also causes the failure: i.e., removing this subset would cause the failure to not occur. Delta debugging repeatedly executes the program with different subsets of i until it finds such a subset. In cases where the fault is a missing input-model element (which is common for model transforms), delta debugging would fail to identify a minimal subset because no subset causes the failure to not occur—the failure would not be observed only if the missing element were to be added to the input model; and delta debugging performs no augmentation of the input. Additionally, delta debugging has efficiency issues as observed in Reference [6] and requires a passing execution. Other classes of fault-localization techniques compare the spectra of passing and failing test executions (e.g., [13,19]), or create program variants via statement removal [21], to identify suspicious statements. These approaches are not applicable to debugging model-transformation failures where the goal is to identify faults in the input models and not in the transforms.

7

Conclusions and Future Work

In this paper, we presented an approach for assisting transform users with debugging their input models. Unlike conventional fault-localization techniques, our approach focuses on the identification of input-model faults, which, from the perspective of transform users, is the relevant debugging task. Our approach uses dynamic tainting to track information flow from input models to the output text. The taints associated with the output text guide the user in incrementally exploring the fault space to locate the fault. A novel feature of our approach is that it distinguishes between different types of taint marks (data, control, and loop), which enables it to identify effectively the faults that cause the traversal of incorrect paths and incorrect number of loop iterations. Our empirical studies, conducted using an implementation for XSL-based transforms, indicate that the approach can be very effective in reducing the fault space substantially. There are several areas of future work that we plan to conduct that will let us perform additional studies, including user studies, to assess the benefits of our approach. Our current implementation analyzes XSL-based transforms. Extensions to accommodate other types of model-to-text transforms, such as JET-based transforms, and even general-purpose programs (for which the goal of debugging is to locate faults in inputs) would enable us to evaluate the wider applicability of our approach. Currently, our debugging approach focuses on fault localization. A useful extension of our approach would be to support fault repair as well. Such a technique could recommend fixes by performing pattern analysis on taint logs collected for

50

P. Dhoolia et al.

model elements that generate correct substrings in the output text. Another technique (applicable for missing substrings) could be to force the execution of not-taken branches in the transform to show to the user potential alternative strings that would have been generated had those paths been traversed. Our approach assumes that the transform user can identify the error markers in the output string. However, identifying the error markers can be challenging in some cases. For example, the failure may not be obvious by examining the output—e.g., the failure may manifest only when the output is used in other computations. In such cases, an approach that helps the user trace back from the observed failure to an error marker in the output text would be useful. Often in MDE, a sequence of transforms are chained together such that the user-provided input goes through multiple transformations before generating an output. It would be interesting to explore how the proposed approach can be extended to debug inputs for chained transformations. Acknowledgements. We would like to thank the anonymous reviewers for helpful comments, which have improved the presentation of the work.

References 1. Agrawal, H., DeMillo, R.A., Spafford, E.H.: Debugging with dynamic slicing and backtracking. Software—Practice and Experience 23(6), 589–616 (1993) 2. Agrawal, H., Horgan, J.R., London, S., Wong, W.E.: Fault localization using execution slices and dataflow tests. In: Proc. of the Intl. Symp. on Softw. Reliability Eng., pp. 143–151 (1995) 3. Aho, A.V., Sethi, R., Ullman, J.D.: Compilers, Principles, Techniques, and Tools. Addison-Wesley Publishing Company, Reading (1986) 4. Baxter, I.D.: Design maintenance systems. ACM Commun. 35(4), 73–89 (1992) 5. Clause, J., Li, W., Orso, A.: Dytan: A generic dynamic taint analysis framework. In: Proc. of the Intl. Symp. on Softw. Testing and Analysis, pp. 196–206 (2007) 6. Clause, J., Orso, A.: Penumbra: Automatically identifying failure-relevant inputs using dynamic tainting. In: Proc. of the Intl. Symp. on Softw. Testing and Analysis, pp. 249–259 (2009) 7. Czarnecki, K., Helsen, S.: Classification of model transformation approaches. In: Proc. of the OOPSLA 2003 Workshop on Generative Techniques in the Context of Model-Driven Architecture (2003) 8. DeMillo, R.A., Pan, H., Spafford, E.H.: Critical slicing for software fault localization. In: Proc. of the Intl. Symp. on Softw. Testing and Analysis, pp. 121–134 (1996) 9. Dinh-Trong, T., Ghosh, S., France, R., Baudry, B., Fleury, F.: A taxonomy of faults for UML models. In: Proc. of the 2nd Workshop on Model Design and Validation (2005) 10. Drivalos, N., Kolovos, D., Paige, R., Fernandes, K.: Engineering a DSL for software traceability. Software Language Engineering, 151–167 (2009) 11. Ferrante, J., Ottenstein, K.J., Warren, J.D.: The program dependence graph and its use in optimization. ACM Trans. Progr. Lang. Syst. 9(3), 319–349 (1987) 12. Halfond, W., Orso, A., Manolios, P.: Using positive tainting and syntax-aware evaluation to counter SQL injection attacks. In: Proc. of the ACM SIGSOFT Symp. on the Foundations of Softw. Eng., pp. 175–185 (November 2006)

Debugging Model-Transformation Failures Using Dynamic Tainting

51

13. Jones, J.A., Harrold, M.J., Stasko, J.: Visualization of test information to assist fault localization. In: Proc. of the 24th Intl. Conf. on Softw. Eng., pp. 467–477 (May 2002) 14. Kolovos, D.: Establishing correspondences between models with the epsilon comparison language. In: Model Driven Architecture-Foundations and Applications, pp. 146–157 15. Korel, B., Laski, J.: Dynamic program slicing. Information Processing Letters 29(3), 155–163 (1988) 16. Masri, W., Podgurski, A., Leon, D.: Detecting and debugging insecure information flows. In: Proc. of the Intl. Symp. on Softw. Reliability Eng., pp. 198–209 (2004) 17. Newsome, J., Song, D.X.: Dynamic taint analysis for automatic detection, analysis, and signature generation of exploits on commodity software. In: Proc. of the Network and Distributed System Security Symp. (2005) 18. Offutt, J., Xu, W.: Generating test cases for web services using data perturbation. Softw. Eng. Notes 29(5), 1–10 (2004) 19. Renieres, M., Reiss, S.P.: Fault localization with nearest neighbor queries. In: Proc. of the 18th Intl. Conf. on Automated Softw. Eng., pp. 30–39 (2003) 20. Schmidt, D.C.: Model-driven engineering. IEEE Computer 39(2), 25–31 (2006) 21. Sterling, C.D., Olsson, R.A.: Automated bug isolation via program chipping. Software—Practice and Experience 37(10), 1061–1086 (2007) 22. Van Gorp, P., Keller, A., Janssens, D.: Transformation language integration based on profiles and higher order transformations. In: Gaˇsevi´c, D., L¨ ammel, R., Van Wyk, E. (eds.) SLE 2008. LNCS, vol. 5452, pp. 208–226. Springer, Heidelberg (2009) 23. Zeller, A., Hildebrandt, R.: Simplifying and isolating failure-inducing input. IEEE Trans. Software Eng. 28(2), 183–200 (2002)