Abstract: In this paper we present an approach that relies on the representation of the debugging problem as a constraint satisfaction problem. We show how ...
On the use of Specification Knowledge in Program Debugging Mihai Nica ∗ J¨ org Weber ∗ Franz Wotawa ∗ ∗ Technische Universit¨ at Graz Institute for Software Technology 8010 Graz, Inffeldgasse 16b/2, Austria Tel: +43 316 873 5724, Fax: +43 316 873 5706 {mihai.nica,jweber,wotawa}@ist.tugraz.at
Abstract: In this paper we present an approach that relies on the representation of the debugging problem as a constraint satisfaction problem. We show how arbitrary sequential programs together with specification knowledge like invariants can be represented as a set of constraints. We provide a formalization of the constraint representation and of the diagnosis problem which clarifies and improves previously published ideas. We also correct a flaw in a previously published paper which could lead to undesired diagnostic results. Moreover, based on the constraint representation we explain how to use a fast constraint solver, which is freely available, for computing diagnoses. Using such a constraint solver we obtain first experimental results for a set of Java programs. The results indicate that the use of specification knowledge reduces the number of diagnoses substantially. This shows that the integration of specifications like pre- and postconditions or invariants, which can be provided as annotations in the program, can significantly help to make model-based debugging applicable in practice. 1. INTRODUCTION
ever, those works did not provide any experimental results which confirmed this claim.
The use of model-based diagnosis (Reiter [1987], de Kleer et al. [1987]) for fault localization in programs has been an active research area for more than one decade (see, e.g., Friedrich et al. [1999], Stumptner et al. [1999], K¨ob et al. [2006]). Recent results indicate that model-based debugging can be effectively used and provides comparable or in many cases even more accurate results than other approaches for programs of smaller size (Mayer [2007]). In order to bring model-based debugging to a stage where it can be used in practice there are still open research problems to be solved. One problem, which we are not tackling in this paper, are performance issues; in particular, the necessary running time for computing diagnoses still prevents the approach from being used in an interactive manner, especially in cases where programs of larger size are considered. A solution to this problem might be the use of appropriate abstraction mechanisms.
In order to integrate specification knowledge into modelbased debugging we rely on a constraint representation of programs. The debugging problem is reduced to a constraint satisfaction problem. The constraint representation is able to express the full semantics of the specification knowledge, and it allows us to use a state-of-the-art constraint solver for computing the diagnoses. Note that constraint representations of programs have been proposed in literature for different purposes like verification (e.g., Collavizza et al. [2007]) and also for debugging (Ceballos et al. [2006], Wotawa et al. [2008]).
In this paper we focus on improving the precision of the diagnostic results. Debugging which is based on a single test case often exhibits a bad discrimination among diagnosis candidates; i.e., a large number of program statements cannot be eliminated as potential fault candidates. Two important approaches for tackling this problem are the integration of multiple test cases (see Peischl et al. [2008]) and the utilization of additional specification knowledge. Specifications can be provided by, e.g., assertion statements in the program, pre- and postconditions, or class and loop invariants. Previous papers suggested that the use of specification knowledge may improve the results of model-based debugging by reducing the number of diagnoses or the number of statements considered as possibly faulty (Ceballos et al. [2006], Nica et al. [2008]). How This research has been funded in part by the Austrian Science Fund (FWF) under grant P20199-N15. Authors are listed in alphabetical order. We thank Iulia Moraru for her support.
In this paper we provide a formalization of the constraint representation and of the diagnosis problem which clarifies and improves previously published ideas. The formalization also describes the integration of specification knowledge. Moreover, note that we correct a flaw in a previously published paper which could lead to undesired diagnostic results (Nica et al. [2008]). Based on the resulting constraint representation we further describe the representation of the model in the language used by the constraint solver MINION (Gent et al. [2006]), and we show how MINION can be directly used to compute diagnoses of a given cardinality. This can be done by counting the fault assumptions during computation and by specifying a constraint which restricts the number of fault assumptions to a fixed value. We also present first empirical results which compare the precision of the diagnostic results with and without specification knowledge. For this purpose we used a set of programs which implement simple arithmetic operations, and we defined assertions like invariants, pre- and postconditions for those programs. We manually injected bugs into the programs, and we found test cases which are not satisfied by the faulty programs. The obtained empirical results are very promising. Using specification knowledge we were able to further reduce the number of statements
to be considered as possibly faulty by about 20% to 60% in the single-fault case. The average reduction was about 30%. The results are even better in the double-fault case. The paper is organized as follows. We start with a brief description of how to obtain an intermediate representation for an arbitrary sequential program which is suitable for the conversion to a constraint representation. Then we formalize the constraint model and the corresponding diagnosis problem, and we show how to perform fault localization based on the constraint representation. Afterwards, we integrate assertions, in particular loop invariants, into our approach. Based on this general constraint model we further describe its representation in the language used by the constraint solver MINION, and we present first empirical results. Finally we provide a conclusion.
Single Assignment (SSA) form, which is an intermediate representation of the program with the property that no two left-side variables have the same name (see Cytron et al. [1991], Brandis et al. [1994], Wegman et al. [1991]). This is achieved by replacing each left-side variable with a new variable whose name is composed of the name of the original variable plus a unique index as suffix, see Fig. 3. In order to obtain the SSA form of a program it is also necessary to convert loops and conditional statements. As loops are, in our approach, represented by nested ifstatements, we only need to consider the conversion of conditional statements of the form if(condexpr) then {...} else {...}
2. BACKGROUND: CONVERSION OF SEQUENTIAL PROGRAMS INTO A LOOP-FREE SSA FORM
Note that the notation xexpr denotes a whole expression rather than a single variable. In brief, this conversion works as follows:
Our work addresses sequential programs with a syntax and semantics similar to well-known languages like Java, but without object-oriented constructs. For simplicity we do not consider procedure calls in this paper, but it should be noted that procedures can be straightforwardly integrated as shown in (Wotawa et al. [2008]). Our approach supports assignments statements, conditional statements, and loops. Figure 1 depicts a program which serves as running example throughout this paper. int power(int a, int exp) 1. int e = exp; 2. int res = 1; 3. while (e > 0) { 4. res = res * a; 5. e = e - 1; }
(1) The value of the evaluated condition condexpr is stored in a new boolean variable cond i, where i is a unique index. (2) The if- and the else-branches are converted separately. For both branches, new variables with unique indexes are introduced. The statements of both branches are concatenated; i.e., the program in SSA form will execute the statements of either branch in every run. (3) New variables are added which have the value of the corresponding variables in the original program after the execution of the if- or else-branch, respectively. The values of these new variables depend on the indexed variables which were introduced for the branches and on the boolean condition condexpr. For the evaluation of those values we define the Φfunction:
Fig. 1. A program for computing aexp , where a and exp are integers. Variable res denotes the result. As loops cannot be directly converted to our constraint representation, past works have proposed to unroll loops, i.e., to create a loop-free program by replacing the loop of the original program by a set of nested if-statements (e.g., see Ceballos et al. [2006] and Nica et al. [2008]). If the maximum number of iterations is known in advance, then the loop-free program is equivalent to the original program. Note that program debugging is based on one or more test cases, and executing the program for those test cases yields the maximum number of iterations in the faulty program for the given test suite. A small number of iterations is often sufficient for locating the fault(s) (see Jackson [2006]). Figure 2 shows the loop-free version of our example program for 2 iterations. Our constraint representation requires that all left-side variables in the program have unique names, i.e., each variable should be defined only once. Hence, we use the Static int power loopfree(int a, int exp) 1. int e = exp; 2. int res = 1; 3. if (e > 0) { 4. res = res * a; 5. e = e - 1; 6. if (e > 0) { 7. res = res * a; 8. e = e - 1; } } Fig. 2. The loop-free version of the program in Fig. 1 for 2 iterations.
def
Φ(v j, v k, cond i) =
v j if cond i = true v k otherwise
For example, the corresponding SSA form of the program fragment if(condexpr) {x = e1expr ;} else {x = e2expr;} is given as follows: cond i = condexpr; x j = e1expr ; x k = e2expr ; x l = Φ(x j, x k, cond i); The SSA representation for the program in Fig. 1 is depicted in Fig. 3. A noteworthy statement is S6 : 6. bool cond 1 = cond 0 ∧ (e 1 > 0); The variable cond 1 is true iff e > 0 holds in the original program after the first loop iteration, i.e., cond 1 is true iff a second loop iteration is executed. This is the case iff e 0 > 0 and e 1 > 0 holds. Obviously, the conversion of the loop-free program into the SSA form has, apart from variable renamings, no influence on the actual program behavior. It can also be seen that the SSA form of a sequential program comprises only assignments of the general form v = Eexpr where v is a variable and Eexpr is either an expression in the syntax of the sequential language or it is an expression of the form Φ(...).
int power SSA(int a, int exp) 1. int e 0 = exp; 2. int res 0 = 1; 3. bool cond 0 = (e 0 > 0); 4. int res 1 = res 0 * a; 5. int e 1 = e 0 - 1; 6. bool cond 1 = cond 0 ∧ (e 1 > 0); 7. int res 2 = res 1 * a; 8. int e 2 = e 1 - 1; 9. int res 3 = Φ(res 2, res 1, cond 1); 10. int e 3 = Φ(e 2, e 1, cond 1); 11. int res 4 = Φ(res 3, res 0, cond 0); 12. int e 4 = Φ(e 3, e 0, cond 0); Fig. 3. The loop-free SSA form of the program in Fig. 1 for 2 iterations. Variable res 4 is the output of the program (i.e., the final result). 3. FAULT LOCALIZATION BASED ON A CONSTRAINT REPRESENTATION We introduce some useful notations which we need for the subsequent definition of our constraint representation: Definition 1. (σ(S) - Mapping original program ↔ SSA). A program is a sequence of statements. Let Π be the original program in the sequential programming language, and let Σ(Π) denote the program resulting from the loop-unrolling and SSA-conversion of Π. Moreover, let ΣΦ (Π) ⊆ Σ(Π) be the loop-free SSA form without those statements which contain Φ. We define a (total) function σ which maps every statement S ∈ ΣΦ (Π) backwards to its corresponding statement S ∈ Π: σ : ΣΦ (Π) → Π
where V is a set of variables, each variable v ∈ V has a domain D(v), and CO is a set of constraints. Each constraint defines a relation between variables. A solution of a CSP assigns values to all variables s.t. all constraints are satisfied. For more details regarding CSPs we refer to Dechter [2003]. Definition 2. (CSP (Π) - Constraint represenation of Π). The constraint representation CSP (Π) of a sequential program Π is a tuple (V, D, CO) with: • V = V AR Σ(Π) ∪ {ab(S) | S ∈ Π} where · V AR Σ(Π) is the set of all variables in Σ(Π). · ab(S) denotes a single boolean variable stating whether statement S of the original program is abnormal (i.e., faulty). • the domain D(v) of a variable v ∈ V AR Σ(Π) is equivalent to the datatype of the variable in the program, and D ab(S) = {true, f alse}. • CO comprises exactly one constraint for every statement in the SSA form. CO is created as follows: · For every S ∈ Σ(Π): (1) if S has the form v = Φ(. . .): add the relation v = Φ(. . .) to CO. (2) otherwise, S has the form v = Eexpr and Eexpr does not contain Φ: add the relation ab(S) ∨ v = Eexpr to CO, with S = σ(S ).
More precisely, S has the general form v = Eexpr , and we distinguish two cases:
Clearly, a constraint of the form ab(S) ∨ (v = Eexpr ) could also be written as ¬ab(S) → (v = Eexpr ).
(1) if Eexpr corresponds to the loop condition of a whilestatement Sw ∈ Π, then σ(S ) returns Sw . (2) otherwise: Eexpr corresponds to the right-hand expression of an assignment statement Sa ∈ Π, and σ(S ) returns Sa .
Example The following constraint representation is extracted from the SSA form in Fig. 3:
We also define a (total) function for the forward mapping: σ : Π → 2 which returns for every S {S | σ(S ) = S}.
ΣΦ (Π)
∈ Π the set σ(S)
def
=
Example We consider the power(a, exp) program which we introduced in Sec. 2. Π comprises 5 statements S1 , . . . , S5 (see Fig. 1), Σ(Π) contains 12 statements S1 , . . . , S12 (see Fig. 3), and ΣΦ (Π) = S1 , . . . , S8 . We obtain σ(S1 ) = S1 , . . . , σ(S5 ) = S5 , σ(S6 ) = S3 , σ(S7 ) = S4 , σ(S8 ) = S5 . Moreover, σ(S5 ) = {S5 , S8 }, etc. 2 An important point is that the set ΣΦ (Π) comprises exactly those statements in the SSA form which our approach may consider as faulty, whereas the Φ-statements cannot be faulty. As a statement in the original program may correspond to several statements in the SSA form, a single fault in the original program may lead to multiple faulty statements in the SSA form. Considering the original program, our approach is able to deal (a) with faults in the boolean expressions of loop conditions and (b) with faults in the expressions on the right side of assignment statements. Now we have the prerequisites needed for defining the constraint representation of a sequential program. A constraint satisfaction problem (CSP) is a tuple (V, D, CO),
• V = {a, exp, e 0, . . . , e 4, res 0, . . . , res 4, cond 1} ∪ {ab(S1 ), . . . , ab(S5 )} • D(a) = Z, D(cond 0) = {true, f alse}, etc. • constraints: ⎧ ab(S1 ) ∨ (e 0 = exp), ⎪ ⎪ ⎪ ab(S2 ) ∨ (res 0 = 1), ⎪ ⎪ ⎪ ⎪ ab(S3 ) ∨ cond 0 = (e 0 > 0) , ⎪ ⎪ ⎪ ⎪ ab(S4 ) ∨ (res 1 = res 0 ∗ a), ⎪ ⎪ ⎪ ⎨ ab(S5 ) ∨ (e 1 = e 0 − 1), CO = ab(S 3) ∨ ⎪ ⎪ cond 1 = (cond 0 ∧ (e 1 > 0)) , ⎪ ⎪ ⎪ ⎪ ab(S4 ) ∨ (res 2 = res 1 ∗ a), ⎪ ⎪ ⎪ ⎪ ab(S5 ) ∨ (e 2 = e 1 − 1), ⎪ ⎪ ⎪ ⎪ res 3 = Φ(res 2, res 1, cond 1), ⎩ ...
cond 0,
[S1 ] [S2 ] [S3 ] [S4 ] [S5 ]
⎫ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎬
[S6 ] [S7 ] [S8 ] [S9 ]
⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎭ 2
The debugging process employs failed test cases for computing the diagnoses. A test case is a specification of the correct program behavior. It states the correct output for a given input. When the program does not compute the expected output, then the test case has failed, and the program must contain at least one faulty statement. In our context a test case is defined as follows: Definition 3. (Test case T ). A test case T is a set of constraints which assign values to those variables in the SSA form which correspond to input and output variables in the original program (i.e, variables which represent values passed to and values returned from the program, respectively). Every constraint τ ∈ T has the form
v in = value or v out = value where v in ∈ V AR Σ(Π) ) and v out ∈ V AR Σ(Π) ). The following notation is useful for the definition of diagnosis below: Definition 4. (Γ(Δ)). Let Δ ⊆ Π be a set of statements of the original program. Then Γ(Δ) denotes the following set of constraints: def Γ(Δ) = {ab(S) = true | S ∈ Δ} ∪ {ab(S) = f alse | S ∈ Π \ Δ} The following definition is analogous to the definition of diagnosis in Reiter [1987], but tailored to the context of program debugging: Definition 5. (Diagnosis). Given a sequential program Π with CSP (Π) = (V, D, CO), a test case T , and the set of statements Δ ⊆ Π. Then Δ is a diagnosis wrt (Π, T ) iff the constraint satisfaction problem DIAG CSP (Π, T, Δ) is satisfiable, where DIAG CSP (Π, T, Δ) = (V, D, CO ) def
and
CO = CO ∪ T ∪ Γ(Δ) A diagnosis Δ is (subset-)minimal iff no proper subset is a diagnosis. Moreover, a diagnosis Δ has a minimal cardinality iff there is no diagnosis Δ with |Δ | < |Δ|. def
We observe that, analogous to Reiter’s definition of diagnosis, every superset of a diagnosis is also a diagnosis. Moreover, note that there is always a diagnosis, given that the test case T does not contradict itself. If there is no contradiction within T , then CO ∪ T is satisfiable, hence Δ = Π is a diagnosis. The definition of diagnosis can be immediately applied to create a simple algorithm which computes all single diagnoses for a program Π and a test case T : for every S ∈ Π, the algorithm generates DIAG CSP (Π, T, {S}) and employs a constraint solver which checks whether or not this CSP has a solution. If yes, then Δ = {S} is a single diagnosis, otherwise Δ is not a diagnosis. Example We inject a fault in statement S1 of the program in Fig. 1 by replacing e = exp with e = 0. The faulty program is depicted in Fig. 4. Accordingly, statement S1 of the SSA form is changed to int e 0 = 0, and the corresponding constraint in CSP (Π) is ab(S1 ) ∨ (e 0 = 0). We consider the test case T = {a = 2, exp = 2, res 4 = 4}, which would lead to 2 loop iterations in the correct program. As the faulty program, which never executes the loop body, returns res = 1 (res 4 = 1 in the SSA form, respectively), this test case fails, and so the empty set {} is not a diagnosis. The algorithm for finding all single-fault diagnoses, which is described above, yields 3 diagnoses: Δ1 = {S1 }, Δ2 = {S2 }, Δ3 = {S3 }. E.g., when the algorithm checks whether the candidate Δ2 = {S2 } is a diagnosis, then we have Γ({S2 }) = {ab(S1 ) = f alse, ab(S2 ) = true, ab(S3 ) = f alse, . . .}, and the constraint solver can determine that the constraint system CO ∪ T ∪ Γ({S2 }) has a solution which assigns the value res 0 = 4. Intuitively, S2 is a single diagnosis because if we replaced S2 in the faulty program (Fig. 4) by int res = 4, then the resulting program would satisfy the test case T . However, other test cases would not be satisfied. Moreover, S3 is also a diagnosis, because the constraint system has a solution with the assignment cond 0 = true and cond 1 = true. In other words, S3 is a diagnosis because the loop condition could be changed s.t.
int power faulty (int a, int exp) 1. int e = 0; 2. int res = 1; 3. while (e > 0) { 4. res = res * a; 5. e = e - 1; } Fig. 4. A faulty program for computing aexp . It is almost equal to the (correct) program in Fig. 1, but statement S1 was changed from e = exp to e = 0. the resulting program has 2 loop iterations for the given test case. 2 4. IMPROVING THE DIAGNOSTIC PRECISION BY INTEGRATING SPECIFICATION KNOWLEDGE In the diagnosis example above we obtained 3 singlefault diagnoses, i.e., less than half of the statements in Fig. 4 could be eliminated as potential single-faults. It is obvious that automated debugging should strive for a higher diagnostic precision in order to be useful in practice. The low precision in the example above results from the fact that the correct program behavior is only specified by a single test case, and that a test case is a black-box specification, meaning that it defines only the input-output behavior of a program, but does not specify expected behavior within the program. In this paper we tackle this problem by integrating assertions and empirically investigating their benefit. With assertions we denote specifications which state conditions which must hold at specific locations within the program. Many programming languages like C or Java have assert(cond)-statements, which can be placed at arbitrary locations in the source code and which are checked at runtime. In recent years the use of assertions has gained wide acceptance among software developers, which has also been fostered by growing tool support. Assertions are also closely related to Design by Contract, see Meyer [1997]. Although programmers often regard assertions as programming-language constructs which are used for fault detection at runtime, the concept of assertions has originally been introduced for the purpose of formal program specification and verification. In Hoare logic assertions are utilized to prove the (partial) correctness of programs, see Hoare [1969]. Common types of assertions are preconditions, postconditions, and loop invariants. In the following we briefly describe loop invariants, which represent properties of loops. Afterwards we will show that a loop invariant can eliminate diagnosis candidates in our example program. A loop invariant IN V states a condition which must hold before and after each iteration of the loop. More precisely, the body of a while-loop preserves the invariant IN V , given that the loop condition C holds before the execution of the body. The following rule of inference, taken from Hoare [1969], formalizes this concept, and it allows us to conclude which conditions hold after the execution of the loop is finished: {C ∧ IN V } body {IN V } {IN V } while(C) do body {¬C ∧ IN V } This rule formalizes the following deductive inferrence: if the Hoare triple {C ∧ IN V } body {IN V } holds, then it can be concluded that the successful execution of while(C) do body leads from a state in which IN V holds to a state in which ¬C ∧ IN V holds (provided that the loop terminates). A Hoare triple has the general form
int power faulty annot(int a, int exp)
P RE : exp ≥ 0 1. int e = 0; 2. int res = 1; 3. while (e > 0) {
IN V : (res == aexp−e ) ∧ (e ≥ 0) 4. res = res * a; 5. e = e - 1; }
P OST : res == aexp Fig. 5. The faulty program from Fig. 4 annotated with three assertions: the program’s pre- and postcondition and a loop invariant. {P } stmt {Q}: if the condition P holds before the execution of stmt, then the execution of stmt establishes the condition Q (provided that stmt terminates). Our approach presumes that the loop invariant is correct, i.e., it specifies the desired behavior of a loop, and a correct implementation of the loop must conform to this specification. For a given program, it can be proven by induction over loop iterations that the program conforms to the loop invariant; i.e., it has to be shown that the loop body preserves the invariant, provided that the loop condition holds. However, if the program contains a bug, then the (correct) loop invariant may contradict the faulty program. Hence, the intention is that a loop invariant is defined before the loop’s body is implemented, because the written code may be faulty, and deriving a loop invariant from a faulty program can lead to an invariant which does not express the desired behavior and which is useless for debugging. Also note that the invariant of a given loop is ambiguous because a loop invariant does not need to state all conditions which are preserved by the loop body. If IN V1 and IN V2 are invariants of a given loop, we say that IN V1 is stronger than IN V2 iff IN V1 |= IN V2 and IN V2 |= IN V1 . For debugging purposes we desire invariants which are as strong as possible in order to improve the diagnostic precision. However, in practice it is often not possible to find the strongest invariant, but our experience has shown that even a ”weak” invariant can often be useful to eliminate diagnosis candidates. Example Figure 5 depicts the faulty example program annotated with assertions. It can be seen that the chosen invariant is strong enough to prove that the postcondition P OST holds after the loop, provided that the loop is correctly implemented and that it terminates. More precisely, ¬C ∧ IN V |= P OST holds, where C is the loop condition (e > 0), because ¬C ∧ IN V |= (e == 0), and so ¬C ∧ IN V |= (res == aexp ). 2 Fig. 6 sketches a schematic depiction of a loop invariant: it can be seen that the invariant must hold before the first iteration of the loop and after each executed iteration. The formulation of a debugging problem as a CSP allows for a utilization of the full semantics of assertions. The integration of assertions in the loop-free SSA form is done as follows. Every pre- or postcondition P of a program is directly mapped to a single SSA-statement of the form assert(Pexpr ); Moreover, for a loop invariant IN V the loop-free SSA form contains n + 1 additional assert-statements, where n is the number of loop iterations considered in the SSA form: regarding the general schema in Fig. 6, the assertion A1 is mapped to the SSA form assert(INV 0expr );
loop with invariant: assert(INV); while(C) { ... [ body ] assert(INV); }
[A1 ] [A2 ]
Fig. 6. General schema showing how a loop invariant can be represented by two assert(cond)-statements. int power SSA annot(int a, int exp) ... ... 13. assertexp ≥ 0 ; 14. assert(res 0 == aexp−e 0 ) ∧ (e 0 ≥ 0) 15. assert (cond 0 == false) exp−e 1 ) ∧ (e 1 ≥ 0) ∨ (res 1 == a 16. assert (cond 1 == false) aexp−e 2 ) ∧ (e 2 ≥ 0) ∨ (res 2 == 17. assert res 4 == aexp ; Fig. 7. The loop-free SSA form from Fig. 3 enhanced with the assertions from Fig. 5. where INV 0expr represents the invariant IN V referring to the variable values before the first loop iteration. A2 is mapped to assert (cond i = false) ∨ INVi+1,expr ; where variable cond i corresponds to the loop condition after i iterations (i.e., if cond i is true, then the (i + 1)th loop iteration is executed), and INVi+1,expr refers to the variable values after the (i + 1)th loop iteration. The SSA form for the example program enhanced by assertions is shown in Fig. 7. The following definition deals with the integration of assertions into the constraint representation: Definition 6. (Integration of assertions into CSP (Π)). The constraint representation CSP (Π), which was defined in Def. 2, is extended as follows. The set of constraints CO comprises exactly one constraint for every statement in the SSA form, including assert-statements. If a statement S ∈ Σ(Π) of the SSA form has the form assert(condexpr), then add the relation condexpr to CO. Otherwise, translate S as defined in Def. 2. Example Integrating the assertions from Fig. 7 adds the following constraints to CO: ⎫ ⎧ ... ⎪ ⎪ ... ⎬ ⎨ exp ≥ 0, [S13 ] CO = exp−e 0 ) ∧ (e 0 ≥ 0), [S14 ] ⎪ ⎪ ⎭ ⎩ (res 0 = a ... ... 2 In the following example we demonstrate the benefit of assertions for improving the diagnostic precision: Example As in the diagnosis example in Sec. 3, we use the test case T = {a = 2, exp = 2, res 4 = 4}, which led without the use of assertions to the single-fault diagnoses Δ1 = {S1 }, Δ2 = {S2 }, Δ3 = {S3 }. Remember that S3 was a diagnosis because the constraint system CO ∪ T ∪ Γ({S3 }) has a solution with the assignment cond 0 = true and cond 1 = true. However, if we integrate the loop invariant as given in Fig. 5 into CO, then this constraint system has no solution: if cond 0 = true, then the invariant in enforces the condition e 1 ≥ 0 (see statement S15 Fig. 7), which contradicts the constraints e 0 = 0 and e 1 = e 0 − 1 (which correspond to statements S1 and
S5 in the faulty program in Fig. 5). Hence, the candidate {S3 } is no longer a diagnosis, and only two single-fault diagnoses remain (Δ1 , Δ2 ). Also note that the invariant IN Vw : (e ≥ 0), which is weaker than the complete invariant as defined in Fig. 5, was here sufficient to eliminate {S3 } as diagnosis candidate. 2
5. USING A STATE-OF-THE-ART CONSTRAINT SOLVER FOR DEBUGGING In this section we show how to apply a state-of-the-art constraint solver to program debugging. For this purpose we have chosen the MINION solver (Gent et al. [2006]). MINION is an open-source constraint solver which, according to the empirical results in Gent et al. [2006], has exhibited a superior performance on a number of large problem instances, compared with modern constraint toolkits like ILOG or GeCode. One particularity of MINION is that it does not perform an in-between transformation of the input constraint system; i.e., the constraint solving algorithms operate directly on the input, whereas many other constraint toolkits transform the input to an internal representation. This property of MINION aims at increasing the performance, but it also imposes some limitations on the way constraints are modelled. E.g., nested quantifications are not supported. Minion allows four type of variable domains. In our experiments we use BOOL domains and DISCRETE domains. The latter are specified as integer intervals. In our example, we restrict the DISCRETE domains to the range −1000.. + 1000. This is sufficient for covering all possible combinations that arise in our test cases, and restricting the domains in general improves the performance. The SSA form of a program to be debugged comprises assignments, conditional statements, arithmetic and boolean operations. In order to convert these programs into CSPs we need arithmetic constraints, conditional constraints and logical constraints. The MINION constraints library provides an implementation of all arithmetic and logical operators which are needed for our purposes. In addition, the MINION library contains a constraint of the form reif y(Condexpr , Condvar ) where variable Condvar is of type BOOL and is true iff the condition Condexpr is satisfied. We need this constraint for converting conditional statements. In Sec. 3 we introduced a boolean variable ab(S) for every statement of the original program Π. In MINION we use an array AB[..] containing boolean values which state the abnormality of the corresponding statement; i.e., the size of this array is equal to |Π|.
Every statement in the SSA form which contains the Φfunction is expressed by 2 constraints in MINION. E.g., statement S9 in Fig. 3 becomes: watched-or({eq(cond 1,0), eq(res 3,res 2)}) watched-or({eq(cond 1,1), eq(res 3,res 1)}) Minion does not support nested quantifications, hence we have to subdivide such a constraint into two or more sim pler constraints. For example, for modelling statement S15 in Fig. 7 in MINION we need to introduce two auxiliary is represented by variables power a and diff exp, and S15 the following 4 constraints: difference(exp, e 1, diff exp) pow(a, diff exp, power a) watched-or({eq(cond 0, 0), eq(res 1, power a)}) watched-or({eq(cond 0, 0), ineq(0, e 1, 0)}) For diagnosis we are interested in different combinations of the boolean values in the AB[..] array. The MINION command VARORDER[AB[ ]] forces the solver to compute all possible solutions of the CSP with the restriction that no two solutions have the same value assignments to the AB[..] array; i.e., for two solutions at least one element in this array must have a different value. A common approach in model-based diagnosis is to focus on (subset-)minimal diagnoses or on minimal-cardinality diagnoses. Although a single call to the MINION solver is not able to deliver all subset-minimal diagnoses, we can achieve that MINION computes all diagnoses Δ with a certain cardinality |Δ| = n in a single call. For this purpose, we introduce an auxiliary variable sum which is equal to the sum of the elements in the AB[..] array (each boolean variable has the value 0 or 1). Then we can achieve our goal by adding a constraint which specifies that sum must be equal to n. The following MINION code leads to the computation of all double-fault diagnoses (note that the constraint sum = 2 must be mapped to two MINION constraints): sumleq(AB, sum) sumgeq(AB, sum) eq(sum, 2) 6. EXPERIMENTAL RESULTS We evaluated our debugging approach on a set of small Java programs which have the property that they contain at least one while-loop. We annotated these programs with pre- and postconditions and with loop invariants. We investigated the benefit of assertions by performing the diagnosis for the annotation-free programs as well as for the annotated programs and by comparing the diagnostic results afterwards. For this purpose we performed the following steps:
For example, the MINION syntax corresponding to statement S3 of the program given in Fig. 3 is: BOOL cond 0 BOOL AB[5] DISCRETE e 0 {-1000..1000} ... watched-or({element(AB,3,1), reify(ineq(0,e 0,-1),cond 0)})
(1) In every program we injected different single- and double-faults (see below). (2) For each faulty program we created two loop-free SSA representations: one with and one without assertions. (3) We converted the resulting programs into a constraint system in MINION syntax. (4) For each program we defined a test case which leads to either 1 or 2 loop iterations. Based on this test case, we computed the diagnoses using the implementation provided at MINION [2009].
The above constraint is satisfied if the third element of the AB array is 1 (true) or if cond 0 ↔ (e 0 > 0).
In the first set of experiments we injected 3 different single faults in every program; i.e., there were three different
faulty versions for each program. The experimental results are given in Tbl. 1. Each fault changes either the righthand side of an assignment statement or the boolean expression of a loop condition. In Tbl. 1 we use identifiers to denote each fault (column Er). For example, in the program power(a, exp) (see Fig. 1) we injected the faults P1-P3, where P1 changes line 2 to int res = 0, P2 changes line 1 to int e = exp, and P3 changes line 4 to res = res + a. Due to space reasons we do not describe the injected faults for all test programs here. When comparing the columns #D and #Dinv in Tbl. 1, it can be clearly seen that the integration of loop invariants significantly increases the diagnostic precision: on average, only 1.8 single-fault diagnoses remain after integrating the assertions. Moreover, we defined the metrics E% and E%,inv which denote the percentage of statements which are eliminated as potential faults by our debugging approach. In case of single-faults they can be simply computed as follows: LΠ − #D LΠ − #Dinv def def E% = 100 ∗ ; E%,inv = 100 ∗ LΠ LΠ As average value of all experiments we obtained avg(E%,inv − E% ) = 30.6%; i.e., the integration of assertions significantly improved the diagnostic precision by further eliminating 30.6% of the statements as fault candidates. In all experiments it took less than 0.1 sec to compute all solutions on a PC with a Pentium 4 2.0 GHz CPU. In the second set of experiments we injected a double fault in each program (one fault before the loop and another one in the loop condition). The results are given in Tbl. 2. A remarkable finding is that without using assertions we obtained several single-fault diagnoses for every program, although the programs contained two faulty statements. The integration of loop invariants greatly improved the diagnostic results: in all of those experiments, the invariant was able to eliminate the single-fault diagnoses, and the number of double-fault diagnoses was significantly smaller. The integration of assertions improved the diagnostic 2 2 precision by further eliminating avg(E%,inv −E% ) = 31.9% of the statements as fault candidates. 7. CONCLUSIONS AND OPEN ISSUES In this paper we provide a formalization of the constraint representation of sequential programs, and we describe debugging as a constraint satisfaction problem. In order to improve the diagnostic precision we integrate specification knowledge into our approach. Such knowledge is provided by program annotations like pre- and postconditions or loop invariants. We also show how the resulting CSP can be modelled in a state-of-the-art constraint solver and how to compute all diagnoses with a specific cardinality. Finally, we provide the first empirical results for model-based debugging based on a constraint representation. In particular, we compare the diagnostic results obtained with and without using loop invariants. Our results clearly demonstrate the advantage of integrating specification knowledge wrt the number of statements which are eliminated as possible fault candidates. This work focuses on the diagnostic precision rather than the performance of our approach. An important open issue it the application of our debugging technique to larger programs in oder to investigate its scalability. In this context it would also be interesting to compare different constraint solvers. Moreover, we plan to compare the results and performance of the constraint representation
with other modelling approaches which have been applied to debugging. REFERENCES M. M. Brandis and H. M¨ ossenb¨ock. Single-pass generation of static assignment form for structured languages. ACM TOPLAS, 16(6):1684–1698, 1994. R. Ceballos and R. M. Gasca and C. Del Valle and D. Borrego. Diagnosing Errors in DbC Programs Using Constraint Programming. Lecture Notes in Computer Science, Vol. 4177, Pages 200-210, 2006. R. Cytron, J. Ferrante, B. K. Rosen, M. N. Wegman, and F. Kenneth Zadeck. Efficiently computing static single assignment form and the control dependence graph. ACM TOPLAS, 13(4):451–490, 1991. H. Collavizza and M. Rueher. Exploring Different Constraint-Based Modelings for Program Verification. In Principles and Practice of Constraint Programming (CP 2007), pages 49–63,Providence, RI, USA, September 2007. R. Dechter. Constraint Processing. Morgan Kaufmann, 2003. J. de Kleer and B. Williams. Diagnosing Multiple Faults. Artificial Intelligence, 32:97–130, 1 1987. G. Friedrich, M. Stumptner, and F. Wotawa. Model-based diagnosis of hardware designs. Artificial Intelligence, 111(2):3–39, July 1999. I. P. Gent, C. Jefferson, I. Miguel. MINION: A Fast, Scalable, Constraint Solver. 17th European Conference on Artificial Intelligence (ECAI-06). Riva del Garda, Italy, 2006. D. Jackson. Software abstractions: logic, language, and analysis. MIT Press, 2006. C. A. R. Hoare. An Axiomatic Basis for Computer Programming. Communications of the ACM, 12(10):576– 583, 1969. D. K¨ ob and F. Wotawa. Fundamentals of debugging using a resolution calculus. In Luciano Baresi and Reiko Heckel, editors, Fundamental Approaches to Software Engineering (FASE’06), volume 3922 of Lecture Notes in Computer Science, pages 278–292, Vienna, Austria, March 2006. Springer. W. Mayer. Static and Hybrid Analysis in Model-based Debugging. PhD Thesis, School of Computer and Information Science, University of South Australia, Adelaide, Australia, 2007. B. Meyer, Object-Oriented Software Construction, OSE Press, 2nd edn., 1997. MINION, 2009. http://minion.sourceforge.net. M. Nica, J. Weber, F. Wotawa. How to debug Sequential Code by means of Constraint Representation. 19th International Workshop on Principles of Diagnosis (DX08), Leura, Australia, 2008. B. Peischl, N. Riaz and F. Wotawa. Model-Based Reasoning with Multiple Test Cases and its Application to Debugging. In 19th International Workshop on Principles of Diagnosis (DX 2008), pages 315–323,Blue Mountains, Australia, September 2008. R. Reiter. A theory of diagnosis from first principles. Artificial Intelligence, 32(1):57–95, 1987. M. Stumptner and F. Wotawa. Debugging Functional Programs. In Proceedings 16th International Joint Conf. on Artificial Intelligence, pages 1074–1079, Stockholm, Sweden, August 1999. M.N. Wegman and F.K. Zadek. Constant propagation with conditional branches. ACM Transactions on Programming Languages and Systems, 13(2), April 1991. F. Wotawa, M. Nica. On the Compilation of Programs into their equivalent Constraint Representation. In Informatica Journal, Volume 32, pages 359-371, 2008.
No 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39.
Name Division Division Division Division Division Division Mult Mult Mult Mult Mult Mult MultV2 MultV2 MultV2 MultV2 MultV2 MultV2 Sum Sum Sum Sum Sum Sum gCD gCD gCD Power Power Power Power Power Power sumPower sumPower sumPower sumPower sumPower sumPower
LΠ 8 8 8 8 8 8 5 5 5 5 5 5 8 8 8 8 8 8 5 5 5 5 5 5 9 9 9 5 5 5 5 5 5 10 10 10 10 10 10
#It 1 1 1 2 2 2 1 1 1 2 2 2 1 1 1 2 2 2 1 1 1 2 2 2 2 2 2 1 1 1 2 2 2 1 1 1 2 2 2
LΣ(Π) Er 12 D1 12 D2 12 D3 18 D1 18 D2 18 D3 9 M1 9 M2 9 M3 15 M1 15 M2 15 M3 16 M21 16 M22 16 M23 41 M24 41 M25 41 M23 8 S1 8 S2 8 S3 14 S2 14 S3 14 S4 22 G1 22 G2 22 G3 6 P1 6 P2 6 P3 11 P1 11 P4 11 P3 13 SP1 13 SP2 13 SP3 21 SP1 21 SP2 21 SP3 average:
#D 4 3 2 5 5 2 4 4 2 5 5 2 6 6 6 6 5 8 4 2 2 5 2 5 3 4 5 2 3 2 2 5 2 3 3 2 3 5 8 3.9
E% 50.0 % 62.5 % 75.0 % 37.5 % 37.5 % 75.0 % 20.0 % 20.0 % 60.0 % 0.0 % 0.0 % 60.0 % 25.0 % 25.0 % 25.0 % 25.0 % 37.5 % 0.0 % 20.0 % 60.0 % 60.0 % 0.0 % 60.0 % 0.0 % 77.7 % 55.5 % 44.4 % 60.0 % 40.0 % 60.0 % 60.0 % 0.0 % 60.0 % 70.0 % 70.0 % 80.0 % 70.0 % 50.0 % 20.0 % 42.4 %
#Dinv 2 1 2 2 2 2 1 2 1 1 2 1 2 4 3 1 1 3 1 1 1 1 1 2 1 4 4 1 2 1 1 2 1 1 2 2 1 2 6 1.8
E%,inv 75.0 % 87.5 % 75.0 % 75.0 % 75.0 % 75.0 % 80.0 % 60.0 % 80.0 % 80.0 % 60.0 % 80.0 % 75.0 % 50.0 % 62.5 % 87.5 % 87.5 % 62.5 % 80.0 % 80.0 % 80.0 % 80.0 % 80.0 % 60.0 % 88.8 % 55.5 % 55.5 % 80.0 % 60.0 % 80.0 % 80.0 % 60.0 % 80.0 % 90.0 % 80.0 % 80.0 % 90.0 % 80.0 % 40.0 % 73.0 %
E%,inv − E% 25.0 % 25.0 % 0.0 % 37.5 % 37.5 % 0.0 % 60.0 % 40.0 % 20.0 % 80.0 % 60.0 % 20.0 % 50.0 % 25.0 % 37.5 % 52.5 % 50.0 % 62.5 % 60.0 % 20.0 % 20.0 % 80.0 % 20.0 % 60.0 % 11.1 % 0.0 % 11.1 % 20.0 % 20.0 % 20.0 % 20.0 % 60.0 % 20.0 % 20.0 % 10.0 % 0.0 % 20.0 % 30.0 % 20.0 % 30.6 %
#D1 4 2 5 2 3 2 3 2.9
#D2 inv 4 3 7 2 6 3 5 4.3
E2 %inv 50.0 % 40.0 % 0.0 % 40.0 % 33.0 % 20.0 % 40.0 % 31.9 %
#D1 inv 0 0 0 0 0 0 0 0
Table 1. Experimental results for the first set of experiments. For each program Name, LΠ is the number of statements in the original program, #It denotes the number of considered loop iterations, LΣ(Π) denotes the number of statements in the loop-free SSA form, Er identifies the injected fault, #D is the number of single-fault diagnoses obtained without loop invariants and #Dinv is the number of single-fault diagnoses when using loop invariants. Moreover, E% and E%,inv denote the percentage of statements of the original program which do not occur in any single-fault diagnosis without assertions (E% ) or when using assertions (E%,inv ); see the text for details. Hence, the last column E%,inv − E% states the percentage of statements which can be eliminated as single-fault candidates when integrating assertions.
Name Division Mult MultV2 Sum gCD Power sumPower
LΠ 8 5 8 5 9 5 10
#It 2 2 2 2 2 2 2
LΣ(Π) 18 15 41 14 22 11 21
Er D1 + D2 M1 + M2 M21 + M22 S1 + S2 G1 + G2 P1 + P4 SP1 + SP2 average:
#D2 19 7 25 7 21 7 27 15.7
E2 % 0.0 % 0.0 % 0.0 % 0.0 % 0.0 % 0.0 % 0.0 % 0.0 %
E2 − E2 %inv % 50.0 % 40.0 % 0.0 % 40.0 % 33.8 % 20.0 % 40.0 % 31.9 %
Table 2. Experimental results for the second set of experiments. #D1 and #D2 denote the number of diagnoses with cardinality 1 and 2, respectively, when no loop invariants are used, whereas #D1inv and #D2inv state the number of diagnoses when using loop invariants. E2% and E2%inv denote the percentage of statements of the original program which do not occur in any double-fault diagnosis without assertions (E2% ) or when using assertions (E2%inv ). The column E2%inv − E2% states the percentage of statements which can be eliminated as candidates when integrating assertions.
