GREEDY ALGORITHMS FOR THE TEST SELECTION ... - CiteSeerX

Journal of Circuits, Systems, and Computers cfWorld Scientific Publishing Company

GREEDY ALGORITHMS FOR THE TEST SELECTION PROBLEM IN PROTOCOL CONFORMANCE TESTING

∗ ¨ TIBOR CSONDES Ericsson Research, Conformance Lab 1037 Budapest Laborc u. 1., Hungary

and ´ BALAZS KOTNYEK Ericsson Research, Conformance Lab 1037 Budapest Laborc u. 1., Hungary

Received (received date) Revised (revised date) Accepted (accepted date)

This paper presents a mathematical model for the test selection problem in protocol conformance testing, the goal of which is to select a suitable test set from a given test suite. The problem is described together with its mathematical formulation including two optimization problems and four different models for the coverage. The test selection problem is shown to be NP-hard. The Integer Programming transformation of the problems is also discussed. We propose greedy algorithms for the selection, which perform much better than the existing methods and can provide very good solutions as experimental results show. Keywords: Conformance test selection, Communication protocols, Integer Programming, Heuristics

1. Introduction Conformance testing is an important stage of the lifecycle of telecommunication protocols 1 . Its purpose is to check whether the implemented protocol is conform to the standards. As well as the protocols, the test suites and the test process are standardized by international standardization institutes 2 . They issue Abstract Test Suites (ATS) for all important protocols, which may contain hundreds of test cases. Ideally an ATS should be complete, i.e. an implemented protocol passes it if and only if the protocol is correct. Since, in general, it is not possible to construct a finite complete test suite, an ATS is required to be sound, which means that all protocols that do not pass it are not correct. A test case is written to check a specific part of the protocol behaviour, which is described in its purpose. A test purpose is associated with a set of test requirements. Test requirements are supposed to ∗Corresponding

author. E-mail: [email protected]

1

2

Greedy Algorithms for the Test Selection Problem in Protocol Conformance Testing

represent basic disjoint parts of the protocol with respect to conformance to the standards. Testing is time intensive and expensive. For this reason test laboratories usually have to omit some test cases and execute only the remaining ones. By this selection, the soundness of the test suite does not change but its completeness may decrease. The coverage is a widely used, though not exactly defined, metric to measure the completeness of a set of test cases with respect to checking the conformance of a protocol 2 . As the whole ATS is usually not complete, the requirements it tests may not cover the entire behaviour of the protocol. However, since the task is to select test cases of a given ATS, the coverage of a set of test cases can be viewed as its relative completeness with respect to the whole ATS, that is, it measures how much the requirements of the test suite are covered by the seleceted test cases. In other words, we assume the coverage of the ATS to be 100%. A natural question arising in test laboratories is which tests should be selected to execution. An ideal choice is a test set that checks as large a part of the protocol (has as big coverage) as possible and needs as few resources as possible. Although this problem arises in almost every case when conformance testing is done, a theoretically well-based solution, which also takes into account the practical aspects, does not exists. In this paper we describe a mathematical model which can help us in solving the test selection problem (Section 2). We also present a possible Integer Programming formulation of the problem (Section 3) and suggest some greedy algorithms as possible methods of selection (Section 4). Finally, we present experimental results in Section 5. 2. Problem Formulation Given a test suite T S = {t1 , t2 , . . . , tn } consisting of the test cases and the set of the test requirements REQ = {r1 , r2 , . . . , rk } related to this test suite. We have a positive cost, c(tj ) assigned to each test case tj measuring the amount of resources its execution needs. A positive weight, w(ri ) is assigned to each requirement which represents the relative importance of ri with respect to the correct behaviour of the protocol. There is a subset of test cases, Tri associated with each requirement ri (i = 1, . . . , k) containing the tests related to the requirement. That is, tj belongs to Tri if and only if the execution of tj provides us with information about the correctness of ri . Let T be an arbitrary set of the test cases, T ⊂ T S. The cost of this test set is P defined as the sum of the costs of the test cases belonging to T : c(T ) = t∈T c(t). Let cov(T ) denote the coverage of test set T , i.e. cov(T ) measures how large part of the protocol is tested by T (later we give formal definition as well). Two problems can be defined according to our purposes and bounds.

Minimal cost problem: Given a lower bound (K) for the coverage. Find the set

Greedy Algorithms for the Test Selection Problem in Protocol Conformance Testing

3

of test cases that satisfies this bound with minimal cost. min subject to

c(T ) cov(T ) ≥ K T ⊂ TS

(1)

Maximal coverage problem: Given an upper bound (L) for the cost. Find the subset of test cases the cost of which is less than L and checks as large a part of the protocol as possible max cov(T ) subject to c(T ) ≤ L T ⊂ TS

(2)

We assumed that the requirements represent disjoint parts of the protocol behaviour and their importance is denoted by their weights. This implies that the coverage of a test set can be expressed as the weighted sum of the individual coverage of the requirements. X wi · reqcov(ri , T ) (3) cov(T ) = ri ∈REQ

where reqcov(ri , T ) measures in what proportion T covers the part of the protocol behaviour which is represented by requirement ri . Of course 0 ≤ reqcov(ri , T ) ≤ 1 for all ri ∈ REQ and T ⊂ T S. We make the assumption that reqcov(ri , T ) depends on only the number of test cases that are related to ri and belong to test set T : reqcov(ri , T ) = fi (|Tri ∩ T |) where fi : {0, 1, . . . , |Tri |} → [0, 1]. For notational convenience, let bi = |Tri |. If test set T contains none of the test cases related to ri , then we have no information about the correctness of the requirement, so fi (0) = 0. Let us suppose that adding one more related test case to T increases the coverage with si , and if m = |Tri ∩T | reaches a given value Di , then T already covers the entire requirement, so fi (m) = 1. Of course if m = |Tri ∩ T | = bi , thus T contains all the related test cases, then fi (m) = 1. With formulas:  msi if m < min{d s1i e, Di , bi } (4) fi (m) = 1 otherwise The exact values of Di and si (i = 1, 2, . . . , k) in a test selection problem depend on the methodology the construction of the test suite followed. Below we list four special cases which model the possible real-life situations the best. We call them coverage models as they determine the coverage of a test set. Let Di ≥ bi in each case. 1. All or Nothing model: si = 0, so fi (0) = fi (1) = . . . = fi (bi − 1) = 0 and fi (bi ) = 1. This model implies that T cannot test anything of ri unless it covers the requirement entirely. In other words, executing bi − 1 test cases from Tri has the same result as executing none of them.

4

Greedy Algorithms for the Test Selection Problem in Protocol Conformance Testing

2. Step and Jump model: si =

1 |T S|

 fi (m) =

and then m |T S|

1

if m < bi if m = bi

In this model we assume that adding one more related test case to T increases the coverage with a relatively small amount until all of them belong to T , when the coverage is 1. If only a few test cases are related to ri , then executing all but one has a small contribution to the coverage, while if |Tri | is big, then we can obtain high coverage by executing only a part of the related test cases. 3. Linear model: si = b1i , so fi (m) = m bi for m = 0, 1, . . . , bi . This models means that the coverage is in direct proportion to the number of related test cases belonging to T . 4. One is Enough model: si ≥ 1, i.e. fi (0) = 0 and fi (1) = . . . = fi (bi ) = 1. That is, we consider the whole requirement to be entirely tested if at least one of the related test cases is in T . The minimal cost problem in the One is Enough model (i.e. if all requirements P follow the One is Enough coverage model) for K = r∈REQ w(r) is equivalent to the Set Covering problem, which is known to be NP-hard. This implies that the general minimal cost problem is NP-hard. Furthermore, if |T S| = |REQ| and P Tri = {ti } for all i, then cov(T ) = {i:ti ∈T } w(ri ) in each model. Thus this special case of the test selection problem (both the minimal cost and maximal coverage) is equivalent to the knapsack problem. Consequently, the test selection problem for general cost and weight functions is NP-hard independently of the coverage model. 3. Integer Programming Formulation In this section, we will show how the minimal cost problem can be transformed to a linear integer programming problem. The maximal coverage problem can be handled very similarly. Let ai be the characteristic vector of Tri for i = 1, . . . , k and A be the 0-1 matrix of size k × n made up of these row vectors. Thus (A)ij = 1 if and only if tj is related to ri . Let x be the characteristic vector of test set T ⊂ T S, c be the cost vector and w be the vector containing the weights of the requirements. Then c(T ) = cx Pk and cov(T ) = i=1 wi fi (ai x) according to (3). Using these notations the minimal cost problem (1) can be written as min subject to

cx k X

wi fi (ai x) ≥ K

i=1

x ∈ {0, 1}n Let us see now how this formula looks like in the four mentioned models.

(5)

Greedy Algorithms for the Test Selection Problem in Protocol Conformance Testing

5

Minimal cost problem in the Linear model: k X

k X

ai x wi fi (ai x) = wi = bi i=1 i=1

k X wi ai i=1

bi

! x

(6)

Thus (5) can be transformed into a knapsack problem. The existing efficient algorithms solving knapsack problems (see e.g. 3 ) can be applied here. Minimal cost problem in the All or Nothing model: Let us define a new variable vector z = (z1, . . . , zk ) in the following manner:  1 if ai x = bi (7) zi = 0 if ai x < bi In other words, z = 1 if ri is tested by the test set represented by x. Using this vector problem (5) can be written as min subject to

cx ai x wz x z

≥ ≥ ∈ ∈

bi zi K {0, 1}n {0, 1}k

i = 1, . . . , k (8)

Minimal cost problem in the One is Enough model: This model can be handled similarly to the previous one. Let z be the following vector:  0 if ai x = 0 zi = 1 if ai x ≥ 1 In this case (5) can be transformed into the following problem: min subject to

cx Ax ≥ wz ≥ x ∈ z ∈

z K {0, 1}n {0, 1}k

(9)

Minimal cost problem
in the Step and Jump model: In this model fi (ai x) = n1 ai x + zi 1 − bni with zi defined in (7) and n = |T S|. Thus (5) is equivalent to min subject to

4. Heuristic Algorithms

cx bi zi ≤ K ≤ x ∈ z ∈

ai x i = 1, . . . , k Pk 1 i=1 wi 1 − n (wA)x + {0, 1}n {0, 1}k

bi n




zi

(10)

6

Greedy Algorithms for the Test Selection Problem in Protocol Conformance Testing

Test laboratories usually use one of the following two simple greedy algorithms for selecting the executed test set T if a lower bound K for its coverage is given. Algorithm ADD Step 1 : Let T = ∅. Step 2 : Find a test case ti in T S \ T with minimal cost. If there are more, then choose the one with the lowest index. Let T = T ∪ {ti }. Step 3 : If cov(T ) ≥ K, then STOP, otherwise go to Step 2. Algorithm DROP Step 1 : Step 2 : If there Step 3 :

Let T = T S and U = T S. If U = ∅, then STOP. Otherwise find a test case ti in U with maximal cost. are more, then choose the one with the lowest index. Let U = U \ {ti }. If cov(T \ {ti }) ≥ K then T = T \ {ti } and go to Step 2.

We propose two more sophisticated algorithms for the minimal cost problem. The biggest problem with the two previous algorithms is that in choosing a test case to add or drop they do not take into consideration the change in the coverage it may imply. To overcome this problem, we calculate the marginal coverage of each test case, i.e. the change in the coverage as a consequence of the change in T . We compare it with the change in cost, and choose the test case to add to or drop from T that prove to be the best. Algorithm ADD-DELTA Step 1 : Let T = ∅. Step 2 : For each ti ∈ T S \ T calculate the increase in coverage and cost if it is added to T : ∆cov(i) = cov(T ∪ {ti }) − cov(T ) and ∆cost(i) = c(T ∪ {ti }) − c(T ). Step 3 : Find a test case ti in T S \ T for which ∆cost ∆cov is minimal. If there are more, then choose the one with the lowest index. Let T = T ∪ {ti }. Step 4 : If cov(T ) ≥ K, then STOP, otherwise go to Step 2. Algorithm DROP-DELTA Step 1 : Let T = T S and U = T S. Step 2 : If U = ∅, then STOP. Otherwise, for each ti ∈ U calculate the decrease in coverage and cost if it is excluded from T : ∆cov(i) = cov(T ) − cov(T \ {ti }) and ∆cost(i) = c(T ) − c(T \ {ti }). Step 3 : Find a test case ti in U for which ∆cost ∆cov is maximal. If there are more, then choose the one with the lowest index. Let U = U \ {ti }. Step 4 : If cov(T \ {ti }) ≥ K, then T = T \ {ti } and go to Step 2. We showed earlier that the minimal cost problem in the Linear model is a knapsack problem. Let min{cx | vx ≥ d, x ∈ {0, 1}n} be this knapsack problem (so

Greedy Algorithms for the Test Selection Problem in Protocol Conformance Testing

v=

Pk

wi ai bi

according to Pl−1 the index for which i=1 vi   xi =  i=1

(6)). Let us suppose that Pl < d ≤ i=1 vi . Then 1 Pl−1 d − i=1 vi 0

c1 v1

≤

c2 v2

≤ ... ≤

cn vn

7

and l is

for i = 1, . . . , l − 1 for i = l for i = l + 1, . . . , n

is the optimal continuous solution. Thus changing xl to 1 gives us a feasible integer solution of the original knapsack problem. In the Linear model ∆cov(i) = vi and ∆cost(i) = ci for every i and every T , so this integer solution is the same as we get by applying algorithm ADD-DELTA or DROP-DELTA. 5. Experimental Results To analyse the performance of the heuristic algorithms we examined two abstract test suites, one for GSM 4 and one for INAP 5 . We developed a software tool that was able to elicit the requirements from the ATSs automatically 8 . The GSM test suite contained 324 test cases and 218 requirements, while the test suite for INAP had 357 test cases and 243 requirements. We used unit cost and coverage as we had no reason for differentiating between the different tests and requirements. The results are for the minimal cost problem in One is Enough, Step and Jump and All or Nothing models for coverage bound K = 1, 2, . . . , cov(REQ). The results of different algorithms were compared with each other and with the best integer solution obtained by applying commercial IP solvers to the problems described in Section . We used two highly developed and commercially available MIP (Mixed Integer Programming) solvers: CPLEX 6 and IBM’s OSL MSLV 7 . In both solvers, we set the node limit at 30,000 and in CPLEX the backtracking parameter was 0.25. Between them, they found the optimal solution for each value of K in the One is Enough model, while in the other two models they were often able to provide only feasible solutions within the node limit. Table 1 gives a summary of the results. For each greedy algorithm the average relative gap between the best integer solution obtained from the commercial solvers and the greedy solution is shown. The average is calculated over the varying coverage bounds. Table 1. The average relative gap between the solution of the heuristics and the commercial solvers (One=One is Enough, Step= Step and Jump, All=All or Nothing) protocol GSM

INAP

model One Step All One Step All

ADD 296% 66% 67% 157% 78% 78%

DROP 92% 118% 105% 86% 122% 110%

ADD-DELTA 4.25% 14% 16% 2.9% 9% 10%

DROP-DELTA 4.72% 42% 82% 5% 24% 50%

As it can be seen, the ADD-DELTA algorithm was the best with respect to the average relative gap, though the other algorithms sometimes gave better solutions

8

Greedy Algorithms for the Test Selection Problem in Protocol Conformance Testing

for specific bounds. For example, for K ≥ 160 in the All or Nothing model for GSM, DROP-DELTA gave better solutions than the other three greedy algorithm. On the other hand, for some specific coverage bounds ADD-DELTA could even find better solutions than the commercial softwares. In the One is Enough model the biggest (absolute) gap between the best commercial and the ADD-DELTA solutions was 3, so in this case this greedy algorithm practically found the best solutions. 6. Conclusions We showed a way for formulating the test selection problem of real-life conformance testing as a mathematical programming problem. We presented an Integer Programming formulation and applied commercial MIP solvers to them. On the other hand, we introduced four greedy algorithms. The two simpler ones describe the heuristics used by test laboratories. By modifying them we got algorithms that were able to provide comparable results to those of commercial softwares. The fact that these simple, fast and easily implementable algorithms (especially ADDDELTA) could give us very good results, which are definitely better than those obtained in practice, means that by applying them to the test selection problem, conformance testing can be made much more economic. References 1. B. Baumgarten, A. Giessler, OSI Conformance Testing Methodology and TTCN, Elsevier Science, 1994 2. ISO/IEC DIS 13245, Information Technology-Framework: Formal Methods in Conformance Testing, 1997 3. L. A. Wolsey, Integer Programming, Wiley, 1998 4. ETSI ETS 300 557, Digital Cellular Telecommunications System (Phase 2), Abstract Test Suite (ATS) 5. ETSI ETS 300 374-4, Core Intelligent Network Application Protocol (INAP), Abstract Test Suite (ATS) 6. CPLEX, version 3.0 1989-1994, CPLEX Optimization Inc. 7. IBM Optimization Solutions and Library for Multi-platforms Version 2 8. T. Cs¨ ondes, B. Kotnyek, “Automated Test Case Selection Based on Subpurposes”, in Testing of Communication Systems-Methods and Applications, eds. Gy. Csopaki, S. Dibuz, K. Tarnay, Kluwer Academic Publishers, 1999, pp. 251–265