TestFilter: A Statement-Coverage Based Test Case Reduction Technique Saif-ur-Rehman Khan1, Aamer Nadeem2 Center for Software Dependabilit, Mohammad Ali Jinnah University, Islamabad, Pakistan.
[email protected],
[email protected]
Ali Awais, COMSATS Institute of Information Technology, Islamabad Campus, Pakistan.
ali.awaisghotmail.com
Abstract- Software testing is an important but expensive phase of software development life cycle. During software testing and retesting, development organizations always desire to validate the software from different views. But exhaustive testing requires program execution with all possible combinations of values for program variables, which is impractical due to resource constraints. For many applications, it is possible to generate test cases automatically. But the core problem is the selection of effective test cases necessary to validate the program during the maintenance phase. This target can only be achieved by eliminating all the redundant test cases from the generated pool of test suites. In this paper, we propose a novel test case reduction technique called TestFilter that uses the statement-coverage criterion for reduction of test cases. To demonstrate the applicability of this approach, we conduct an experimental study. The results show that our technique is beneficial in identifying non-redundant test cases at a little cost. Ultimately it is beneficial to optimize time & cost spent on testing and it is also helpful during regression testing. Keywords: Software testing, testing efficiency, test case reduction. I.
INTRODUCTION
Software testing is an important but expensive phase of Software Development Life Cycle (SDLC). Exhaustive testing provides more confidence about the quality and reliability of the developed software during maintenance phase. In this technique, Test manager executes the programs with all infinite combinations of values for program variables [1]. Generally the domain of a program is infinite and cannot be used as a test data. According to Rothermel et. al. [2] the product of about 20,000 lines of code requires seven weeks to run all its test
1-4244-0794-X/06/$20.00 ©2006 IEEE
275
cases and costs several hundred thousands dollars to execute them. Generally development organizations cannot afford such exhaustive testing, rather than they are forced to accommodate the inherent complexities of other phases of SDLC also in the testing phase. Ultimately they choose test case optimization as a solution of all these problems. A test case is defined in IEEE standard as [3]: "A set of test inputs, execution, and expected results developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement". While a test suite is a combination of test cases. The quality of a test suite is measured through following four factors: 1) its fault coverage, 2) its code coverage, 3) its size, and 4) the number of faults detected by the most effective test contained in it [4]. For many applications, it is possible to generate test cases automatically. Formalization of requirements and design documents additionally permits automation of test execution and evaluation process [6]. But core problem is selection of effective test cases necessary to validate the program during maintenance phase. This can be done by identifying and eliminating redundant test cases from all generated test suites. Note, a test case is redundant, if it tests the same portion of code as tested by other test case (s) of that test suite [3]. Test manager can eliminate these redundant test cases using random or adhoc techniques. But this discarding capability can be challenged and there is an arbitrarily high risk of eliminating all tests that reveal an error of high severity from the code. Test Case Reduction (TCR) techniques helps test manager to find out representative set of test cases at little cost [5]. By doing this, we can reduce the test case execution, management, and storage cost. Ultimately, it helps to accommodate the delays and cost overrun of other phases of SDLC. In this paper, we propose a novel test case reduction technique, called TestFilter. This technique selects test cases based on their statement-coverage (i.e., weight). Note, weight
refers to the number of occurrences of a particular test case that covers different statement of the program under test. The technique first calculates weight of all generated test cases. Next it selects test cases of higher weight and marked all of its corresponding requirements as satisfied. Again this process continues until all requirements are satisfied. In case of tie between test cases (i.e., test cases having same weight), random selection strategy is used. The remainder of the paper is organized as follows: In Section 2, we discuss background of test case reduction techniques; section 3 presents the basic algorithm of TestFilter; section 4 describes our experimental study related to TestFilter and the results obtained; finally Section 5 concludes the work and discusses future directions. II.
requirements that are next hardest to satisfy and if it is still a tie, one of the tests is chosen at random. Ronne [3] presents a new test reduction heuristic called as Multi-Hit Minimization algorithm. This heuristic first determines different probabilities known as PIE (Propagation, Infection, and Execution) analysis for statement's sensitivity. III.
PROPOSED TEST CASE REDUCTION TECHNIQUE
Development organizations always desire such test case reduction techniques that identify optimal test cases using minimal testing resources. TestFilter is a simple technique that picks non-redundant test cases very efficiently based on their weights. The working procedure of TestFilter is as follows: Inputs: Set of requirements (SR): SR = {Reqi i c N, i < m}
BACKGROUND
Test case reduction (also known as minimization or filtration) technique help to find out effective test cases during maintenance phase and drastically reduce the testing cost. It focuses on reducing test suites to obtain a subset that yields equivalent coverage with respect to some criteria [7]. The optimal test suite reduction problem may be stated as follows [8]: Given: Test suite TSj, a set of test cases requirements Reql, Req2, ..., Req11 that must be satisfied to provide the desired test coverage of the program, and subsets of TS,, TS2, TS3., TSm, one associated with each of the Reqi's such that any one of the test cases tci belonging to TSj can be used to test Reqi. Problem: Find a representative set of test cases from TSj that satisfies all of the Reqi.
The Reqi in the foregoing statement can represent various test case requirements, such as source statements, decisions, definition-use associations, or specification items [3]. A representative set of test cases that satisfies all of the Reqi must contain at least one test case from each TSj; such a set is called a hitting set of the group of sets TS,, TS2... TSm. To achieve a maximum reduction, it is necessary to find the smallest representative set of test cases [4]. The main advantageous of TCR techniques are, that test manager, 1) spend less time in executing test cases, 2) examining test results, and 3) managing the data associated with testing. All these savings in time are dependent on the extent to which TCR technique reduces test suite size [2]. However, the most costly risk associated with TCR technique is reduction of fault-revealing test cases during filtration process. Rothermel et. al. [10, 11] conduct some experimental studies to investigate this risk. They found that this loss is so minimal as compared to the benefits that test manager can get by using these techniques. In literature, a number of test case reduction heuristics have been presented by different researchers [9, 12, 13, 14, 15, 16, 17, 18]. Harrold et. al. [8] presents a heuristic which selects test case that hits most of the requirements that are hardest to satisfy. In case of tie, it picks the test case that hit most of the
276
Set of test cases (STC): each test case completely satisfies one or more requirements. STC = ttci i c N, i < n} Set of test suites (STS). STS = tTSi i c N, i < m} Where each test suite in the set of test suites is a function from one or more test cases to exactly one requirement. i.e.,
TSk: (tCi,, tCn -* Req) TSk means that each test case tci,..., tc, in the test suites satisfies the requirement Reqk. Output: Representative Set (RS).
/* initially empty *
Step 1: Calculates a Weighted Set (WS) of test cases. The weighted set is a function from test cases to their weights. The weight of a test case is the number of its occurrences in the set of test suites. The weight of a test case tck is: Weight (tck)
n
Contains (Domain (TS, ), tCk)
WS: (tck, weight (tck)), where k varies from 1 to n and tck is not in RS.
Step 2: Select the first test case (tch) from the WS that has the highest weight. In case of a tie between test cases, use a random selection.
Step 3: Move tCh to the RS, and mark all test suites from STS, which contain tch in their domain. If all test suites of STS are marked then exit, otherwise go back to step 1. Fig. 1 presents the basic algorithm of TestFilter.
algorithm TestFilter input: set of requirements; set of test cases; set of test suites; output: RS:
SR STC STS
i C N, i i C N, i i C N, i
{Reqi {tci {TSi
ml
#include
n} m}
void main(
declare: weighted set WS: {STC ->
O}
{
cout
