2012 12th International Conference on Quality Software
Generating Test Data for Structural Testing Based on Ant Colony Optimization Chengying Mao1,2,
Xinxin Yu1
Jifu Chen1
1. School of Software and Communication Engineering, 2. The State Key Laboratory of Software Engineering, Jiangxi University of Finance and Economics, Wuhan University, 330013 Nanchang, China 430072 Wuhan, China
Jinfu Chen3 3. School of Computer Science and Telecom. Eng., Jiangsu University, 212013 Zhenjiang, Jiangsu, China
Corresponding Email:
[email protected]
outperforms the traditional test data generation methods in several aspects such as cost, efficiency and fault-revealing ability [3]. At present, meta-heuristic search algorithms including genetic algorithm (GA) [4], simulated annealing (SA) [5], Tabu search (TS) [6] and particle swarm optimization (PSO) [7] have been widely used to assist software testing activities. GA and SA, in particular, become the two most well-known methods. Ant colony optimization (ACO), one of MHS algorithms, has been widely used to obtain the optimal solution in management and industrial field. But its application for generating test data has not been deeply studied until today. In the paper, the basic ACO is modified to handle the structural test data generation problem, and some strategies are used to speed up the convergence.
Abstract — Software testing has been always viewed as an effective way to ensure software quality both in academic and industry. In fact, the quality of test data set plays a critical role in the success of software testing activity. According to the basic line of search-based software testing, we introduced ant colony optimization (ACO) to settle this problem and proposed a framework of ACO-based test data generation. In our algorithm TDG_ACO, the local transfer rule, global transfer rule and pheromone update rule are re-defined to handle the continuous input domain searching. Meanwhile, the most widely-used coverage criterion, i.e., branch coverage, is adopted to construct fitness function. In order to validate the feasibility and effectiveness of our method, five real-world programs are utilized to perform experimental analysis. The results show that our algorithm outperforms the existing simulated annealing and genetic algorithm in most cases.
II. ACO-BASED TEST DATA GENERATION
Keywords — Test data generation; ant colony optimization; branch coverage; fitness function; meta-heuristic search
A. Overall Framework of Test Data Generation The basic ACO algorithm is mainly used to settle the discrete optimization problem. Here, we have to reform it so that it can generate test data set with high coverage. Without loss of generality, the test data generation problem can be formally described as follows: Given a program under test P , suppose it has n input variables X ( x1 , x2 , " , xn ) , and each input variable xi takes its values in a domain Di (1 d i d n) . Thus, the corresponding input domain of program P can be expressed as D D1 u D2 u " u Dn . For a given coverage criterion C , it should produce a test data set which can traverse all elements w.r.t. C . In our work, we adopt branch coverage as coverage criterion. The corresponding fitness function is denoted as f ( X ) , and the test data generation problem is converted to a maximum problem. When we want to generate test data based on ACO algorithm, it should ensure that the test driver and ACO algorithm work together collaboratively. At first, the program under test (PUT) is analyzed to extract the static construct information. Meanwhile, the input parameters are encoded into the position vector of ant colony.
I. INTRODUCTION In general, test data automated generation is a key and challenging task in automated software testing [1]. A good test data set can reduce testing execution times and increase fault exposure probability. Apart from the above two issues, coverage ratio is also a critical item to evaluate the quality of test data set. In structural testing (white-box testing), the coverage usually focuses on the construct elements of program code, such as statement, branch, definition-usage pair and path etc. In our work, we mainly consider the structure-oriented test data generation problem. In recent years, search-based software testing (SBST) technology has attracted great attention and gained extensive popularity in testing activity [2]. The basic idea of SBST is to adopt some meta-heuristic search (MHS) algorithms to generate test data with high coverage ratio and fault-revealing ratio. In general, the structure-oriented test data generation is firstly converted into a fitness function optimization problem. Subsequently, the search algorithm is used to produce test inputs, which can gradually gain upon the full coverage. Experimental results show that, SBST 1550-6002/12 $26.00 © 2012 IEEE DOI 10.1109/QSIC.2012.12
98
On the other hand, ACO algorithm uses the initial position vector to trigger the evolution of ant colony. During this process, local search and global search are performed to update the pheromones on the search paths to find the best solution at the current iteration. At the same time, it also needs to update the global optimal solution. Once the current iteration is finished, the position vector of ant colony is decoded into the input data of PUT. Then, the test driver will execute the PUT with such input data to record the traces and output results. Afterwards, the traces are utilized to count the coverage information. Accordingly, the fitness function can be calculated in the light of the coverage criterion. Subsequently, the value of fitness function can be used to direct the next iteration of ant colony evolution. The search process can be terminated when one of the following two conditions is satisfied. (1) the max iteration time is reached, or (2) the test data set achieves the full coverage for some specific criterion.
02 for d o 1: n 03 initialize the position vector ant[i]. x[ d ] of ant i ; 04 calculate the fitness ant[i]. fitness of ant i ; 05 endfor 06 ant[i ].W 0 1 , ant[i ].count 0 ; 07 for k o 1: L 08 ant[i ].record [ k ] 0 ; 09 endfor 10 endfor 11 get gbest from all ants’ fitness; Stage 2: Optimum Solution Search 12 while ( gen MaxGen or doesn’t reach full coverage of criterion C ) 13 apply local search strategy to each ant i , and calculate the fitness of ant i ; 14 call Procedure GlobalSearch(); 15 reacquire gbest after the search process; 16 call Procedure UpdatePheromone(); 17 decode position ant[i]. x[ d ] into a test case tci TS ; 18 collect the coverage information by executing program with tci ; 19 endwhile 20 return TS {tci } ;
B. Test Data Generation Algorithm As mentioned above, test data generation algorithm is essentially a combination of test driver and ACO. The details are expressed by the pseudo-code TDG_ACO. In the algorithm, the population size of ant colony is L , and the number of PUT’s input variables is n . The first stage is to perform the initialization operations on ant colony and the related parameters. In the segment of line 06 to 09, W 0 is the initial pheromone of each ant, where we set it to 1. count represents the number of ants which have changed their positions after one time of iteration. record [ L] stores the ID numbers of these ants. Line 13 conducts the local search for each ant, and the transfer rule is to make each ant randomly search the solution in the neighborhood of the maximum neighbor radius rmax . Here, the value of rmax is set in the range of 5 to 10. In line 14-18, the algorithm executes global search to get the best fitness gbest , and updates the pheromone information at the same time. The global search and pheromone update are implemented by the following two procedures: GlobalSearch() and UpdatePheromone(). After these two tasks are finished, the position of each ant is decoded to a test case tci . Then, use these test cases as input to gather the coverage information. Algorithm’s execution process is controlled by the coverage criterion C or the max generation MaxGen .
The details of two key sub-procedures, i.e., GlobalSearch() and UpdatePheromone(), can be addressed as below: (1) Global Search Sub-procedure For ant i , if it satisfies the condition: f ( xi ) ! f avg q q0 , it will perform the global stochastic search. Here, f avg is
the average fitness of ant colony, q is a random number from 0 to 1, and q0 is a pre-set threshold. Otherwise, it should transfer to ant j with the following probability. d /T if (dij 0 i z j ) W i, j e ij (1) p (i , j ) , diu /T (dij 0 i j ) ¦uN W i, u e i
where dij f ( xi ) f ( x j ) , and T is a pre-set slope. The amount of pheromone trail W (i, j ) represents the learned desirability of choosing ant j w.r.t. ant i . Ni is the set of nodes in the neighborhood of node i which can be visited in next step. When ant i is searching according to the above rule, it will find a new source food (i.e., a good solution), so its new position can be calculated via formula (2). ° random value of the neighbor of x j , if U U0 (2) xic ® otherwise °¯M x j (1 M ) xi , where U is a random value form 0 to 1, U0 and M are two pre-set values. (2) Pheromone Update Sub-procedure For the test data generation problem, we re-write the pheromone update rule here. Suppose the number of ants
Algorithm. TDG_ACO Input: (1) the program under test P , and its input variable list X ( x1 , x2 ," , xn ) ; (2) structural coverage criterion C ; (3) algorithm parameters, i.e., D , E , M , U0 , q0 , T , L and rmax ; (4) the maximum evolution generation MaxGen . Output: test data set TS satisfying the criterion C . Stage 1: Initialization 01 for i o 1: L
99
TABLE 1. THE BENCHMARK PROGRAMS USED FOR EXPERIMENTAL ANALYSIS Program
Argument Number
Branch Number
LOC
triangleType
3
5
31
type classification for a triangle.
by authors
gcd
2
5
38
greatest common denominator.
refers to [9]
calday
3
11
72
calculates the day of the week.
refers to [10]
isValidDate
3
16
59
checks a date is valid or not.
cal
5
18
53
computes the days between two dates.
Description
TABLE 2. THE PARAMETER SETTING FOR EXPERIMENT AND ALGORITHMS
Here, 'W k f ( xk ) , 1 d k d cnt , D is the pheromone decay parameter. Of course, 'W k can also be directly assigned with the fitness ( f ( xbest ) ) of the current best solution. C. Fitness Function According to the previous testing experience, branch coverage is the most cost-effective approach in structural testing. Here, we adopt branch coverage as criterion for experimental analysis. The value of branch coverage is the ratio of the covered branches to all branches. Meanwhile, the fitness function can be constructed in accordance with branch distance. In fact, so called branch distance is the deviation for a given branch predicate when the variables are assigned to the input values. The metrics of branch distance in our experiments refer to the definitions in [5] and [8]. Suppose a program contains s branches, the fitness function of the whole program can be defined as below.
s
Parameter population size
30
Common Issues
number of max generations
100
number of runs
100
SA
selection strategy
gambling roulette 0.90
mutation probability
0.05
initial temperature
1.00
cooling coefficient
0.95
maximum radius rmax
6 0.7 1.0 0.5
balance coef. E ACO
Value
crossover probability
pheromone decay para D . adjustment coef. M threshold of global random search control q0 threshold of neighborhood transfer control U0
2
i 1
Item
GA
s ª º (4) fitness 1 « 0.01 ¦ wi f (bchi ) » i 1 ¬ ¼ where f (bchi ) is the branch distance function for i-th branch in program, and wi is the corresponding weight of this
¦
by authors refers to [11]
test data. The parameters of these three algorithms in our experiment are shown in Table 2.
which transfer to the position of ant j in an iteration is cnt , the pheromone update rule can be modified as below: cnt (3) W i, j m (1 D ) W i, j ¦ k =1 'W k
branch. Obviously,
Source
0.5 0.3
B. Comparison Analysis In this section, we mainly compare GA, SA and ACO on the ability to generate test data. While considering the effectiveness and efficiency of these three algorithms, the following four issues are adopted as evaluation metrics. (1) Average coverage (AC), i.e., the average of achieved branch coverage of all test inputs in 100 runs. (2) Successful rate (SR), i.e., the probability of all branches can be covered by the generated test data. (3) Average (convergence) generation (AG), i.e., the average evolutionary generation of realizing all branch coverage. (4) Average time (AT), i.e., the average execution time (millisecond) of realizing all-branch coverage. According to the experimental results as shown in Table 3, test data generation algorithm TDG_ACO has obvious advantage over the other two algorithms, i.e., TDG_GA and TDG_SA. Specifically, the metric AC and SR of TDG_ACO algorithm can attain 100% for triangleType and calDay. Meanwhile, the average (convergence) generation (i.e., AG) of TDG_ACO is also shorter than the corresponding metrics of TDG_GA and TDG_SA, nearly 70 percent
wi 1 . The weight should be assign-
ed according to the reaching difficulty of each branch. In general, the harder reachable branch the higher weight value. III. EXPERIMENTAL ANALYSIS A. Experiment Setup In order to validate the effectiveness of ACO-based test data generation, we performed experimental analysis on five well-known benchmark programs. The details of these programs are shown in Table 1. Some of them are available at http://tracer.lcc.uma.es/problems/testing/index.html. The experiment is performed in the environment of MS Windows 7 with 32-bits and runs on Pentium 4 with 2.4 GHz and 2 GB memory. The algorithms are implemented in C++ and run on the platform of MS Visual Studio .Net 2010. In our experiments, we mainly compare ACO, GA and SA in terms of the ability to generate
100
TABLE 3. EXPERIMENTAL RESULTS OF THREE ALGORITHMS FOR TEST DATA GENERATION
TDG_GA
TDG_SA
TDG_ACO
Program
avg cov.
succ. rate
avg gens
avg time (ms)
avg cov.
succ. rate
avg gens
avg time (ms)
avg cov.
succ. rate
avg gens
avg time (ms)
triangleType
0.95
0.76
13.79
10.83
0.9988
0.994
42.17
3.77
1.00
1.00
5.76
6.22
1.00
0.76
0.85
1.00
1.00
2.69
0.15
1.00
1.00
0.36
1.37
gcd
1.00
calDay
0.9631
0.65
35.80
35.73
0.9997
0.996
28.29
1.79
1.00
1.00
9.51
12.84
isValidDate
0.9995
0.994
21.69
11.68
0.9821
0.963
15.16
2.43
0.9998
0.998
15.16
19.94
cal
0.9902
0.987
15.24
11.41
0.9927
0.965
10.26
0.73
1.00
1.00
9.57
11.18
Grant No. GJJ10433, the Open Foundation of State Key Laboratory of Software Engineering under Grant No. SKLSE2010-08-23, and the Program for Outstanding Young Academic Talent in Jiangxi University of Finance and Economics. We are grateful to Prof. Ilkka Tervonen for providing some useful comments and suggestions.
improvement. For the program cal, the metric AC and SR of TDG_ACO show better than those of other two algorithms. While considering the convergence generation, TDG_ACO is 5 generations faster than TDG_GA, and about one generation than TDG_SA. For the program isValidDate, algorithm TDG_ACO and TDG_GA have no significant difference for the metric of AC, but are better than TDG_SA. For the metric of AG, TDG_ACO is faster than TDG_GA, but is about the same as TDG_SA. For the program gcd, the above four metrics of three algorithms are very close to each other. Only the AG of TDG_ACO has little decrease w.r.t. other two algorithms. In a word, TDG_ACO has obvious improvement to other two meta-heuristic algorithms for the metric AC, SR and AG. For the metric of AT, TDG_ACO is faster than TDG_GA but little slower than TDG_SA. This phenomenon means that single-step evolution of TDG_ACO will cost more time than that of TDG_SA. The possible future improvement can be focused on reducing the time of global/local search and pheromone update.
REFERENCES [1]
D. C. Ince, “The Automatic Generation of Test Data,” The Computer Journal, vol. 30, no. 1, 1987, pp. 63-69.
[2]
P. McMinn, “Search-Based Software Testing: Past, Present and Future,” The 4th International Workshop on Search-Based Software Testing (SBST’11), in conjunction with the 4th IEEE International Conference on Software Testing (ICST’11), 2011, pp. 153-163.
[3]
S. Ali and L. C. Briand, “A Systematic Review of the Application and Empirical Investigation of Search-Based Test Case Generation,” IEEE Transactions on Software Engineering, vol. 36, no. 6, 2010, pp. 742-762.
[4]
B. F. Jones, H. H. Sthamer, and D. E. Eyres, “Automated Structural Testing Using Genetic Algorithms,” Software Engineering Journal, vol. 11, no. 5, 1996, pp. 299-306.
[5]
N. Tracy, J. Clark, K. Mander, and J. McDermid, “An Automated Framework for Structural Test-Data Generation,” Proc. of the 13th International Conference on Automated Software Engineering (ASE’98), Hawaii, 1998, pp. 285-288.
[6]
E. Diaz, J. Tuya, and R. Blanco, “Automated Software Testing using a Meta-heuristic Technique Based on Tabu Search,” Proc. of the 18th IEEE International Conference on Automated Software Engineering (ASE’03), 2003, pp. 310-313.
[7]
A. Windisch, S. Wappler, and J. Wegener, “Applying Particle Swarm Optimization to Software Testing,” Proc. of the 9th Annual Conference on Genetic and Evolutionary Computation (GECCO’07), 2007, pp. 1121-1128.
[8]
B. Korel, “Automated Software Test Data Generation,” IEEE Transactions on Software Engineering, vol.16, no. 8, 1990, pp. 870879.
[9]
A. Bouchachia, “An Immune Genetic Algorithm for Software Test Data Generation,” Proc. of the 7th International Conference on Hybrid Intelligent System (HIS’07), 2007, pp. 84-89.
IV. CONCLUDING REMARKS The quality of test data set has a great impact on the fault-revealing ability of software testing activity. In the paper, we presented an ACO-based test data generation framework. The basic ACO algorithm is reformed to collaboratively work with software test driver. The main work is to re-define three rules: Local transfer rule, global transfer rule, and pheromone update rule. In order to validate our proposed solution for generating test data, five well-known benchmark programs are utilized to perform experimental analysis. The results show that our algorithm outperforms the existing simulated annealing and genetic algorithm for the metrics, such as average coverage (AC), successful rate (SR) and average convergence generation (AG). ACKNOWLEDGMENT
[10] E. Alba and F. Chicano, “Observations in Using Parallel and Sequential Evolutionary Algorithms for Automatic Software Testing,” Computers and Operations Research, 2008, vol. 35, pp. 3161-3183.
This work was supported in part by the National Natural Science Foundation of China (NSFC) under Grant No. 60803046 and 61063013, the Natural Science Foundation of Jiangxi Province under Grant No. 2010GZS0044, the Science Foundation of Jiangxi Educational Committee under
[11] P. Ammann and J. Offutt, “Introduction to Software Testing,” Cambridge University Press, 2008, pp. 149-150.
101
