Multi-deterministic Prioritization of Regression Test Suite Compared ...

Multi-deterministic Prioritization of Regression Test Suite Compared: ACO and BCO Shweta Singhal, Shivangi Gupta, Bharti Suri and Supriya Panda

Abstract. Regression Test Suite Prioritization has become a very prominent area of research in software engineering due to the advancements in the field of technology. Software development budget generally keeps very little room for the software maintenance phase. Hence instead of developing new test cases for any version of the software, it is intelligent to prioritize the available test suite to check the correctness of the available code. Researchers have come across many actual natural systems that are remarkable examples of solving any problem efficiently. In this paper we have compared the work of two nature inspired systems: Ant Colony Optimization (ACO), Bee Colony Optimization (BCO). The comparison has been analyzed using eight examples used to solve the regression test prioritization problem. The effectiveness of the two techniques discussed here have been compared using several metrics namely Average Efficiency (AE) and Average Percentage of Test Suite Size Reduction (ASR), Percent Average Execution Time Reduction (AETR).

Keywords. Regression testing Ant colony optimization (ACO) Bee colony optimization (BCO) Average efficiency (AE) Average percentage of test suite size reduction (ASR) Percent average execution time reduction (AETR)

S. Singhal (&) B. Suri University School of Information and Communication Technology, G.G.S. Indraprastha University, Dwarka, New Delhi, India e-mail: [email protected] B. Suri e-mail: [email protected] S. Gupta S. Panda The Northcap University, Gurgaon, Haryana, India e-mail: [email protected] S. Panda e-mail: [email protected] © Springer Science+Business Media Singapore 2016 R.K. Choudhary et al. (eds.), Advanced Computing and Communication Technologies, Advances in Intelligent Systems and Computing 452, DOI 10.1007/978-981-10-1023-1_19

187

188

S. Singhal et al.

1 Introduction Any project, be it commercial or professional, hardware or software, civil or mechanical has certain budget allocated to it in terms of finance, time, man power etc. Out of these sanctioned budget factors, the majority of the portion (80–90 %) is spent in its development that is from planning to delivery of the product and a small portion (10–20 %) is left for the maintenance phase. On the contrary, the Project Development phase lasts negligibly for a shorter duration of time as compared to its maintenance phase. Thus, thorough and faster mechanisms are required to check the modified project with minimal possible rework to be done in the maintenance phase. In software engineering, prioritization of existing test suite is done to expedite the maintenance phase processes. Many researchers have done enormous work in this context so as to make the software maintenance phase easy and efficient. Ant Colony and Bee Colony, both these evolutionary computation techniques have been explored by the researchers for solving problems in many diverse domains. Time-constrained test case prioritization is a deterministic problem. This paper compares the already applied ACO and BCO for test case prioritization [1, 2]. These techniques have been applied with added constraint of maximum fault coverage and minimum execution time taken by the prioritized regression test suite. Hence, we call this as multi-deterministic prioritization. The current study includes experimental comparison of ACO and BCO applied to multi deterministic prioritization for eight sample programs averaged over ten sample runs on each. The paper is organized in the following sections: Section 2 gives the background and related work ensuing in ACO, BCO and Regression Testing. Sections 3 and 4 provide brief explanation of the ACO and BCO techniques. This is followed by Sect. 5 elaborating on the comparison and finally the last section concludes the findings.

2 Related Work Prioritization for software development has been tackled in various ways by Rothermel et al. [3]. Li et al. [4] performed an empirical study by using various greedy algorithms for test case prioritization. Time-aware test suite prioritization was proposed by Walcott et. al. [5] where testing is bounded within a predefined period of time. ACO is a nature inspired meta heuristic approach introduced by Dorrigo in [6]. It has been applied successfully to solve various NP hard optimization problems. Artificial ants find their applicability to a considerable number of optimization problems. Quadratic assignment [5], prioritization [7], scheduling [8], sequential ordering [9], routing in networks etc. have benefited from the ACO technique. Singh et al [10] proposed an algorithm for Test Case Prioritization using ACO in a time restricted environment. This algorithm was then implemented as ACO_TCSP tool in [1]. This tool is one of the tools used to gather the comparison

Multi-deterministic Prioritization of Regression Test Suite …

189

details in the current study. Suri and Singhal [11] also accomplished a survey of application of the ACO technique in various fields of software testing such as test case generation, selection, prioritization etc. ACO on test case prioritization problem has also been individually tested and analyzed on various data sets and in varying parameters in [12, 13]. The current paper has taken the earlier implementation [1] of the ACO technique for comparison. An automated software testing application using the BCO by producing the CFG of the original code and the modified code for comparison of test suite has been presented in [14]. An Artificial Bee Colony Algorithm (ABC) was implemented in [15] in which cooperation and cooperative behavior of bees have been used to optimize the large scale projects. It optimizes the high dimensional variables considering them as graphs. Furthermore, A Test Case Generation algorithm for BPEL processes has been given in [16] where a transition coverage criterion has been used as the fitness function. Regarding regression test suite prioritization, for total fault coverage in minimum execution time of a test case and total code coverage in give time, algorithm has been designed and implemented in [2, 17] respectively. A secure Bee Colony optimization algorithm [18] for pairwise test set generation has been presented, implemented and analyzed w.r.t. the parallel techniques available. An efficient algorithm to generate test cases using Unified Modelling Language has been proposed in [19], where minimum possible timeframe of execution is ensured using (BCO-mGA), software testing technique ensures maximum coverage by executing the final minimized test suite. The technique proposed in [20] ensures full path coverage, by generating paths from cyclomatic complexity and hence it ensures of a reliable test of software under test. Furthermore, A Regression testing technique has been proposed in [21] which uses traceability parameter to check agent error of the modified code. Next, in [22] a novel approach using Bee Colony Optimization (BCO) based intelligent search algorithm for Criticality Analysis based on Sensitivity and Severity metrics and Testing has been proposed for the generating test cases efficiently.

3 ACO ACO is a set of steps based on search algorithms of artificial intelligence for finding optimal solutions; this includes ANT System, proposed by Colorni, Dorigo and Maniezzo [23–25]. Natural ants are blind and tiny in size, yet still they are able to explore the shortest route between their nest and the food source. Ants can accomplish this by using their antennas and the pheromone liquid so as to keep informing their community ants about their followed paths. ACO technique is inspired from this very behavior of ants, i.e., capability of being in synchronization while exploring solutions for a local problem based on some previous information gathered and distributed by each ant to the community. In addition, [14] ACO involves two important processes of Pheromone deposition and pheromone trail evaporation. Pheromone deposition, as the name suggests, is the process of ants laying down the pheromone on every path being

190

S. Singhal et al.

followed. Pheromone trail evaporation, on the other hand, is the natural process of depleting the pheromone deposited by evaporation (in case of natural ants), on every path corresponding to the passing time since the pheromone had been deposited. This trail update is achieved when the ants either finish their search or have got the probable shortest path between their nest and the food source. Each multi-deterministic problem defines its own pheromone updating/deletion criteria based on their local search merged with the global search eventually. Artificial ants are programmed to leave a virtual pheromone trail on the digital path segment they follow. The path to be followed by each ant is chosen at random including the support from the amount of “trail” deposited on the probable paths initiating from node of the current ant. Increased pheromone trail on any path would increase the probability of that particular path being followed. Ant would then reach the next node in the path and would again select the further path using the same process. This process would continue until the ant reaches back its starting node. This finished tour followed by each ant gives us a local solution for the possible shortest path which is then analyzed for optimality or explored further by other ants. The complete algorithm and tool for ACO applied to test case selection and prioritization problem has been given by Suri and Singhal [1]. The same has been explained using various examples in [26, 27].

4 BCO BCO is a population based search algorithm in which the bees are the main working agents. Any given problem is solved using the food foraging behavior of honey bees. There are three types of bees in a honey bee comb: Queen Bee—It is a unique bee holding the responsibility of laying eggs, which in turn helps in building the hive population; Drone Bees—These are few in number, which are responsible for mating with the queen bee and help in growing the hive; Working Agents— Thousands of Worker Bees are the main working agents in any bee colony. They do all the maintenance work, bring food for the hive members, guide the fellow bees to the rich food source etc. There are two types of worker bees namely Scouts and Foragers which follow the Exploration and the Exploitation process respectively [2]. The BCO algorithm for maximum fault coverage has already been developed [2] and has been implemented and analyzed in [10]. The same has been used as the basis of our BCO technique compared in the current paper.

5 Analysis and Comparison The ACO and BCO algorithms have been analyzed and compared for the prioritization of fault based regression test suite on eight sample examples. Subsequently, the details are presented in Table 1 and the results for the both are plotted in Figs. 1, 2 and 3.


191

Table 1 Input programs details for ACO and BCO Program no.

Program name

Language

Size (LOC)

No. of faults

Test suite size

Total test suite execution time

P1 P2 P3 P4 P5 P6 P7 P8

CollAdmission HotelMgmnt triangle quadratic cost_of_pub calculator prev_day railway_book

C++ C++ C++ C++ C++ C++ C++ Java

281 666 37 38 31 101 87 129

5 5 6 8 8 9 7 10

9 5 19 19 19 25 19 26

105.32 49.84 382 441 382 82.5 468 177

Fig. 1 ACO versus BCO plot for ASR (average % of size reduction in test suite)

ASR for ACO and BCO

100 80 60 40 20 0 1

2

3

4

ACO

Fig. 2 ACO versus BCO plot for AETR (average % of execution time reduction for the test suite achieved using ACO and BCO)

5

6

7

8

7

8

BCO

AETR for ACO and BCO

150

100

50

0 1

2

3

4

ACO

5

6

BCO

The proposed techniques have been analyzed by running them ten times for each example. We compute the Average Percentage of Test Suite Size Reduction (ASR), Percent Average Execution Time Reduction (AETR) and the Average Efficiency (AE) of the proposed algorithm (ACO/BCO) using following formulas (1), (2), and (3) respectively.

192

S. Singhal et al.

Fig. 3 Radar plot for ACO’s versus BCO’s AE (average efficiency % of efficient runs yielded perfect results on each ACO and BCO)

AE for ACO and BCO 1 100 8 2 50 7

0

3

ACO BCO

4

6 5

ASR =

jTSj jPTSj 100 jTSj

ð1Þ

where, TS is the Total Test Suite Size, PTS is Prioritized Test Suite Size. Figure 1 demonstrates the comparison for ASR computed on the sample eight programs using ACO and BCO respectively. As can be inferred from the graph, ACO and BCO, both techniques provide very high and almost similar average percentage of the reduction in the size of the test suite. This provides motivation for using these techniques in test case prioritization. TET AET AETR = 100 TET

ð2Þ

where, TET is Total Execution Time, AET is Average Execution Time A line graph comparison for the average percentage reduction in the execution time (AETR) achieved using ACO and BCO techniques have been plotted in the Fig. 2. BCO technique is found to achieve comparatively better AETR than the ACO technique. This result is due to the fact that the ACO technique includes complete fault detection as a second deterministic criterion in the path finding using the algorithm; whereas the BCO technique does not necessarily consider complete fault detection as a path finding criterion. Hence, paths found using BCO which are incapable of killing all the faults would be shorter thus are having lesser execution time. BR AE = 100 TR

ð3Þ

where, BR is Best Run of the respective algorithm, TR is the Total no. of Runs Radar plot was chosen to represent the Average percentage efficiency of both the ACO and BCO techniques, multi-variation of the data being the sample eight programs, represented by 1–8 on the radar boundary in Fig. 3. The plot clearly represents better AE of the ACO technique for programs P1–P6, better AE of BCO technique for P8 and same AE for both the techniques for P7. Thus for the current


193

sample data, ACO technique was in general found to represent better average percentage efficiency than the BCO technique for the problem of test case prioritization.

6 Conclusion and Future Work Fault coverage based test case prioritization for minimum execution time has been analyzed and compared using ACO and BCO. It is clear from the above tables and plots that ASR, AERT and AE are mixed for both the approaches. ASR was found to be almost similar for both ACO and BCO. Hence a considerable reduction in the test suite size can be achieved using either of the techniques. AETR was found better for the BCO technique while AE was found better for the BCO technique. These techniques have been tested experimentally on the eight sample programs, and the very nature and size of these programs does not allow the authors to generalize these results. More examples need to be tested and analyzed to give a clearly better technique for test case prioritization. However, the results obtained still provide excellent results for both these techniques and thus the software researchers can benefit from either of the technique for solving the problem of test case prioritization.

References 1. Suri, B., Singhal, S.: Implementing ant colony optimization for test case selection and prioritization. Int. J. Comput. Sci. Eng. 3(5), 1924–1932 (2011) 2. Kaur, A., Goyal, S.: A bee colony optimization algorithm for fault coverage based regression test suite prioritization. Int. J. Adv. Sci. Technol. Korea 29, 17–29 (2011) 3. Rothermel, G., Untch, R.J., Chu, C.: Prioritizing test cases for regression testing. IEEE Trans. Softw. Eng. 929–948 (2001) 4. Li, H., Peng Lam, C.: Software test data generation using ant colony optimization. Trans. Eng. Comput. Technol. (2005) 5. Walcott, K.R., Soffa, M.L., Kapfhammer, G.M., Roos, R.S.: Time aware test suite prioritization. In: Proceedings of ACM/SIGSOFT International Symposium on Software Testing and Analysis, pp. 1–11 (2006) 6. Dorigo, M., Maniezzo, V., Colorni, A.: The ant system: optimization by a colony of cooperating agents. IEEE Trans. Syst. Man, Cybern. Part B: Cybern. 26(1), 29–41 (1996) 7. Gambardella, L.M., Taillard, È.D., Agazzi, G.: A multiple ant colony system for vehicle routing problems with time windows. In: New Ideas in Optimization, pp. 63–76 (1999) 8. Stützle, T., Dorigo, M.: ACO algorithms for the quadratic assignment problem. New Ideas in Optimization McGraw Hill, pp. 33–50 (1999) 9. Singh, Y., Kaur, A., Suri, B.: Test case prioritization using ant colony optimization, association in computing machinery. In: Newsletter ACM SIGSOFT Software Engineering Notes, New York, USA, pp. 1–7 (2010) 10. Kaur, A., Goyal, S.: Implementation and analysis of the bee colony optimization algorithm for fault based regression test suite prioritization. Int. J. Comput. Appl. 41, 1–9 (2012) 11. Suri, B., Singhal, S.: Literature survey of ant colony optimization in software testing. In: The Proceedings of the CSI Sixth International Conference on Software Engineering, Indore (2012)

194

S. Singhal et al.

12. Suri, B., Singhal, S.: Analyzing test case selection and prioritization using ACO. ACM SIGSOFT Softw. Eng. Notes 36(6), 1–5 13. Suri, B., Singhal, S.: Understanding the effect of time-constraint bounded novel technique for regression test selection and prioritization. Int. J. Syst. Assur. Eng. Management. (2014) 14. Jeya Mala, D., Mohan, V., Kamalapriya, M.: Automated software test optimization framework—an artificial bee colony optimization based approach. Inst. Eng. Technol. 4, 334-348 (2010) 15. Liang, Y., Liu, Y.: An improved artificial bee colony (ABC) algorithm for large scale optimization. Int. Symp. Instrum. Measur. Sensor Network Autom. 2, 644–648 (2013) 16. Daghaghzadeh, M., Babamir, M.: An ABC based approach to test case generation for BPEL processes. In: International Conference on Computer and Knowledge Engineering, vol. 3 (2013) 17. Kaur, A., Goyal, S.: A bee colony optimization algorithm for code coverage based regression test suite prioritization. Int. J. Eng. Sci. Technol. 29, 2786–2795 (2011) 18. Dahiya, S.S., Chhabra, J.K., Kumar, S.: Application of artificial bee colony algorithm to software testing. Australian Softw. Eng. Conf. 21, 149–154 (2010) 19. Dalal, S., Chhillar, R.S.: A novel technique for generation of test cases based on bee colony optimization and modified genetic algorithm (BCO-mGA). Int. J. Comput. Appl. 68(19), 0975–8887 (2013) 20. Karnavel, K., Santhoshkumar, J.: Automated software testing for application maintenance by using bee colony optimization algorithms (BCO). In: 2013 International Conference on Information Communication and Embedded Systems (ICICES), Chennai, 21–22 Feb 2013, pp. 327–330 (2013) 21. Srikanth, A., Kulkarni, N.J., Venkat, K., Singh, N., Ranjan, P., Srivastava, P.: Test case optimization using artificial bee colony algorithm, advances in computing and communications. Commun.Comput. Inform. Sci. 192, 570–579 22. Dharmalingam, J., Balamuruga, M., Nathan, S.: Criticality analyzer and tester—an effective approach for critical components identification and verification. In: ICT and Critical Infrastructure: Proceedings of the 48th Annual Convention of Computer Society of India, Advances in Intelligent Systems and Computing, vol. I, pp. 663–670 23. Colorni, A., Dorigo, M., Maniezzo, V.: Distributed optimization by ant colonies. In: Proceedings ECAL’91, European Conference Artificial Life. Elsevier Publishing, Amsterdam (1991) 24. Dorigo, M.: Optimization, learning and natural algorithms. Ph.D. thesis, Politecnico diMilano, Milano (1992) 25. Dorigo, M., Maniezzo, V., Colorni, A.: The ant system: an autocatalytic optimizing process. Technical Report TR91-016, Politecnico di Milano (1991) 26. Dorigo, M., Socha, K.: An introduction to ant colony optimization. IRIDIA Technical Report Series, 10 (2006) 27. Suri, B., Singhal, S.: Test case selection and prioritization using ant colony optimization. In: International Conference on Advanced Computing, Communication and Networks, Chandigarh (2011)