2014 8th Malaysian Software Engineering Conference (MySEC)
Development Test Case Prioritization Technique in Regression Testing Based on Hybrid Criteria 1
Mohammad V. Malakooti
Mojtaba Raeisi Nejad Dobuneh, 2Dayang N. A. Jawawi, 3Masitah Ghazali
Department of Computer Engineering, Islamic Azad University UAE Branch
[email protected]
Department of Software Engineering, Faculty of Computing, Universiti Teknologi Malaysia (UTM), Skudai 81310, Johor, Malaysia, 1
[email protected],
[email protected], 3
[email protected]
during the software development cycle. It is a complicated process for web applications based on modern architectures and technologies. User session based testing has the benefit that tests can be automatically constructed from web logs for use in regression testing and they contain sequences of actions that real users have performed. User session based test cases also have the benefit that testers do not need to specify input for test cases. For instance, web applications are accessible through the Internet and each HTTP POST and GET request that a user makes is written to a log file. The logs can then be passed into test cases by using the IP addresses, cookies, and time stamp for each POST and GET request in order to identify the steps of each user and to create the respective test cases [3].
Abstract—Test case prioritization techniques improve the performance of regression testing, and arrange test cases in order to obtain maximum available fault that is going to be detected in a shorter time. In this research the priority is given to test cases that are performed based on multiple criteria and hybrid criteria to enhance the effectiveness of time and cost for proposed technique. This paper shows that our prioritization technique is appropriate for regression testing environment and show that our prioritization approach frequently produces a higher average percentage of fault detection rate value, for web application. The experiments also reveal fundamental tradeoffs in the performance of time aware prioritization. In this technique some fault will be seeded in subject application, then applying the prioritization criteria on test cases to obtain the effective time of average percentage fault detection rate.
In this paper, the technique is developed for test case prioritization based on hybrid criteria for web application testing. By applying hybrid criteria compare the number of tet cases execution with rate of fault detection. Also the time execute will be decrease by running less number of test cases.
Keywords— web application; regression testing, prioritization criteria;
I. INTRODUCTION Web applications have served as critical tools for different business. Failure of these critical tools means the loss of millions of dollars for the organizations that are using them [1]. Web application testing, a software testing technique exclusively adopted to test the applications that are hosted on web in which the application interfaces and other functionalities are tested.
The paper is organized in the following way. Section II briefly describes existing technique for web application regression testing Section III describes the details of the proposed technique. Section IV gives a metric for evaluation average percentage of fault detection rate. Experimental results and their analysis are presented in Section V, and Section VI concludes the paper
Web applications are presentations of organizations and in front of a large number of audience faces. Most of the web applications must run without any interruption during the day and night. This requires that Software testers to detect the software bugs and software engineers to fix those bugs immediately and release the new versions. Under such circumstances the execution of regression tests is performed in order to make the performance of the new version as per the requirements of the client or organization. The fixing of bugs in web applications requires a short time span, so the test suites can help the testers to perform its best in detection of new faults during the testing phase [2].
II. RESEARCH BACKGROUND In simplest form we can easily execute all the existing test cases in the test suite without any extra handling. However, software engineers know about the gradually growing size of the test suites due to software modifications. Thus, executing the entire test suite would be expensive. This leads the software engineers to think about deploying efficient techniques to reduce the effort that is required for regression testing in different ways. In the life cycle of an application, a new version of the application is created as a result of (a) bug fixes and (b) requirements modifications [4]. A large number of reusable test cases may be accumulated from different application
The purpose of regression testing is to provide confidence that the newly introduced changes do not obstruct the behavior of the existing, unchanged part of the software. Regression testing is one of the largest maintenance costs
978-1-4799-5439-1/14/$31.00 ©2014 IEEE
301
the approach is to generate the test cases without any awareness web application’s internal structure. The test cases that are generated by user sessions are not too much dependent on different technologies that are required for web based applications.
versions suitable to test newer versions of the application. However, running all the test cases may take a significant amount of time. An example that may spend weeks in the execution of all the test cases of an earlier version [5], Regarding the time restrictions, software testers need the selection and ordering of a covering subset of test cases for execution. The three major approaches for regression testing are test suite minimization, test case selection and test case prioritization [6].
III.
HYBRID TEST CASE PRIORITIZATION
Priority test cases are one approach to schedule the test cases in an order based on criteria that increases the effectiveness in meeting a performance goal. Scheduling test cases in this manner are defined as test case prioritization. To reduce the cost of regression testing and the time involved in it, software testers may prioritize their test cases so that those which are more important, by some measure, are run earlier in the regression testing process [5].
Test case prioritization (TCP) helps us to find out more faults by using different ordered of the test cases. In prioritization process, it is assumed that all test cases must be executed. But, it tries to get the best schedule running of test cases in a way that if the test process is interrupted or early halted at an arbitrary point, the best result is achieved in which more faults are detected. TCP is introduced by Wong et al. [7].
A hybrid prioritization process is associated with the selection process of criteria for schedule test cases in order to better fault detection rate with time effectiveness. Combining criteria have the potential to improve the effectiveness of regression testing techniques over those with a single criterion. In TCP all test cases to be executed, but to achieve the best results, it’s applying the criteria to prioritize the test cases. This effort is for helping to find out the different optimal combinations of the test cases, and it tries to best schedule running of test cases in a way that if the test process is interrupted or early halted at an arbitrary point, the best result that is finding more faults is achieved. Hybrid criteria for prioritization in regression testing have been used by researchers.
Structural coverage is the most commonly used metric for prioritization [8]. The logic of this criterion is that faster structural coverage of the whole software code leads to maximum detection of faults in a limited time. Therefore, the aim of this approach is achieving higher fault detection rates, via faster structural coverage. Although the most common prioritization technique is about structural coverage in different forms, some prioritization techniques are presented that have different criteria [9], [10], [11]. An approach which is presented by Sampath et al., for TCP in order to test the web applications with different user sessions of previous software versions is recorded. The best test cases are session-based for a web application testing due to the reflection of real user patterns, and make the testing process quite realistic [3].
In [7] Sampath presented a uniform framework to formalize the notion of combining multiple criteria into a hybrid. Walcott et al.[1] has been performed an approach for time-aware test case prioritization using two criteria: 1) execution time and 2) code coverage (in particular, block and method coverage) [1]. Their technique used a genetic algorithm to prioritize the regression test suites based on these two criteria.
User-Session based techniques are new light weight useful mechanisms of testing. Just applying for web applications, Automating test process is more feasible and simpler by user sessions. In user-session approaches the collection of the interactions of the users with the server is collected and the test cases are generated using a suitable policy. Client's requests transported as URLs composed of page addresses and name value pairs are the data to be captured. These data that can be found in the log files stored in web servers or cookies left in clients machines. Captured data about user sessions can be used to generate a set of HTTP requests and turn into a real test case. The benefit of
Test cases for web application is come from log file which has some parameters such as date & time %t, HTTP version %H, request method %m, resource address %U, session ID %S, response code %s.The log file format example shows in Table I.
TABLE I. LOG FILES COLLECTION %t- Date & Time
[30/Aug/2013:22:49:38] [30/Aug/2013:22:49:48] [30/Aug/2013:22:49:55] [30/Aug/2013:22:50:01] [30/Aug/2013:22:51:11]
%H – Request protocol HTTP/1.1 HTTP/1.1 HTTP/1.1 HTTP/1.1 HTTP/1.1
%m – Request method
%U – Requested URL path
%S – User session ID
GET POST GET GET GET
/Login.jsp /Login.jsp /Default.jsp /Books.jsp /ShoppingCart.jsp
[F8430392AB9E841D281E2D7FBD3B1B6C] [F8430392AB9E841D281E2D7FBD3B1B6C] [F8430392AB9E841D281E2D7FBD3B1B6C] [F8430392AB9E841D281E2D7FBD3B1B6C] [F8430392AB9E841D281E2D7FBD3B1B6C]
302
%s response code [200] [302] [200] [200] [200]
www.gotocode.com. For test case generation, the test cases come from the real user session transaction for web application testing, in server side then convert each log file to the test cases. Online Book Store is a portal for shopping books. Users are allowed to register at the bookstore, logging in, searching for books by keyword, browse for books, add books to a shopping cart, rate the books, update personal information, and log out. Bookstore comprises classes use Java scripts for its front end and a create database with MySQL for the back end. Bookstore application has introduced to undergraduate students of University Technologi Malaysia (UTM) to collect log files on server side. The log files have been collected for 60 days.
In this research the prioritization criteria which used for running test cases and combining any two criteria into hybrid are: first ordered by number of most common HTTP requests in pages, Second ordered length of HTTP request chains to better rate of average percentage fault detection (APFD). Third ordered dependency of HTTP requests helps to improve the rate of average percentage fault detection (APFD). Using hybrid criteria for better average time for execution test cases. Test cases performed by random priorities, create a reasonably effective test order with APFD comparable to the other techniques. IV.
EVALUATION
The rate of fault detection is defined as total number of faults detected in a given subset of the prioritized test case order [6]. For a test suite, T with n test cases, if F is a set of m faults detected by T, then let TFi be the position of the first test case t in T', where T' is an ordering of T, that detects fault i. Then, the APFD metric for T' is given as: APFD
1
TF
TF
TF
TF
The subject book store is selected in order to verify it by applying criteria to test cases of web application. The result of applied criteria to the Bookstore is shown in Table II. as it can be seen the third criteria which ordered the test cases base on dependency HTTP requests has been detected more fault at early stage of running test cases by 50 % of whole test cases detected 28 out of 30 faults. The 28 numbers of total faults have been detected by first criteria (number of most common HTTP requests in pages) with executed 60% of existing test cases. The length of HTTP request chains prioritization criteria as a second criterion performed poorly for detecting faults by run more than 90 % of test cases. The effectiveness of fault detection rate is increased while it can be found most of the faults (95% percent) with running half of all test cases, by hybrid criteria together. Test cases executed by random priorities shows a low fault detection rate for web application testing.
(1)
For 100% detection of faults, the time used by each prioritized suit is measured properly. The best possible option to calculate the detection of the total faults is to detect them in earlier stages of the tests. V.
RESULTS
An open source web application online book store chooses as a case study that available on
TABLE II. RESULTS FOR HYBRID TEST CASE PRIORITIZATION Test case execution percentage 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Average number of fault detection for Bookstore Fault detected by numbers of common HTTP requests
Fault detected by length of HTTP requests
Fault detected by dependency HTTP requests
Fault detected by hybrid criteria
Fault detected by random
23 faults
21 faults
23 faults
24 faults
16 faults
25 faults
24 faults
23 faults
26 faults
16 faults
25 faults
26 faults
25faults
26 faults
20 faults
25 faults
26 faults
25 faults
27 faults
24 faults
26 faults
26 faults
28 faults
28 faults
24 faults
28 faults
26 faults
28 faults
28 faults
24 faults
28 faults
26 faults
28 faults
28 faults
24 faults
28 faults
28 faults
28 faults
28 faults
26 faults
28 faults
28 faults
28 faults
28 faults
28 faults
28 faults
28 faults
28 faults
28 faults
28 faults
303
Test case execution
90% 80% 70% 60% 50% 40% 30% 20% 10% 0%
Time / second
The graph in Fig. 1. Indicates the percenntage of running test cases by different prioritization criterria. The hybrid criteria and dependency HTTP request crriteria are more efficient with less numbers of running testt cases and find more faults at an early stage. In other hand more than 80% percent of test cases run for testing applicatiion by length of HTTP request criteria, although the prioritty test cases by frequency http request has 60 percent of ruunning test cases with acceptable fault detection rate.
2.5 2 1.5 1 0.5 0
1.9 1.5
1.2
1.2
Priooritization criteria
60%
90%
80% 50%
Fig. 2. Effects of tim me prioritization criteria
% 50% According to the results using u three criteria is faster than one criteria in some cases, it need to be tested more case studies to evaluated hybrid crriteria in different environment. VI.
Fig. 1. Results of prioritization criteriia
The entire test cases are 291, and avverage time for executing 100% of test cases is 2.48 secondds. The average time of execution test cases for prioritizationn shown in Table III. by each criteria. Prioritization by hybrid multi m criteria are the best time for test cases running with a minimum m average time 1.2 seconds. TABLE III. TEST CASES EXECUTION TIIME Average time of fault detection for Bookstore By first criterion
By second criterion
By third criterion
By hybbrid criteriia
By random
1.5 Sec
1.9 Sec
1.2 Sec
1.2 Seec
2.2 Sec
CONC CLUSIONS
The regression testing domain d has an advantage that actual user-sessions can bee recorded and used for web application testing. In this paper, we examined a new technique for using hybrid crriteria for test case prioritization of such user-sessions web appplications. By utilizing several new prioritization criteria to t these test suites to identify whether they can be applieed to increase the rate of fault detection. Prioritization byy dependency HTTP request criteria and the hybrid criteriia has increased the rate of fault detection for web applicationns. Experimental results indicate that our prioritization approoach frequently yields a higher average percentage of faultss detected (APFD) values, for a web application. The experim ments also revealed fundamental tradeoffs in the performancce of time aware prioritization. This paper proves that our o prioritization technique is appropriate for many regresssion testing environments and explains how the baseline approach can be extended to operate in additional time connstrained testing circumstances.
Prioritization criteria
Time
2.2
ACKNOW WLEDGMENT
The numbers of test cases running for dependency d http request criteria and hybrid criteria have about a the same adequacy and take around the same amoount of time to execute, as observed in Fig. 1. different teest prioritization criteria have the same APFD. The time awaree of running test cases by multiple prioritization criteria has been b concern as part of the result. As it can be seen in Figg. 2. The graph depicts the effectiveness of time execution test t case which, given the priority to test them by using hybrid multiple criteria and dependency http request criteria from other criteria.
We are thankful to Ministry M of Education (MOE) Malaysia for supporting thiis research under Fundamental Research Grant Scheme (FR RGS) by R.J 130000.7828.4F334 and to Universiti Teknologi Malaysia (UTM) for providing resources. REFEERENCES [1]
[2]
[3]
[4]
304
K.R. Walcott, M.L. Soffa, G.M. Kapfhammer, and R. Roos, “TimeAware Test Suite Prioritizatioon,” Proc. Int’l Symp. Software Testing and Analysis, pp. 1-12, July 20006. G. Rothermel, R. H. Untch, C. C Chuand M. J, Harrold, “Prioritizing test cases for regression teesting”, Software Engineering, IEEE Transactions on. 27(10), 929-9948, 2001. S. Yoo, and M. Harman, , “Reegression testing minimization, selection and prioritization: a survey. Software Testing, Verification and Reliability”, 22 (2), 67-120, 20012. S. Elbaum, A. G. Malishevvsky, and G. Rothermel, “Test case prioritization: A family of em mpirical studies”, Software Engineering, IEEE Transactions on. 28(2), 159-182, 1 2002.
[5]
[6]
[7]
[8]
Proceedings of the 2003 Software Reliability Engineering: IEEE, 442-453, 2003. [9] W. E. Wong, J. R. Horgan, A. P. Mathur, and A. Pasquini, “Test set size minimization and fault detection effectiveness: A case study in a space application”, Journal of Systems and Software. 48(2), 79-89, 1999. [10] S, Manchester. “Combinatorial-Based Prioritization For UserSession-Based Test Suite”. Master Thesis, Utah State University, 2012. [11] M. Raeisi Nejad Dobuneh, D.N.A. Jawawi,s V.M. Malakooti, “Web application regression testing: A session based test case prioritization approach”, The International Conference on Digital Information Processing, E-Business and Cloud Computing, 107-112, (2013).
H. Srikanth, L. Williams, and J. Osborne, “System test case prioritization of new and regression test cases”, Proceedings of the 2005 Empirical Software Engineering, International Symposium on: IEEE, 10. 2005. S. Sampath, R, C. Bryce, “Improving the effectiveness of test suite reduction for user-session-based testing of web applications”, Information and Software Technology 54. 2012.pp. 724–738. S. Sampath, R. Bryce, and A. M. Memon, “A Uniform Representation of Hybrid Criteria for Regression Testing,”, IEEE transactions on software engineering, vol. 39, no. 10, October 2013. D. Leon, and A. Podgurski,” A comparison of coverage-based and distribution-based techniques for filtering and prioritizing test cases”,
305
