2008 International Conference on Computer Science and Software Engineering
Performing Combinatorial Testing on Web Servicebased Software Chengying Mao School of Software, Jiangxi University of Finance and Economics, 330013 Nanchang, China E-mail:
[email protected]
form complete test cases [3]. It is not hard to find that, this method can achieve satisfactory coverage via the possible minimum number of test cases. Thus, it is a rational way for increasing efficiency and effectiveness of testing. In the paper, we adopt this testing strategy to test the WSBS according to the characters of this new kind of software. Furthermore, a debugging method based on rough set reasoning is proposed to locate faults.
Abstract—Although Web service-based software (WSBS) possesses a lot of merits such as loose coupling and good interoperability, its some new characters also bring challenges to latter maintenance activities inevitably. In the paper, we proposed a combinatorial testing framework for the WSBS, which mainly includes test cases generation and knowledge-based debugging. For the former, naïve method and “greedy +backtracking” method are detailedly addressed. At the stage of debugging, rough set reasoning is adopted to generate association rules, which can help maintainers to locate faults in the WSBS by using dynamic slicing technology. In addition, the feasibility and effectiveness of our approach is validated by some working examples and experiments.
The rest of the paper is organized as follows: In the next section, we address the foundation and running example for testing. In Section 3, the method of performing combinatorial testing on Web service-based software is discussed, and the framework of whole testing process is also introduced. The debugging method based on rough set reasoning is presented in Section 4. Section 5 presents the conclusions and future work in this area.
Keywords-Web service; combinatorial testing; test cases generation; rough set; debugging
I.
INTRODUCTION
Web services (WS) technique has been widely adopted in constructing new-style software systems during it development. Although Web service-based software (WSBS) possesses a lot of merits such as good interoperability, easy integration, reuse and evolvement, its some new characters also bring challenges to latter maintenance activities inevitably. Therefore, exploring new testing methods and strategies is a hot issue in the field of software engineering [1].
II.
Combinatorial testing has been extensively used across many disciplines including medicine, agriculture, manufacturing, etc., and was recently applied to software testing [4]. As shown in Figure 1, we use X {x1, x 2, ", xn} to indicate the set of all input variables identified for use in testing WSBS. Let pi denotes the number of parameters that have vi values (1 d i d k ) . Note that
Web service is an independent software component with a specific function, thus developers of Web service-based software can access its interface merely through the related standards and protocols such as WSDL, SOAP and UDDI. Furthermore, the whole system may contain several kinds of heterogeneous service units (i.e., components) and run in an open context. If testing problems of this system is taken into account, black-box testing (also called functional testing) [2] should be a preferred method during testing process. The service component or integrated system of some service units can be viewed as a black box, and testers can only change the value of its interface parameters during testing process. In order to ensure its high quality, the generated test cases should have the ability of covering combinatorial pairs of input parameters as many as possible, which is so-called combinatorial testing.
n ¦ i k1 pi . Without loss of generality, we assume that vi are listed in descending order (i.e., vi ! vj if i ! j ). Thus, the testing problem can be uniquely represented as PCT [v1 p1 , v 2 p 2 ,", vk pk ] .
Combinatorial testing (CT) strategy defines ways to select values for individual input parameters and combines them to
978-0-7695-3336-0/08 $25.00 © 2008 IEEE DOI 10.1109/CSSE.2008.1284
PRELIMINARY KNOWLEDGE AND RUNNING EXAMPLE
Figure 1. WSBS with set of input variables X
755
How to generate the minimum combinatorial test suite is a NP-hard problem [7], and hasn’t been perfectly resolved until today. For WSBS combinatorial testing issues, the paper provides two naïve methods first, and then gives our primary settlement.
In combinatorial testing, the conservative strategy is to test all possible combinations of input variables. For the model [45 ] , it needs 1024( 45 ) test cases. Since the size of test suite will grow large very quickly, this strategy is not practical in most cases. Instead one can create test suites that guarantee tway ( 2 d t d n ) coverage or pair-wise (a.k.a. 2-way testing) [5]. In n-way testing, a set of test cases is generated to cover a subset of the possible combinations of the system’s input variables, rather than attempting to cover all possible combinations. While the trade-off between fault detection effectiveness (FDE) and cost is considered, pair-wise testing method seems better, and this paper only discusses this case.
(1) Naïve methods for WSBS testing In fact, test cases that cover all pair-wise interface interactions can be viewed as a 2-factor design in experimental design theory. Therefore, the validated orthogonal tables (arrays) [8] can be treated as references to generate test suite. For the working example, testers can look up a 4-factor and 3level orthogonal table in mathematics handbook, and then convert each row in the table into a test case according to the candidate values of parameters. The limitation of this method lies in that, this algebraic approach mainly handles the case of all parameters having the same number of values, and nearly can’t deal with the case where parameters have different numbers of values.
In order to address the steps of our method clearly, we take a simple Web service-based application as an example. The application only contains one Web service component, with main function of date judgment and calculation. It has four input parameters: the former three are year, month, date, and flag. The value types of input parameters are listed as following.
D. M. Cohen et al. proposed a heuristic search-based approach to generate pair-wise test suite, and designed an automatic efficient test generator (AETG) [9] to help worldwide testers to perform combinatorial testing. This system can be accessed via Web, so we can input testing indicators into it to gain the combinatorial results for the WSBS under test. This method has been widely adopted and proven to be high effectiveness. However, the AETG adopts a greedy search technique to find a pool of test cases, so it doesn’t always produce test suites with known covering array numbers. For this reason, we add a backtracking procedure to greedy algorithm to avoid Local optimization problem.
TABLE 1. VALUES OF INPUT PARAMETERS IN EXAMPLE Para. Year (Y) Month (M) Date (D) Flag (F)
Abbr.
Description
Y1 Y2 Y3 M1 M2 M3 D1 D2 D3 F1 F2 F3
Y%100==0 && Y%400!=0 Y%400==0 Y%100!=0 M in the set {1,3,5,7,8,10,12} M in the set {2} M in the set {4,6,9,11} D
