A Parallel Tree Based Strategy for 3-way Interaction Testing

Available online at www.sciencedirect.com

ScienceDirect Procedia Computer Science 65 (2015) 377 – 384

International Conference on Communication, Management and Information Technology (ICCMIT 2015)

A Parallel Tree Based Strategy for 3-way Interaction Testing Mohammad F. J. Klaib College of Computer Sciences and Engineering, Taibah University, Madina P.O.Box 344, Kingdom of Saudi Arabia [email protected], [email protected]

Abstract Software testing is one of the most important phases of the software development life cycle. However, software testing is traditionally seen as a difficult and time consuming activity that is hard to embed in the software development process. Software and hardware testers concentrate on how to minimize the testing time, as well as ensure that the system is also tested well and made acceptable. Basic combinatorial interaction (i.e. pairwise or 2-way) testing has been one of the commonly used methods in achieving the above goal with 50-97 percent of errors detection. However, empirical evidence has proved that the 2-way interaction testing is a poor strategy for testing highly interactive systems. Therefore there is a need for going beyond pairwise testing to uncover these errors. To speed up the process of solving problems, researchers have applied parallel algorithms to various large computationally expensive optimization problems and have succeeded in solving these problems in an acceptable time. Therefore, in this paper we have enhanced our previous strategy “A tree based strategy for test data generation and cost calculation for pairwise combinatorial interaction testing” to work effectively in parallel and to support a 3-way interaction testing. The correctness of the strategy has been proved, and the performance evaluation shows the efficient of the strategy in reducing test size. © 2015 2015The TheAuthors. Authors.Published Published Elsevier © byby Elsevier B.V.B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of Universal Society for Applied Research. Peer-review under responsibility of Universal Society for Applied Research Keywords: parallel

algorithms, combinatorial interaction testing, software and hardware testing.

1. Introduction In software development life cycle (SDLC), Testing plays an importance role, which helps to improve the quality, reliability and performance of systems. To insure a high quality enhancement of software and hardware products, a good testing is required, where testing is the process of executing programs or systems with the goal of finding errors thereby assuring the correctness, completeness and quality of developed hardware or software systems [2, 3, 8, 19, 25]. Lack of testing can lead to disastrous consequences including loss of data, resources, and sometimes lives [5, 18]. A well-tested product or service is necessity to ensure customer's satisfaction. However, exhaustive testing is

1877-0509 © 2015 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of Universal Society for Applied Research doi:10.1016/j.procs.2015.09.099

378

Mohammad F.J. Klaib / Procedia Computer Science 65 (2015) 377 – 384

unaffordable due to combinatorial explosion problem [10-13, 20, 24] which occurs when a small increase in the number of elements that can be combined increases the number of combinations to be tested. To illustrate this problem consider testing the IC7489, a 16 by 4 random access memory [23]. Pin assignment of inputs is shown in Figure 1, we have 10 parameters: four address inputs, four data inputs, memory enable and write enable. Each of these pins has two values L or H.

Fig. 1. IC7489, 16 X 4 random access memory

Table 1. Function table for 7489. ME

WE

Operation

Data Outputs

L

L

Write

Complement of data inputs

L

H

Read

Complement of selected word

H

X

Disable

High impedance

In order to test this IC exhaustively there are 210 (i.e. 1024) combinations of test cases which are to be evaluated. If the time required for one test case to be evaluated is 2 minutes, then it would require nearly 4.26 working days for a complete test to be done for only one IC. If 1000 such IC’s needs to be tested then imagine the time and resources consumed for exhaustive testing to be done. Therefore, it is very clear that combinatorial explosion is a serious issue which has to be considered, and testing most hardware and software systems face this problem. Combinatorial explosion in testing may occur for configurable systems. When systems under test have many configuration parameters, each with several possible values, testing each configuration is sometimes infeasible. Thus, to bring a balance between exhaustive testing and lack of testing, combinatorial interaction testing [28-42] has demonstrated to be an effective technique to achieve reduction of test suite size. Although 2-way interaction testing can detect 50-97 percent of errors [1, 7, 9, 18, 21, 26, 27], empirical evidence has proved that 2-way interaction testing is a poor strategy for testing highly interactive systems [43, 18]. Therefore there is a need for going beyond pairwise testing to uncover these errors. Concerning the issue of speed up gained from parallelization, researchers have applied parallel algorithms to various large computationally expensive optimization problems and have succeeded in solving these problems in an acceptable time [22, 44-46]. Therefore, in this paper we have enhanced our previous strategy “A tree based strategy for test data generation and cost calculation for pairwise combinatorial interaction testing” to work effectively in parallel and to support 3-way interaction testing. The correctness of the strategy has been proved, and the performance evaluation shows that the strategy is efficient in test size reduction.

Mohammad F.J. Klaib / Procedia Computer Science 65 (2015) 377 – 384

2. Methodology Two algorithms were designed; a parallel tree generation algorithm which generates the lists of test cases and a cost calculation algorithm which is used to construct in parallel the test suite with minimum number of test cases. Algorithm 1: Tree Generation Algorithm Input: A set of parameters and their possible values Output: Tree generation Begin X = number of values of first parameter p1 {For the first parameter p1} Ti=Vi, where i=1,2,3,……..,X ; parameter p1 has X values If N=1 then stop and exit Create X threads with unique thread ids. Assign each Ti to a separate child thread and execute all the child threads in parallel Wait for the termination of all the threads. End

Algorithm 2: Cost Calculation and Test Suite Generation Input: Lists of test cases. Each list holds the test cases generated by the tree in one particular branch of that tree. Output: T-way test suites with minimum number of test cases Begin Tempb = Tb (where b is the number of lists of test cases) X = number of values of parameter p1 B=min (Value(p1), Value(p2), …….Value(pn) For i = 2 to N-1 do Begin Generate the i-way covering array for the given parameters. Wmax = N!/((i!)*((N-i)!)) // N – is the number of parameters Let T’ be an empty set where i-way test suites are stored. For a = 1 to B do Begin Testa = concatenate the ath values of all the parameters to form a test case. End For each Testa do Begin Delete all the T-way combinations that Testa covers in the covering array Delete Testa from the Ti Lists T’ = Testa End Creates a set of temporary lists Yi corresponding to the Ti lists, where i= 1,2,…..X, X is the number of values of parameter p1 or the number of lists. Create X threads with unique thread ids. Assign every child thread Th i with one Ti lists, the corresponding Yi lists, i value and Wmax value, and execute all the child threads in parallel. Wait for the termination of all the child threads. Store the i-way test suite generated in the list T’ Tb = Tempb End End

379

380

Mohammad F.J. Klaib / Procedia Computer Science 65 (2015) 377 – 384

The first algorithm constructs the tree based on the parameters and values given. It constructs every branch of the tree in parallel. The number of branches the tree has depends on the number of values of the first parameter i.e. if the first parameter has 3 values then the tree also would have 3 branches. Therefore every branch construction starts by getting one value of the first parameter i.e. branch T1 gets the first value, T2 gets the second value and so on. After the base branches are constructed one child thread is assigned to every branch and the further construction takes place in a parallel manner. Each of the branches considers all values of all the other parameters two, three,…..N where N is the total number of parameters. All the branches consider the values of the parameters in the same order. Once the parallel tree construction is over, at first, the algorithm starts by constructing the covering array, for all possible 3-way combinations of input variables. Then we are ready with all the test cases to start the parallel iterative cost calculation (algorithm 2). In this strategy the cost of the leaf nodes in each of the lists are calculated in parallel in order to reduce the execution time. The cost of a particular test case is the maximum number of 3-way combinations that it can cover from the covering array. The second algorithm starts to include all tree branches, which might definitely give the maximum cost (Wmax) into the test suite. Then these test cases are deleted from the tree branches lists, and the corresponding 3-ways covered by it in the covering array are also deleted. In the next step, the main thread in the algorithm invokes a number of child threads equal to the number of values of the first parameter and calculates the cost of all the test cases in each of the branches in a parallel fashion. Each child thread stores all the test cases with the Wmax value from its corresponding branch into a separate sub-list. The child thread that finishes calculating the cost of all the test cases in its branch first locks the covering array. This thread then looks into its sub-list and includes the test cases stored in it into the test suite only after confirming that the test case definitely has the maximum cost or Wmax value. Then the test cases included in the test suite are deleted from the tree branches list and sub-list, and the corresponding 3-ways that these cover are deleted from the covering array. The other threads wait in a queue until the execution of the first thread is over, after which these threads resume their execution in the order in which they are queued. These threads on resumption re-evaluate the test cases in their sub-list to confirm that these test cases have the Wmax value before including these into the test suite. Thus in the first iteration all the test cases with the maximum Wmax value from all the branches are included in the test suite. Now the Wmax value is decremented by one and the same parallel execution of all the threads continue until all the pairs in the covering array are covered. 3. Results To prove the correctness of the above algorithms and to illustrate how the algorithms work consider the following simple system with parameters and values: x Parameter A has two values A1 and A2 x Parameter B has one value B1 x Parameter C has three values C1, C2 and C3 x Parameter D has two values D1 and D2 To illustrate the minimum test suite construction of 3-way combinatorial interactions testing using our algorithm, for the system mentioned. The algorithm starts constructing the test-tree by considering the first parameter. As the first parameter has two values the tree is said to have two main branches with the first branch using A1 and the second branch using A2. Then each of the branches is constructed in parallel by considering all the values of the second parameter, then the third and fourth and so on. When the branches are fully constructed the leaf nodes gives all the test cases that has to be considered for cost calculation. Since all of the branches are constructed in parallel a significant reduction in time will be there. Figure 2 below shows how the test-tree would be constructed. The test cases generated by the first branch are stored in the lists T1 and the test cases generated by the second branch are stored in T2 respectively. i.e. (A1,B1,C1,D1), (A1,B1,C1,D2), (A1,B1,C2,D1), (A1,B1,C2,D2), (A1,B1,C3,D1), (A1,B1,C3,D2) are stored in T1, and (A2,B1,C1,D1), (A2,B1,C1,D2), (A2,B1,C2,D1), (A2,B1,C2,D2), (A2,B1,C3,D1) and (A2,B1,C3,D2) are stored in T2.

Mohammad F.J. Klaib / Procedia Computer Science 65 (2015) 377 – 384

Fig. 2 Test-Tree Construction

Once the parallel tree construction is over we are ready with all the test cases to start the parallel iterative cost calculation based on algorithm 2. In this strategy the cost of the leaf nodes in each of the lists are calculated in parallel in order to reduce the execution time. The cost of a particular test case is the maximum number of 3-way combinations that it can cover from the covering array. At First, the algorithm starts by constructing the covering array, for all possible 3-way combinations of input variables, i.e. . [A, B, C], [A, B, D], [A, C, D] and [B, C, D], for the example in Fig. 2. The covering array (in table 2) for the above example has 28 3-way interactions which have to be covered by any test suite generated, to enable a complete 3-way interaction testing of the system. Once the covering array is generated (Table 2), the algorithm starts to include all tree branches, which might definitely give the maximum Wmax cost into the test suite. For the above example all the test cases which are included in the test suite are identified in three iterations and there are 12 such test cases shown in Table 3.

Table 2

3-way interaction covering array.

A, B, C

A, B, D

A, C, D

B, C, D

A1, B1, C1

A1, B1, D1

A1, C1, D1

B1,C1, D1

A1, B1,C2

A1, B1,D2

A1, C1, D2

B1,C1, D2

A1, B1,C3

A2, B1,D1

A1, C2, D1

B1,C2, D1

A2, B1,C1

A2, B1,D2

A1, C2, D2

B1,C2, D2

A2, B1,C2

A1, C3, D1

B1,C3, D1

A2, B1,C3

A1, C3, D2

B1,C3, D2

A2, C1, D1 A2, C1, D2 A2, C2, D1 A2, C2, D2 A2, C3, D1 A2, C3, D2

381

382

Mohammad F.J. Klaib / Procedia Computer Science 65 (2015) 377 – 384

Table 3

Generated test suite for 3-way combinatorial interaction. Max

Test Case No.

Test Case

Iteration/ Child Thread No.

T1

A1,B1,C1,D1

1/1

4

[A1,B1,C1][A1,B1,D1][A1,C1,D1][B1,C1,D1]

T4

A1,B1,C2,D2

1/1

4

[A1,B1,C2][A1,B1,D2][A1,C2,D2][B1,C2,D2]

T8

A2,B1,C1,D2

1/2

4

[A2,B1,C1][A2,B1,D2][A2,C1,D2][B1,C1,D2]

T9

A2,B1,C2,D1

1/2

4

[A2,B1,C2][A2,B1,D1][A2,C2,D1][B1,C2,D1]

T5

A1,B1,C3,D1

2/1

3

[A1,B1,C3][A1,C3,D1][B1,C3,D1]

T12

A2,B1,C3,D2

2/1

3

[A2,B1,C3][A2,C3,D2][B1,C3,D2]

T2

A1,B1,C1,D2

3/1

1

[A1,C1,D2]

T3

A1,B1,C2,D1

3/1

1

[A1,C2,D1]

T6

A1,B1,C3,D2

3/1

1

[A1,C3,D2]

T7

A2,B1,C1,D1

3/2

1

[A2,C1,D1]

T10

A2,B1,C2,D2

3/2

1

[A2,C2,D2]

T11

A2,B1,C3,D1

3/2

1

[A2,C3,D1]

Weight

Covered pairs

To evaluate the efficiency of the proposed strategy for 3-way test data generation, we consider two different configurations. The first configuration has non-uniform parametric values. The second configuration has a uniform number of values for all parameters. The two systems configurations used are summarized as follows: x

S1: 5 parameters with 3, 2, 1, 2 and 2 values respectively.

x

S2: 4 3-valued parameters. Table 4 System

Exhaustive number of test cases

TBGCC 3-way results. TBGCC-3 3-way 3-way Test Reduction % suite size

S1

24

16

33.33%

S2

81

31

61.73%

Also to evaluate the efficiency of the proposed strategy in test size reduction by comparing its results with available testing strategies which is also based on parametric interactions. We have identified the following existing strategies that support interaction testing: AETG [4,6], IPO[7], AllPairs [14], TConfig [15], Jenny [16], TVG [17], G2Way[1], GTWay [18] tool. We consider the same two configurations as described above. Table 5

Comparison of PTBGCC-3 with other strategies for generated 3-way test suite size. System

TConfig

Jenny

TVG

GTWay

PTBGCC-3

S1

NS

NS

13

12

16

S2

32

34

33

32

31

NS – Not Supported

Mohammad F.J. Klaib / Procedia Computer Science 65 (2015) 377 – 384

4. Discussion In this paper, we have enhanced our previous strategy “A tree based strategy for test data generation and cost calculation for pairwise combinatorial interaction testing” to work effectively in parallel and to support a 3-way interaction testing. The correctness of the proposed strategy has been proved in section 3. Table 2 showed how to construct the generated tree in parallel. Furthermore, Table 3 showed how the cost calculation algorithm works iteratively to generate the test suite with the order in which the various test cases are actually included, also showed the covered 3-ways interactions in each test case, and they showed that all 3-ways combination have been covered in the generated test cases, which prove the correctness of the proposed strategy. Referring to experiments and evaluation part in section 3. The second column in Tables 4, showed the exhaustive number of test cases for each system, and the third column displayed the generated test suite size for 3-way interactions using the proposed algorithm. The last column showed the percentage of reduction achieved. The presented evaluation showed that the proposed strategy is efficient in test size reduction for uniform and nonuniform parameter’s values. Referring to comparison part in section 3. Tables 5 compared the generated test suite size of the proposed strategy with other strategies. Note that the minimum test suite size is highlighted. The proposed strategy (PTBGCC-3) has produced the best results for the second system (S2), while in the first system (S1) the Gtway strategy has performed better than PTBGCC-3. However, test reduction is a NP-complete problem, for which no strategy may perform the best for all cases. The proposed algorithms could be extended in future to work for general higher multi-way (i.e. 4, 5, …N) and new different models for shared and distributed memory architectures can be used. References 1. M. F. J. Klaib, K. Z. Zamli, N. A. M. Isa, M. I. Younis, R. Abdullah, G2Way – A Backtracking Strategy for Pairwise Test Data Generation, in: Proc. of the 15th IEEE Asia-Pacific Software Engineering Conference, Beijing, China, 2008, pp. 463-470. 2. L. Copeland, A Practitioner’s Guide to Software Test Design. Boston, MA: Artech House, 2004. 3. M. Roper, Software Testing, 3 ed. vol. 15. California, USA: Encyclopedia of Physical Science and Technology, Academic Press, 2002. 4. D. M. Cohen, S. R. Dalal, M. L. Fredman, G. C. Patton, The AETG System: An Approach to Testing Based on Combinatorial Design, IEEE Transactions on Software Engineering, vol. 23, pp. 437-444, July 1997. 5. K. Z. Zamli, M. D. A. Hassan, N. A. M. Isa, M. F. J. Klaib, Developing Automated Testing Tool - SFIT: Experiences and Lessons Learned, Academy of Science Malaysia (ASM) Journal, December 2007. 6. D. M. Cohen, S. R. Dalal, A. Kajla, G. C. Patton, The Automatic Efficient Test Generator (AETG) System, in: Proc. of the 5th International Symposium on Software Reliability Engineering, Monterey, CA, USA, 1994, pp. 303-309. 7. Y. Lei, K. C. Tai, In-Parameter-Order: A Test Generation Strategy for Pairwise Testing, in: Proc. of the 3rd IEEE Intl. High-Assurance Systems Engineering Symp, Washington, DC, USA, 1998, pp. 254-261. 8. T. Shiba, T. Tsuchiya, T. Kikuno, Using Artificial Life Techniques to Generate Test Cases for Combinatorial Testing, in: Proc. of the 28th Annual Intl. Computer Software and Applications Conf. (COMPSAC’04), Hong Kong, 2004, pp. 72-77. 9. K. C. Tai, Y. Lei, A Test Generation Strategy for Pairwise Testing, IEEE Transactions on Software Engineering, vol. 28, pp. 109-111, 2002. 10. M. Grindal, J. Offutt, S. F. Andler, Combination Testing Strategies: a Survey, Journal of Software Testing, Verification and Reliability, vol. 15, pp. 167-200, 2005. 11. Mats Grindal: Handling Combinatorial Explosion in Software Testing. Linkoping Studies in Science and Technology, Dissertation No. 1073, Sweden, 2007. 12. K. Z. Zamli, M. F. J. Klaib, N. A. M. Isa, Combinatorial Explosion Problem in Software Testing: Issues and Practical Remedies. in: Proc. of the 3rd Malaysian Software Engineering Conference (mysec07), pp. 24--28, Selangor, Malaysia, 2007. 13. K. Z. Zamli, N. A. M. Isa, M. F. J. Klaib, Z. H. C. Soh, C. Z. Zulkifli, On Combinatorial Explosion Problem for Software Configuration Testing. in: Proc.of the International Robotics, Vision, Information and Signal Processing Conference (ROVISP2007), Penang, Malaysia, 2007. 14. Bach, J.: Allpairs Test Case Generation Tool, http://tejasconsulting.com/opentestware/feature/allpairs.html 15. TConfig, http://www.site.uottawa.ca/~awilliam/ 16. Jenny, http://www.burtleburtle.net/bob/math/ 17. TVG, http://sourceforge.net/projects/tvg

383

384

Mohammad F.J. Klaib / Procedia Computer Science 65 (2015) 377 – 384

18. Kamal Z. Zamli, Mohammad F.J. Klaib, Mohammed I. Younis, Nor Ashidi Mat Isa, and Rusli Abdullah," Design and implementation of a tway test data generation strategy with automated execution tool support." Information Sciences Volume 181, Issue 9, Pages 1741-1758, 2011 19. Luciano Baresi, Mauro Pezz`, An Introduction to Software Testing, Journal of Parallel and distributed computing, Elsevier, Electronic Notes in Theoretical Computer Science 148 pp. 89–111, 2006. 20. Klaib, M. F. J., S. Muthuraman, N. Ahmad, and R. Sidek, Tree Based Test Case Generation and Cost Calculation Strategy for Uniform Parametric Pairwise Testing, Journal of Computer Science, Journal of Computer Science, 6 (4): 425-430, 2010. 21. Klaib, M. F. J., S. Muthuraman, N. Ahmad, and R. Sidek, "A Tree Based Strategy for Test Data Generation and Cost Calculation for Uniform and Non-Uniform Parametric Values," in International Symposium on Frontier of Computer Science, Engineering and Applications (CSEA2010), Bradford, UK, 2010. 22. Klaib, M. F. J., S. Muthuraman, N. Ahmad, and R. Sidek, "A Parallel Tree Based Strategy for Test Data Generation and Cost Calculation for Pairwise Combinatorial Interaction Testing," in The Second International Conference on Networked Digital Technologies (NDT2010) Charles University in Prague, Czech Republic: Springer, 2010. 23. IC7489 Datasheet, http://www.datasheetcatalog.org/datasheets/134/499326_DS.pdf 24. R. C. Bryce, S. Sampath, J. B. Pedersen, and S. Manchester. "Test suite prioritization by cost-based combinatorial interaction coverage". International Journal of Systems Assurance Engineering and Management, vol 2(2), pp.126–134, 2011. 25. S. Varshney, M. Mehrotra. Search based software test data generation for structural testing: a perspective, in ACM SIGSOFT Software Engineering Notes archive, vol 38 (4),pp. 1-6, NY, USA , 2013. 26. M. B. Cohen, "Designing Test Suites for Software Interaction Testing," Computer Science, Ph.D. New Zealand: University of Auckland, 2004. 27. S. Khatun, K. F. Rabbi, C. Y. Yaakub and M. F. J Klaib, "A Random Search Based Effective Algorithm for Pairwise Test Data Generation" in proc. of IEEE International Conference on Electrical Control and Computer Engineering 2011, Kuantan, Malaysia, 2011. 28. M. Grindal, B. Lindstrom, J. Offutty, S. F. Andler, “An Evaluation of Combination Strategies for Test Case Selection”, Technical Report HSIDA-TR-03-001, Department of Computer Science, University of Skövde, 2003. 29. D.R. Kuhn, R.N. Kacker and Y. Lei, Measuring and Specifying Combinatorial Coverage of Test Input Configurations, submitted toInnovations in Systems and Software Engineering: a NASA journal, 2014. 30. L. S. Ghandehari, J. Czerwonka, Y. Lei, S. Shafiee, R. Kacker and R. Kuhn, An Empirical Comparison of Combinatorial and Random Testing, Third International Workshop on Combinatorial Testing (IWCT 2014), in Proceedings of the Seventh IEEE International Conference on Software, Testing, Verification and Validation (ICST 2014), Cleveland, Ohio, March 31-April 4, 2014, pp. 68-77. 31. J. Hagar, R. Kuhn, R. Kacker, and T. Wissink, Introducing Combinatorial Testing in a Large Organization: Pilot Project Experience Report, Third International Workshop on Combinatorial Testing (IWCT 2014), in Proceedings of the Seventh IEEE International Conference on Software, Testing, Verification and Validation (ICST 2014), Cleveland, Ohio, March 31-April 4, 2014, p. 153. 32. R. Kuhn, R Kacker and Y. Lei, Estimating Fault Detection Effectiveness [poster], Third International Workshop on Combinatorial Testing (IWCT 2014), in Proceedings of the Seventh IEEE International Conference on Software, Testing, Verification and Validation (ICST 2014), Cleveland, Ohio, March 31-April 4, 2014, p. 154. 33. J. Torres-Jimenez, I. Izquierdo-Marquez, Survey of Covering Arrays, 15th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC 2013), Timisoara, Romania, September 23-26, 2013, pp. 20-27. 34. R.N. Kacker, D.R. Kuhn, Y. Lei, and J.F. Lawrence, Combinatorial Testing for Software: an Adaptation of Design of Experiments,Measurement, vol. 46, no. 9, November 2013, pp. 3745-3752. 35. X. Niu, C. Nie, Y. Lei, A.T.S. Chan, Identifying Failure-Inducing Combinations Using Tuple Relationships, 2nd International Workshop on Combinatorial Testing (IWCT 2013), in Proceedings of the Sixth IEEE International Conference on Software, Testing, Verification and Validation (ICST 2013), Luxembourg, March 18-22, 2013, pp. 271-280. 36. D.R. Kuhn, I. Dominguez Mendoza, R.N. Kacker and Y. Lei. Combinatorial Coverage Measurement Concepts and Applications, 2nd International Workshop on Combinatorial Testing (IWCT 2013), in Proceedings of the Sixth IEEE International Conference on Software, Testing, Verification and Validation (ICST 2013), Luxembourg, March 18-22, 2013, pp. 352-361. 37. L.S.G. Ghandehari, M.N. Borazjany, Y. Lei, R.N. Kacker and D.R. Kuhn,Applying Combinatorial Testing to the Siemens Suite, 2nd International Workshop on Combinatorial Testing (IWCT 2013), in Proceedings of the Sixth IEEE International Conference on Software, Testing, Verification and Validation (ICST 2013), Luxembourg, March 18-22, 2013, pp. 362-371. 38. M.N. Borazjany, L.S.G. Ghandehari, Y. Lei, R.N. Kacker and D.R. Kuhn,An Input Space Modeling Methodology for Combinatorial Testing, 2nd International Workshop on Combinatorial Testing (IWCT 2013), in Proceedings of the Sixth IEEE International Conference on Software, Testing, Verification and Validation (ICST 2013), Luxembourg, March 18-22, 2013, pp. 372-381. 39. D.R. Kuhn and R.N. Kacker, Measuring Combinatorial Coverage of System State-space for IV&V [extended abstract], NASA IV&V Workshop, Morgantown, West Virginia, September 11-13, 2012. 40. S. Manchester, R. Bryce, S. Sampath, N. Samant, D. R. Kuhn and R. Kacker, Applying Higher Strength Combinatorial Criteria to Test Prioritization: a Case Study, to appear in: Journal of Combinatorial Mathematics and Combinatorial Computing. 41. D.R. Kuhn, R.N. Kacker and Y. Lei. Combinatorial Testing, Encyclopedia of Software Engineering. Taylor & Francis, 2012. 42. D.R. Kuhn, R.N. Kacker and Y. Lei, Combinatorial Coverage as an Aspect of Test Quality, Journal of Defense Software Engineering,2014. 43. Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, and J. Lawrence,“IPOG/IPOD: Efficient Test Generation for Multi-Way Software Testing,” Journal of Software Testing, Verification,and Reliability, vol. 18, pp. 125-148, 2009. 44. M. I. Younis, K. Z. Zamli, MC-MIPOG: A Parallel t-Way Test Generation Strategy for Multicore Systems, ETRI Journal,vol. 32, no. 1, 2010. 45. D. A. Bader, W. E. Hart, C. A. Phillips, Parallel Algorithm Design for branch and bound, Tutorials on Emerging Methodologies and Applications in Operations Research. Mathematics and Statistics, vol. 76. Springer, New York , 2006. 46. R. Setia, A. Nedunchezhian, S. Balachandran, A New Parallel Algorithm for Minimum Spanning Tree Problem. in: Proc. of the 16th Annual IEEE International Conference on High Performance Computing. Cochin, India, 2009.