Using the Computational Grid to Speed up Software Testing

Using the Computational Grid to Speed up Software Testing Alexandre Nóbrega Duarte, Walfredo Cirne, Francisco Brasileiro, Patricia Duarte de Lima Machado Departamento de Sistemas e Computação Universidade Federal de Campina Grande Avenida Aprígio Veloso, s/n. Bodocongó, CEP 58.109-970 – Campina Grande – PB – Brazil {alex, walfredo, fubica, patricia}@dsc.ufcg.edu.br

Abstract. Software testing practices can improve software reliability. This is a fact that the long history of disasters caused by poorly tested software should not allow us to forget. However, as software grows, its test suite becomes larger and its execution time may become a problem to software developers. This is especially stronger for agile methodologies, which preach a short develop/test cycle. Moreover, due to the increasing complexity of systems, it is not enough to test software only in the development environment. We need to test the software in a variety of environments. In this paper we investigate how the Grid can help with both problems and present a framework to use the intrinsic characteristics of the Grid to speed up the software testing process.

1. Introduction

Software testing practices can improve software reliability. For instance, Ben-Ari discusses how a poorly tested software could brought down a rocket, costing hundreds of millions of dollars and delaying the European Space Agency space program by a year (Ben-Ari, 1999). The cause of the crash was an attempt to convert a 64-bit floating-point number greater then 32,768 to a 16-bit signed integer. Other examples of disasters caused by untested software are also showed in (Mellor, 1994). A test suite that catches the expected behavior of the software can be used to check if its implementation works as it is supposed to. Furthermore, when a software defect is fixed, the test suite can be used to check whether the modification has collateral effects in other parts of the software. Large software projects usually involve several development teams. In these situations, a test suite can be used by a development team to check if a software module provided by another team has an intended behavior, hiding the implementation details (Jeffries, Anderson and Hendrickson, 2000). This way, large software projects usually have large test suites. There are industry reports showing that a complete regression test session of a 20.000 lines software could take as much as seven weeks of continuous execution (Rothermel, Untch and Chu, 2001). In order to be effective, the software testing process should be carried out in an automatic and fast way. It should be automatic because the same test suite must be executed many times along the software life cycle, including every time a new functionality is added and every time a software defect is fixed. Additionally, a test suite that executes fast can be executed more frequently during the development process, catching problems sooner.

Another issue that must be dealt with when testing a software is environment heterogeneity. Although software components may be exhaustively tested in the development environment before going into production, we believe that the ability to run tests in a variety of environments can be very useful. This is because the environment and its configuration change from development to production environments. In fact, due to the complexity of systems, nowadays, each environment is unique. Therefore, running the tests in a heterogeneous production-like environment allows for finding configuration errors as well as software defects that were not detected while the software was tested within the homogeneous development environment. There are well established solutions (Gamma and Back, 1999) (The Open Group, 2003) to automate the software testing process. There are also a few efforts (Kapfhammer, 2001) (Hughes, Greenwood and Coulson, 2004) (SpiritSoft, 2005) to speed up the software testing process by distributing the execution over a set of processors. But, to the best of our knowledge, nobody has explored the unique characteristics of the Computational Grids (Foster and Kesselman, 2004) (Berman, Fox and Hey, 2003) yet, such as its massive parallelism and huge heterogeneity, to improve software testability. In this paper we investigate the use of the Grid as a testing environment and present an open source solution to execute automated software tests in the Grid. Our solution is an extension of the widely adopted JUnit testing framework (Gamma and Back, 1999). We start by presenting, in Section 2, the Grid as a test environment. Then, in Section 3, we present the GridUnit1, a java-based framework than can be used to run JUnit tests in the Grid and also some preliminary performance evaluation of its utilization. Finally, in Section 4, we conclude the paper with our final remarks and proposals for future work.

2. The Grid as a Test Environment

The Grid is the computing and data management infrastructure that will provide the electronic underpinning for a global society in business, government, research, science and entertainment (Berman, Fox and Hey, 2003). Grids have the potential to revolutionize computing by providing ubiquitous, on demand access to computational services and resources, provided by multiple independent sources. Grids can also provide unprecedented levels of parallelism for high-performance applications. These possibilities create fertile ground for entirely new applications, much more ubiquitous and adaptive, maybe with huge computational and storage requirements. Grid software is highly complex distributed software however, we reach a conclusion that seems to be, at first, contradictory: The same caracteristics that make Grid software a very complex piece of code make the Grid an excelent test environment. The very high levels of paralelism provided by a Grid can speed up the test execution, increasing productivity and making the testing process of large software a less expensive task. The Grid can also lower the costs of acquisition and maintenance of the test environment. For instance, a large company, with several software factories distributed over the world, working on different time zones, can use the Grid to share its idle resources among its factories. By creating an enterprise Grid, a company does not 1

The framework can be freely downloaded from http://www.lsd.ufcg.edu.br/~alex/gridunit.

need to buy additional machines to improve test speed and, therefore, does not need to spend more money with new hardware and new software maintenance. Additionaly, as the. Grid is a highly heterogeneous environment, the software can be tested using varied software and hardware configurations, bringing the software test process closer to the software production environment. This can minimize test result contamination by specific development configurations. Moreover, the Grid can provide resource virtualizations for its nodes, creating empty and well defined environments for the execution of the user tasks. This bounded environment impedes that the execution of test ti alters the normal outcome of test tj.

3. The GridUnit Framework

The GridUnit framework is an extension of the widely adopted JUnit testing framework (Gamma and Beck, 1999) able to run JUnit test suites in a Grid with minimum user intervention. Its main functionalities were pointed by (Kapfhammer, 2001) as important aspects that a test distribution tool must consider in order to improve the costeffectiveness of the testing process. 1. Transparent and Automatic Distribution: The GridUnit considers each JUnit test as an independent task, scheduling its execution in the Grid without any user intervention; 2. Test Case Contamination Avoidance: Each test is executed using the resource virtualization provided by the Grid, and, as explained in Section 2, this impedes that the execution of test ti alters the normal outcome of a test tj; 3. Test Load Distribution: The GridUnit relies on the Grid broker scheduler to provide load distribution; 4. Test Suite Integrity: The default JUnit test runner runs each unit test as an independent task. For each test, it creates an instance of the test class, calls the setUp() method, calls the testMethod(), calls the tearDown() method and then destroys the instance. The GridUnit reproduces the same behavior with the difference that each test can be executed in a different location in the Grid; 5. Test Execution Control: The GridTestRunner and GridTestListener interfaces provide a mechanism to build a centralized test execution and monitoring point. The GridUnit graphical user interface provides controls to start and stop the execution of the tests of a given test suite. It also monitors the execution of the tests and presents the result of the execution of each test as soon as it is available. Figure 1 shows the high level architecture of the GridUnit framework. The framework is composed by four entities: GridTestSuite, GridTestRunner, GridTestListener and GridTester. The GridTestSuite is an extension of the JUnit TestSuite and provides a way to specify which tests should be executed in the Grid. GridTestRunner is a Java interface that represents an entity able to coordinate the execution of the tests in the Grid. GridTester is a Java class that uses the JUnit framework to execute one TestCase in a Grid node. The GridTestRunner sends each of the TestCases (and all files needed to execute the test) in a GridTestSuite to a Grid node and uses a GridTester to run it and to collect the results of the execution. GridTestListener is a Java interface that should be implemented by each class that wants to monitor the execution of the tests.

Figure 1: The GridUnit high level architecture

The only Grid-dependent part of the GridUnit framework is the GridTestRunner. So it needs to be specialized in order to provide support to specific Grid software and middleware, like Globus (The Globus Alliance, 2005), OurGrid (Cirne et al, 2005) or Condor (Thain, Tannenbaum, and Livny, 2004). In addition to the core framework, we created the graphical user interface (GUI) showed in Figure 2 to provide a centralized user friendly way to run and to monitor the execution of JUnit tests on the Grid. It implements the GridTestListener interface in order to receive information about the status of the execution of the tests on the grid. The input for the GridUnit GUI is a JUnit TestSuite containing all tests that should be executed on the Grid. Like the JUnit TestRunners, the tool must be executed inside the directory where all files used by the tests are located. This is necessary because GridUnit need to rebuild the application directory on the remote grid node prior to the execution of each test.

Figure 2: The GridUnit graphical user interface

Figure 2 shows an example of the execution of a test suite in a Grid. In this example, the GridTestSuite has 250 TestCases. The status bar, in the lower corner of the window, shows that at that moment 6% (or 15) of the 250 TestCases were executed, five of them failed due to unsatisfied assertions and five of them failed due to unanticipated errors. The progress bar is gray because there are tests that failed due to unanticipated

errors, indicating that some tests could not be executed. It would be red if all failed tests failed due to unsatisfied assertions and green if no tests failed at all. The tree at the left corner represents the test hierarchy and shows the status of the execution of the tests using colored icons. It provides an overview of the execution process. For detailed information the user can look at the Failures and the Errors tabs. The Execution Trace tab provides an event driven log of the test execution. The user can, at any time, save the contents of each tab to a file for posterior analysis. 3.2 Preliminary Results In order to validate the GridUnit framework we have instantiated it to run tests in a real Grid environment. We decided to start using the OurGrid (Cirne et al, 2005) toolkit based on our knowledge and past experiences with the solution and in the fact that the OurGrid is a free-to-join Grid, so we could start to use it immediately, without previous access negotiations. To do so, we have implemented the MyGridTestRunner.The MyGridTestRunner is able to convert a JUnit TestSuite into a set of Grid tasks and jobs that can be executed using the OurGrid toolkit. Note that the graphical user interface could be used to run tests over any kind of Grid, given that there is an appropriate GridTestRunner available. We have created also the LocalTestRunner that simulates the behavior of JUnit, running all tests in the local machine. So the GridUnit GUI can also be used as a replacement for the JUnit graphical test runners. We have created a synthetic application, with 250 TestCases, and compared the execution time of the tests using JUnit and using GridUnit with some different Grid configurations. The chart in Figure 3 shows the results of the experiments. GridUnit Speedup 12,00

Speedup

10,00 8,00 6,00 4,00 2,00 0,00 0

10

20

Grid Size

30

40

50

Figure 3: GridUnit performance evaluation

In the chart, the X axis represents the number of machines used to run a test suite during one execution of the experiment while the Y axis shows the speed up obtained by using GridUnit instead of JUnit. As we can see the GridUnit provided a speedup of almost 12x over JUnit when using a grid with 45 machines. Due to some network instability observed during the experiments, there were some unexpected results, like a speedup of only 3x using a grid with 30 machines.

4. Conclusions and Future Work

In this paper we showed that the computational power provided by Computational Grids can be used to parallelize the execution of the tests, speeding up the overall process.

Additionally, we discussed how the Grid can bring the software testing process a step closer to the software production environment. We created an open source Java-based framework, called GridUnit, able to distribute the execution of a JUnit test suite over a Grid, providing a cost-effective improvement to the testing processes. We showed how the framework addresses the five considerations pointed by (Kapfhammer, 2001) as key aspects to build a reliable test distribution mechanism. The framework was validated by using OurGrid (Cirne et al, 2005) as the Grid environment to run the tests of a synthetic application. There is much room for future work. First, we need to use GridUnit to test some real applications in a Grid and characterize how the heterogeneity of the Grid could help find defects in the software. Next, it is important to study the relation between test granularities and file transmission delays measuring the network overhead introduced by the Grid. Finally we want to study how the use of the Grid as a software environment can improve the quality of Grid software itself.

Acknowledges

This work has been partially developed in collaboration with HP Brazil R&D. Authors would also like to thank the financial support from CNPq/Brazil.

References

M. Ben-Ari. The bug that destroyed a rocket. Journal of Computer Science Education, 13(2):15–16, 1999. P. Mellor, CAD: Computer-Aided Disaster, High. Integr. Syst., 1(2):101-156, 1994. F. Berman, G. Fox, A. J. G. Hey, Grid Computing: Making the Global Infrastructure a Reality, (John Wiley & Sons Inc., 2003). W. Cirne, F. Brasileiro, N. Andrade, R. Santos, A. Andrade, R. Novaes and M. Mowbray, Labs of the World, Unite!!!, Submitted for publication in May 2005. http://www.ourgrid.org/twiki-public/bin/view/OurGrid/OurPublications D. Thain, T. Tannenbaum, and M. Livny., Distributed Computing in Practice: The Condor Experience, Concurrency and Computation: Practice and Experience, 17(2):323-356, 2004. E. Gamma, K. Beck. JUnit: A cook's tour. Java Report, 4(5):27-38, May 1999 R. E. Jeffries, A. Anderson and C. Hendrickson, Extreme Programming Installed. (Addison-Wesley, 2000). G. M. Kapfhammer, Automatically and Transparently Distributing the Execution of Regression Test Suites, in: Proceedings of the 18th International Conference on Testing Computer Software, 2001. G. Rothermel, R. H. Untch and C. Chu. Prioritizing test cases for regression testing, IEEE Transactions on Software Engineering, 27(10):929-948, 2001. D. Hughes, P. Greenwood,G. Coulson. A Framework for Testing Distributed Systems. in: Proceedings of the 4th IEEE International Conference on Peer-to-Peer computing (P2P’04), 2004. I. Foster and C. Kesselman. The Grid: Blueprint for a New Computing Infrastructure, 2nd Ed, Morgan Kaufmann, 2004. SpiritSoft, SysUnit, http://sysunit.org, 2005. The Globus Alliance, Globus, http://www.globus.org, 2005. The Open Group, TETware, http://tetworks.opengroup.org, 2005.