TSINGHUA SCIENCE AND TECHNOLOGY ISSN 1007-0214 09/49 pp51-56 Volume 12, Number S1, July 2007
Novel Software Automated Testing System Based on J2EE* PEI Songwen (ଂഡำ)**, WU Baifeng (๑̈́)ף, ZHU Kun (ᅋ ន), YU Qiang (ဥ ஜ) Department of Computing and Information Technology, Fudan University, Shanghai 200433, China Abstract: Software automated testing is one of the critical research subjects in the field of computer application. In this paper, a novel design of architecture called automated testing system (ATS) is proposed. Based on techniques relating to J2EE including MVC design pattern, Struts framework, etc, ATS can support any black-box testing business theoretically with relevant APIs programmed using Tcl script language beforehand. Moreover, as the core of ATS is built in Java, it can work in different environments without being recomplied. The efficiency of the new system is validated by plenty of applications in communication industry and the results also show the effectiveness and flexibility of the approach. Key words: automated testing system (ATS); J2EE; Tcl
Introduction Software testing[1,2] is a critical phase within the software lifecycle, currently more and more companies are paying attention to it. Especially, the software automated testing is becoming the most disputed subject in the software industry. Therefore, it is now a popular research problem in the computer application research area. The increased complexity of systems as well as short product release schedules makes the task of testing challenging. Therefore, how to improve the software qualities and how to advance the testing efficiency are difficult. As a result, software testing is the biggest challenging work during the software lifecycle, especially the software automated testing. Subsequently, test-driven development is advocated by engineers. At the same time, software capability maturity model[3] is put forward for controlling the process of software development and for guarantying the software quality, but it Received: 2007-02-01 γ Supported by the AMD’s University Cooperation Project γγTo whom correspondence should be addressed. E-mail:
[email protected]; Tel: 86-21-61445936
imposes limited effects on software automated testing. To our knowledge, most of the present automated testing tools or products concern how to partition cases or how to implement a single target of testing without uniform testing system, not to mention the distributed and cross-platform testing functions. However, ATS is a black-box testing system based on J2EE[4] theories such as MVC design pattern, Struts framework, Hibernate persistence layer, etc, which comprises two function modules: one is a front-end module developed in Java script language, the other is a back-end module ü the core of ATS ü developed in pure Java language. Thus, the two function modules abide by browser/server model naturally. Under the strong support of front-end module, testers without any prerequisite knowledge can master it in a short training time. Besides, the core of ATS can support crossplatform execution due to pure Java code, and it can support distributed execution due to the separation of job controller and execution server. Every testing business is supported theoretically by ATS due to the excellent inherent capability – self-defined APIs which can be developed by advanced testing developers quickly with Tcl[5] script language aiming at different special targets, and the APIs is transparent to common testers
Tsinghua Science and Technology, July 2007, 12(S1): 51-56
52
who work on the front-end system, what they need to do is just filling suitable data into blanks of web interface according to specific cases.
1
Related Work
The most popular methods about automation test include record-playback, functional decomposition, datadriven[6], and keywords-driven[7]. Record-playback is applied in GUI test products usually, such as Winrunner1, etc. Functional decomposition method is an infant version of data-driven or keywords-driven method, which separates data from functions and uses data-files to provide both the input and the expected-results verification. However, tester using this method must not only maintain the detail test plan with specific data, but also re-enter this data in the various required data-files. Data-driven method records data by a spreadsheet file such as Excel and the executions of test scripts depending on various APIs and key data. Keywords-driven is an advanced data-driven method, which uses the actual test case document developed by the tester using a spreadsheet containing special keywords. In this method, the entire process is data-driven as well and the keywords control the processing. However, ATS has strong programming function supported by various fundamental and business’s APIs and flexible management and control function due to combining datadriven and keywords-driven methods. In recent years, there are many products and architectures surged out. Blackburn, for instance, proposed a model-based test automation method called test automation framework (TAF)[8] which uses an interface-driven approach that combines requirements modeling to support automated test-case and testdriver generation. The model focuses on how test engineers can develop more reusable models by clarifying textual requirements as models in terms of component or system interfaces. However, due to uncanonical requirement documents and un-strict business control, it is very difficult to extend it to many big software companies, not to mention those middle-small companies. Meanwhile, the methods based on component test are proposed, such as CBSFG[9], etc. Unfortunately, they care little about the architecture of automated testing framework but pay attention to how to generate test cases. iSoftTech2 test automation system and AXE3 automated testing framework are excellent
automation systems, but without distributed and crossplatform functions. The dominating commercial products such as Winrunner and Robot4 are designed without much consideration about compatibility and portability. These traditional testing products care little about the operations on cross-platform, distributed control, and distributed execution capabilities.
2
ATS Architecture
ATS is developed according to Struts and MVC design pattern based on J2EE. Server end is built on the tomcat server while client end is presented on Web pages programmed in JSP, and back-end business logic module is implemented by Java beans or even enterprise Java beans. The general ATS architecture is shown in Fig. 1[10-13].
Fig. 1 Architecture of ATS
The ATS is made up of user interface, request handler manager, job controller, normal request handlers, execution server, and suite execute layer. The user interface module provides front-end user with convenient operational interfaces to send the user’s operation requests to the request handler manager module. However, the request handler manager module will not deal with these requests by itself, but just forward them to some specific handler or action threads. In fact, there are two kinds of user request. One is related to control logic belong to test suite or test job, the other is miscellaneous request except from the first class. All the requests related to control logic are dealt with by job controller. To support distributed execution, job controller and execution server can run on different servers independently. A job controller can control sev-
PEI Songwen (ଂഡำ) et alġNovel software Automated Testing System Based on J2EE
eral execution servers synchronously. And execution server is designed by a finite states machine, which can run test suites queue in a test job. When all these test suites in a job queue is finished, the execution server will notify job controller to change the states of job. Of course, the suite execute layer is responsible for compiling and executing suites or cases to implement a specific test task, which is an execution core of ATS. Meanwhile, it provides a unified testing framework with various APIs for programming test scripts at ease. While LRMS is a library resource management system interfaced with ATS for promoting the management of test tasks and resources allocation. TMS is a testing management system interconnected with ATS for advancing the management of test results and testing version control. They are important auxiliary systems supporting ATS. 2.1
Request handler manager
The front-end interfaces of ATS are Web pages. Tester can program test scripts or drag methods from the basic library programmed in Tcl scripts to form a complete program segment. All these requests will be dispatched to server. Generally, every request from frontend interface has its own handler function in the execution server. If Web browser sends every request to the corresponding handler immediately, the Web browser must store all these relationships between request and handler function that will decrease the flexibility and extensibility of the whole system. To solve this problem, ATS creates a request handler manager module to store these relationships between request and handler function. Figure 2 shows the task and the role of request handler manager in the ATS.
Fig. 2 Request handler manager in ATS
Thus, Web browser gets rid of this perplexity. It just needs to hand on requests to the request handler
53
manager which is responsible to dispatch them to corresponding handler threads. Then, handler processes these quests through hibernate between handler and database, which returns results to Web browser finally. The request event sequence is shown in Fig. 3.
Fig. 3 Request event sequence diagram
When the request arrives, the request handler manager will retrieve information from the requests pool and construct an object of ActionForm bean with the information. Then, it will search an action from action mapping file for the request. If the action is found, the manager will forward it and get the corresponding database files. If not, it will dispatch the request to handler which is responsible for getting or updating corresponding database files. The basic function of ATS is to build Web system based on struts framework and operating database through hibernate, which is our infant version of ATS called ATF. However, the ATF lacks modules to deal with the logic relationship between requests and handler events. We create job controller run as a finite states machine to solve this problem. 2.2 Job controller In ATS, there are three kinds of test events—job, suite, and case. Job is an integrator of suite queues, which can be scheduled and allocated to an appointed execution server. Suite is an implementing entity, which is assigned to suite execute layer by execution server. Case is an executable scripts designed in accordance with a certain business in Tcl script language. Firstly, we overview job event in Fig. 4, then describe the finite states machine working in the job controller which is responsible for job control, such as job create, job copy, job schedule, job kill, and job query. The suite execution server is explained in the next segment.
Tsinghua Science and Technology, July 2007, 12(S1): 51-56
54
Fig. 4
Job event flow in ATS
Figure 4 shows a job controller maps n job executers, and a job executer maps n suite executer. Job controller, job executer, and suite executer can update or query their files storied in database, respectively, through different interfaces. Obviously, job controller is responsible for controlling the states of jobs and assigning jobs to different job executers for distributed execution. Soˈthe job controller plays a key role in the whole working, which guarantees the entire system to run smoothly. Generally, different phases have different states. To control all these states well, we design a finite states machine. Figure 5 shows the states transition of job controller’s finite states machine.
from one to another. In order to bind jobs’ states with corresponding operations and to discriminate from the same or similar operations occurring in other states, the state pattern[14] is adopted to deal with these problems according to the theories of design pattern. We design a JobStateMachine class to drive the transition of job’s states. The class holds its instance all the time, which is designed in accordance with skeleton pattern. JobStateMachine stores all states and other information of jobs into database. Besides, we design a basic class JobState, several subclasses derived from it to handle various jobs’ operations such as kill job, suspend job, etc. Another duty of job controller is to retrieve a corresponding execution server according to the respective configure file information of jobs. Specially, job controller and execution server can run on different servers simultaneously through RMI communication bridge, as a result, ATS can work in distributed mode naturally. 2.3 Execution server
Fig. 5
Job state transition
As shown in Fig. 5, there are new state, suspended state, wait_start state, running state, closed state, and done state here. The contents contained in rectangle linked by broken line telling why job’s state transmits
As shown in Fig. 1, the execution server is an intermediary of job controller and suite execute layer. Their relationships are shown in Fig. 6 which exposes the processes of execution server totally. Job controller launches a job to execute according to its state. Execution server receives execution job command and builds a suites queue for it, which will call suite execute layer to execute these suites in turn. When execution server receives the results returned
PEI Songwen (ଂഡำ) et alġNovel software Automated Testing System Based on J2EE
Fig. 6 Relationship among job controller, execution server, and suite execute layer
from suite execute layer, it will check whether the suite queue is null. If not, it will notice the next suite to execute in the suite execute layer. Otherwise, the execution server will inform job controller that all suites belong to the job have been done. Obviously, the suite execute layer is constituted by some executable scripts or commands to implement the task of a case. These scripts can be designed in Tcl language in a certain templates which can speed up the rate of testing development and guarantee the qualities of testing programs. The other function of the suite execute layer is that it provides APIs and external interfaces such as GUI API, Socket API to support a third tool -- Winrunner or QTP. Therefore, ATS has powerful flexibility and expansibility.
3
Experiments
As a matter of fact, we consider that all testing tasks can be classified into two classes: one is a testing about back-end program or procedure, the other is Web test or GUI test. For simplicity, the experiment about GUI test will be omitted here. So, we will only introduce an experiment about back-end program here. Our experiment environment is listed as follows: the database is Oracle 9i.2.0, testing script is programmed in Tcl and executed on ActiveState ActiveTcl 8.5.0.0, Web server is tomcat 5.0. All of these are installed and running on Redhat operation system separately. The tested object is ShpptCDR which transforms a pure text file to another format file. Our testing task is to check whether a certain tag in the transformed file cohering with what we expect according to transformation rules. The critical testing script about the transformation of 281th tag of source file is shown and explained below.
55
TestCase 1105432426903405902 0237243 {} { } { #Parameter section, this section is used by declaring and setting global or local #parameters. }{ # Setup section, commands in this section are to #prepare for test, such as reserved the environment #variables. For example, S "" remoteoper GetRemoteFile "$CfgPath" "$Config File"; # The command is getting a remote configure file to local host for the purpose of protecting configure #file from changed by testing. ... }{ #Step sections, commands in this section are main executable scripts. For example, M macro0; # Calls macro macro0 to pave the way for testing S "" remoteoper RunRemoteCmd "$Command&" "$Prompt" ""; # Runs ShpttCDR on the remote host, not local. ... S "" remoteoper RunRemoteCmd "$CtrlCCmd" "$Prompt" ""; # Executes Ctrl+C command to end the ShpttCDR program. ... S "" toolkit CheckListValue "$ExpectTag281" "$Tag281" "1" "EQUAL"; #checks the tag281 whether equal to what we expect. }{ #Cleanup section, commands in this section is to recover test environment. For example, M macro1; #Call macro macro1 to recover test environment to the situation that testing program have not been run. }
The script is not a pure Tcl script language, the instructions like “S” and “M” will be transformed into pure Tcl scripts according to an instruction configure file. The result of this case running on single execution server is shown in Fig. 7.
Fig. 7 Testing result of ShpptCDR
The result shows that there is a case in testing the function suite – patch for centrex of ShpttCDR. The result log shows that this test is “PASS”, which demonstrates that the function bit tested tag281 is verified
Tsinghua Science and Technology, July 2007, 12(S1): 51-56
56
correct. And, the detailed log can be obtained from the step log label by clicking the left mouse key. Meanwhile, we design another supplementary experiment that three jobs run on different execution server from execution servers 1 to 3, respectively, to test the function of ShpttCDR. The testing statistic results are shown in Fig. 8.
the scripts of regression test can execute on ATS without revising any command line but just revising global input variables. In a sense, the ATS is a general automated testing framework. However, the function of ATS should be improved in tracking of run-time instructions. And, the function of importing and exporting scripts between Web editor and database files should be created to gear to many testers who are not good at coding. References [1]
Shan Jinhui, Jiang Ying, Sun Ping. Research progress in software testing. Acta Scientiarum Naturalium Universitatis Pekinensis, 2005, 41(1): 134-145. (in Chinese)
[2] Fig. 8
4
An empirical study of regression test selection techniques.
Experiment results statistics graph
As shown in the graph, there are 30 different cases corresponding to 30 different testing tags scattered in the transformed file. The 30 cases are arranged into a job with one suite running on three different servers by a distributed way. The statistical results show that, atexecution servers 1 and 2, the same results indicate that the suite run on servers 1 and 2 in a distributed way successfully. While the situation at the execution server 3, a case killed and another right case edited to error additionally on purpose, has been indicated in the graph correctly compared with the situations 1 and 2. In addition, by checking the detail log, all the cases have been executed in the purpose we expected. All these results prove that ATS can run in a distributed way, being controlled randomly and work perfectly.
Conclusions
Compared with traditional automated testing tools, ATS is a novel software automated testing system applied to a communication company by plenty of successful test (such as GUI test with the help of Winrunner and back-end program test, etc.). It has been regarded as an effective and novel testing system with distributed function, resource management function, and strong expansibility. Specially, we can design different APIs to implement different testing business and
Graves Todd L, Harrold Mary Jean, Kim Jung-Min, et al. ACM Transactions on Software Engineering and Methodology (TOSEM), 2001, 10(2): 184-208.
[3]
Paulk M, Weber C. The Capability Maturity Model Grideline for Improving the Software Process. MA: AddisonWesley, 1995.
[4]
http://java.sun.com/javaee/, 2006.
[5]
Welch B B. Practical Programming in Tcl and Tk. New York: Prentice Hall PTR, 1997.
[6]
Keith Zambelich. Totally data-driven automated testing, http://www.sqa-test.com/w-paper1.html, 2006.
[7]
Feng Yucai, Tang Yan, Zhou Chun. Theory and implementation of keyword-driven test automation. Computer Applications, 2004, 24(8): 140-142. (in Chinese)
[8]
Blackburn M, Busser R, Nauman A. Interface-driven, model-based test automation. Crosstalk: The Journal of Defense Software Engineering, 2003: 27-30.
[9]
Smai Beydeda, Volker Gruhn. An integrated testing tecnique for component-based software. In: Proceedings of the ACS/IEEE Int. Conf. on Computer Systems and Applications (AICCSA'01). Aiccsa, 2001: 328-334.
[10] http://www.mercury.com. [11] http://www.isofttech.com/. [12] http://www.odin.co.uk/. [13] http://www-306.ibm.com/software/rational/. [14] Erich G, Richard H, Ralph J, John V. Design Patterns Elements of Reusable Object-Oriented Software. MA: Addison-Wesley, 1995.
