EATIS 2007
Web-based tool for automatic acceptance test execution and scripting for programmers and customers Bruno C. Araújo, Anne Caroline Rocha, Arthur Xavier, Ana Isabella Muniz and Francilene P. Garcia Federal University of Campina Grande {brunocoitinho, annecaroline, arthur.barbaro, isabella.muniz}@gmail.com
[email protected] Abstract Software testing is a software engineering practice that improves the quality of developed computer systems. It is a testing discipline strongly related to the agile software development method Extreme Programming (XP) [2], thus this paper examines acceptance tests mostly from a XP viewpoint. Nowadays, there are many online portals in which a large database of software projects can be found. If developers intent to run acceptance tests in those software, they must always have a complete development environment installed. In this paper, we present FLOAppTest, a visual online (remote) tool integrated to a collaborative environment (like Source Forge) that makes Java projects testing easy by offering a fashion to generate executable acceptance scripts with EasyAccept framework support. Keywords— Acceptance tests, Software engineering, Software testing, WEB components, WEB systems.
1. Introduction Using a good software engineering process to develop computer systems is proved to be a necessary choice for a team of programmers to improve the chances of having successful projects. Software tests, or more specifically Acceptance Tests, compound one of the good practices used in most software processes, like Extreme Programming (XP) [4]. The task of idealizing and writing acceptance tests (unlike unit tests) is assigned as a customers’ role [2]. It happens that in most cases customers are not familiar to software engineering or software processes, so probably the executable tests scripting will not be in fact wrote by the customers, but by programmers. Customers may provide a rustic document, in which there is a description of the expected software behavior. An environment where even a customer who doesn’t know the programming language used in the product is able to write and execute acceptance tests, in a fashion as practically as accessing a webpage, is highly desirable. Collaborative environments, such as Source Forge (http://www.sourceforge.net), have a project repository. Usually, this kind of environment provides tools for history information management, versions download and
Control Version Server (CVS). A very desirable tool which could be integrated to collaborative systems, as these tools were, is one that executes acceptance tests in a visual and simple fashion. Any system which implements WebServices defined by FLOAppTest could be used as “collaborative environment systems”. In the next section, some considerations about related works are done. In section 3, both general automated tests and acceptance tests are briefly reviewed. Section 4 presents the basics of EasyAccept framework, used as support by our tool, which is described in details in section 5.
2. Related Work Much effort may be found in literature to propose or analyze languages [6], tools [7] or frameworks [3,8,9] to make acceptance tests easy or comprehensive. However, visual tools for easing their test execution in a customer viewpoint are often focused for WEB or GUI (Graphical User Interface) based applications. There are some webbased frameworks meant to test applications in general, like FitNesse [10] or FIT [11]. They frequently use scripts defined by expected result tables. Longing for a more interactive environment, FitNesse framework uses a Wiki system for writing the scripts. One can use FLOAppTest not only to test WEB or GUI applications, but for all kinds of Java application. Their functionalities may be properly assigned in a Façade class, before or after the test execution itself. Since FLOAppTest is based on EasyAccept, scripts are composed of a set of little test pieces (lines), which represent the full acceptance test. Expected result tables are part of the possible commands used in FLOAppTest, which includes others more. FLOAppTest is an AJAX (Asynchronous JavaScript and XML) web application.
3. Automated software testing Automated software testing is a process used to identify the correctness, completeness, data integrity, security and quality of developed software. This process is intended to reveal quality-related information about the product by means of bug (error) findings.
EATIS 2007
There are many ways of automatically testing software: unit tests, integration tests and acceptance tests, to cite some. As with any other type of testing, acceptance testing can be done manually. But in order to get the maximum benefit from the tests, automate tools must be used for the execution of the tests, carrying and automated testing in a continuous integration cycle, according to the software development process. Our tool executes Acceptance Tests on software. In the following section, this kind of testing will be shortly reviewed.
3.1. Acceptance Tests Acceptance tests are part of the software development method Extreme Programming (XP). They operate on the highest level of a software application: customer interface. If an acceptance test fails, it means the user story implementation made is not acceptable for the customer. So, acceptance tests are not meant to test internals or technical elements of the code, they are meant to test customer requirements which are in form of user stories [4]. As a means of testing, acceptance tests help to identify quality-related information about the product in development by finding software errors. It verifies if the software system behaves as expected in user stories. Techniques used to find those errors are based on several asserts and assigns, which represent a simulation of the real context in which the software will interact when it gets in production. Acceptance tests, which are made based on functional requests indicated by the customer, provide the following advantages: • guarantee customer's project scope is attended fully and correctly; • avoid that programmers unintentionally develop more functionalities than are actually needed; • make developers have a basic idea of how much implementation is still unfinished; • make developers previously think about system design and modules integration in a realistic manner. Acceptance tests reflect product’s functionalities that are still in the customer’s mind. Usually it is the customer who writes, using natural language, the features which the programmers will implement. EasyAccept [3] is a tool that helps customers to create and run acceptance tests in an easy, quick and clean way. In the next section, this tool will be better explained.
4. Create and Run Acceptance Tests using EasyAccept EasyAccept framework [3] is a tool that eases the creation and execution of acceptance tests for Java programs. For the EasyAccept framework to be able to execute acceptance tests using the Java programming language, it requires a Façade-like class, which will be used as a bridge between EasyAccept and the program’s business logic. The Façade provides the methods which names will define commands for the test scripts. EasyAccept has many built-in commands. These commands can be used to handle several kinds of tests, for instance: • expect: expects a string from a determined method. Ex: expect "Orange Street" getAdress name=Joe; • expectError: expects an exception with a certain text message. Ex: expectError "Invalid e-mail" setEmail name=Joe
[email protected]; Other built-in commands: expectTable, equalFiles, stackTrace, quit, etc. Although EasyAccept tests are actually scripts written in a close to natural language, they are still a low-level way of making acceptance tests, as they have to be programmed and executed under an environment properly prepared for that. This environment will include everything related to Java Development Kit, the EasyAccept framework itself and a good Integrated Development Environment (IDE), avoiding the use of some rustic command line editor or prompt. The proposal of this work is to present a test tool capable of create, edit and execute acceptance tests on software hosted in a collaborative environmnet, through WebServices communication.
5. FLOAppTest The FLOAppTest tool was developed using the software development process XP1 [5], which is slightly different from XP, from which it derives. FLOAppTest offers an acceptance tests development and execution environment for Java projects using EasyAccept, a good acceptance tests execution framework. One of the main advantages of FLOAppTest is that it is a web application, so there’s no need to install any additional platform, like JVM (Java Virtual Machine), in the user’s machine. FLOAppTest can retrieve Java projects from an external project database. This makes unnecessary for the user to supply directly to the system any Java application. Yet, this is useful because the tool can execute tests from
EATIS 2007
different repositories indistinctly, through WebServices, since they give direct access to jar files or resourceful compacted files (.zip and .tar.gz), following a given specification. The architecture of FLOAppTest is shown in Figure 1. It uses Web Services in order to retrieve needed projects information, as well as saving the scripts themselves.
Figure 1. FLOAppTest architecture.
The tool provides a scripts handler visual interface for the user to create new acceptance tests or to edit old ones. This web interface was built using Ajax technology, what makes it much more interactive than standard web-based applications, as an Ajax application is close to a desktop one. In particular, we use GWT (Google Web Toolkit) for implementing Ajax technology. For FLOAppTest to be able to execute acceptance tests, user must inform both the Façade class’s name and which acceptance tests scripts he wants to run. The following steps describe how the Façade class, the bridge between acceptance tests and the program world, may be selected: 1. The user must choose which Java project he wants to test from a list. This will bring to the user a list of releases; 2. So he chooses the release / version he wants to test from the release list; 3. Then, a tree list is made based on the project’s Java Classes. The Façade must be selected (an example of this tree is seen in Figure 2); 4. After selecting the class, the methods of the Façade chosen will be loaded in the main test editing screen, as seen in Figure 3.
Figure 2. Choosing a Façade class and scripts.
The main test editing screen has two interchangeable views: the visual and the code view. Even before selecting the class on which the tests will be based, the user can use the code view to write new acceptance tests for the chosen project release or edit old ones. At the top of the screen, one can upload a script file. Below, in the “Create/edit script” area, some fields represent the creation of new tests in a non-programming way. The New Script form is composed from information found in the Façade class, as methods or parameters. The combo box Select a Test in Figure 3 contains the list of methods of the Façade class. In the example, the parameters shown Name, Address and Salary are for the method changeEmployerData. This customizable behavior is possible thanks to the Java Reflection API.
Figure 3. Scripts creation screen.
For expert users, there is an alternative for creating or editing scripts, by changing to the Script Code Mode, as shown in Figure 4. In this case, one can insert new commands and comments using EasyAccept script language directly.
Figure 4. Scripts creation screen.
EATIS 2007
After pressing the test execution button, the tool displays a report with information about the results of the acceptance tests execution, such as number of successful and unsuccessful tests for each selected test script, as we can see in Figure 5, in the Results area.
for the project programmers. Since they are already in a environment capable of execute the acceptance tests through batch scripts, delegate the test execution task to web-based systems may slow the process, making it less practical.
7. References
Figure 5. Tests execution results.
When any block (blocks are defined as a set of “test lines”) inside an acceptance test succeeds 100%, the number of succeeded test lines and an OK message is shown in the report. When a test (or test block) does not succeed, the report presents the number of unsuccessful tests (or test blocks), together with the obtained results in contrast to the expected ones, and shows further information about the failure (exception caught etc.).
6. Conclusion In this paper, we have presented a tool that makes and executes acceptance tests in an automated and easy fashion, so that the customer does not need to concern about the script language itself or install any additional software to be able to execute any of his projects, which are deposited, together with those from other customers, in a database. A FLOAppTest contribution is that it allows nonprogrammers users of the collaborative environment to participate directly in the writing of executable acceptance tests, instead of writing indirectly executable documentation or even non-executable documentation, what would make him less involved with the real behavior of the system. Furthermore, FLOAppTest is important not only from the script editing perspective, but also in the accessibility level, since acceptance tests can be archived all together with each Java project and executed by authorized persons anywhere, from any computer with internet access. Nevertheless, the on-line testing approach used for FLOAppTest has advantages for project clients because it simplify the accenptance test execution and scripting. On the other hand, our approach may not be as much useful
[1] K. Beck, Test Driven Development: By Example, Addison Wesley, 2003. [2] R. Jeffries, A. Anderson, C. Hendrickson, C. Extreme Programming Installed. Addison Wesley, 2001. [3] O. L. Abath., J. P. Sauvè, W. Cirne, “EasyAccept: A Tool to Easily Create, Run and Drive Development with Automated Acceptance Tests”, Workshop on Automation of Software Test, Shanghai. Proceedings of the Workshop on Automation of Software Test. , 2006. v.1. p.1 – 10. [4] K. Beck and C. Andres, Extreme Programming Explained: Embrace Change (2nd Edition). Addison-Wesley Professional, 2004. [5] J. P. Sauvé, A. F. da Silva, F. R. A. Neto and F. W. L. Cabral. “XP1: Um Processo de Desenvolvimento”. Avaliable from http://www.dsc.ufcg.edu.br/~jacques/projetos/common/ xp1/xp1.html. Accessed in December 2006. [6] E. Aranha, P. Borba. "Uma linguagem para testes de aceitação de sistemas web". In VI Brazilian Symposium on Programming Languages, p. 128-141, Rio de Janeiro, Brazil, 2002. [7] C. Lowell, J. Stell-Smith. Successful Automation of GUI Driven Acceptance Testing. Extreme Programming and Agile Processes in Software Engineering. p. 331-333, 2003. [8] R. Miller, C. Collins. “Acceptance Testing”. Proc. XPUniverse 2001, July, 2001. [9] R. Mugridge, B. MacDonald, P. Roop. “A Customer Test Generator for Web-Based Systems”. Proc. XP2003, LNCS, Vol.2675, Springer Verlag: 189-197, 2003.
[10] FitNesse Acceptance Testing Framework, http://fitnesse.org
[11] FIT: Framework for Integrated Test, http://fit.c2.com/ [12] AutAT at BEKK Open Source Software, http://boss.bekk.no/
