Extension of Selenium RC Tool to Perform ... - ACM Digital Library

Extension of Selenium RC Tool to Perform Automated Testing with Databases in Web Applications Andreza M. F. V. de Castro Nokia Institute of Technology Manaus, Amazonas, Brazil [email protected]

Gisele A. Macedo

Eliane F. Collins

Arilo C. Dias-Neto

Nokia Institute of Nokia Institute of Institute of Computing Technology Technology Federal University of Amazonas Manaus, Amazonas, Brazil Manaus, Amazonas, Brazil Manaus, Amazonas, Brazil [email protected] [email protected] [email protected]

Abstract—This paper presents one extension of the Selenium RC tool to perform tests in web applications that require checking data in databases. This verification is performed using new functions implemented into the Selenium framework’s core. These functions allow opening and closing database connection and comparing test data with data stored in the database used by the web application. This solution aims to contribute to the system quality by reducing the effort during the testing process, since the verification of UI and database elements will be performed at the same time during execution of the test scripts. A case study is described to analyze the impact of the proposed tool in terms of effort and rate of automation in the development of a new web application project. The results suggest a significant reduction (92% and 88%) in the effort to execute automated tests in database when compared to, respectively, manual and semiautomated execution. Index Terms—Selenium RC, Software Testing, Functional Testing, Test Automation, Database Testing.

I. INTRODUCTION Nowadays, a high number of software systems have been created as web-based applications (the ones executed in a web browser). As with any software, these applications must have their quality assured before their deployment to final users. In this scenario, testing is an essential activity to guarantee the quality of a system. As part of their attempt to do more with less, organizations want to test software adequately, but as quickly and thoroughly as possible. To accomplish this goal, organizations are turning to automated testing [1]. According to Dustin et al. [2], the definition for Software Test Automation means to automate software testing activities including the development and execution of test scripts, verification of testing requirements, and the use of automated testing tools. The use of this practice may reduce the effort to perform testing in a software project, that is, we will be able to execute a highest number of test cases in less time [3]. As mechanism to support this task in software projects, there are several commercial and opensource tools available to automate tests for web-based applications, such as Watir [4], Canoo WebTest [5], JMeter [6], Selenium [7], amongst others. The Selenium framework is considered the most popular opensource functional test tool for web-based applications [7].

c 2013 IEEE 978-1-4673-6161-3/13/$31.00

Selenium comprises a set of tools to support the development of automated tests scripts for web-based applications. This framework can be executed in several web browsers and operating systems. Furthermore, it can be controlled using several programming languages and other test frameworks. Each of Selenium’s tools (e.g.: Selenium IDE, Selenium RC, Selenium Grid) has a specific role in the general framework, aiming to contribute to automated tests. However, despite all advantages in using Selenium (and its tools), it has some limitations when testing a web-based application. One of the major limitations is the lack of asserts to check data in database’s tables after doing any operation in a web-based application that manipulates this database. Its asserts are restricted to GUI (Graphical User Interface) elements. This limitation makes the automation of test cases harder (and sometimes unfeasible) in database-based information systems, which are most of web-based applications currently developed. Databases are the core of most modern web-based applications. The interaction between these applications and the database imposes some difficulties during the testing activities in a software project [8]. The differences of states/contents in a database among different software versions may help to identify potential failures in regression testing [9]. However, the manual verification of these differences is a hard and timeconsuming activity. Thus, automated tests that check data in databases could perform this task quickly and efficiently. Based on this scenario, this work presents an extension of the Selenium RC tool (one of Selenium framework’s tools), called SeleniumDB, to automatically test web-based applications with asserts that check data in databases. New (assert) functions were implemented in Selenium RC’s core to connect/disconnect to databases and check data in its tables. Thereby, we have a new and efficient mechanism to automatically test web-based applications that use databases to store their information. SeleniumDB was used in an industrial case study and the results indicated a reduction of 92% and 88% in the effort (time) to execute test cases using the proposed tool when compared to, respectively, manual and semi-automated strategies.

125

AST 2013, San Francisco, CA, USA

Besides this introduction, this paper is organized as follows: Section 2 summarizes some related tools to perform database testing. Section 3 describes the extension of the Selenium RC tool developed to support automated database testing. Section 4 reports a case study performed to apply the proposed tool in a real software project and the comparison between two other strategies (manual and semi-automated tests execution). Finally, Section 5 presents the conclusions and future work. II.

RELATED WORKS

In [10], the authors claimed that tools developed for software testing today do not work well for database applications. The interaction between applications and underlying database(s) imposes a number of difficulties. These testing tools only work well if applications are stateless or tests can be designed in such a way that they do not alter the state. Some proposals to support functional testing in databasebased applications are available at the technical literature. In [10], the authors describe the SIKOSA Project, a capture-andreplay tool for functional regression testing in database-based applications. This tool checks the state of databases after test cases execution. As drawback, this tool is restricted to check input-output relationships in these applications. Another proposed solution is the DART tool [9], a tool to support regression testing in database-based applications. This tool works executing the system under test (SUT) twice with the same input data and database’s initial state. The first execution is done using the system’s original version and the second one is done using its changed version. After that, it computes the differences in the database’s state considering the two executions. The found differences would be caused by a valid change or a failure in the regression tests. DBUnit [11] is a JUnit framework extension that can be used to perform unit tests using database operations. It puts the database into a known state between test runs. The framework can also verify if a database data match an expected set of values. This tool can repopulate the database with sample data, perform unit testing of the Java application and it provides a simple XML based mechanism for loading test data. As drawback, this tool does not support functional tests. SWAT [12][13] is a test tool that allows users to automate web application testing in multiple browsers. This tool has a UI recording and SQL tools, integrated with Fitnesse. Another proposed solution is the Apollo tool [14]. It is a supporting technique to adequately test DB applications (in particular, interactions application-DB). It generates tests automatically, runs the tests capturing logical constraints on inputs, and minimizes the conditions on the inputs to failing tests so that the resulting bug reports are small and useful in finding and fixing the underlying faults. In [15], the authors introduce a test adequacy criterion that is based on coverage of the database commands generated by an application and specifically focuses on the applicationdatabase interactions.

III.

EXTENSION OF SELENIUM RC TO ALLOW DATABASE DATA CHECKING

A. Extensible Selenium’s Architecture The creation of new assert functions was feasible because of the infrastructure provided by the Selenium framework, which supports extensions using different programming languages. First of all, the file selenium-java-2.25.0.zip was downloaded (as indicated previously, we decided to use Java in this extension) available at http://seleniumhq.org/download/. It comprises the file selenium-java-2.25.0-srcs.jar, where all .java files within Selenium RC source code can be found. The next step was to extract the files from selenium-java2.25.0-srcs.jar, to create a new project in Eclipse IDE (environment used to implement the new assert functions) and import all extracted files. After creating the project, all .java files was analyzed in the extracted packages in order to identify in which class we could implement the new assert functions proposed in this work. We discovered that we should update only two files in the package com.thoughtworks.selenium: DefaultSelenium.java and Selenium.java. The file Selenium.java implements an interface, which is an abstract type that comprises the signatures of methods to be implemented in child class DefaultSelenium. The new proposed assert functions (and supporting methods) were implemented in the second file and their signatures were inserted in the first one. B. New Selenium’s Asserts to Check Databases’ Data Due to the previous observed limitation in the Selenium framework to support automated testing in web-based applications that access databases, we extended the Selenium RC tool to deal with asserts that query data in these databases. This extension of the Selenium RC tool (we called it SeleniumDB) was developed using Java, so the tests can be executed using the JUnit framework. This technology was selected because it is very popular technologies in the software engineering community and the Selenium RC tool’s source code is available in Java. Regarding the choice of database management system (DBMS), SeleniumDB would be able to connect to any DBMS. In this case, we need to create a specific function to perform the connection to the database. In the first moment we decided to implement the connection function for two most popular DBMS’s: MySQL and PostgreSQL. These functions must receive some parameters (server, username, password, port number, database name) to perform the connection by JDBC (Java API that supports connection to databases). Moreover, we need to create a function to disconnect the database. In this case, only one function is necessary, because it just receives the object connection and finalizes it. The signatures of these three functions are listed in Fig. 1.

126

public Connection connectionMySQLDB(String username,String password,String server,String portnumber,String dbName) throws SQLException;

public void assertKeyDBPresent(Connection con,String table,String column,String data){ Statement stmt = null; try{ stmt = con.createStatement(); String q = ""; q += "Select * from "+table+" WHERE "+column+" = '"+data+"'"; ResultSet rs = stmt.executeQuery(q); if(rs.next()){ rs.getString(column); assertTrue(true); }else { fail("Test Failed!"); } } catch (SQLException e){ e.printStackTrace();} }

public Connection connectionPostgreSQLDB(String username,String password,String server,String portnumber,String dbName) throws SQLException; public void closeConnectionDB(Connection con) throws SQLException;

Fig. 1. Extract of file Selenium.java with the signature of the database connection/disconnection methods.

To support automated testing with databases, six new assert functions were implemented in Selenium RC’s core. These functions compare data in tables’ columns under test with texts submitted by testers in test scripts or texts filled in GUI elements of the web-based application under test. The signatures of these assert functions are listed in Fig. 2. public void assertKeyDBPresent(Connection con,String table,String column,String data);

Fig. 3. Extract of the assertKeyDBPresent function’s source code.

public void assertKeyDBNotPresent(Connection con,String table,String column,String data); public void assertColumnDBPresent(Connection con,String table,String keycolumn,String keydata, String column,String data); public void assertColumnDBNotPresent(Connection con,String table,String keycolumn,String keydata, String column, String data); public void assertLastRecordDBPresent(Connection con, String table,String column,String idColumn String data ); public void assertLastRecordDBNotPresent(Connection con,String table,String column,String idColumn,String data);

Fig. 2. Extract of file Selenium.java with the signature of the new assert methods.

3) assertColumnDBPresent(con,table,keycolumn,keydata, column, data) Function developed to search for a test data in a column that does not represent a table’s primary key. In this case, we cannot check the test data only in the indicated column. As it accepts duplicated data, the test data could be in another record, which could affect the test results. Therefore, the function searches for an identifier submitted as parameter in the table’s primary key column and then checks the existence of the test data in the desired column. Its implementation is very similar to the function assertKeyDBPresent. However, the query used to search the test data in the database is a little different (Fig. 4). public void assertColumnDBPresent(Connection con,String table,String keycolumn,String keydata, String column, String data){

From now, each assert method and their role in automating database’s data check is presented. 1) assertKeyDBPresent(con, table, column, data) This function searches for a data that represents a table’s primary key in a specific column (Fig. 3). When the test data is found in the database’s table, the assertTrue method is called (test case passes). Otherwise, the fail method is called with an error message (test case fails).

Statement stmt = null; try{ stmt = con.createStatement(); String q = ""; q += "Select * from "+table+" WHERE "+keycolumn+" = '"+keydata+"' AND "+column+" = '"+data+"'"; ResultSet rs = stmt.executeQuery(q); if(rs.next()){ rs.getString(column); assertTrue(true); }else { fail("Test Failed!"); } }catch (SQLException e){ e.printStackTrace(); } }

2) assertKeyDBNotPresent(con, table, column, data) The inverse of the previous function, this one checks if a test data does not exist in the table’s primary key column. Its implementation is very similar to the previous function, however when the test data is found, the fail method is called (test case fails). Otherwise, the assertTrue method is called (test case passes).

Fig. 4. SQL Query used in function assertColumnDBPresent.

127

4) assertColumnDBNotPresent(con, table, keycolumn, keydata, column, data) The opposite of the previous function, this one checks if a test data does not exist in a table’s column. Its implementation is very similar to the previous function. However, when the test data is found, the fail method is called (test case fails). Otherwise, the assertTrue method is called (test case passes). 5) assertLastRecordDBPresent(con, table, idcolumn, column, data ) This function searches for a data in the last inserted record in a database’s table column other than the primary key. This function should be appropriate when we do not have access to the primary key that identifies uniquely the last record stored in a table (e.g.: insertion operation in a database’s table) and the table has an id column. When the test data is found in the database’s table, the assertTrue method is called (test case passes). Otherwise, the fail method is called (test case fails). Fig. 5 presents the code used in this function to get the last record in a database. public void assertLastRecordDBPresent(Connection con,String table,String idColumn,String column,String data){

7) Example of Using the New Assert Functions To show how to create a test script using the new assert functions, a database was created in PostgreSQL called records with one table (users) that comprises 4 columns: id (primary key), name, email and phone. In this table, we inserted five records to support test execution with the new asserts functions implemented into SeleniumDB (Fig. 6).

Fig. 6. Database configured in PostgreSQL to support test automation with the new implemented assert functions.

The test script created using the new implemented assert functions is shown in Fig. 7. Firstly, the function connectionPostgreSQLDB is called to establish a connection with the database (line 12) in the method setUp, which is a configuration method executed before all test cases in the JUnit framework. Thus, each test case could access the content of the selected database.

ResultSet rs = null; PreparedStatement pstmt = null;

1. 2. 3. 4. 5.

try{ String q =""; q += "Select * from "+table+" ORDER BY "+idColumn+" DESC LIMIT 1"; pstmt = con.prepareStatement(q); rs = pstmt.executeQuery();

import import import import import

com.thoughtworks.selenium.*; org.junit.After; org.junit.Before; org.junit.Test; java.sql.Connection;

6. public class SeleniumDBTestClass01 extends SeleneseTestBase { 7. Connestion connec; 8. @Before 9. public void setUp() throws Exception { 10. selenium = new DefaultSelenium("localhost", 4444, "*chrome", "https://testapp.com/"); 11. selenium.start(); 12. connec = selenium.connectionPostgreSQLDB ("postgres","rootroot","localhost","5432","records"); 13. }

if(rs.next()){ String columnData = rs.getString(column); if(data.equalsIgnoreCase(columnData.trim())){ assertTrue(true); } else{ fail("Test Failed!"); } }else { fail("Test Failed!"); } } catch (SQLException e){ e.printStackTrace(); } }

14. @Test 15. public void testClass01() throws Exception { 16. selenium.assertKeyDBPresent(connec,"users", "email", "[email protected]"); 17. selenium.assertColumnDBPresent(connec,"users", "email", "[email protected]", "name", "John"); 18. slenium.assertLastRecordDBPresent(connec, "users", "phone", "555-1143"); 19. } 20. @Test 21. public void testClass02() throws Exception { 22. selenium.assertKeyDBNotPresent(connec,"users", "email", "[email protected]"); 23. }

Fig. 5. Extract of the assertLastRecordDBPresent function’s code

24. @Test 25. public void testClass03() throws Exception { 26. selenium.assertColumnDBNotPresent(connec, "users", "email", "[email protected]", "name", "John"); 27. }

6) assertLastRecordDBNotPresent(con,table,column,data) The inverse of the previous function, this one checks if a test data does not exist in the last inserted record in a database’s table column. Its implementation is very similar to the previous function, except that when the test data is found, the fail method is called (test case fails). Otherwise, the assertTrue method is called (test case passes).

28. @Test 29. public void testClass04() throws Exception { 30. selenium.assertLastRecordDBNotPresent (connec, "users",”id” "phone", "555-1143");31. } 32. @After 33. public void tearDown() throws Exception { 34. selenium.closeConnectionDB(connec); 35. selenium.stop();36. } 37. }

Fig. 7. Test Script using the new proposed assert functions.

128

In the line 16 (Fig. 7), the function assertKeyDBPresent is called to search for the email ([email protected]) in the primary key column email. In the next step, the function assertColumnDBPresent is called to search for the name (John) in the non-primary key column name, using also as parameter the text [email protected] to be searched into the primary key column email in order to find uniquely the desired record (line 17). In the line 18 (Fig. 7), the function assertLastRecordDBPresent was used to check the content of column phone in the last inserted record in table users. Tests were also created using the other proposed assert functions in the same database. In lines 22 to 23, it will search for the email “[email protected]”, that does not exist in the database, using the assert function assertKeyDBNotPresent. This script must pass during the tests execution. In the lines 26 to 27, the name “John” with the email (key) “[email protected]” will be searched using the assert function assertColumnDBNotPresent. This script must fail when executed, because the name “John” is associated to two emails, and “[email protected]” is one of them. The idea is to observe the behavior of this tool when a test script fails. Finally, in the lines 30 to 31, it will search for the phone number “555-1143” using the assert function assertLastRecordDBNotPresent. This script must also fail, since this phone number is located in the last record. The connection to the database is finalized in the method tearDown (line 33). The next step is to run the created test scripts and verify if they have passed or failed by Selenium framework. It’s expected that two of them pass (testClass01 and testClass02) and two of them fail (testClass03 and testClass04), as indicated in Fig. 8.

as instrument to increase the quality and reduce the effort of testing. In this section, a case study was presented where the proposed tool was applied to support automated database testing in a web-based application module responsible by register users, institutions, groups and user access profile. The complete software project comprises two modules: web and mobile (JavaME platform). The web module has 5 functionalities and in the mobile module has 6 functionalities. During the development 128 test cases were designed and executed. This system supports people who are visually impaired. It sends audio information to cell phone with the description of places in the public institutions as libraries, hospitals, markets and etc. In this case study, two functionalities developed/tested were analyzed in this project (Fig. 9): CRUD for User and CRUD for Institutions. Thirteen test cases were designed to test these scenarios: 4 test cases for User’s CRUD and 9 test cases for Institution’s CRUD.

Fig. 9. Functionalities tested in the case study (in Portuguese).

Fig. 8. Result after to execute the created test scripts.

It’s possible to access these examples of test scripts in the URL www.icomp.ufam.edu.br/experts/seleniumdb_project.zip. IV.

CASE STUDY

The extension of Selenium RC developed in this work (SeleniumDB) has been used in several real software projects

Three strategies of test execution were performed: manual, semi-automated and automated. In the manual execution, the tester should open the browser, login into the system, do the action indicated in the test case (e.g.: add user, edit institution, etc.), open the database tool and access the database table to check the data. In the semi-automated execution, some steps were done by the test script implemented using Selenium RC (open the browser, login into the system, do the action indicated in the test case ) and other steps were done manually by the tester (open the database tool and access the database table to check the data). Finally, in the automated execution, all steps were performed by the test scripts built by the testers using the proposed tool, including the access to the database’s table to check the test data.

129

TABLE II. RESULTS OF TEST CASES EXECUTION (IN SECONDS)

Test Cases ID

Description

TC-81 TC-47

Manual Execution’s Time

Semi-automated Execution’s Automated Execution’s Time (Using Selenium RC) Time (Using Selenium DB)

Functiona l Test

Functional Test

DB Check

Total

DB Check

Total

Functional and DB Test

Total

Add Institution (success) Add Institution (invalid characters) Add Institution (limit of TC-40 characters) TC-39 Add Institution (required fields) TC-82 Edit Institution (success) TC-48 Edit Institution (invalid characters) Edit Institution (limit of TC-42 characters) TC-41 Edit Institution (required fields) TC-83 Delete Institution TC-62 Add User TC-61 Edit User TC-60 Delete User TC-119 Add User (invalid characters)

75.0 70.0

93.0 93.0

168.0 163.0

14.5 14.1

93.0 93.0

107.5 107.1

12.5 13.7

12.5 13.7

73.0

93.0

166.0

14.7

93.0

107.7

14.1

14.1

57.0 71.0 68.0

93.0 93.0 93.0

150.0 164.0 161.0

13.5 14.8 15.8

93.0 93.0 93.0

106.5 107.8 108.8

12.9 14.2 14.8

12.9 14.2 14.8

70.0

93.0

163.0

13.8

93.0

106.8

13.5

13.5

55.0 18.0 98.0 89.0 16.0 93.0

93.0 93.0 93.0 93.0 93.0 93.0

148.0 111.0 191.0 182.0 109.0 186.0

14.2 11.2 11.7 11.5 9.5 11.8

93.0 93.0 93.0 93.0 93.0 93.0

107.2 104.2 104.7 104.5 102.5 104.8

13.9 10.9 11.3 10.8 9.1 11.30

13.9 10.9 11.3 10.8 9.1 11.3

TOTAL

853.0

1209.0

2062.0

171.1

1209.0

1380.1

163.0

163.0

Using both strategies, 13 test cases were executed. The results are presented in Table II. The total time spent for manual tests execution was 2062 seconds (around 34 minutes) and 58% of this time (1209 seconds) was spent checking the test data in the database. The time spent for test cases execution using the semi-automated strategy was 1380,1 seconds (around 23 minutes) and 87% of this time (again 1209 seconds) was spent checking the test data in the database. This result indicates a reduction of 33% in the final time spent for test cases execution using a semi-automated functional testing tool when compared to manual tests execution. Analyzing the use of SeleniumDB (automated functional and database testing), it was possible to observe that the total time spent for test execution was 163 seconds (around 3 minutes). In this strategy, the check of test data in the database is performed in the same script used for functional test. Thus, they cannot be divided into two steps. This result indicates a reduction of 88% in the final time spent for test cases execution when compared to the semi-automated test execution and a reduction of 92% in the final time spent for test cases execution when compared to manual tests execution. Moreover, using Selenium DB, the automation rate of test cases designed for these two functionalities was 100% (13 automated test cases ÷ 13 designed test cases). It was not necessary to perform any manual step to execute these test cases. V.

CONCLUSIONS

Automated Database Testing can be used to support software engineers in the evaluation of database-based web

applications, reducing the effort to perform this task when compared to manual or semi-automated strategies. Some problems in database testing would be the use of other tool to automate specific tests and the effort to create additional test scripts to deal with this aspect. In order to solve these problems, this paper proposes an extension of a very popular functional testing tool for user interface (Selenium RC) to support database testing in webbased applications. We called this solution SeleniumDB. The proposed tool implements new assert functions into the Selenium framework’s core to deal with test data checking in databases. These functions are applied to different scenarios where data need to be checked in databases, such as: searching for primary keys, non-primary keys columns, or the last record inserted into a table. An interesting and very relevant aspect of this proposed tool from a practical viewpoint is that it would be able to be used now in real software projects. The tool proposed was evaluated by applying it in a real software project developed by Nokia Institute of Technology located in Brazil and comparing the obtained results to other two strategies (manual and semi-automated tests execution) regarding the effort (time) required to execute a test case suite. The results indicated a reduction of 88% and 92% in the effort to execute a test case suite when compared, respectively, to semi-automated and manual tests execution strategies. Currently, the proposed assert functions work only in MySQL and PostgreSQL databases. As future works, we intend to extend SeleniumDB management systems, such as Oracle or SQL Server.

130

As another extension, we will implement new assert functions to provide different tests in a database including, for instance, assert functions that perform a join among different tables in a database. Finally, we are providing a mechanism to offer the new assert functions developed in this work using the Selenium IDE (Firefox plugin) tool. The Selenium framework is being updated by its developers and we are waiting for its new version to perform this extension.

[8]

[9]

ACKNOWLEDGMENT The authors would like to thank INdT, FAPEAM, CNPq (process 575696/2008-7) and INCT-SEC (processes 573963/2008-8 and 08/57870-9) for supporting this research.

[10]

REFERENCES [1]

[2]

[3]

[4] [5] [6] [7]

Graham, D. Fewster, M. (2012), “ Experiences of Test Automation: Case Studies of Software Test Automation”, (1st ed.). Addison-Wesley Professional. Dustin, E.; Rashka, J.; Paul, J. (2008) “Automated Software Testing: Introduction, Management, and Performance”. Addison-Wesley Professional. Karhu, K.; Repo, T.; Taipale, O.; Smolander, K.; (2009) "Empirical Observations on Software Test Automation," In: International Conference on Software Testing Verification and Validation (ICST '09), vol., no., pp.201-209, 1-4 April 2009 doi: 10.1109/ICST.2009.1. Watir.Com | Web Application Testing In Ruby. Available in: http:// watir.com. Access in: October, 30th 2012. Canoo Web Test. Available in: http://webtest.canoo.com/. Access in: October, 30th 2012. Apache Jmeter - Apache Jmeter™. Available in: http:// jmeter.apache.org/. Access in: October, 30th 2012. Introduction - Selenium Documentation. Available at: http://seleniumhq.org/docs/01_introducing_selenium.html#intro ducing-selenium. Access in: October, 24th 2012.

[11] [12] [13]

[14]

[15]

131

Chays, D.; Dan, S.; Frankl, F. G.; Vokolos, F. I.; Weber, E. J.; (2000); “A framework for testing database applications”. In Proceedings of the 2000 ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA '00), Mary Jean Harold (Ed.). ACM, NY, USA, 147-157. DOI=10.1145/347324.348954.M. Young, The Technical Writer’s Handbook. Mill Valley, CA: University Science, 1989. Rogstad, R.; Briand, L. C.; Dalberg, R.; Rynning, M.; Arisholm, E. (2011); "Industrial Experiences with Automated Regression Testing of a Legacy Database Application"; In: 27th IEEE International Conference on Software Maintenance (ICSM), pp. 362-371. DOI: 10.1109/ICSM.2011.6080803. Haftmann, F.; Kossmann, D.; Lo, E. (2007); “A framework for efficient regression tests on database applications”. In: The International Journal on Very Large Data Bases, Vol. 16, (January2007), 1nas45-164. DOI=10.1007/s00778-006-0028-8. ABOUT DBUNIT. Available at: http://www.dbunit.org. Access in: December, 19th 2012. Simple Web Automation Toolkit. Available in: http://ultiswat.wikispaces.com/. Access in: March, 7th 2013. Alshahwan, N.; Harman, M; (2011); “Automated web application testing using search based software engineering”. In: Automated Software Engineering (ASE), pp.3-12. DOI=10.1109/ASE.2011.6100082. S. Artzi; Kieżun, Adam; Dolby, Julian; Tip, Frank; Dig, Danny; Paradkar, Amitkumar M.; Ernst, Michael D. (2010); “Finding Bugs in Web Applications Using Dynamic Test Generation and Explicit-State Model Checking”. Journals & Magazines, Vol. 36, pp.474-494. DOI=10.1109/TSE.2010.31. Halfond, William G J; Orso, Alessandro; (2006); “CommandForm Coverage for Testing Database Applications”. In: Automated Software Engineering, Vol. 16, pp.201-209. DOI=10.1109/ASE.2006.27.