AsT -A Simple Automated System Testing Tool Leckraj ... - IEEE Xplore

AsT-A Simple Automated System Testing Tool

Leckraj Nagowah and Purmanand Roopnah Computer Science & Engineering Department, University of Mauritius, R6duit, Mauritius. [email protected], [email protected]

participation or interaction in unskilled, repetitive or redundant tasks [2]. An automated testing tool is basically a computer-controlled tool that tests software for functionality and performance. There exist a number of automated test paradigms [1,14] among which the main ones are: load and performance testing, endurance and stress testing, automation of test input generation, regression testing, automated unit testing, browser testing and functional-testing. Although many automated testing tools exist, people tum to automated testers for designing their test cases and test suites and execute them in due time as building the required tests usually requires knowledge of the automated tool. Playback and record is very common for automated testing tools. This form of test creation is used to create throwaway test scripts. Creating automated tests by this method is faster but cannot be maintained for a long period of time. From our experience of the software industry in Mauritius, we understand that data is very important for testing. Therefore we have opted for the design of a testing tool to perform an overall coverage, as compared to unit testing which cater only for small units. Most testers rewrite or copy and paste repeated flows across their test scripts. Record/playback misleads many testers into using automation to separately re-record each instance of a repeated flow. After great consideration following the main existing automated tools, we decided to design a Browser-based Functional automated testing tool. This type of testing has not been so much developed in software industries and we believe that there would be tremendous scope for our tool because of the boom in web technologies. The rest of this paper is structured as follows: Section 2 talks about well-known automated testing tools. Section 3 presents a number of design issues to be considered in the implementation of our automated testing tool, AsT. Section 4 presents the system architecture and the actual implementation of AsT. Section 5 provides testing results and evaluates our testing tool, AsT. Finally Section 6 concludes with future works on AsT.

Abstract-In an age of rapid development in increasingly complex environments, automation is the only real hope for achieving any acceptable level of test coverage. Yet, even with all of the tools and technology available, the overwhelming amount of testing is still being done manually. Test automation is a vital component of every test project. While testing seems to take a new dimension, the big question is how can an automated testing tool be used to improve the overall testing process. Test Automation develops and customizes automation test suite that are implemented across projects. With the rapid updates of software and constant update of applications and limited time to test a whole system, testing automation is the way that can move software development model towards completion coverage of testing in less time and less labour. We propose a list of features that we believe any automated testing tool should contain and we develop a simple, automated testing tool which can aid analysts testers and developers to design and develop their test cases, execute them with minimum human intervention and to generate reports to determine the results of the test cases.

I.

INTRODUCTION

Testing allows companies to discover bugs in an early stage and allows them to be corrected in due time resulting in companies preventing losses in projects. Testing can however be viewed as a boring task. This is the reason why automation testing is moving towards success as its use enable to reduce time, cost and also labour. Test Automation develops and customizes automation test suite that are implemented across projects. The fast pace of modem software development creates tremendous challenges for the test team. Online software updates and frequent release cycles tum out to be a "moving target" for test automation efforts. Test organizations that have not yet advanced automation, beyond simple record/playback, soon discover that recorded test scripts happen to be obsolete within a few project iterations of the software being tested. Many organizations acquiring a test automation tool begin the automation process without any consideration about maintenance, scalability, or effectiveness with however great expectations on immediate payback [1]. "Automated Testing" is simply the automation of the manual testing process currently in place. This requires that a formalized "manual testing process" currently exists in the organisation. Automation is the use of strategies, tools and artifacts that enhance or reduce the need of manual or human

II.

RELATED WORK

An automated testing tool should be designed such that

data can be stored for future reuse. While some open source testing tools are barely known to people, other functional testing tools are known to be constantly improving to reflect the continuous changes in the world of testing. However not

978-1-4244-5540-9/10/$26.00 ©2010 IEEE

301

all of them have similar functionalities. Below is an outline of some common automated testing tools: IBM Rational Functional Tester [4] is an automated functional testing and regression testing tool which uses java language as scripting language. IBM Rational Functional Tester uses Excel as database to store the data that is being inputted into the web application to be tested. An HTML log is displayed as result at the end of the tests. It uses ODBC connection to connect to and retrieve values from the Database (Excel). Selenium IDE [5] is much known in terms of testing automation. It is implemented as a Firefox extension, and allows the tester to record, edit, and debug tests. Selenium is a record and playback testing tool, i.e. a complete testing tool. Selenium converts the data recorded during the test case into XML and generates an HTML report for comparison of data. Selenium can be used for both functional capability and regression testing. QuickTest Professional [6] also known as 'QTP' is an automated testing tool provided by HP/Mercury Interactive. QTP uses VB scripting language to build its flows. It integrates easily with other Mercury testing solutions including TestDirectorlQuality Center, WinRunner and LoadRunner. QTP utilizes add-in architecture for compactness and extensibility. QTP can be used for both types of Testing: Manual:- Usage of steps defined within test cases to manually test the application to ensure that it functions properly; Automated:-Automation of the manual testing process to all unattended execution and ensures repeatability Sahi [7] is an automated testing tool for web applications, with the facility to record and playback scripts. Developed in Java and JavaScript, it uses simple JavaScript to perform events on the browser. Sahi then injects JavaScript so that it can access elements in the webpage. IBM Rational functional tester and QTP both use scripting language to build flows. Each screen in the application requires a separate class for the test flow. Therefore, any non IT person cannot use the program as coding knowledge is required for proper use of these automated tools. Selenium IDE which is a Firefox extension does not support the latest version of Firefox. The main advantage of selenium is indeed the quick ability to build the test cases. However, when a new field is be inserted in the web application for an existing flow, the flow has to be built again. III.

importance. The user interface should guide the user throughout the process. The automated testing tool should enable users to automate regression tests. Existing features such as: play/stop/pause for easy manual creation of tests, execution of test scripts, comparison of test results with expected results and the ability to connect with different data sources should be included in the tool. The user should be able to view results in an understandable format that can also be printed. Another important feature of an automated testing tool is the flexibility in allowing some test steps to be skipped or run at specific instances. The automated testing tool should also allow users to debug errors and allow breakpoints in the application. The automated testing tool should be able to run automated tests by feeding the system with data directly from the data source/file. The system should be able to process the data obtained from the file, compare the data from the file to those found on a database table and log the comparison in a report. The testing tool should be able to read web applications as well as Java applications. IV.

SYSTEM STRUCTURE AND IMPLEMENTATION

Taking into account the limitations of existing automated testing tools, we chose to design and implement a simple Automated fu'stem Iesting tool abbreviated as AsT, rather than to make modifications to existing open source automated testing tools. AsT tool has hence been developed from scratch to avoid any risks of existing framework limitations in the event of eventual software updates. Figure 1 shows the overall layout of our automated testing tool, AsT and figure 2 shows its structural view. The automated tester first views the web application and analyzes the different elements (fields) that are to be automatically populated. For each screen, there can be a number of elements that can be used to create an action. Based on this analysis, the tester then builds the test flows in the data source, which in AsT is an Excel sheet. The AsT tool then retrieves the data from the data source, processes them sequentially and populates them into the web application's fields. The AsT tool assigns and inserts the values in the corresponding fields of the web application as per the designed flow in the data source. The testing party interacts with the data source which stores all the different data inputs and the test flows. This same Excel sheet is used to display the current results in each row of each flow. The tester records a test flow in the Excel sheet along with the data inputs. The idea of such a system is to give the tester more flexibility while testing and to allow testing to be done at any time. No human intervention is needed for the execution of the flows. The application is designed to support the basic web application language, i.e. PHP, Ajax, HTML and other web development languages.

DESIGN ISSUES

The challenges to be considered for the design and development of an automated testing tool are very critical for a high performance, easily learnable and effective testing tool. In this section of the paper, we present some design issues that should be considered in the design and implementation of an automated testing tool. We formulate these design issues after an analysis of the existing automated testing tools. Novices and people, who don't have much knowledge of web development and testing in general, should be able to use the testing tool. A user friendly system and a menu driven interface are therefore of utmost

There two types of test mode that can be done are Verification and Assertion.

302

R.....

Sequence Number is the Test Case number that shall be run one by one by the AsT tool; Fields (Names of textbox, radio buttons on the webpage) stores the different elements of the testing web application that need to be populated by the tool; Values (Indicate the values to be inserted in the blank fields) stores the different values that the tool populates into the website elements; Commands (cancel button, submit button, link) contains the different actions for a website; Results (Failed or passed): displays the results of the execution of scripts. AsT Tool enables users to automate regression tests. The tests consist of different methods such as easy manual creation of tests, execution of test scripts, and comparison of test results with expected results. AsT allows testers to conceive and create a test case as well as a complete test suite in Excel sheet. Results of the tests are compiled in a report. The detailed report and the results on excel sheet allows users to debug the cause of any failures. The test cases developed on the Excel sheet can also be saved for future use. Regression tests can be done based on the tests cases just by connecting the existing excel sheet. A larger coverage of the system can hence be covered in low manpower, less time and less cost. Object Orientation is a programming paradigm whereby the use of objects and features are emphasized regarding the object properties and interaction [15]. This means that real world objects can effectively be modeled in the system and features can be added and extended. Hence an excellent object oriented programming language would be advantageous to the design and development of our AsT tool as all the components of the AsT tool can be regarded as objects. The Java programming language has been chosen as it provides for the best features which can be implemented more rapidly compared to programming language like C++ or VB.net [16]. For instance, the Write-once-Run-Anywhere capability makes Java an ideal choice. Also, Java provides other valuable features like Automatic Garbage Collection. The user interface is made up of Swing and AWT components since Java Swing helps to take full advantage of the Swing classes, providing detailed descriptions of every class and interface in the key Swing packages which enables the building of state-of-the-art user interfaces [10]. By making a comparison between the different databases, Microsoft Excel has been chosen as the data source since Microsoft Excel provides for easy and quick creation and population of tables, a feature that is hard to obtain in most of the back end databases. Microsoft Excel is portable and as compared to other database and no additional software is needed [8]. ODBC-JDBC connection shall be used to connect the Excel sheet to the AsT tool. This type of connection works very well with Excel and it allows data to be inserted, updated and deleted easily [9]. Concerning the report generated, our choice was between HTML file and PDF report. Most automated testing tools generate an HTML file but we chose to create PDF in order to ensure that no formatting features are lost if used on different browsers and platforms. All PDF files support data encryption hence providing an easy and secure way of data transfer and sharing [11]. API used to generate the PDF file

AsT testing 1001

Figure 1.

Layout of Overall System

f--.l

1

[email protected],I. .

·

. . ·

:

·

-

- - -

-

-

-

-

{- l

-

-

-

-

-

- -

-

•

-

- - - - -!r-

-

-

-

�

Extel ,�t as datastore

Figure 2.

I I

-_ ...

-

.

i- --,

"'-"'.......- {"'--) "'etaill!

,

� ,

Figure 2: Structural View of Overall System

The verification occurs during the execution of commands if one of the commands fails, the execution continues till the end of the flow and upon completion of the tests suites, results are displayed informing that the execution has failed at specific point(s) as well as the others results of the test cases. In the other hand, Assertion mode occurs during the execution of commands if one of the commands fails, the execution ends at once and the results are displayed. The test cases that follow are not run. An example of how the test flows are built is given in figure 3. Sequence nwnber

1 1 J 4 j

Fields

Values

Txt nam:!

John

Subntit(Bunon)

Figure 3.

Conunarxh

Results

1234

Txt...,pissword

Click

Design for Excel Spreadsheet

303

containing the data and the built flows. On selecting Play button, the AsT tool connects to the Excel file, which comprises of the different designed test cases, and starts the test procedure. As shown in figure 5, the column 'TestCase' includes the names of all the designed test cases. The next column (Test or Skipped) links the test case to the associated value that is 'T' implies that Test! need to be tested whereas 'S' implies Test case is skipped as in the case listed below

is iText API, the latter being an ideal library for developers looking to enhance web and other applications with dynamic PDF document generation and/or manipulation. By making a comparison between Cobra framework HTML parser and htmlUnit HTML parser, choice was towards htmlUnit HTML parser 2.6 [12] since the Cobra framework in spite of being a complete API Cobra framework HTML parser does not allow reading JavaScript and PHP scripts. HtmlUnit is still in pioneer phrase but with the help of our developed methods, it reads HTML files as well as JavaScript files. The Integrated Development Environment chosen was Eclipse since it provides Essential tools to design and build the AsT tool in less time and efficiently. Eclipse is a free and open source software and is very well supported by a global community. This Tool successfully combines coding, editing and execution of Java codes. This pack also works well with the connection of databases. Maven2.2 has also been used since it allows a project to be built using its project object model (POM) and a set of plug-ins that are shared by all projects using Maven, providing a uniform build system. Maven makes the build process easy and allows transparent migration to new features. V.

(TestUpdateuser). C13 A

OpUons

I[�I I II

Pauu

T

Passed

4

Test U pdateuser

S

Skipped

Passed

List of test cases in the TestScenarios sheet

assigne� to t�e c�rresponding field's name. S

C

-0'

F

G

1 •

.

t T"st_,-c1rtrltlol'l • Tflt_'e&iSltIUon

"

I " 2

9 Test_,qlslrltlol'l rj 10 Tfll_'tCiw.tlon .. 11 Tesl_'-clslrllloI'I

12 r"SI_'qlstrf'Uon 13 Testj.,.str.loon

Figure 6.

-

'5 '6

."

"

1

Test Data sheet consisting of the different test cases

The flow for test! was built by analyzing a typical login page as shown in Figure 7. The txt_name is the name of the text field for the username and txt�assword is the text field name given to the password field.

I [E]

semame: Password:

Figure 7.

As eltion Figure 4.

T

TestReg istrati on

The AsT tool selects the value of the first test case Test! in the "TestScenarios" sheet and links Test1 to the TestData sheet which is as shown in figure 6. This sheet consists of all the details about the flow of events. As shown in figure 6, Test 1 has 4 test steps; all the tests steps have a value 'T' which means that the test have a status 'Tested'. The column headers such as txt name and others are the different names in the HTML page that are to be tested. The row values are the values that are to be retrieved by the AsT tool and

TESTING & EVALUATION

Ii�'

Testl

3

Figure 5.

As we moved towards the development of the project, verification and validation was performed on the specific requirements of the AsT Tool to remove almost all uncertainties concerning the requirements and to ensure that almost all requirements are achievable. Upon completion of the AsT tool, a number of testing processes were performed on the AsT tool. Among these tests, unit testing of different components was done. Before the different components were integrated, unit testing was performed on the main components: retrieving values from the database (read/write/update), generation of the PDF report that consisted of the specific details based on the test cases that were run and finally the retrieving the fields' values from an HTMLIPHP page and setting values to the existing fields. The second type of test performed was a full system testing: a number of test scenarios were built to test the flow of events while using the AsT tool. Fli.

2

Assertion screen

Login screen being tested

As the AsT tool is launched by pressing the Play button, the tool selects the value Test1 from the TestScenarios sheet and then chooses the first test step of Test! from the TestData sheet. The AsT tool retrieves the first column header 'txt name' and its rowvalue 'manish' under the

Figure 4 shows the AsT tool interface, consisting of 3 main buttons Play, Pause and Stop. Before launching the AsT tool, an ODBC connection is made with the Excel sheet

304

comparisons, 5 being the best possible rating and 0 being the worst. We agree that AsT tool is still way behind the Existing Automated tools in terms of features. However, the main advantage of the AsT tool is the time for the creation and maintenance of test cases is relatively smaller compared to other existing tools, hence a higher rating for AsT tool. In IBM Rational Functional Tester, test cases are created in the regression test sheet and another separate excel sheet, named the Application map, is also required. In this particular case, two sheets are needed which means more time to build the flows. With Selenium, in case of an event where a new field is to be inserted in the tested application, the whole flow has to be captured again. Our Ast Tool only requires 1 Excel sheet and in the event that a new field needs to updated in the web application, only 1 additional row value and 1 additional column header value has to be inserted in between the flow of events, the rest of the flow remaining unchanged. Therefore less time is lost in making the scripts in AsT tool. Instant update of results is also a great piece of work accomplished by the AsT tool. No coding is required as compared to QTP and IBM rational functional tester.

column header. The AsT tool identifies the column header in the HTML page provided, by making the connection between the AsT tool and the Web application and assigns the value 'manish' to the txt name and move to the next test step. Results are updated in the 'Results' columns in the Excel sheet instantly as tests are being run as shown in figure 6. A PDF report is also displayed, as in figure 8, and can be used to debug the test case if there are any failures. This procedure continues till all the test steps for Testl in TestData sheet are completed and then it moves to the other test case TestRegistration according to the TestScenarios sheet as show nm fiIgure 5 .

Test ID

Test Step

Test Result

Test 1

1

Pass

Test 1

2

Pass

Test 1

3

Pass

Figure 8.

VI.

Results diplayed on PDF file.

No matter how rich is a product in functionalities, if it fails to meet the performance expectations of customers, the product is branded as a failure. In terms of performance testing, AsT is rapid and it creates its output instantly. In terms of security, the AsT tool is without doubt safe since the PDF report which is generated at the end of the test cases cannot be modified or created otherwise. AsT tool also caters for error messages that are displayed on the console of the AsT tool in cases of failures. We also found that once the connection with the excel sheet is done; the testing process is done much quicker, as compared to manual testing. As compared to other existing testing tool, our simple automated testing tool, AsT, is only a novice tool but it does have the same functionalities as the other existing automated tools discussed earlier. AsT can basically read an HTML page, set the appropriate values to the appropriate elements and automatically generate reports in a PDF file, the results of which are also included in the Excel sheet. 6

,-----

t,.r------------------4 +i.r----5

3

• AsT tool

2

.IBM RFT

1 0

• Selenium

Time for

Ability to

Time of

No of

Creation of

display

execution

features

Test cases

resu Its

Figure 9.

CONCLUSION AND FUTURE WORKS

The main functionality of our AsT tool is to facilitate the testing of functional components of a system. Some successfully implemented issues are the assertion mode, verification mode and the easy creation of the test flows in Excel. The user interface guides the user throughout the execution of the test cases. AsT allows the quick generation of test flows and eventually generates a report at the end of the run. This tool also generates instant results in Excel. We also believe that AsT tool also executes more rapidly than other functional tools. The work achieved till now is only the beginning of a new era to produce the best possible testing tool. We present some potential improvements that could be brought to the existing system. Implementing these features could help to either solve the current shortcomings or improve the functionalities of AsT tool. Firstly, in case of errors, no log is currently displayed in a separate file. Reference has to be done on the AsT console to be able to debug the error. A log file in HTML or even PDF file can be created for advanced debugging mode. Secondly, the PDF report designed currently only prints values, the current test step and current tests that have passed or failed. For a more precise report, the report may consist of the columns' header name that is being passed, the 'rowvalues' and the pages that it is passing these values. Thirdly, when the tests run, the values that are passed are not seen, only the results of the test cases are returned. By using specific commands such as selenesse commands, this tool can be made more interactive by providing a better simulation of the tests. Fourthly, AsT tool was primarily designed for HTML and PHP scripts. The system can be further enhanced by allowing tests to be carried out on more environments such as Java, Ruby, C# and other web languages. Lastly, in the current version of AsT, the database connection has to be done manually. We also plan to

Comparison offunctionalities with other existing software

Figure 9 shows our comparison of the testing tool AsT to two well-established testing tools, Selenium and IBM Rational Functional Tester. Ratings were assigned to the

305

establish this connection to the excel spreadsheet from the AsT interface itself in the future.

REFERENCES

[1] [2]

Hoffman D., "A Course on Software Test Automation Design",

2003.

[3]

Kanglin Li and Mengqi Wu. "Effective GUI Test Automation Tool: Developing an automated GUI testing", 2004.

[4]

IBM Coporation, "Rational Functional Tester- Validate application quality withfunctional and regression test automation", 2008.

[5] [6]

Selenium Project, "Selenium IDE", 2009.

[7] [8]

Hewlett-Packard Development Company, "Best practices implementing automatedfunctional testing solutions, 2009. Sahi Automation http://sahi.co. inlw/sahi

Tool

for

Web

Application

[9]

for

Testing.

McCaffrey, James, "TestRun: Using Excel for Test Data", MSDN

Magazine, Vol, No 21(12), November 2006, pp.

Eckstein R, Loy M., Wood D. and Elliott J., Java Swing, O'Reilly Media, 2002.

[II]

Rider C. H., Edgar Filer Handbook: A Guide for Electronic Filing With the Sec, 4th Edition, Aspen Law & Business, 1994.

[12]

HtmlUnit - A Java unit http://htmlunit.sourceforge.netl

[13]

Exforsys.com, "Automated Testing Advantages, Disadvantages and Guidelines".

25-3\.

Network Computing, "Network Design Manual - Data Access Via ODBC and JDBC ODBC".

306

2nd Edition

[10]

testing

framework,

Sourceforge.

Clark R, "How to Test Software", Rubric Quality Consultants Ltd,

2009.

[14] [15]

Jiantao Pan, Software Testing, Carnegie Mellon University,

1999.

[16]

Ahuja S. P., Eggen, R, and Daucher, C., "Performance Evaluation of Java And C++ Distributed Applications In A CORBA Environment", Proceedings of the International Symposium on Performance Evaluation of Computer and Telecommunication Systems (SPECTS 2002), San Diego, CA, July 2002.

Lee R C. and Tepfenhart W. M., Practical Object-Oriented Development with UML and Java, Prentice Hall, Pearson Education, Upper Saddle River, New Jersey, 2003.