Lessons learned from automating tests for an ... - Semantic Scholar

SOFTWARE—PRACTICE AND EXPERIENCE Softw. Pract. Exper. 2002; 00:1–23

Prepared using speauth.cls [Version: 2000/03/16 v2.12]

Lessons learned from automating tests for an operations support system ∗ Mariusz

A. Fecko and Christopher M. Lott

Applied Research Area, Telcordia Technologies, Inc., Morristown, New Jersey, USA

SUMMARY We present experience gained in automating tests for an operations support system. A major portion of the effort was devoted to extending a commercial test tool so that testers could easily manipulate GUI objects on two implementations of the application. For this purpose, we developed a test automation library as support infrastructure for writing tests. The challenges and tradeoffs are discussed such as simplicity/complexity for a tester vs. a library developer, hiding/exposing window hierarchy to the tester, providing common methods for different types of GUI objects, transparently manipulating custom GUI widgets, and coping with data-dependent test cases. We discuss the requirements of test code reusability, maintainability, and portability, and describe the solutions we found. In addition, we offer observations about benefits and pitfalls of test automation, our recommendations for maximizing return on investment, and results from automating a variety of tests. key words:

1.

operations support system; test automation; test library

INTRODUCTION

We present experience gained in automating tests for an operations support system (OSS) [1]. The OSS we worked with provides back-office functionality for telephone subscribers who wish to change service providers but retain their telephone numbers (commonly known as number portability). Users interact with the system using form-driven GUIs that display and manipulate tabular data stored in a relational database. Before this project began, testing practices were largely dedicated to manual tests executed via these GUIs. In addition, the project had recently added a web implementation (a GUI

∗ Correspondence

to: Applied Research Area, Telcordia Technologies Inc., 445 South St. MCC-1G332R, Morristown NJ 07960, USA.

c 2002 John Wiley & Sons, Ltd. Copyright

Received December 2001 Revised July 2002 Accepted ??

2

M. A. FECKO, C. M. LOTT

implemented as a Java† applet) of the existing legacy system (a GUI based on PowerBuilder technology). The Java GUI was functionally equivalent to the legacy GUI, but introduction of a second GUI effectively doubled the workload of the test group. Our efforts towards test automation focused on business-flow tests that involved manipulating the system’s GUIs. A business-flow test corresponds to a sequence of actions taken by a user that accomplish a common business task; e.g., adding a service for a customer. Business-flow tests differ sharply from tests that only focus on the GUI and not the back-end functionality; e.g., verifying that a certain GUI edit field is present and only allows numbers. The primary business goal of the test automation project was to improve customer satisfaction by detecting more defects than were being caught by the testing staff (as opposed to reducing the number of testing staff). To meet this goal, we developed technology to support automated execution of test cases on the GUIs. A major portion of our effort was devoted to extending a commercial test tool (SilkTest ‡ from Segue Software [2]) so that testers could easily manipulate GUI objects on either implementation of the application. This extension took the form of a library that customized the test tool for use on the two GUIs. The library made it possible for testers to write maintainable and portable tests. More concretely, the library allowed a tester to write a test script in such a way that it was highly readable and therefore maintainable. Further, tests were portable because library-based test scripts could be executed against either GUI. The paper is organized as follows. Section 2 presents the main challenges in this test automation project, and discusses the tradeoffs we made to meet these challenges. Section 3 discusses the OSS system and the test tool. A survey of related work appears in Section 4. The last two sections focus on our response to the challenges of the project. In particular, Section 5 describes the library that we developed to support automated tests, and Section 6 discusses technology transfer, qualitative impact of the technology, and return on investment recommendations.

2.

CHALLENGES OF TEST AUTOMATION

Our main technical objective was to automate tests using technology that could be easily transferred to testers and that would support two functionally equivalent GUIs. We had to face several basic challenges. As with most test automation projects, we had to find a maintainable way to assist in automating tests, and to convince testers that automating their manual practices can bring real benefits rather than create additional burdens. With this issue in mind, we set several technical goals for ourselves: • Develop a test infrastructure that is easy to use by the testers. In particular, the GUI should be manipulated through high-level methods that hide the low-level details required for that manipulation.

† Java

is a trademark of Sun Microsystems, Inc. is a trademark of Segue Software, Inc.

‡ SilkTest

c 2002 John Wiley & Sons, Ltd. Copyright Prepared using speauth.cls

Softw. Pract. Exper. 2002; 00:1–23

LESSONS LEARNED FROM AUTOMATING TESTS. . .

3

• Help testers meet their commitments of testing the legacy application (PowerBuilder§ ) as well as the new implementation (Java). The library methods should be transparent to the GUI type, thereby offering support for test scripts that are portable between the two GUIs. • Demonstrate the benefits of the library and test automation by developing and delivering a wide variety of tests, from basic GUI sanity tests to business-flow tests involving several systems’ components. • Identify existing tests that are especially cumbersome and time-consuming to run manually, and explore the benefits of automation for these tests. Here we had a few hundred test cases developed internally by the product testers, and externally by the Number Portability Administration Center. • Teach testers how to write and run automated tests as a regular part of their responsibilities. Test automation projects can use capture-reply techniques, scripting techniques, or a combination to produce tests [3]. An organization’s first attempts at automating tests often rely on using a testing tool’s capture-replay feature. Companies usually begin with capturereplay to see “if test automation will work for them,” and in general this approach yields useful, automated tests very quickly. However, in the long run this approach is usually unsuccessful and therefore eventually abandoned [4]. The capture-replay concept is poorly suited for the realities of software development [5]. If the test team only uses a GUI test automation tool in capture-replay mode, they have little protection from “test death;” i.e., the next major revision of the user interface will “kill” many of the prerecorded tests [6]. In our case, the capture-replay technique was not a match for the need to record tests on both GUIs, the large number of test cases, and the differences between the GUIs. Recording a potentially large number of test cases would take much time, and recorded test cases are rather unreadable. Because the capture-replay approach would not help us meet our goals, we avoided it. In contrast, a library can provide facilities for code portability and reuse, at the same time yielding test scripts that are readable and maintainable. These benefits are achieved because a library can offer data and access abstractions between the test tool and the system under test. We chose to develop a library as the most promising approach to achieving our goals. Having decided upon a library, we faced several design and implementation tradeoffs. A point often brought up in the testing community is that testers are not developers. As such, they may be uncomfortable developing test scripts; i.e., writing code. We sought to minimize product implementation details that are irrelevant from a testing point of view. This issue brought about the following considerations that we had to address: • Accessing fields in opaque objects: Testers should not be concerned with the aspects in which the test automation tool lags slightly behind the software development technology used in the application. It is common to have custom objects on the GUI that the test tool cannot “see into” (i.e., does not know how to manipulate). We had to dig rather deeply into the test tool details to provide a common abstraction to the testers in this case (also see Section 5.4).

§ PowerBuilder

is a trademark of Sybase, Inc.



4


• Exposing window construction hierarchy: Simplicity vs. complexity for a tester often creates an opposite tradeoff for the test library developer. Initially, we tried to simplify the use of the library by providing a separate method for each (object, action) pair on the GUI. In this approach, a tester need not be concerned where in the window construction hierarchy the given object resides. For example, to perform a “set value” action in edit text field TelephoneNumber on window SubscriptionData, a tester would simply issue this method invocation: SubscriptionData.setValueTelephoneNumber(‘300-100-1234’) Given a huge number of objects (∼720 fields etc.) and desired actions (e.g., set or get value, set invalid value, check enabled status, set focus, etc.), this approach proved to be practically infeasible to implement. As a result, we had to expose almost all objects and their place in the window construction hierarchy to the tester (also see Section 5.5). • Data abstraction: The values shown in the fields on the GUI are often different from the ones stored internally in the database (e.g., the database uses short codes for long string values in the drop-down windows/boxes) or the ones read by the test tool’s native methods (e.g., formats of date/time values differ). In some cases, these methods do not return meaningful values (e.g., when a check box is cleared, the native method returns garbage). Ideally, a tester should use meaningful values to fill out the fields and to retrieve with the library’s “get value” method, regardless of the field contents such as NULL or cleared (also see Section 5.6). • Providing common methods: Different types of GUI objects (text edit fields, drop-down list boxes, check boxes, etc.) offer different methods for dealing with setting and getting their values. We chose to hide these idiosyncrasies from testers by providing common methods such as “get” (to fetch the current value) for every object. As we later found out, in some cases type-specific methods were also needed (also see Sections 5.6 and 5.7). • Handling of dialog boxes: Another problem related to test portability is the handling of dialog boxes. Dialog boxes typically appear with messages such as Error, Warning, Status, Confirm, Information, etc., and must be handled as part of the normal flow of a test script. The problem here was that dialog boxes of the same type were sometimes placed by the developers in different places in the window hierarchy. Unless we wanted to test a dialog box’s placement in the window hierarchy, we needed to hide this fact from the testers and provide common, portable methods for detecting, verifying, and dismissing dialog boxes (also see Section 5.8). • Testing data-dependent system: Last but not least, the System Under Test (SUT) contained large amounts of highly structured data in several local and remote databases. We found that this feature made automated testing (as a matter of fact, any type of testing) of business flows difficult. To properly set up the test data in the databases required a significant amount of domain knowledge, and even with this knowledge it was a time-consuming activity. The dependence of test scripts on data in this system was much higher than in some other testing projects in which we participated [7, 8] (also see Section 5.9). Consider testing a web application with a back-end database that contains simple, static data. Such an application may be, for example, a set of web-based calculators for performing statistical data analysis. A user typically inputs data in the appropriate database form, and the server side computes results based on the input data and static tables with test parameters. A test case prerequisite is simple, and based almost exclusively on input data. Although there may be many tables in the server database, they are unrelated and thus data are easy to set up.




5

In the case of the number portability OSS, the databases reflected the complexity of the system. Most test cases had several data-dependent prerequisites. For example, to execute a test case requesting to port a telephone number, the prerequisites may involve verifying (1) the old service provider as the current owner of the NPA-NXX ¶ , (2) the existence of other telephone numbers already ported for this NPA-NXX, (3) validity of the new service provider’s Location Routing Number, (4) the correct due dates for the request, and (5) the existence of other subscription versions of this telephone number.

3.

SYSTEM OVERVIEW

In this section, we briefly present the system under test (SUT) and the test automation tool. The SUT is a service management system for local number portability [1]; the test automation tool is SilkTest [2] from Segue Software. 3.1.

System Under Test

Figure 1(a) presents an overview of the system under test, and depicts its architecture. The system’s functionality is divided into Service Order Activation (SOA) processing on the service management end of the system for handling service requests, and Local Service Management System (LSMS) processing on the network management end of the system to manage subscriptions in the local network when downloaded from the NPAC SMS (Number Portability Administration Center SMS). Local Number Portability (LNP) functions coordinate updates from NPAC SMS to multiple Network Element Managers (NEMs). The NEM is the piece of equipment that assists with processing real phone calls; all others are back-office functionality. Version 4 of the OSS offers two functionally equivalent GUIs: a PowerBuilder client and a Java applet. Figure 1(b) shows the testing environment. The automated testing described in this paper uses SilkTest scripts as the drivers of the OSS GUIs. These GUIs supply data for issuing requests and queries to the local SOA/LSMS database. Both NPAC and NEM are simulated in the test system, with simulators responding to SOA/LSMS requests and audits. Although it may be technically feasible to drive both NEM and NPAC simulators GUIs by SilkTest (as shown by dashed lines), our tests focused on the OSS in both initiating the test sequences (i.e., applying inputs and data to OSS) and verifying the test results (e.g., “a subscription version with the status of Pending created in the NPAC simulator”). In particular, the developed test cases are self-checking in that they verify their results by doing local or remote queries to the SOA/LSMS database or the NPAC simulator, respectively. The results of these queries are then automatically checked on the OSS GUI against the expected ones by the appropriate SilkTest script. In this way, back-end flow testing uses the GUI for supplying test input data and retrieving test output data from the SUT.

¶ NPA-NXX

is a three-digit Numbering Plan Area followed by a three-digit exchange.



6


(a)

(b) PB GUI

PB GUI

NPAC

SOA LSMS IBM or HP Web GUI

Simulator NPAC

SilkTest

SOA LSMS

Autom. library

IBM or HP

Web GUI

NEM

NEM Simulator

Figure 1. Overview of System Under Test (a) and the testing environment (b).

3.2.

SilkTest

To meet our goals, we required a test tool that would recognize the product’s GUI controls and be able to manipulate them effectively; allow scripting of data-dependent via a programmable language; and have built-in error logging and recovery to return an application to a base state and continue with the next test case [3]. A test tool that satisfies these requirements is SilkTest, which is a widely used testing product for e-business applications. It can perform regression and functional testing for a web browser, Java applet, or traditional client/server application. SilkTest also offers test planning and management, direct database access and validation, a record/playback feature, a flexible and robust object-based scripting language, a built-in recovery system for unattended testing, and an ability to test across multiple platforms, browsers and technologies. In SilkTest, Tests are written in the 4Test k scripting language (4GL). The 4Test language includes a set of statements, operators, and data types that add structure and logic to recorded test cases, and also provides a library of built-in functions for performing common support tasks. We made extensive use of SilkTest scripting language—essentially extending the tool using its own API. We call the extensions our “test automation library,” which is discussed in Section 5.

k 4Test

is a trademark of Segue Software, Inc.




7

4. RELATED WORK Several reports on practical experiences with test automation either explicitly adopt or discuss benefits of a test automation library [3, 4, 6, 9, 10, 11, 12] This paper presents significantly more details on applying the idea of a test automation library to a real-life system. A common approach to test automation may include the following [9]: • give low priority to test automation by developing tests only during the staff’s spare time; • start test automation by recording scripts using a test tool’s capture and replay feature, without designing the proper test library; • add some test parameterization later by factoring out common code; • as time passes and the number of scripts increases, replace some of the redundant code with functions. The test automation infrastructure presented in this paper helps eliminate the above steps, which are known to cause serious problems in test automation efforts [9]. A common problem with not having a standard for test automation (defined for example by a test automation library) is maintaining multiple copies of test automation software by the members of a test team [12]. It is recommended in the literature that each project have just one library, and that there be no duplication of code or functionality in the library. If a tester needs some new feature for generating random numbers, either an existing function in the library should be modified in a way that will not break any script currently using it [12]. Marick reports that test libraries filter out irrelevant information [6]. They allow tests to specify only and exactly the data that matters: on input, they add additional information required by the intervening code; on output, they condense all the information from the intervening code down to the important nugget of information actually produced by the code under test. Many user interface changes will require no changes to tests, only to the test library. Since there is (presumably) a lot more code affected by these changes in the test code than the library code, the cost of change is dramatically lowered. The process of combining the capture-replay feature with test scripts using a library may be enhanced further. One approach equips the test harness with additional modules for facilitating regression testing [11]. A test data retriever separates test data and output data in the recorded script and the captured screen output from the template script. A data assigner then creates new test scripts by associating thus retrieved test data with the template script.

5. OVERVIEW OF THE TEST AUTOMATION LIBRARY As mentioned in Section 2, we chose to develop a library, thereby providing an abstraction layer between the tool and the tested application. This library allows rapid development of automated test cases, which are implemented by a test script using the SilkTest 4Test language. When the library methods are used, a test script manipulates the GUI objects by calling methods on them.



8

5.1.


Capture of System Under Test Features

SilkTest requires a definition of the objects on a GUI and the methods on those objects before test automation can begin. In brief, a SilkTest object corresponds to a GUI feature. Each SilkTest object has information, for example a window title, that allows SilkTest to match its own object with the corresponding GUI feature at run time. Methods are defined within SilkTest objects to manipulate GUI features. For example, a method might select a menu item, insert text into a field, or fetch the value of a text field. SilkTest can generate a rudimentary version of object definitions and methods as the first step of automating tests. The library that we developed is a sophisticated version of this information, including user-friendly object names, uniform method definitions, and other extensions to enhance functionality and improve test maintainability. As mentioned before, the main design principle followed in construction of the library was ease of use by testers. Our library (in contrast to an automatically generated version) hides from the testers many of the low-level details required for successful GUI manipulation. For example, a library method that allows a tester to place the value “1” in a field (e.g., setValue(1)) hides details of setting the input focus to that field, typing the value, and causing the value to be accepted and validated.

5.2.

Reusability

The library code is roughly divided into two parts: application-independent class and method definitions, and application-specific objects and methods. The first part identifies common building blocks that appear on the GUIs (e.g., text fields, data windows, check boxes), and defines appropriate classes and methods for manipulating these building blocks. For example, check box objects might be defined by a “JDataWindowControlCheckBox” class that includes a set of methods for setting or getting a value, checking if the check box is enabled, etc. The second part uses these classes to define specific objects on the 40 GUI windows for each GUI implementation. For example, a check-box called “AuthorizePort” is an instance of the class “JDataWindowControlCheckBox” and is defined on a window called “SOASubscriptionData.” If we were to develop a similar library for another application, then the first part should be reusable to the extent that this application’s GUI consists of the same types of building blocks (GUI objects). The type of a GUI building block is determined by the block’s basic type (e.g., drop-down list, edit field, check box) and the GUI development tool used to build this block. For example, to test an application that uses a drop-down list component created with the help of PowerBuilder or PowerJ 3.0∗∗ , the testers could directly reuse the corresponding library section. The above analysis suggests that up to 30% of the library could be reused. Time-wise, these numbers are even better because developing common classes and methods is a time-

∗∗ PowerJ

is a trademark of Sybase, Inc.




9

Table I. Hand-crafted vs. recorded test case. Hand-crafted test case (using a library) SOASelection.init () SOASelection.SearchDW.FromTN.SetValue ("300-100-0000") SOASelection.runQuery () Recorded test case (no library) MainWin("OSS Desktop|$w lnpsms frame").Menu("Sub Admin|#2").MenuItem ("SOA Selection|#1|$10036").Pick () MainWin("OSS Desktop|$w lnpsms frame").ChildWin("Subscription Administration SOA Selection|$w lnpsms subs selection").DataWin("#1|$dw search").DWEditField ("From TN:|#2|$from tn").SetValue ("300-100-0000") MainWin("OSSDesktop|$w lnpsms frame").Menu("Subscription|#2").MenuItem ("Query |#1|$10083").Pick ()

consuming effort, whereas writing the code that utilizes them (i.e., defining GUI windows) is a more routine task. Reuse is also promoted by associating standard functions with a window class rather than with each specific instance of the class. In this way, generic test functions can be used by any script testing any application of the product [3]. The application-specific part of the library contains functions specific to application windows. 5.3.

Maintainability and portability

The library provides an abstraction layer to enhance portability of test script actions across the two GUIs. As mentioned in Section 2, an alternative to developing a library between a generic test tool (e.g., SilkTest) and a specific application (e.g., OSS for local number portability) is to use a capture/replay technique. The former approach, which offers data and access abstractions for the GUI objects, allows testers to develop hand-crafted test scripts that are highly readable. The scripts consist of object declarations and calls of predefined methods. Low-level details, such as absolute pixel locations and internal tags for tracking GUI windows that make test scripts unmaintainable and often unreadable, are not required. Consider a test case that first selects SOASelection item from SubAdmin menu on the main window, inserts data into FromTN field the on SearchDW data window, and then runs a database query. Table I shows these actions in two versions: the first one uses the test library, and the second one is recorded by the SilkTest capture feature. The first test case is both portable and readable, whereas the second one clearly does not have these advantages. Accessing the fields on the GUIs is straightforward using the test library, and since 99% of the fields are the same on both interfaces, portable field access is naturally provided for transitioning from the legacy GUI to the new Web GUI. However, tasks like opening a window, running a query, or closing a window are done differently on the two GUIs. Methods are provided in the test library to accomplish these tasks, and should always be used to enhance the portability of a test case. A frequently seen example is running a query and coping with



10


the cases of no result or too much result (too many rows); both cases cause dialog boxes to appear. The test library allows a test script to dismiss the dialog boxes easily without being specialized for either GUI. The test library is robust to a large class of common GUI changes, such as moving an item up or down on a menu, moving an input control like a text field around on the screen, or changing the shape of an input control. Other changes to the GUI, such as adding or deleting an input control, require library changes. For maximum benefit, the test library must evolve in parallel with the application. This investment is required to attain the benefits of portable, maintainable test cases. 5.4.

Methods for opaque objects

We designed and implemented a technique to hide from a library user some specific aspects in which the test automation tool lags behind the software development technology used in the application [13]. In this project, SilkTest was not (yet) able to identify and track certain objects embedded in the Java GUI. The Java applet contained a custom JavaBean data window whose fields were not directly visible to the test tool in the usual way (such as the PasswordLifetime field on the DataDW data window shown in Figure 2). To allow Web test scripts the same easy access that was possible on the PowerBuilder interface, we declared a class that represents the apparently hidden field objects, and defined methods in that class to access those objects. The JavaBean provides an API that we used to implement a set of methods on the invisible widgets. We do not present the full solution here, because it is rather detailed and tool-specific. Sample library code implementing the above ideas is presented in the Appendix, and a full discussion can be found in [13]. 5.5.

Exposing window construction hierarchy

Unfortunately for the GUI tester, writing a test script frequently requires considerable knowledge about how the system under test is built. Modern GUIs are built from small components, which means that a GUI screen consists of a set of input controls, each of which is a full-fledged object with methods and properties. A person looking at the GUI on a computer screen sees only a collection of pixels and such, but a test tool sees the individual components. Because a test script must reference the GUI components appropriately, a common problem we encountered was the need to discover the construction hierarchy that results from parent-child and sibling relationships among window elements. Consider performing a “set value” action in edit text field “PasswordLifetime” on the window “PasswordParameters” that is shown in Figure 2. To access this field, the test tool must know that the text field is a child of the “DataDW” data window, which itself is a child of the “PasswordParameters” window. To allow a tester to set a value, a library might provide the method†† “PasswordParameters.DataDW.PasswordLifetime.setValue(‘90’).”

†† Convention

A.B(c) means calling method B with input parameter c on object A.




Parent window: PasswordParameters

11

PasswordParameters

DataDW

Password Lifetime

Password Expiration

Maximum Login Attempts Child data window: DataDW

Grandchild edit field: PasswordLifetime not visible to SilkTest!

Figure 2. Example window construction hierarchy.

This issue leads to a central design decision: should the library attempt to conceal all details about window-construction relationships or should it expose all of them to the user? Continuing the example, alternately we might have implemented library functionality to offer a simpler interface such as “PasswordParameters.setValuePasswordLifetime(‘90’)” for accessing the “Lifetime” field on the Password Parameters window. This approach would hide the hierarchy from the tester, but at a considerable cost. Alternately, we could have chosen to hide some details about window construction relationships but expose others. For example, we might have hidden the data window object and allowed something like “PasswordParameters.PasswordLifetime.setValue(‘90’).” The presence of windows with identical data window objects argued against this approach. On these windows, the same field (e.g., “Telephone Number”) occurred in two separate data windows, and the only way to reference them unambiguously was to reference the parent data window. Hiding certain data-window objects might have achieved local improvements in usability of the library, but only at the cost of added complexity and of having a tester frequently be uncertain about whether to reference a data window or not. Further, recall from Section 2 that we found it infeasible to implement a set of several methods for each GUI field. Because we relied on a core set of methods for objects, a tester needed to know how to find each object in order to invoke those methods. Ultimately, this choice made it necessary for a tester to know the exact place of each GUI object (other than dialog boxes) in the window construction hierarchy. Happily for us library developers, in the



12


vast majority of cases, the window construction hierarchy was identical in the two GUIs under test. Although the decision to expose the window hierarchy details resulted in slightly increased complexity as seen by the tester, we were able to dramatically cut down on the size of the library and the time needed to develop the library. The common methods that we needed for each editable field are as follows: setValue, getValue, setValueInvalid, isEnabled, hasFocus, and setFocus. Instead of providing a separate method for each of the ∼720 fields, it was sufficient to implement these methods for each object type: text edit fields, drop-down data windows, drop-down list boxes, check boxes, etc. An added benefit to the testers is that they do not have to worry about any internal mechanisms these different objects use for setting and getting their values. 5.6.

Abstraction of data

In addition to the functional abstractions of Section 5.3, the library provides a certain data abstraction for test scripts. Put simply, every piece of data is a string. One of the goals was to provide testers with a consistent way of setting and getting a field value, regardless of its type and format. The common methods described in the previous section achieve this purpose, since they accept and return all values as strings visible on the GUI or their reasonable equivalent (e.g., string “Y” for a marked check box). In the cases where the values shown in the fields are different from the ones stored internally in the database, the library methods provide the necessary translation. The cases where the test tool’s native methods do not return meaningful string values are detected, and an appropriate value is returned. Thus input data can be put into any field as a simple string, e.g., the calls to setValue(‘Y’) and setValue(‘N’) check and clear a check box, respectively. The corresponding getValue method always returns “Y” or “N.” Special attention was needed for the date/time fields. The classes of these objects were extended to understand both the displayed format and the internal format. The corresponding access methods provide formatting functionality that enhances ease of use by the tester. For example, the tester can also request the data to be retrieved in a format other than the format visible on the GUI. This sometimes proved convenient for the purpose of date/time comparison and sorting in test cases (e.g., when simple string comparison is used). 5.7.

Object-specific methods

The set of common methods mentioned previously turned out to be insufficient for GUI edit controls such as drop-down data windows, list boxes, and column fields in a results area with multiple rows (many instances of a data item, see Figure 3). We encountered special problems when trying to verify results of various actions by reading the contents of the various controls. • Drop-down data window/list box: There is often a need to loop through the values selectable from the window/box or to verify the contents of the window/box. Therefore, we had to provide some type-specific methods such as selecting next/previous value that can be used for these purposes.




13

Figure 3. Selecting rows in a result area.

• Column fields: These objects contain the results of database queries, and therefore are heavily used for verifying the results of test cases. For this purpose, methods were needed for finding a row/selecting all rows with the desired value, and extracting a list of row values. The latter method returns a list of string values in a format appropriate for sorting. For the query-result windows that contained lists of records, methods were required for selecting or de-selecting specific rows or a range of rows, and checking whether specific rows or a range of rows are selected or de-selected. In addition, we found it useful to provide methods for navigating through retrieved records: “go to the previous/next/first/last record” and “scroll to a given record” so that it would become visible on the GUI (the details of navigation, such as using Previous/Next push buttons or a vertical scrollbar, were hidden from a tester). The primary need for these methods was to verify the results of test cases. For example, a tester may want to verify that a test case supposed to select rows 3 and 6–8 on the GUI actually highlighted the appropriate rows (as in Figure 3). 5.8.

Handling of dialog boxes

Since dialog boxes appear as separate windows, a desirable feature for a tester is to manipulate the dialog boxes using their titles (e.g., ErrorDialogBox.dismiss() or WarningDialogBox.accept()), without consideration for their place in the window hierarchy. By window hierarchy we mean whether two windows are implemented as peers or as a a parentchild relationship. We were not able to hide this implementation detail completely; however, we managed to reduce the problem to just two different classes of dialog boxes. In general, dialog boxes may be children of the Microsoft Windows‡‡ root or any other window currently open. By exploiting certain tagging features of the test tool, we provided common methods for detecting, accepting, and dismissing dialog boxes on two levels: root and application. Thus, a tester needs to know only if a dialog box is a child of the root or of any

‡‡ Windows

is a trademark of Microsoft Corporation.



14


application window. For example, to detect an information dialog box, a tester calls either RootInformationDialogBox.Exists() or InformationDialogBox.Exists(). Appropriate declarations were also added so that a test script could be completely portable; i.e., a dialog box might not ever appear on the GUI under test, but checking for it could be done with no ill consequences. Multiple instances of the same dialog box at the same time could be handled via mechanisms provided by the test tool; no special extensions were necessary. Example: Portable dialog box This example includes the source code (Figure 4) of two test cases that use the test automation library. The test cases check identical GUI features on both the Powerbuilder (pbgui r40 p340 ngn fh) and Web (webgui r40 p340 ngn ah) interface. As can be seen from the sample code, both test cases consist of just two lines of code: the first one sets up the value of the test case body parameter; the second one is a call to the test case main body. Thanks to the test automation library, the test case body is entirely portable between the two interfaces. The test case body iterates over the edit boxes for setting the application’s parameters, and, for the min/max border values, performs the following actions: (1) get valid value, (2) set valid value, (3) set invalid value, (4) restore valid value. The valid values should be accepted by the GUI; entering invalid values should cause an error dialog box to appear. The test case body achieves portability by accepting the name of this error dialog box as an input parameter. However, the Web interface uses the dialog box that is a child of MS Windows root, whereas the Powerbuilder GUI displays the dialog box that is a child of the current window. This problem is overcome by the special features of our test automation library—test scripts can reference any dialog box by the name only, regardless of its place in the window hierarchy. 5.9.

Coping with data-dependent systems

We use the term “data-dependent” for systems that involve frequent database access. When testing data-dependent systems, data becomes as important in a test suite as the sequence of actions. As said before, the SUT is essentially a big database—test cases beyond sanity tests require use of valid data and therefore are highly sensitive to the database contents. When the database is created manually on a periodic basis, each run must be preceded by setting up a proper data set. This gets more difficult when (1) a typical test case involves many prerequisites and actions, and (2) producing test data requires deep domain knowledge. The testers, who often have other workload demands, are effectively prevented from running the automated regression tests without adjusting the data. Although we did not arrive at the general solution to this problem, we experimented with several approaches to cope with this difficulty, as follows: • Check test-case prerequisites using the automation library to prevent spurious test failures. • Isolate test data to the top few lines of a test script where they can be easily found (and adjusted as needed) by the tester. • Build test cases that choose their data at run time; e.g., select a valid telephone number from the currently loaded set, then use it as needed.




15

// Powerbuilder GUI test case testcase pbgui r40 p 340 ngn fh () { STRING sErrorDialogBox = ”ErrorDialogBox”; gui r40 p 340 ngn ah body (sErrorDialogBox); } // Web GUI test case testcase webgui r40 p 340 ngn ah () { STRING sErrorDialogBox = ”RootDataWindowErrorDialogBox”; gui r40 p 340 ngn ah body (sErrorDialogBox); } // Shared code void gui r40 p 340 ngn ah body (STRING sErrorDialogBox) { // Field Description of Application Parameters AppParameters.init (); INT iWaitForDialog = 5, iMinValue = 1, iMaxValue = 60; STRING sDefaultValue, sParameterName; LIST OF STRING lsParameterName = { ”SOANotification”, ”SOAError”, ”LSMSTLog”, ”LSMSError” }; // need to clear error dialog boxes for invalid value testing for each sParameterName in lsParameterName { sDefaultValue = AppParameters.RetentionDW.@(sParameterName).getValue (); AppParameters.RetentionDW.@(sParameterName).setValue (Str (iMinValue)); AppParameters.RetentionDW.@(sParameterName).setValueInvalid (Str (iMinValue-1)); Verify (@(sErrorDialogBox).Exists (iWaitForDialog), true, ”(no dialog box in response to invalid value)”); @(sErrorDialogBox).accept (); AppParameters.RetentionDW.@(sParameterName).restoreValueValid (Str (iMinValue)); AppParameters.RetentionDW.@(sParameterName).setValue (Str (iMaxValue)); AppParameters.RetentionDW.@(sParameterName).setValueInvalid (Str (iMaxValue+1)); Verify (@(sErrorDialogBox).Exists (iWaitForDialog), true, ”(no dialog box in response to invalid value)”); @(sErrorDialogBox).accept (); AppParameters.RetentionDW.@(sParameterName).restoreValueValid (Str (iMinValue)); AppParameters.RetentionDW.@(sParameterName).setValue (sDefaultValue); } }

Figure 4. Sample code implementing a portable dialog box.



16


• Create necessary data entries as part of a test case (establish precondition), then clean up at the end of a test case. This sort of functionality can be placed in a subroutine called by every test case in a suite to prevent each case from becoming bloated. • Develop methods and procedures to ensure that a test data set is always loaded and ready for testing, and promptly restored after a new software release is installed.

6.

QUALITATIVE IMPACT

When this project began, the OSS testing project had limited use of automation. The introduction of the test automation library and methodology was expected to develop staff competency and bring a substantial return on investment. Once GUI automation was in place, automation of other types of testing such as back-end could proceed. Automated test cases also contain a significant amount of domain knowledge, which can be viewed as their marketable value. Automation of tests allowed testers to find more faults on their existing (tight) schedules. Helping testers run complex tests, especially in unfamiliar areas, freed up staff time for developing better tests. According to Hayes [14], the above benefits outweigh questionable gains from reducing either test resources or test cycle time, since the goal of test automation should be to decrease the risk and cost of software failure by increasing test coverage and quality. 6.1.

Practical results

We demonstrated the benefits of the library and test automation by developing and delivering a wide variety of tests. The advantages are clear for in-house data-independent regression tests that are run after each software delivery. As part of the regression test suite, we developed and delivered a set of so-called “sanity” cases to check windows and fields (4 per window, for total of 160 per GUI). Sanity test cases are the ones applied to a new release, offering quick, broad, and shallow testing. These cases are run often by the testers, and frequently discover major problems immediately after a new release is made available. An advantage of using the library is the ability to make field tests robust to changes in the field status (grayed-out, read-only, read/write). Since the navigation between fields employs direct calls to field objects (instead of, e.g., utilizing the “TAB” key), if read/write status is changed then the tests will not require modification. We also demonstrated the ability to automate a wide range of tests by delivering a set of 75 cases, including in-house GUI usability and business flow tests. The GUI usability tests included input data validation, single window navigation (e.g., TAB order), display and dismissal of proper message boxes, autocompletion feature for edit fields, autopopulation/enabling/disabling of certain fields based on the data entered in other fields, prepopulation of fields on a window launched from another window, presence of newly required GUI elements, basic back-end data retrieval and display, etc. In addition, a subset of about 50 test cases from NPAC 2.0 turn-up and regression test plan [15] exercising common business flows were automated. The entire set of turn-up tests




17

takes up a few hundred pages in the NPAC-released document. These exhaustive tests are highly data-dependent and time-consuming to run manually. They have great potential business value for the company, since customers (service providers) are required to run the tests against the third party (NPAC). Exhaustive tests typically explore many variations of the same behavior—thus lend themselves naturally to automation. For example, in an effort to automate about 45 turnup tests, we achieved about 90% reduction in test size after automation by viewing test cases as different data-dependent execution paths of a relatively small test script. The time needed to automate tests in this manner also dropped dramatically after the common behavior was identified and coded (first 15 tests/10 days; remaining 30 tests/2 days). One of the problems that we faced during the technology transfer, besides testers’ tight schedules, were difficulties with a stable system configuration. Thus time gained through automatic test execution is frequently offset by the time needed to configure/set up the testing environment. One more practical result was that we found problems in the system under test during development and trial runs of the test automation library. We wrote about 35 software Modification Requests (MRs) to address these issues. 6.2.

Achieving a positive return on investment

The development effort for automated testing is worthwhile only if a positive return on investment can be expected. As we mentioned in the last section, we achieved some immediate benefits from automating a suite of sanity tests. Kaner [10] confirms this result, similarly claiming a big payoff in automating a suite of acceptance-into-testing (also called “smoke”) tests. A tester might run these 50 or 100 times during development of a release. Even if it takes 10 times as long to develop each test as to execute each test by hand, and another 10 times cost for maintenance, this still creates a time saving equivalent to 30-80 manual executions of each test case. Similarly, Hayes [14] reports 80-90% drop in post-production incidents thanks to “sanity check” that ran 150 test cases every day in production to check for problems with the infrastructure, data tables, interfaces, and other potential sources of failure. If the product is being tested through a GUI (graphical user interface), and the adopted automation style is to write scripts (essentially simple programs) that drive the GUI, an automated test may be several times as expensive as a manual test [6]. As reported in Ref. [10], it usually takes between 3 and 10 times as long (and can take much longer) to create, verify, and minimally document the automated test as it takes to create and run the test once by hand. Our experience shows that automation of first few test cases is expensive; beyond that they become much cheaper to develop. By exploiting the commonalities, automated test cases can be made concise, and therefore easier to run and maintain. Some of the rules of thumb applied when a decision on test automation is made are as follows [3]: automate regression tests; automate tests for stable applications; automate tests that are not time-dependent: do not automate tests with complex timing issues; automate repetitive tests; automate detailed, already designed tests; limit your scope: don’t try to automate everything. Many tests will be worth automating, but for all the tests that are run only once or twice, this approach is inefficient.



18


Based on our experience, the real benefits of test automation are most visible when: • Tests are often rerun for regression. • There exists a subset of test cases independent of stored data—these tests are much easier to automate and run on a regular basis than the data-dependent ones. • An already developed paper-based test suite is large and its manual execution requires many tedious, repetitive tasks to perform—automation eliminates human errors due to the monotony of manual testing. • Testing must be performed on multiple platforms through various front-end interfaces, and it is possible to develop a common API for coding tests. • Complex tests, once automated, can be run by testers in unfamiliar areas, thereby freeing up staff time for developing better tests. • Commonality among test cases allows test automation efforts to be amortized across many tests. Our recommendations for maximizing long-term return on the investment that was provided to the test team are summarized as follows: (1) run a set of sanity test cases after every software delivery; (2) establish a stable database data set for automated testing; (3) tailor the automated cases for the test data set; (4) facilitate configuration of the system for testing (ideally, a tester should be able to set up a test environment quickly when needed); and (5) view automated test suites as a marketable product. The last point is worth a closer look. Manual execution of a test case requires a significant amount of application knowledge, and so does the process of its automation. However, in the process of automation, this type of domain knowledge becomes permanently embedded in the test case. As a result, an automated test case might be viewed as a product that could be marketed to customers.

7.

CONCLUSION

Test automation requires a mix of expertise: the ability to use test tools, some software design and development skills, and a knowledge of the system under test. Testers today have the first and the last skills, but might not be ready to dedicate the effort required to design and write effective, maintainable test scripts. Testers should be provided with a means to perform an effective test campaign without a need to learn these extra skills, which are essentially beyond the core of their expertise and technical interests. Therefore, we believe that an applicationspecific test infrastructure, such as the one described in this paper, can significantly reduce the extra effort that test automation requires from testers. Test infrastructure can be realized through test automation facilities such as common data and access abstractions for different types of GUI objects, methods to transparently manipulate custom GUI widgets, and stable test data for automated testing. The investment in developing these facilities leads to test scripts that are reusable, maintainable, and portable. The same testing staff can then improve customer satisfaction by detecting more defects than it is possible through manual testing.

REFERENCES




19

1. Telcordia Technologies, Inc., Piscataway, NJ, TelcordiaTM Operations Support Systems. (http://www.telcordia.com/products services/oss). 2. Segue Software, Inc., Lexington, MA, SilkTest User’s Manual, 2000. 3. B. Boehmer and B. Patterson, “Software test automation: Planning and infrastructure for success,” in StarEast’01 [16]. 4. S. K. Allott, “Automate your tests—you won’t regress it!,” in Proc. PNSQC: Pacific N.W. Softw. Qual. Conf., (Portland, OR), pp. 132–154, Oct. 1999. 5. B. Pettichord, “Alternatives to capture replay,” in Proc. SQE STAR: Int’l Conf. Softw. Test. Anal. Rev., (San Jose, CA), Oct. 2000. 6. B. Marick, “When should a test be automated?,” in Proc. QW: Int’l Softw. Qual. Week, (San Francisco, CA), May 1998. 7. S. R. Dalal, A. Jain, N. Karunanithi, J. M. Leaton, and C. M. Lott, “Model-based testing of a highly programmable system,” in Proc. IEEE ISSRE: Int’l Symp. Softw. Reliab. Eng., (Paderborn, Germany), Nov. 1998. 8. M. A. Fecko, M. U. Uyar, P. D. Amer, A. S. Sethi, T. J. Dzik, R. Menell, and M. McMahon, “A success story of formal description techniques: Estelle specification and test generation for MIL-STD 188-220,” in FDTs in Practice (R. Lai, ed.), vol. 23(12) of (Elsevier) Comput. Commun., pp. 1196–1213, July 2000. 9. D. Johnson, “Designing an automated Web test environment,” in StarEast’01 [16]. 10. C. Kaner, “Improving the maintainability of automated test suites,” in Proc. QW: Int’l Softw. Qual. Week, (San Francisco, CA), 1997. 11. H. Suganuma, K. Nakamura, and T. Syomura, “Test operation-driven approach on building regression testing environment,” in Proc. COMPSAC: Int’l Computer Softw. Appl. Conf., (Chicago, IL), Oct. 2001. 12. B. Tervo, “Standards for test automation—a case study,” in StarEast’01 [16]. 13. C. M. Lott, “Automating tests on a PowerJ Web applet with SilkTest,” (Segue Software, Inc.) Segue@Work Newsletter, Oct. 2000. (http://www.segue.com, http://www.cs.umd.edu/∼cml). 14. L. Hayes, “The business case for test automation,” in Proc. SQE STA: Softw. Test Automat. Conf. Expo, (Boston, MA), Sept. 2002. 15. NeuStar, Inc., Chicago, IL, NPAC SMS: Service Provider Certification and Regression Test Plan Matrix and Test Cases, rel. 2.0, r3.0.1 ed., July 2000. 16. Proc. SQE STAR: Int’l Conf. Softw. Test. Anal. Rev., (Orlando, FL), May 2001.

APPENDIX A. SAMPLE LIBRARY CODE The following code sample includes a portion of the library for the Java applet, including class definitions, an object definition for the window shown in Figure 2, and a test case definition. // Winclass for DataWindow objects (containers). winclass PowersoftDatawindowJDataWindowControl : AnyWin { tag "[powersoft.datawindow.JDataWindowControl]"; // Omits the many properties and methods that were recorded by // SilkTest. // This is the winclass for objects that SilkTest cannot see directly. winclass JDataWindowControlEditField { STRING sDataWindowTag; STRING sColumnName; STRING sColumnDescriptor; void init () {



20


sColumnDescriptor = ""; STRING sTag = WindowTag (this); // Search from the right-hand side for a descriptor. // Note that there might be slashes in it. // Remove and save the descriptor if present. INT iPositionOfLastPlusSign = StrPos ("+", sTag, TRUE); if (iPositionOfLastPlusSign > 0) { sColumnDescriptor = SubStr (sTag, iPositionOfLastPlusSign + 1); sTag = SubStr (sTag, 1, iPositionOfLastPlusSign - 1); } // Find the last slash - delimits the last component INT iPositionOfLastSlash = StrPos ("/", sTag, TRUE); if (iPositionOfLastSlash < 1) { raise 1, "No slash in tag string >{sTag}

Lessons learned from automating tests for an ... - Semantic Scholar

Lessons learned from automating tests for an ... - Semantic Scholar

Suggest Documents

Learning from Lessons Learned - Semantic Scholar

Lessons Learned from Transferring Software ... - Semantic Scholar

Lessons Learned from Aligning two ... - Semantic Scholar

Lessons Learned from Hurricane Katrina - Semantic Scholar

Article Lessons Learned from Undergraduate ... - Semantic Scholar

Lessons Learned from the Numerical ... - Semantic Scholar

Lessons learned from England's national ... - Semantic Scholar

An Intelligent Lessons Learned Process - Semantic Scholar

Blast injury: lessons learned from an autopsy - Semantic Scholar

Ten lessons learned from conducting an ... - Semantic Scholar

Lessons Learned from an Extensive Spectrum ... - Semantic Scholar

Blast injury: lessons learned from an autopsy - Semantic Scholar

Oncoshare: Lessons Learned from Building an ... - Semantic Scholar

microBEnet: Lessons Learned from Building an ... - Semantic Scholar

Lessons Learned from Evaluation

Lessons Learned from an Emergency Bridge ... - CiteSeerX

Barcode Medication Administration: Lessons Learned from an ...

Lessons learned from an ecosystem-based ...

Appreciating Lessons Learned lessons learned

six lessons learned from software maturity criteria - Semantic Scholar

Message in a bottle: lessons learned from ... - Semantic Scholar

Lessons Learned from Development of De ... - Semantic Scholar

Lessons Learned from Modelling the NHL Playoff ... - Semantic Scholar

Session 7D8 LESSONS LEARNED FROM A ... - Semantic Scholar