A Practical Approach for the Regression Testing of IP-based Applications Tiziana Margaria1,2, Oliver Niese1, and Bernhard Steffen1 1 Chair of Programming Systems, University of Dortmund, Germany {Tiziana.Margaria, Oliver.Niese, Steffen}@cs.uni-dortmund.de 2 METAFrame Technologies GmbH, Dortmund, Germany
[email protected]
Abstract. Modern IP-based applications are multitiered, distributed applications that typically run on heterogeneous platforms. Their correct operation depends increasingly on the interoperability of single software modules, rather than on their intrinsic algorithmic correctness. We present a scalable, integrated test methodology capable of granting adequate coverage of functional testing, yet still easy to use. It bases on our previous work on system level test of Computer Telephony Integrated (CTI) and Web-based applications, where our coordination-based coarse grain approach has been applied very successfully.
1
Introduction
Modern IP-based solutions are very complex multitiered, distributed solutions that typically run on heterogeneous platforms. Their correct operation depends on steadily growing measure on the interoperability of single software modules, rather than on their intrinsic algorithmic correctness. Accordingly, multipoint coordination and distribution issues rather than computation capabilities define the heart of the application challenge: interoperability with the environment at large (including platform, middleware, communication networks, interfaces) and interaction with other applications and with human users constitute the central themes for correctness, reliability, and business criticality. Additionally, the accelerated pace of technology evolution in all the mentioned subsystems endangers the stability of the offered services. This particular profile has a huge impact on the test and validation requirements: it is increasingly unrealistic to restrict the consideration to single units, e.g. the client side only, since complex subsystems affect each other. Thus whenever a subsystem is updated (new hardware, new operating system release, new application server release, ...) a system-wide regression test has to be carried out, in order to check for the correct interplay between the various components. We propose an Integrated Test Environment (ITE) for the test of IP-based solutions: it is capable of addressing the distributed solution in a whole and covers mostly the global functional aspects. Figure 1 shows a typical test architecture for a complex system under test, which is composed of several subsystems that communicate with and affect each other through a wide area network (WAN) or a local area network (LAN). To test this kind of complex systems we add a Test Coordinator to the system under test: it drives the generation,
Test Coordinator
Complex System under Test
LAN/ LAN/ WAN WAN
Figure 1: Test Architecture for the test of complex systems
execution, evaluation and management of the system-level tests in a highly heterogeneous landscape. This introduces the required flexibilization of the overall architecture of the test environment: it leads to a modular and open environment, so that diverse test tools and units under test can be added at need. The Test Coordinator has access to all the involved subsystems and can manage the test execution through a coordination of different, heterogeneous test tools. These test tools, which locally monitor and steer the behavior of the software on the different clients/servers, are technically treated just as additional units under test. In the following we first give a rough overview of the proposed process for the test of complex systems in Section 2. Afterwards we discuss in Section 3 how the test environment supports the user friendly design of test cases. In Section 4 we illustrate the usage of the test environment along two real life examples of IP-based solutions: the test of a web-based application (Conference Service) and the test of an IP-based telephony scenario (Personal Call Manager). Finally in Section 5 we draw some conclusions.
2
Test Process for Complex Systems
Design
Analysis
Integrated Test Environment
Verification
Execution
Figure 2: Process model for the test of complex systems
The iterative test process for complex systems supported by the ITE is roughly presented in Figure 2. The cycle starts with the design of test cases by a test engineer. In the design phase the test engineer defines test cases on a coarse granular level. This means that test cases treat the system from the end user point of view, which implies that test cases can be designed by experts of the overall application domain (e.g. web-based applications or CTI systems) and no expertise about the technical aspects (like the underlying communication protocols) of the system is needed. Key to the user-level test case design are the test blocks [3, 7, 8], which provide a coarse granular access to test stimuli and checkpoints. They form the basis for the graphical design of complex test cases (see e.g. Fehler! Verweisquelle konnte nicht gefunden werden.) and, additionally, establish the basis for user-level verification: consistency conditions, guaranteeing the executability of the test cases or capturing the test purpose are collected in a library and automatically checked via model checking. The library of constraints captures the essence of the expertise of designers and system experts about do’s and don’ts of test case creation, e.g. which test blocks are incompatible, or which can or cannot occur before/after some other test blocks. Such properties are rarely straightforward,
sometimes they are documented as exceptions in thick user manuals, but more often they are not documented at all, and have been discovered at a hard price as bugs of previously developed tests. They are perfect examples of the kind of precious domain-specific knowledge that expert designers accumulate over the years, and which is therefore particular worthwhile to include in the test design environment for successive automatic reuse. The verification process can be done in a push-button fashion and the results, which provide concrete information concerning the mistake and its possible location, guide the test engineer through the redefinition of the test cases. Test cases that pass the verification phase can directly be executed within the ITE in a control flow driven way. For this purpose an execution semantic is defined for each test block, which enables the ITE to steer and observe the corresponding subsystems through the (dedicated instance of their) assigned test tool. The execution engine can then proceed from test block to test block. Here, the return values of the current test block execution determine the next test block. When executing a test case, a detailed report is prepared in form of a (XML based) test protocol. For each test block executed by the execution engine, all relevant data (its execution time, its name, the version of the files associated with the test block, the block’s parameter assignments, and the processed data) are written to the protocol for later analysis and reuse. During the analysis phase the test execution results are investigated. For this purpose the ITE provides a web-service that offers a role and permission based access to the database of the test protocols. Since the test protocols are stored in XML format it is easy to provide a large variety of different views on the protocols, e.g. depending on the level of detail. This enables a test engineer to analyze the results of the test execution of a single test case in detail, whereas a test manager may only be interested to get a quick overview of the overall success of a complete test series. Additionally the web-service provides advanced features for the management and documentation of the related test protocols. The results of the analysis typically lead to a revision of the test cases themselves. This is either because incomplete or faulty test cases are detected or because adaptations to the test cases are necessary as a result of modifications in the system under test. We are now going to discuss the four phases of the process model in detail.
3
User Friendly Design of Test Cases for Complex Systems
The process of testing complex systems, and in particular IP-based applications is complex and error prone. It is therefore extremely important to provide a flexible and powerful yet easy to use formalism for the test case design. It turns out, that at least three key aspects are of central importance to achieve this aim: graphical design, reuse, and guidance. The following three subsections discuss these aspects.
3.1
Graphical Design of Test Cases
System level testing typically treats the system under test from an end user's point of view. This Test should be maintained when moving to an auto Block mated test execution. In fact, not only the test execution, but also the test design should happen at this level of expertise and intuition. In particular, they should not require programming Figure 3: Test Block expertise. The ITE therefore provides the concept of test blocks [3, 7, 8], which can either stimulate the system or validate certain properties of the system state. These test blocks, whose granularity can flexibly be adjusted to the considered scenario, can then be used to compose new test cases in graphical manner. To provide a
fully aspect-oriented test case design it is not prescribed in the ITE which properties of the system state are checked. In order to support reuse, test blocks are hierarchical (cf. Section 3.2) and parametric (cf. Figure 3): besides its (unique) identifier and its name, each test block is characterized through a set of formal parameters and its branching potential. This information is complemented by an informal description, executable code for the test execution and some validation code that checks concrete test block instances for their branching (cf. Figure 4) and their parameters. The integration of a new system under test into the ITE typically starts with the specification and/or identification of the test blocks required for testing the new application. Subsequently, the graphical test case design and the implementation of the executable code for the new test blocks1 can proceed in parallel. The graphical test case design simply proceeds by connecting the icons of the required test block instances through their branches (cf. Figure 4) and by setting the internal parameters. We used two significantly different ways for the test block implementation: • Programming by domain experts, e.g. for testing a CTI application at the protocol level. • Wrapping the results of 'recording and replaying'-tools. E.g. the Rational Robot [9] has been made available in the ITE in a flexible way for driving an application's GUI and to build dedicated test blocks. For the test of web-based applications it turned out that a comparably small set of test blocks, which enables a test engineer e.g. to follow a hyperlink or to validate certain properties of a HTML page, became sufficient. In fact, very soon our test block library became stable and no more test blocks had to be added for integrating a new system under test. The whole process of integrating new systems under test into the ITE, which supports an expertise-driven division of labor, is described in more detail in [3]. In particular, it enables test engineers to set up new test cases very quickly on an intuitive level, which allows us to express the execution semantics of test cases in terms of coarse granular control flow graphs.
3.2
TB1
TB2
Passed
Failed
Figure 4: Test Case
Reuse through Hierarchical Test Case Design
Test cases have recurrent structures, e.g. when testing a web-based application typically a browser must be started during the initialization phase and a user must log on to the system. Hierarchical test case design allows test engineers to specify these tasks as test cases once and for all and to make them available as generic test blocks (or macros) for later reuse. This accelerates the design of new test cases and it supports the maintenance of the test suites: modified macros automatically update all their instances. The ITE supports a truly hierarchical design, where macros are allowed to make full use of other already existing macros. Figure 5 shows how this works in practice (cf. [11] for a detailed discussion): Within an abstraction step a (part of a) test case is stored as a macro, which directly becomes available as a new generic test block. Beside the identifier and the name of the macro the formal parameters and the outgoing branches have to be specified. The parameters of the macro can be mapped to (selected) parameters of the underlying test blocks. E.g. in Figure 5 the parameter P12 of test block TB1 and P21 of test block TB2 become parameters of the macro. Similarly, the set of (unset) outgoing branches of the underlying test blocks defines the outgoing branches of the macro. 1
A quite large set of generic test blocks is available for the test of all systems, e.g. for the general handling of test cases, or the control of generic GUI items (buttons, windows, …).
E.g. in Figure 5 the unset branches of test block TB2 and TB4 (denoted through the dashed edges) will constitute the outgoing branches of the macro. As usual, the resulting hierarchy is a design and management aid without any influence on the execution time: during the execution macros are automatically unfolded (concretized) and the underlying graph structure will be executed.
TB1 (P11,P P12) Abstraction
TB2 (P21)
TB3
Macro (P12, P21)
TB4
Concretization
Figure 5: Hierarchical test case design using macros
3.3
Guidance through Verification of Test Cases
Test cases for complex systems typically become complex themselves, as they must capture the interplay between the involved components and subsystems. Therefore verification of test cases provides a strong aid to reliable test case design, where vital properties concerning the usage of parameters (local properties) and the interplay between the stimuli and verification actions of a test case (global properties) are automatically checked. Design decisions that conflict with the constraints and consistency conditions of the intended system or the current test purpose can thus be immediately detected. There is a number of consistency conditions that every valid test case must fulfill, independent of the particular application. Straightforward examples are: • all outgoing branches are connected properly, • there exists at least one possible execution of the test case that ends up with a verdict `passed´, • every possible execution of the test case encounters `passed´ or `failed´, and • the parameters are set correctly. Additionally, system specific consistency properties have to be checked which state the expertise of test case designers and system experts about the do’s and don’ts of test case creation. Figure 6 sketches a (global) consistency property which states that every Login to the system must be followed on all possible execution path by a Logout. This property is crucial when considering web-based applications in order to avoid side effects on the next test case: all resources that are requested during a test case must be freed after execution. This kind of properties is adequately specified in temporal logics [2], e.g. Login ⇒ F Logout specifies the described property.
Login
TB3
Logout
…
…
…
TB2
Logout
Logout
Figure 6: Illustration of a consistency property (Every Login must be followed by a Logout)
Whereas temporal logic specifications are not adequate for application experts and test engineers to express their informal consistency requirement, it turns out in practice, that a few simple patters of formulae are sufficient in order to capture most of the vital properties, cf. [1]. Thus it is possible to provide a GUI controlled pattern system, which allows even inexperienced users to conveniently express their requirements.
4
Testing IP-based Applications
The ITE supports the test process proposed in Section 2 and 3. Whereas the general philosophy and especially the coarse granular, graphical design of test cases is discussed in previous papers (cf. [3, 7, 8]), we will illustrate in this section the reuse and guidance aspects along two complex IP-based solutions: The Online Conference Service [6] and the Personal Call Manager [5].
4.1
Testing the Online Conference Service
The Online Conference Service is a complex web-service that supports the online management of the scientific program of professional conferences. It proactively helps authors, Program Committee chairs, Program Committee members, and reviewers to cooperate efficiently during their collaborative handling of the composition of a conference program. This application provides a timely, transparent, and secure handling of the submitted papers and of their related delegation, review, report and decision management process. The online service is a powerful application, including in particular a role-based access and rights management feature that is reconfigurable online, which accounts for a high flexibility and administration comfort, but which is responsible for potentially disruptive behavior in connection with sensitive information. Because of the complexity of the underlying workflows (and of course therefore the complexity of the application itself) intensive testing is clearly unavoidable. However state of the art of webbased application testing tools is still dominated by static approaches focusing on a posteriori structure reconstruction and interpretation [4, 10] rather than on functional aspects directly related to the applications workflows. Figure 7 sketches the architecture of the service. The browser, located on a client, plays the role of the ''classical'' application GUI and interacts with a Webserver. The Webserver itself communicates with an Application Server which builds dynamically the requested web pages through an interaction with a database and (several) Back-end services. Often staging servers, firewall solutions, load balancing and redundant architectures increase the speed and availability of the offered application, and raise the complexity of the required test setting. Due to the architecture of such a system it is obviously not sufficient to test the user interface only but it is of central
importance to consider the server side as well (i.e. the web-server, application-server and the back-end services).
Client
Client
... Internet/ Intranet
Test Coordinator Webserver
Application Server
Back-end Services
Database Figure 7: Test Setting for the Online Conference Service
The test case shown in Figure 8 tests whether an article, submitted by an author, is delegated correctly to a member of the program committee. From the submission to the start of the review process of an article at least three different roles are involved: the Author himself, the Conference Chair (PCChair), who must delegate the article to members of the program committee, and finally the members of the program committee (PCMember), who are responsible for the reviews. So we need at least three concrete users to run the test, whereby each user is associated with a distinct role. At the beginning of the test case for every user a browser must be started to perform the required tasks. The single steps to be executed are here encapsulated in a macro, cf. Figure 8 (upper right), which is available as a test block. As described in Section 3.2. this allows us to use in the corresponding test cases a single test block (Prepare) which can be expanded at need. This highly improves the readability of test cases, and, if for some reason the initialization procedure changes, only the macro has to be adapted and the change will be propagated to all its occurrences, as macros Figure 8: Test case for the Online Conference Service have a call-by-reference semantics. The considered test case is distributed, in the sense that it covers several subsystems involved in the application. In particular, test actions are executed on the clients marked with Author,
PCChair and PCMember, as well as on the database and on the webserver in our test environment. Nevertheless, in the ITE, the design of such a complex test is still quite easy and intuitive, since the tester (or test engineer) is not concerned with instantiation details like e.g. the concrete IP-Addresses of test clients.
4.2
Testing the Personal Call Manager Test Coordinator Rational Robot HTTP HUSIM
Hipermon
HUSIM
Internet/ Intranet
Hipermon
Application PC‘s
CSTA II/III HTTP
HUSIM
Hipermon
HUSIM
Hipermon
Application Server
Figure 9: Test Setting for the Personal Call Manager
In this scenario we are considering in particular the Personal Call Management application, which allows the clients to comfortably reconfigure their settings at any time and independently from their actual location. In particular it allows the management of a qualified call-forwarding: this takes into account characteristics like the ID of the caller or the current time, it can forward the call with respect to those conditions to different numbers, and it can handle user-defined profiles connected with specific forwarding conditions (forwarding policies) as well as a well organized user-defined exception handling to those policies. Additionally it supports a user management that allows protecting the access via personal passwords, and a role management that takes care of the association of specific rights to different user groups. The Personal Call Manager is connected to a network of switches which are themselves connected through an IP-infrastructure. This multi-node solution can be seen externally as a 'virtual' switch: the system can grow and be reconfigured at runtime, and the handling of the up to 250 users can be modularly organized over the physical switches. In Fehler! Verweisquelle konnte nicht gefunden werden. (left) a test case is shown which sets up a profile in the Personal Call Manager and tests the impact of it with an interaction of several telephones.
Figure 10: User friendly specification of consistency properties
Especially for such complex solutions it is important to guide the test engineer as much as possible during the test case design. As described in Section 3.3, the ITE provides a GUI controlled pattern system, which allows even inexperienced users to express their requirements in temporal logic. Our constraint editor manages the (concrete) consistency properties in a XML based database, as well as the underlying pattern system. This allows experts in temporal logic to extend the used pattern system in an easy fashion, whenever this is needed. Figure 10 illustrates the specification of the //: LoginLogout consistency property: Every user that logs in // Every 'Login must be followed // by an 'Logout for this user! to the Personal Call Manager has to logout constraint LoginLogout later on. This can be expressed using the ex{ Forall X in model ( istence after pattern, i.e. a given test block Q 'Login[role==X] => AF_F('Logout[role==X]) ) must occur after a test block P. After the test } engineer has chosen the pattern he can specP Q ify a name for the property and an informal Figure 11: Concrete consistency property description, which will be displayed to the user once the error occurs. After that Q and P have to be specified, i.e. a logout (Q) must follow a login (P). As the test block `login´ is parameterized, i.e. one can specify which user will be logged into the system, we have to take care of the concrete values of the parameter user as well. The special usage forall ensures that, whatever concrete value for a parameter is used, it will be checked that the value is the same for all occurrences of the corresponding test blocks, e.g. it will be checked, that a login for a concrete user user1 will be followed by a logout for user1. There exists three different representations for such a property: a formal one in terms of a temporal formula, an informal one that states the `meaning´ of the property (cf. Figure 11, first three lines) and an internal representation suitable for the ITE (cf. Figure 11 (bottom)).
4.3
Analysis of the test execution
The ITE web service is designed as a role-based, pro-active web service, which supports the cooperation process of teams of test engineers. Currently, we focus on the role-based and permission-based management of the test protocols: the web service provides several roles with different rights to access test reports and functionalities. This ensures that all users of the service get a tailored view onto the scenario of their interest, which guarantees data security and an easy handling: only the functionalities (e.g. show test report, search, delete, and modify) are presented to a user that are supported by the current role.
Figure 12 presents some screenshots of the ITE web service. It provides a comfortable search mask for accessing the database, e.g. concerning the date of test runs, the test result, the system under test, etc. (cf. Figure 12 (1)). Test reports can be modified, so that different attributes can be set (e.g. the status) and comments can be added (cf. Figure 12 (2)). Due to the XML based storage of the test protocols the ITE web service is able to present different views onto the same underlying sources: the service provides different XSL style sheets that generate different HTML outputs for the presentation of the data. E.g. Figure 12 (3) shows a test report at the level of detail as required for a test engineer, whereas Figure 12 (4) shows a guest user's view onto the same data, which is provided by another style sheet.
(1)
(2)
(3)
Figure 12: Analysis of test runs
(4)
5
Conclusions
In this paper, we have presented an approach to the overall test management of complex distributed systems like large IP-based applications, which addresses functional testing. Our approach is characterized by providing a coarse granular level for the test case design, which allows users to design test cases by directly mimicing the manual testing process. Together with our graphical notation in terms of control flow graphs, this enables test engineers to formulate even extremely complex test cases without any programming knowledge. The combination with our macro facility, which supports a truly hierarchical design, constitutes a scalable test method which can be operated solely on the basis of testing expertise. This, together with the testautomation, is particularly important for systems, where the frequent modification of parts (new hardware, new operating system release, new application server release, ...) requires a systemwide regression test, in order to check for the correct interplay between the various components. We have illustrated the generality of our approach by discussing two concrete real life examples: the test of a web-based application (Conference Service) and the test of an IP-based telephony scenario (Personal Call Manager). We are convinced that, due to the structural similarities, the success in telecommunications and CTI can be repeated on a larger scala for applications which exploit the converging media phenomenon. Here the planned elaborations in the support of cooperative (business) processes by to-do lists, time and resource management, as well as as fault management will be of major importance.
References 1
M. Dwyer, G. Avrunin, and J. Corbett: Patterns in property specifications for finite-state verification, Proc. of the Int. Conference on Software Engineering, ACM Press, 1999, pp. 411-420. 2 E. Emerson: Temporal and Modal Logic, Handbook of theoretical computer science, Elsevier, 1990. 3 A. Hagerer , T. Margaria , O. Niese, B. Steffen, G. Brune, H.-D. Ide: Efficient Regression Testing of CTI-Systems: Testing a complex Call-Center Solution, Annual Review of Communication, Vol. 55, Int. Engineering Consortium (IEC), 2001. 4 C.H. Liu, D.C. Kung, P. Hsia, and C.T Hsu: Structural Testing of Web Applications, Proc. of Int. Symposium on Software Reliability Engineering, 2000, pp.84-96. 5 T. Margaria , O. Niese, B. Steffen, A. Erochok: System Level Testing of Virtual Switch (Re-) Configuration over IP, Proc. of IEEE European Test Workshop, IEEE Press, 2002. 6 T. Margaria, O. Niese, B. Steffen: Automated Functional Testing of Web-based Applications, Proc. of the 5th Int. Conference On Software and Internet Quality Week Europe, 2002. 7 O. Niese, B. Steffen, T. Margaria, A. Hagerer, G. Brune, H.-D. Ide: Library-based Design and Consistency Checking of System-level Industrial Test Cases, Proc. of Int. Conference on Fundamental Approaches to Software Engineering, LNCS 2029, Springer Verlag, 2001, pp.233-248. 8 O. Niese, T. Margaria, A. Hagerer, M. Nagelmann, B. Steffen, G. Brune, H.-D. Ide: An Automated Testing Environment for CTI Systems Using Concepts for Specification and Verification of Workflows, Annual Review of Communication, Vol. 54, Int. Engineering Consortium (IEC), 2000, pp. 927-935. 9 Rational, Inc.: The Rational Suite description, http://www.rational.com/products. 10 F. Ricca, and P. Tonella: Building a tool for the analysis and testing of web applications, Proc. of Int. Conference on Tools and Algorithms for the Construction and Analysis of Systems, LNCS 2031, Springer Verlag, 2001, pp. 372-388. 11 B. Steffen, T. Margaria, V. Braun, and N. Kalt: Hierarchical service definition, Annual Review of Communication, Int. Engineering Consortium (IEC), 1997, pp. 847-856. 12 B. Steffen, T. Margaria: METAFrame in Practice: Intelligent Network Service Design, In Correct System Design -- Issues, Methods and Perspectives, LNCS 1710, Springer Verlag, 1999, pp.390-415.
