Hybrid Test of Web Applications with Webtest Harald Raffelt
Bernhard Steffen
Chair of Programming Systems TU Dortmund
Chair of Programming Systems TU Dortmund
[email protected]
[email protected]
Tiziana Margaria
Maik Merten
Chair Services and Software Engineering Universit¨ at Potsdam
Chair of Programming Systems TU Dortmund
[email protected]
[email protected] Categories and Subject Descriptors: D.2.4 [Software Engineering]: Software/Program Verification — Validation; D.2.5 [Software Engineering]: Testing and Debugging — Testing tools; I.2.6 [Artificial Intelligence]: Learning General Terms: Verification, Reliability Keywords: Testing, web applications, automata learning
the SUT (System Under Test) to determine whether the output behaviour matches the expectations. Development time testing is still often handled in practice via manual testing, a time consuming activity. Test-inputs generated during manual testing, however, can be recorded and used for regression-testing using replay. Replay-testing is an important methodology to ensure that single system components comply to their specification, but this approach is inherently limited to testing functionality for which test input sequences exist. Model-based testing has the potential to overcome this limitation by generating test cases from a model representation of the SUT. This way extensive test suites can be obtained automatically, significantly increasing the test coverage. However, what can be done if there is no model, or only a partial one? This is in practice very often the case. In this paper, we present hybrid testing, a method that combines
ABSTRACT In this paper, we present hybrid testing, a method that combines replay-testing (static testing) with automata learning techniques that generate models of black box systems (dynamic testing). This combination helps bridging the gap towards model based testing also for legacy systems. Webtest is an implementation of hybrid testing that builds on top of preexisting technology such as the LearnLib, a framework for automata learning, and the jABC, a framework for modeldriven and service-oriented design, that we use here for modelling, executing, and managing test suites for and models of the web applications under analysis. In particular we intend to move towards Rich Internet Applications (RIAs), that include e.g. advanced client side capabilities and access to heavy resources (e.g. database access) over the Web.
1.
• record-and-replay testing (static testing), a technique often used in industrial contexts and supported by widespread tools like HP’s (formerly Mercury) WinRunner and IBM’s Rational Robot, with
MOTIVATION
• automata learning techniques to generate models of black box systems (called dynamic testing),
Testing is a central part of both development and quality assurance processes. While a number of formal specification and design approaches that help enforcing higher quality of software show good potential within their bounds of applicability, in general they fail to replace testing as primary mean to ensure the required quality level. Testing is generally used at development time to confirm that new features work as intended, and later on as regression test, to discover whenever already existing functionality does no longer behave es expected after a version change. The latter aspect is often dealt with via record-and-replay techniques, where input data is recorded and injected into
this way bridging the gap towards model based testing. Dynamic testing [15] is a method that exploits automata learning to systematically test (black box) systems, e.g. component-based systems with third party components, and legacy systems, almost without prerequisites. Automata learning for black box components has been extensively studied in the recent years [22, 20, 8] from different points of view. In contrast to passive learning, via observation methods such as in Behaviour Capture and Test (BCT, [12]), dynamic testing uses an active approach to system exploration: Based on interface descriptions, this method successively explores the system under test, and at the same time it extrapolates the minimal behavioral model consistent with the observations. This model is in turn used to steer the further exploration process. In essence, dynamic testing can be regarded as a method for on-the-fly conformance model generation and testing process [1]. In the context of the current study, which aims at application in a standard industrial environment, we had the following requirements:
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. TAV-WEB – Workshop on Testing, Analysis and Verification of Web Software, July 21, 2008 Copyright 2008 ACM 978-1-60558-052-4/08/07 ...$5.00.
1
• support a smooth transition from manual test to modelenabled and later model-driven testing for existing web applications • seamless integration in the current practice (captureand-replay, Mozilla as browser) Figure 1: Very basic example of an executable graph
• support a wide range of users: test experts in the laboratory, but also end users, who may submit traces
[17, 3] towards a ’one-thing-approach’ [18] that uses and successively refines a single model as the central thing along the entire lifecycle, from the requirement capture to the testing an evolution phases. In hybrid testing, we use the jABC to capture runnable test cases as models, and to combine them by dynamic testing into behavioural models of the web application under consideration. To this aim we use the ITE (Integrated Test Environment) specialization of the jABC plugin and the LearnLib for automata learning [14]. Realizing the dynamic and hybrid testing process in the jABC directly provides us with:
• support regression. The result is Webtest, a hybrid testing environment especially customized for web applications. This solution is developed in cooperation with our industrial partner T-Systems, where it is already in productive use, supporting their preexisting quality assurance structures. Our focus is here on testing web applications which implement services using (X)HTML and client side scripts running in a remote browser application. It took only a few years for web applications to make a decisive impact onto the software landscape, in many cases replacing classic clientside applications. Prominent examples are web mail services (that make classic desktop mail clients obsolete for many users), calender services or even office solutions, that offer means to create documents and spreadsheets. Thus web applications conquer an increasing market share, eventually turning web browsers into a central runtime environment for all kinds of application domains. Webtest an implementation of hybrid testing that builds on top of preexisting technology such as LearnLib, a framework for automata learning, and jABC, a framework for model-driven and service-oriented design, that we use here for modelling, executing, and managing test suites for and models of the web applications under analysis. In particular we intend to move towards Rich Internet Applications (RIAs), that include e.g. client side capabilities and access to heavy resources (e.g. database access) over the Web.
• Customizability. Test cases can easily be customized: their process structure can be modified conveniently using a graphical interface, their organization is done in terms of taxonomies, that group and structure also large test suites, and their look and feel can be modified at ease, since it is possible to adopt the notational and symbolic standards of the application domain. These three points are central here for the seamless integration with preexisting practices and conventions. • Executability. jABC’s built-in execution environment allows one to execute both the overall hybrid and dynamic test process comprising complex test suites and the individual test runs generated during the extrapolation process. As an example, Fig. 1 shows a very simple instance of an executable test case. The jABC models can be interpreted as control flow graphs, where nodes represent atomic actions that are executed on the SUT, and the edges control the execution by determining which next step will occur. Atomic actions are implemented as SIBs (Service Independent building Blocks). This example initializes a web client, a process which includes loading an initial page, and proceeds to fill in a text input field and to activate a submit button.
In the remainder of the paper, we first summarize in Sect. 2 the key features and benefits of jABC, to which Webtest is a plugin. Webtest itself is presented in Sect. 3. We then consider a concrete application example in Sect. 4, where we focus on the extrapolation of models, the kernel of the dynamic testing technology. The paper closes with some conclusions and perspectives in Sect. 5.
2.
OVERVIEW OF THE JABC
• Verification. jABC provides model checking and checks for local consistency. We apply them to the overall hybrid and dynamic testing process to guarantee compliance to predefined rules. Such rules may express how to avoid that the testing process gets stuck. As a typical example, test runs should leave the system under test in an orderly state, so that further test runs can continue from there.
The jABC is a framework [6, 11] for modelling, development, and execution of applications in a model-driven, service-oreinted way. Predecessors of jABC have been used since 1995 to design, among others, industrial telecommunication services [11], Web-based distributed decision support systems [7], and test automation environments for ComputerTelephony integrated systems [5]. jABC allows users to easily develop services and applications by composing reusable building-blocks to (flow-) graph structures. This development process is supported by an extensible set of plugins that provide additional functionality in order to adequately support all the activities needed along the development lifecycle like animation, rapid prototyping, formal verification, debugging, code generation, testing, and evolution. It does not replace but rather enhance other modelling practices like the UML-based Rational Unified Process
Model checking can also be applied to enforce behavioural properties of the models extrapolated during the dynamic testing process. This may be e.g. advantageous to steer the testing/learning process in a certain direction. In the testing approach described here, we use the ITE specialization of the jABC to model, verify, and enact the
2
single test cases, as shown in Fig. 1 and 3, as well as the learned models (Fig. 5. We also use the LearnLib plugin to steer the learning process that underlies the dynamic testing.
tested progresses, classical test suites, without manual intervention, will increasingly fail to adequately cover all aspects of the application under development. It is thus necessary to continuously extend test suites, which can be an undertaking of considerable dimensions. This can lead to a slow down in development, either because further development is postponed until the test suite ”catches up”, or because test coverage is declining to dangerous levels where critical flaws are discovered too late. Dynamic testing, however, can in many cases (semi-)automatically explore new functionality introduced into applications, thus reducing the urge to manually maintain large sets of test cases.
The ITE. The Integrated Test Environment (ITE) is a specialization of the jABC for testing purposes, which includes SIBs tailored to meet testing demands. This includes functionality to execute tests as well as means to document the test results in reports. Being a direct descendant of jABC, the ITE inherits all its features and concepts.
The LearnLib.
3.1
LearnLib [14] is a framework for automata learning. Originally, LearnLib has been designed to systematically infer finite state machine models of unknown real world systems. In the meantime, it has become a platform for experimentation with different learning algorithms and to statistically analyze their characteristics in terms of learning effort, run time and memory consumption. The LearnLib consists of: • an automata learning library, containing the basic learning algorithms, • a filter library, providing several strategies to suppress redundant tests during the dynamic testing process, and • a library based on ideas from conformance testing. It is essential for steering the model-based selection of individual test cases during the dynamic testing process. The LearnLib can be used in different modes, as detailed in [16], depending on the purposes of usage. This is why it meanwhile constitutes a framework on its own. We use here its learning process modelling mode [16], which enables the user to control the entire learning process, comprising the context-specific choice of optimizations, strategies of search, as well as setting interaction points for a truly interactive learning process.
3.
Input philosophy
The number of ways to interact with a web-application is limited: Most of the navigation is done using mouse clicks, either on links or on buttons, and keyboard input goes e.g. into text input fields of forms. Thus one can abstract the interaction capabilities into a rather small set of basic actions, e.g. to go to an URL, to click onto an element or to fill in a text input of a form. This approach, which successively builds up a library of interaction components library, is still valid also for RIAs: they offer enhanced interaction capabilities due to their rich clients, but also these capabilities can be easily captured for test in libraries of elementary actions. Seen from the end-user point of view, however, there is no difference whether the actions are performed at the client or at the server side: what the user sees is just a ”page”. This is the point of view we take, thus we refer simply to pages and page actions in the following. Webtest works with such a basic set of actions, each action allowing specification of on what element of the webpage the action should be performed. Therefore Webtest guides the user through the process of configuring the desired actions to achieve the intended effect. It is important however that the action library is easily extensible, at need.
3.2
Recording test cases
Webtest offers two distinct ways to generate test cases: by modelling and by harvesting.
THE WEBTEST ENVIRONMENT Modelling mode.
The Webtest testing solution is a jABC plugin that offers means to record and execute tests of web applications on top of the ITE and is thus a pre-processor to the dynamic approach for testing web applications that uses automata learning. Webtest is developed in cooperation with our industrial partner T-Systems. It is already in productive use, supporting their preexisting quality assurance structures. Its purpose is to offer an environment for comfortable, scalable test recording and execution for web applications, to be directly adopted in an industrial environment where regressions must be efficiently and extensively run. It must thus easily fit in the current practice. At the same time it is meant to pave the way to enabling dynamic testing, ideally with no extra effort. Central issues for Webtest are therefore the coverage of typical test actions (the inputs), the recording facility, the ease of replay, and the easy integration of all the collected information into the dynamic testing process. Webtest, enhanced with dynamic testing, is expected to overcome restrictions seen with passive exploration schemes like BCT [12] and classic test management solutions like Mercury’s [13]: As the development of the system to be
In the modelling mode, the available actions for a given page are displayed in a tree representation, which follows the structure of the underlying document. The test-engineer can select one action, configure it and append it to the executable graph, which forms a test case. Figure 2 shows how to configure an action: it fills in the search input field of a web application offering web searches. This mode gives an extensive amount of control over how an action is supposed to be executed. It is useful for experienced testers, and to enhance previously defined test cases with additional actions.
Harvesting mode. In the harvesting mode, actions are recorded while the test-engineer browses freely. In our case, harvesting is carried out with an integrated browser component based on Mozilla technology. This is a very intuitive way of recording, but gives only limited control over how e.g. page elements are supposed to be addressed. The generated defaults, however, cover much of the desired functionality in practice. The same general concept of capturing user interaction
3
Figure 2: Configuring an action to fill in an input field. was already employed in e.g. the BioMoby platform for bioinformatics workflows [2]. There, the aim is to record the (erratic) user actions of an end user (typically biologists) to make the sequence repeatable. We support this mode too, with the idea of including also non-experts in the test production process.
3.3
For instance, to test if a database frontend is returning correct query results it is necessary to specify a fixed set of inputs. Similar considerations apply to systems where certain functionality is only accessible after an authentication process. Without providing such information the learning process cannot possibly explore the whole system, meaning that the learned model will remain incomplete. Executable test cases recorded as described above consist of a set of actions configured for the structures exposed by the web-application, including values for such input fields. Once a suite of manually recorded test cases is available, the set of configured actions can be used as input alphabet for the learning algorithm. This complements the alphabet symbols dynamically discovered during dynamic testing. For example, actions for clicking on links that were not covered by the manually recorded test cases would be discovered dynamically during the dynamic test, and included in the alphabet. This ensures that even areas of the web application that are not yet covered by tests can be visited by the learning algorithm, as long as no additional information is needed (e.g. for authentication).
Replaying test cases
The recorded test cases are executed via a browser component that exposes an interface to trigger the desired actions on a webpage. Extensive reporting is provided by collecting the status of each test case and generating a HTML page contained within a compressed archive. Fig. 3 shows the report for the execution of one single test case. As one can see the parameters for each action and the verdict are recorded. Also included is a snapshot of the loaded webpage after every action, making it easy to track the progress of testing. This is particularly important for RIAs: simple actions can trigger complex executions, and it is thus central to be able to examine exactly and stepwise the response they produce. In Webtest it is also possible to trigger the automatic execution of a given set of test cases, e.g. to execute the complete test suite at fixed time intervals. The reporting for the automated execution includes statistics on e.g. how many test cases executed successfully and how many failed.
3.4
3.5
Learning: Configuration and Execution
Fig. 4 shows on the left the configuration options for the dynamic testing module of Webtest. Major options concern the alphabet of actions and inputs to be used (option ”samples”: this can be extracted from a graph assembled via record-and-replay as described above), and various additional processes to be executed during different stages of the learning. The option ”after test case”, for example, contains post-processing execution, i.e. instructions on how to restore a specified initial state on the SUT to ensure that no side effects impact onto the execution of further test cases. In the next section we show how hybrid testing can be applied to a well known application, the Mantis bugtracking
Retrieving a configured alphabet for automata learning
Most actions applicable on an HTML-page delivered by a web-application can be immediately deduced from the elements contained within that page. E.g., all links are clickable, as are buttons and checkboxes. This is often sufficient to navigate through applications that do not have restricted areas, but additional information is needed to test functionality that depends on meaningful, non-random input data.
4
Figure 3: Execution report of one trace.
Figure 4: Configuration of the learning setup (left) and execution status (right). system. It is not a RIA, but it is heavily database-oriented and it offers all the significative traits of Webtest’s target applications.
4.
rithms require a reliable and fast reset. Fortunately, Mantis can use a PostgreSQL database backend and one can use PostgreSQL in order to clone databases. This enables us to realize the required reset operations via snapshots. Having recorded test cases that log into a local Mantis instance and perform various actions we reused those recorded tests to provide the alphabet needed for the learning phase. The system’s reset operations are very complex, since they require a reset of the underlying database. These reset operations were modelled as jABC graphs, to be executed after every test case to preserve consistency of behaviour during the learning process. Figure 5 shows the result of a learning session. One can observe an overall circular topology. The various states of the web application are represented by nodes with fin-like structures attached. Each fin of the fans represents an action that can be executed in the corresponding area, with the execution result annotated at the edge.
MANTIS: A CASE STUDY
Mantis [9] is a popular web-based bugtracking system. It is written in the PHP scripting language and requires a MySQL or PostgreSQL database and a webserver. Mantis is platform-independent and runs on Windows, Mac OS, OS/2, and a variety of Unix operating systems. Almost any web browser is able to function as a client. It is released under the terms of the GNU General Public License (GPL). Reactive systems and in particular web applications are typically designed to run forever. Thus they do not need and therefore do not support an efficient reset operation. This also applies to Mantis. On the other hand, most automatic testing approaches as well as automata learning algo-
5
Figure 5: Learned model of the Mantis case study. The areas of the web application are connected by those actions which trigger transitions. Those actions are usually the activation of hyperlinks or submit buttons. Looking at the learned model it is immediately visible which areas are having a ”neighborhood relation” and which actions trigger a transition.
5.
of model structures, and last but not least, with the option to dynamically/interactively enlarge the alphabet underlying the learning process. The applicability of this approach has been demonstrated with an example employing Mantis, a popular web application for bugtracking. For the moment we have worked with own palettes of test SIBs. It would be possible for instance to integrate other test scripting elements, e.g. Watij scripts [23]: Since it is just a Java API, Watij does not prescribe a test environment, and it could therefore use our ITE. It provides capabilities to find suitable user level test element that allow Watij scripts to find, access, and control any HTML element on a page, which could become new SIB palettes in the Webtest tool. Similarly, we could easily integrate HttpUnit functionalities [4], which also provides a Java API to define, access, and assess testing operations on web applications. A great deal of effort is spent today towards test of Web
CONCLUSION AND PERSPECTIVES
We have presented hybrid testing, a method combining the classic testing using capture and replay with an automata learning approach (dynamic testing). Dynamic testing extrapolates a behavioral model, based on interface descriptions and a systematic exploration of the system under test. This model is in turn used to steer the further exploration process. Using the LearnLib, our framework for automata learning, our method can elegantly be combined with numerous optimizations of the learning procedure, various choices
6
services, assuming the knowledge of the WSDL descriptions. Recent work has started to cast standard-based approaches, e.g. those basing on ETSI’s TTCN-3 test specification language [24] and corresponding test environments, towards the test of Web services. [21] for example shows an extension of the TTworkbench Basic [19], a TTCN-3 test development and test execution environment, for Web service testing. This encompasses a test adapter and a codec that enable the execution of TTCN-3 abstract test suites derived from WSDL descriptions, a wizard that map a WSDL document to a TTCN-3 abstract test suite, and a wizard that creates a TTCN-3 project that is configured for the testing of Web services (using the implemented test adapter and codec). The web applications we envisage here are however black boxes, for which no description like a WSDL is available. We have however already realized TTCN-3 capabilities in the jABC in a different contexr, and are considering an extension of Webtest towards inclusion of TTCN-3 elements, in order to support mixed settings. Benefits of hybrid testing with Webtest include the ability to conduct regression tests using the well-known method of replay-testing and the ability to reuse data sets to seed automatic exploration, expanding into areas not yet covered by replay-testing. This can speed up the testing process, while at the same time increasing test coverage and offering as of yet unseen views onto the SUT. We expect this capability to significatively improve the productivity of test engineers in the field.
6.
[9] [10]
[11]
[12]
[13] [14]
[15]
[16]
[17]
REFERENCES
[1] Berg, T., Grinchtein, O., Jonsson, B., Leucker, M., Raffelt, H., Steffen, B.: On the correspondence between conformance testing and regular inference. Proc. FASE’05. LNCS 3442, Springer, pp. 175–189. [2] Dibernardo, M., Pottinger, R., Wilkinson, M.: Semi-automatic web service composition for the life sciences using the BioMoby semantic web framework. J Biomed Inform. 2008 Mar 4 [3] H¨ ormann, M., Margaria, T., Mender, T., Nagel, R., Schuster, M., Steffen, B., Trinh, H.: The jABC approach to collaborative development of embedded applications. In: CCE ’06, Int. Workshop on Challenges in Collaborative Engineering - State of the Art and Future Challenges on Collaborative Design. (April 2006) (Industry Day), Prag (CZ). [4] Kulvir Singh Bhoga: HttpUnit: A Civilized Way to Test Web Applications in WebSphere Studio. 2003 http://www.ibm.com/developerworks/websphere /library/techarticles/0303 bhogal/bhogal.html. [5] Hungar, H., Margaria, T., Steffen, B.: Test-based model generation for legacy systems. Proc. of 2003 International Test Conference (ITC 2003), Charlotte, NC, IEEE CS, pp. 971–980. [6] J¨ orges, S., Kubczak, C., Nagel, R., Margaria, T., Steffen, B.: Model-driven development with the jabc. Proc. of Haifa Verification Conference 2006 , (HVC 2006). LNCS 4383, pp. 92-108, Springer [7] Karusseit, M., Margaria, T.: Feature-based modelling of a complex, online-reconfigurable decision support service. Electr. Notes Theor. Comput. Sci. Vol. 157(2) (2006) pp. 101–118 [8] D. Lo, S. Maoz, S.-C. Khoo: Mining modal
[18]
[19] [20] [21]
[22]
[23] [24]
7
scenario-based specifications from execution traces of reactive systems. Proc. ASE 2007: 465-468. Mantis bug tracker. http://www.mantisbt.org (seen Jun. 2007) Margaria, T., Raffelt, H., Steffen, B.: Knowledge-based relevance filtering for efficient system-level test-based model generation. Innovations in Systems and Software Engineering 1(2) (September 2005) 147–156 Margaria, T., Steffen, B., Reitenspieß, M.: Service-oriented design: The Roots. Proc. ICSOC 2005, LNCS 3826, pp. 450-464, Springer Verlag. Mariani, L., Pezze, M.: Behaviour capture and test for controlling the quality of component-based integrated systems. Proc. ESEC/FSE Worksh. on Tool Integration in System Development, Helsinki, 2003. Mercury TestDirector. http://www.fronde.com /products mercury.htm (seen April 2008) Raffelt, H., Steffen, B., Berg, T.: Learnlib: A library for automata learning and experimentation. Proc. FMICS’05, Lisbon (P) ACM Press (2005) pp. 62–71. Raffelt, H., Steffen, B., Margaria, T.: Dynamic Testing Via Automata Learning. Proc. Haifa Verification Conference 2007, LNCS 4899, pp.136–152 Raffelt, H., Steffen, B., Berg, T., Margaria, T.: LearnLib: A library for automata learning and experimentation. Appears in STTT, Int. Journ. on Software Tools Technology Transfer. Rational Unified Process. http://www-306.ibm.com /software/awdtools/rup/ (seen Jun. 2007) Steffen, B., Narayan, P.: Full Life-Cycle Support for End-to-End Processes. Computer (2007), IEEE Computer Society Press, Los Alamitos, CA, USA Testing Technologies: TTworkbench Basic http://testingtech.de/products/ttwb basic.php E. Torlak, D. Jackson. Kodkod: A Relational Model Finder, Proc. TACAS 2007. S. Trosch¨ utz: Web Service Test Framework with TTCN-3, MSc. Thesis, Univ. G¨ ottingen, June 2007, http://www.swe.informatik.uni-goettingen.de /publica tions/ST/WebServiceTestFrameworkWithTTCN-3.pdf A. Vardhan, K. Sen, M. Viswanathan, G. Agha. Actively learning to verify safety for FIFO automata. Proc. FSTTCS’04, Chennai, India, LNCS 3328, pp. 494-505, 2004. Watij User Guide http://watij.com/wiki:user guide. Willcock, C., T. Deiß, S. Tobies, S. Keil, F. Engler, S. Schulz: An Introduction to TTCN-3, First Edition, John Wiley & Sons Ltd, 2005.
