Automation of Test Case Generation and Execution for Testing Web Service Orchestrations Vera Stoyanova
Dessislava Petrova-Antonova, Sylvia Ilieva
Faculty of Mathematics and Informatics Sofia University “St. Kliment Ohridski” Sofia, Bulgaria
[email protected]
Institute of Information and Communication Technologies Bulgarian Academy of Science Sofia, Bulgaria
[email protected],
[email protected]
Abstract—Testing service orchestrations is a challenging research domain due to necessity of additional testing efforts, complementing traditional software testing. Although various testing approaches and tools have been proposed, most of them provide partial solutions covering single testing activities such as test path analysis, test case generation, web service emulation, fault injection and so on. Following the current research direction, we have developed an integrated testing framework, called TASSA, which aims to provide end-to-end testing of Business Process Execution Language (BPEL) orchestrations. This paper introduces its core capability for automation of test cases generation, execution and management implemented in an open source tool for WSDL-based testing of both single web services and composite web services, described with BPEL. The tool’s functionality includes identification of web service operations as well BPEL variables in case of service composition testing, generation of SOAP request templates, data driven testing, definition of assertions at different levels (HTTP, SOAP and BPEL variable), execution and management of test cases. Keywords— BPEL, test case, web service testing, WSDL
I.
INTRODUCTION
Service-Oriented Architecture (SOA) is a widely adopted computing paradigm for development of distributed applications. Its vision is preferably implemented using web services, which are built and deployed on heterogeneous platforms and orchestrated to achieve a particular business goal. The web service orchestrations are usually described with Business Process Language for Web Services (WS-BPEL) that is a complex standard presenting additional challenges to software development and testing. Testing business processes brings various challenges at infrastructure, service as well as orchestration level [6]. Since the design information and the source code of both infrastructure components and web services are missing, the usage of white box testing techniques is not applicable. The reusability of web services allows them to be consumed by multiple clients with different quality requirements in variety contexts, which are unknown at the development phase. This in turn complicates the testing, since the coverage of all use cases is practically impossible. Furthermore, the web services are developed by different software teams and controlled by different providers. That is why establishment of common
semantic, fault handling and transactional models is very hard to achieve. This leads to difficulties when testing service compositions. Additionally, some of the orchestrated web services could be unavailable or under development at the time of business process testing. The lack of control over partner web services and their distributed nature cause performing of emulations, generation of stubs, simulation of communication failures, and so on. Although various approaches and tools for WS-BPEL testing have been proposed, most of them provide partial solutions covering single testing activities such as test path analysis, test case generation, web service emulation, fault injection and so on. However, in order to facilitate the testing process, it is important to integrate all testing activities in a common automated platform. Therefore, the current research efforts focus on the development of testing frameworks that complements existing development environments and can be used jointly to achieve end-to-end testing. Following this research direction, we have developed our flexible TASSA framework for testing of BPEL processes [9]. This paper introduces its core capability for automation of test case generation, execution and management implemented in Test Case Generation and Execution Tool (TCGET). The rest of the paper is organized as follows. Section II analyses current tools for testing SOA implementations. Section III presents the architecture of TASSA framework focusing of the module structure of the TCGET. Section IV describes the main features of the TCGET related to test case generation and execution. A case study showing the usage of the TCGET to testing a sample BPEL process is presented in Section V. Section VI gives directions for future work. II.
RELATED WORK
This section presents a comparative analysis of the current software tools for testing of SOA implementations. Note that the goal of the analysis is to examine the solutions providing similar functionality to that of TCGET. Here, the software platforms that are competitive to TASSA framework as a whole are not covered. The criteria for comparison of the testing tools are chosen according to their common features described in the available documentation. They cover the generation of test cases based
on WSDL or BPEL file (Feature 1); the definition of SOAP requests to particular operation from the WSDL file (Feature 2); the support of data driven tests (Feature 3); the definition of assertions for the SOAP response (Feature 4); the definition of assertions for the WS-BPEL process’s variables (Feature 5); the possibility for integration with widely used Integrated Development Environments (Feature 6); the support of free software license (Feature 7); the providing of standalone Graphical User Interface (Feature 8). The core functionality of the software tools for testing is defined by the testing artifacts. Therefore, in order to provide objective comparison, all the tools under analysis must cover the first criterion, namely their main testing artifact should be WSDL or BPEL file. As a result five testing tools were examined: SoapUI, BPELUnit, SOAtest, Oracle BPEL Test Framework and SoapSonar. SoapUI is an automated testing solution with rich functionality [1]. It supports leading technologies and standards such as SOAP and REST web services, Java Message Service (JMS), Adobes ActionScript Messaging Format (AMF), Java Database Connectivity (JDBC), etc. Its core capabilities include functional testing, service mocking, security testing and load testing. Unfortunately, some essential features such as data driven testing, multi environment support, coding free test assertion and tree-based input for easy manual testing are provided only with the professional edition of the tool, which is not free for use. In addition, the SoapUI does not allow definition of assertions for WS-BPEL process’s variables when the WSDL under test describe a business process interface. Such feature is presented in Oracle BPEL Test Framework provided by Oracle BPEL Process Manager [2]. It is fully integrated with Oracle JDeveloper, providing functionality for simulation of web service interactions, validation of process actions with test data, calculation of activities’ coverage, etc. Parasoft SOAtest is an integrated solution for end-to-end testing of SOA implementations [3]. Since version 4.5, it supports automated testing of business processes, described with WS-BPEL. Along with this, the SOAtest performs checking of the reliability, security, and maintainability of web services developed against WS-I, W3C, OASIS, and WSstandards. However, since it is distributed with proprietary software license, which is quite expensive, only the big software vendors can take its advantages. In contrast, the BPELUnit is a testing framework developed within an open source project [4]. It provides white-box testing of business processes described with WS-BPEL, supporting service mocking and calculation of path coverage. Although the framework allows execution of data driven tests, it does not provide a GUI for data source selection and variable definition for the request message. In addition, it supports only XPath matching assertions. Assertions related to HTTP status code, fault response message, valid response message and existence of string token in response message, covered for example by the SoapUI, are not provided. SoapSonar is a testing tool providing functional, performance, security and compliance testing of web services [5]. It simulates web service interaction with clients by automated generation of client requests. The free version of the tool supports only functional testing. Features such data driven tests, capture of dynamic response HTTP header value and
conditional test execution are available only in licensed versions. The results from the analysis are summarized in Table I. TABLE I.
TESTING TOOLS FOR SOA IMPLEMENTATIONS
Testing Tools Feature Feature 2 Feature 3 Feature 4 Feature 5
SoapUI
+ + Eclipse, Feature 6 IntelliJ Idea, NetBeans Feature 7 + Feature 8 +
Oracle BPEL SOAtest Test Framework
BPELUnit SoapSonar
+ + +
+ + + -
+ + + +
+ + -
JDeveloper
Eclipse
Eclipse
-
+ -
+
+ -
+ +
Our solution covers all features mentioned so far. It supports testing of SOA implementations, which interface is described with WSDL. Therefore, it can be used for testing both single web services and web service compositions, described with WS-BPEL. Furthermore, TCGET provides well-defined project structure, in which each test case is described in separate XML file. In contrast, the BPELUnit stores all test cases that belong to given testing project in a single file. This in turn hampers the testing activities, especially in case of large number of test cases. Additionally, TCGET is implemented as open source solution, which is fully integrated with NetBeans IDE. III.
ARCHITECTURE OF TASSA FRAMEWORK AND TCGET
The TASSA framework provides a methodology and set of tools for testing business processes described with WS-BPEL. This section describes the general concept behind it as well as the architecture of the TCGET. A. TASSA framework The TASSA framework supports functional as well as nonfunctional testing of business processes. The functional testing includes path coverage according to functional requirements, full branch coverage with shortest path length of the business process, and path coverage according to new functionality. The non-functional testing is achieved through robustness and scalability testing. The robustness testing is conducted by simulations of possible failures of unreliable service(s) and tests compositions of both reliable and unreliable services. The scalability testing covers many configuration changes. The testing capabilities of the TASSA framework are provided by a set of tools each of which covers different testing issues. The Isolation Tool (IT) provides the functionality needed to isolate the BPEL process under test from outside dependencies. It works by transforming the WS-BPEL file describing the business process so as to remove one or more dependencies, and replace them with activities that are internal to the BPEL and can mimic the output of the original activity. For a given set of WS-BPEL activities, the Data Dependency
Analysis Tool (DDAT) finds a path that goes through all activities in the set starting from one initial activity [7]. Also, it finds all control activities on the discovered path and calculates the condition that should be met in order for the process to continue execution along the path. The Fault Injector Tool (FIT) injects delays, errors, etc. in the message exchange for a particular WS-BPEL activity. The main function of FIT is to make modifications in a way that the BPEL file describing the process can be tested against fault injection [8]. The Value Generation Tool (VGT) provides appropriate values that DDAT needs for the variables of the conditional activities. It also communicates with FIT, when faults in message data need to be simulated. The Test Case Generation and Execution Tool (TCGET) automates the test case generation and execution and along with other TASSA tools provide end-to-end testing of web service compositions. B. TCGET Architecture TCGET is a solution for WSDL-based testing of both single and composite web services. In other words, the main testing artifact of the tool is a WSDL description of the service under test. Its core functionality supports:
Identification of web service operations as well BPEL variables in case of business process testing;
Generation of SOAP request templates;
Execution of single test case, all test cases in a test suite and all test cases in a test project;
Management of test cases and file organization of testing project including grouping of test cases in test suits, updating of test cases and test suites, and so on;
Data driven test cases;
Assertions at HTTP, SOAP and BPEL variable level;
Integration with NetBeans IDE.
The TCGET consists of nine modules depicted with a component diagram in Fig. 1.
TestSuite, BpelOperation, etc. The Test Definition Engine module manages test cases and test suites. The Persistence Engine module performs serialization and deserialization of test cases. The SOAP Request Generator module handles the generation of template SOAP requests based on a selected operation from WSDL file. The Test Execution Engine module performs execution of test cases. The Template Test Processor module provides support for generation data driven tests. The Settings module is responsible for the tool configuration related to connection timeout, request charset, request content type, proxy server address and port, address of OpenESB server and port, user name and password. The Assertion Manager module implements different types of assertions, described in the next section, and provides GUI for definition and modification of an assertion in a test case. The NetBeans plug-in packages the functionality of the TCGET in a NetBeans plug-in. IV.
TEST CASE GENERATION AND EXECUTION
The core functionality of the TCGET is related to test case generation, organization and execution. This section describes the key aspects of these issues. A. Test case generation The BPEL Test Project, created by the TCGET, is related to a single WSDL file that corresponds to the business process or web service under test. During the creation of a new project the wizard automatically extracts the WSDL file from a specified location and creates an empty project with a node named WSDL. The WSDL node provides a context menu, which is populated with all available operations under test. After the tester selects an operation for testing, he/she is prompted to specify a name for a new test case and a test suite, where the test case will be placed. Next, the Test Case Editor (TCE) of the TCGET is opened. The newly created test case is automatically populated with an empty SOAP request template relevant to the selected operation. In order to facilitate test data definition, the generated SOAP request is supplied with hints for the data types of the SOAP body elements. Each test case is stored in a separate XML file with extension .BTC. The test case structure is displayed with UML object diagram in Fig. 2.
Figure 2. Test case structure
Figure 1. Test Case Generation and Execution Tool Architecture
The Test Model Definition module implements the main interfaces and classes of the domain model, namely TestCase,
As is shown in Fig. 2, the test case has two attributes – “name”, defining the test case name, and “template”, specifying whether the test case is data driven. The main XML elements of the test case are “description”, “operation, “testInput”, “datasource” and “assertions”. The element “description” provides narrative description of the test case.
The element “operation” specifies the WSDL operation under test including service name, port name and operation name. The element “testInput” stores the input data described with elements “body”, “header” and “authorization”. The “datasource” element defines the datasource when data driven test case is created. Currently, two types of datasource are supported, namely CSV and JDBC. The last element “assertions” is responsible for definition of test case assertions. The TCGET supports seven types of assertions. The “HTTP Status Code” assertion verifies the status code of the HTTP response. The “Response Time” assertion verifies that the response is received in a given period of time. The “SOAP Fault” assertion verifies that the HTTP response contains a valid SOAP Fault message. The “SOAP response valid” assertion verifies that the HTTP response contains a valid SOAP message. The “XPath Equals” assertion verifies that when a certain XPath expression applied to a SOAP body or a BPEL variable, its result is equal to a particular value. It is possible to make it case insensitive and/or match it with regular expression. The “XPath Exists” assertion verifies if a certain XPath expression applied to a SOAP body or a BPEL variable is not empty. The “Contains” assertion verifies that a certain string exists within a SOAP body or a BPEL variable. In addition, the TCGET provides Negated assertion, which can be applied to all assertions defined above. B. Test case organisation The test cases related to a certain WSDL file are organized in a custom NetBeans project. The project’s files are visualized in a tree structure. The project contains a WSDL file and an optional XSD file that corresponds to the web service/business process interface. The test cases are grouped in test suites. The test suites are stored in separate directories, which are subdirectories of one root directory, called “TestSuites”. As it is already mentioned, each test case is stored in a separate XML file. The metadata of the project is saved in a XML file, named tassa-tcme-project.xml. C. Test case execution The execution of test case includes two steps: (1) sending of SOAP request to the proper web service address defined in the WSDL file of the business process, and (2) receiving and assessment of obtained result according to the assertions defined in the test case. On the first step the information stored in the test case is used to retrieve the web service address, where the SOAP request should be sent and the message parts such as HTTP headers, user name and password if HTTP authentication is needed, and data of SOAP request body. The obtained result from test case execution consists of HTTP headers of the received response, data of SOAP response body, values of process variables and execution time. The actual data of the SOAP request body and the expected data in the SOAP response body are replaced with variables when data driven tests are defined. In such case, first a connection to the data source needs to be established. Second, for each record in the data source a separate test case is created, where the variables are replaced with concrete values. After, the generated test cases are executed ordinarily.
The most important task with respect of testing is analysis of the testing result on which all assertions defined in the test case should be applied. The obtained data from test case execution are encapsulated in a single object that is processed by the Test Execution Engine module of the tool. The following test execution statuses are used by the module to assess the test result:
Success – The result satisfies the test assertions.
Failure – The result does not satisfy the test assertions.
Error – The assertion is not verified. For example, an XPath assertion is defined and the XPath statement is not valid according to the XPath specification.
Unknown – The status before checking of the assertion.
The Test Execution Engine module provides functionality for subscription to events related to the test case execution. This functionality is used in the implementation of the GUI panel, which is responsible for visualization of the test cases in a tree structure with icons indicating their statuses. V.
CASE STUDY
This section presents the TCGET capabilities through testing a manually generated business process providing functionality for withdraw and transfer of amount in a bank system. A. Business process under test The business process under test (BPUT) has eight partner web services providing the functionality as follows: (1) Login, logout and registration of users; (2) Risk assessment of the performed transaction related to withdraw and sending amount; (3) Deposit, withdraw and transfer of amount, and checking of account balance; (4) Currency conversion; (5) Calculation of commission. The commission is calculated based on FX Rate of the currency and amount due for payment provider routing; (6) Logging; (7) Automated notification via email, SMS, etc.; (8) Finding payment provider. The web service returns a unique identifier of the provider. The BPUT is deployed on a GlassFish application server and communicates with its partner web services through SOAP messages. B. Test cases In this case study, we have used the GUI of the TCGET to define the following test cases, which cover all types of assertions supported by the TCGET:
TC1: The BPEL process should revoke any attempts to transfer money without valid login credentials. The test sends a valid SOAP request but with incorrect password and checks that the response is a valid SOAP response with the message that the login credentials are not correct.
TC2: The valid transaction types for the BPEL process are “sendMoney” and “depositWithdraw”. Any unknown transaction type should be revoked. The test sends a valid SOAP request with correct login credentials but with wrong transaction type and checks
that the response is a valid SOAP response with the message that the transaction type is incorrect.
TC3: The BPEL process should deny using more money than a person actually has. The test sends a valid bank operation request with an amount bigger than the available for the user and checks that the money transfer is being revoked. TC4: The test sends a valid SOAP request with “sendMoney” transaction type and verifies that the transaction has been executed successfully. The test also asserts the money conversion has passed successfully by verifying the BPEL variable: SendMoneyIn and the amount in euro.
TC5: The test sends a valid SOAP request with “depositWithdraw” transaction type and verifies that the transaction has been executed successfully.
TC6: The BPEL process should revoke any attempts to send money to nonexistent recipients. The test sends a valid request with invalid recipient identifier and checks that the transaction was not successful.
TABLE III.
SOAP request element
Test case recipientUserId
amount
country
130.00
Bulgaria VISA
BGN
TC2 4
130.00
Bulgaria MASTER
BGN
TC3 4
1300000.00 Poland
VISA
PLN
TC4 4
10.00
USA
MASTER
USD
TC5 4
10.00
Poland
MASTER
PLN
TC6 2
10.00
USA
VISA
USD
The test case outputs are presented in Table IV. The table shows assertions defined for each test case as well as their statuses after test case execution. TABLE IV. Test case
TEST CASE RESULTS
Assertion description HTTP Status Code (200) Response Time = (650 ms)
TC1
SOAP response valid Contains (case sensitive: "true", is regex: "false") text "Incorrect" in Response Response XPath Equals (xpath: "Envelope/ Body/bankOperationResponse/response/test ", value:"Incorrect username or password", case sensitive: "true", is regex "false") HTTP Status Code (200) Response Time (700 ms) SOAP response valid Not SOAP Fault
TC2
Figure 3. SOAP request template
The actual values of SOAP body elements for each test case are presented in Table II and Table III. TABLE II. Test case
TEST CASE INPUT DATA SOAP request element
transactionType
username
[email protected]
XPath Exists in Response ("descendant::bankOperationResponse") Response XPath Equals (xpath: "descendant::test", value:"Invalid Transaction Type", case sensitive: "true", is regex "false") Not Contains (case sensitive: "false", is regex: "false") text "success" in Response HTTP Status Code = 200
password
WRONG PASSWORD
paymentType currency
TC1 4
Fig. 3 shows the format of the SOAP requests, which is sent to the BPUT during test case executions. ... ... ... ... ... ... ... ...
TEST CASE INPUT DATA
TC3 SOAP response valid
Assertion Result
Status
Value: “200”
Passed
Actual time: 572 Passed ms Valid SOAP Passed response Value found
Passed
Value: “Incorrect username or password”
Passed
Value: “200”
Passed
Actual time: 572 Passed ms Valid SOAP Passed response No SOAP Fault Passed in response XPath found
Passed
Value: “Invalid Transaction Type”
Passed
Value missing
Passed
Value: “200”
Passed
Valid SOAP response
Passed
TC1
sendMoney
TC2
INVALID
[email protected] TRANSACTION TYPE
johnspass
Not Contains (case sensitive: "false", is regex: "false") text "success" in Response
Value found
Failed
TC3
sendMoney
[email protected]
johnspass
HTTP Status Code = 200
Value: “200”
Passed
TC4
sendMoney
[email protected]
johnspass
Response Time = 600 ms
TC5
depositWithdraw
[email protected]
johnspass
SOAP response valid
TC6
sendMoney
[email protected]
johnspass
Not SOAP Fault
TC4
Actual time: 517 Passed ms Valid SOAP Passed response No SOAP Fault Passed in response
Test case
Assertion description Contains (case sensitive: "true", is regex: "false") text "success" in Response Response XPath Equals (xpath: "Envelope/Body/bankOperationResponse/r esponse/test", value:"Transaction was successfully executed", case sensitive: "true", is regex "false") BPEL variable "SendMoneyIn" XPath Equals (xpath: "message/part/sendMoney/amount", value:"5.088205128205129", case sensitive: "true", is regex "false")
Value found
Status Passed
Value: “Transaction Passed was successfully executed” Value: ”5.08820512820 Passed 5129” Value: “200”
HTTP Status Code = 200
Passed
Actual time: 540 Passed ms Valid SOAP Passed response No SOAP Fault Passed in response
Response Time = 600 ms SOAP response valid TC5
Assertion Result
Not SOAP Fault Contains (case sensitive: "true", is regex: "false") text "success" in Response Response XPath Equals (xpath: "Envelope/Body/bankOperationResponse/r esponse/test", value:"Transaction was successfully executed", case sensitive: "true", is regex "false")
Value found
Value: “Transaction Passed was successfully executed” Value: “200”
HTTP Status Code = 200
Passed
Passed
by various tools and frameworks, there is still lack of tool support for testing SOA implementations. The presented solution in this paper provides functionality for identification of web service operations as well as BPEL variables in case of business process testing, generation of SOAP request templates, data driven testing, definition of assertions at different levels (HTTP, SOAP and BPEL variable), and execution and management of test cases. As a part of TASSA framework, it contributes to achieve end-to-end testing of BPEL orchestrations, which is a challenging task, given the distributed nature of the system under test. Since the tool has its own GUI and is fully integrated with NetBeans IDE, it can be used outside TASSA framework automating functional testing of web services as well as business processes, described with WS-BPEL. The future work includes further improvement of the TCGET in two directions: support of more types of assertions and implementation of GUI allowing usage of the tool as standalone application outside of NetBeans IDE. In addition, we will evaluate the performance of the proposed solution in comparison with other existing testing tools for web services. ACKNOWLEDGMENT This work has been supported by the National Scientific Fund, Bulgarian Ministry of Education and Science under grant agreement number DО02-182.
Actual time: 442 Passed ms
TC6 Response Time = 600 ms Not Contains (case sensitive: "false", is regex: "false") text "success" in Response
Value found
Failed
REFERENCES [1] [2]
The experiment results show the ability of the TCGET to facilitate testing of SOA implementations by supporting variety test case assertions. As is shown in Table III, TCGET provides assertions related to HTTP status code, SOAP response, content of SOAP body and BPEL variable, and Response time. It is worth noting that the last one concerns the non-functional testing of the BPUT and is not provided by Oracle BPEL test framework, BPELUnit and SoapSonar. Furthermore, three types of assertions for content verification of the SOAP body and BPEL variable are provided. Two of them are based on XPath expressions. Additionally, all supported assertions can be negated. For comparison, Oracle BPEL test framework supports only three assertions, namely Simple Value Assert, XML Assert and Activity Executed Assert. The BPELUnit provides only XPath assertions, which consist of an XPath query and an expected value. Rich functionality with respect of assertions is presented in SoapUI. Since the main purpose of the SoapUI is to test web services, the assertions over BPEL variables are not supported. VI.
[3]
[4]
[5]
[6]
[7]
[8]
CONCLUSION
The present work addresses test case generation, execution and management on single web services and BPEL orchestrations. While traditional software testing is supported
[9]
soapUI v.2.5.1, Project Documentation, 2009, http://www.whiteboxqa. com/StudentMaterial/Books/D%20-%20 soapui.pdf L. Dikmans, “Testing BPEL in the Real World,” 2007, http://www.oracle.com/technetwork/issue-archive/2007/07-nov/o67bpel100400.html. Parasoft Corporation, “Parasoft SOAtest Best Practices,” http://www.jameslewiscoleman.info/jlc_stuff/project_research/SOAtest_ Best_Practices.pdf. Ph. Mayer and D. Lübke, “Towards a BPEL unit testing framework,” in Proceedings of the 2006 workshop on Testing, analysis, and verification of web services and applications (TAV-WEB’06), July 17, 2006, Portland, Maine, USA, pp. 33-42. D. Esposito, “SOAPSonar Measures the Distance Between Your Web Services and the Real World,” 2006, http://www.codeproject.com/ Articles/15182/SOAPSonar-Measures-the-Distance-Between-YourWeb-S. Morris E., W. Anderson, S. Bala, D. Carney, J. Morley, P. Place, S. Simanta, “Testing in Service-Oriented Environments,” Technical Report CMU/SEI-2010-TR-011, Carnegie Mellon University, 2010. Spassov I., D. Petrova-Antonova, V. Pavlov, S. Ilieva, “DDAT: Data Dependency Analysis Tool for Web Service Business Processes,” Int. Workshop on “Software Quality” within ICCSA, June 20-23, 2011, Santander, Spain, Part V, LNCS 6786, Springer, pp. 232-243. Manova, I., D. Manova, V. Pavlov, S. Ilieva, D. Petrova-Antonova, “Fault Injection Testing of Web Service Business Processes,” International Journal on Information Technologies & Security, Year 3, No. 2, ISSN 1313-8251, 2011, pp. 3-12. Pavlov V., B. Borisov, S. Ilieva, D. Petrova-Antonova, “Framework for Testing Service Compositions,” Proceeding of the 12th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing , Timisoara, Romania, September 23-26, 2010, pp. 557-560.