Open Web Services Testing

Open Web Services Testing

Nabil El Ioini, Alberto Sillitti

Free University of Bolzano, Faculty of Computer Science Piazza Domenicani 3, 39100 Bolzano, Italy {nelioini, asillitti}@unibz.it

Abstract—The advance in web services (WS) and service oriented architecture (SOA) applications has led to the development of new types of systems in which different heterogeneous components and platforms can connect and collaborate to solve a business problem. This advancement adds a new level of abstraction to the existing stack of technologies and development methodologies, but it adds also new challenges. One sucha big challenge is testing. From the client prospective, a Web Service is a black box that needs to be tested before being used, mainly to confirm the claims of services' providers about the quality of their services.By calling a WS, we delegate part of our business logic to an external provider to do it for us.Thus, we have no control of what could happen during the execution of that part of the business logic. To this end, many testing approaches and techniques have been proposed in the literature to address various aspects related to WS testing. In our effort to improve WS testing infrastructures, we propose a framework forservice integrators to collaborate during Web Services testing by making test suites open to the public and share testing results. Keywords: Web Services; Testing

I.

INTRODUCTION

The advance in web services (WS) and service oriented architecture (SOA) applications has led to the development of new types of systems in which different heterogeneous components and platforms can connect and collaborate to solve business problems. This advancement adds a new level of abstraction to the existing stack of technologies and development methodologies, but it adds also new challenges. One of these challenges is testing. In terms of web services, we are facing mainly two issues concerning testing. The first issue is that the only type of testing we talk about in a realistic scenario is black box testing. This is mainly due to the fact that it is very unlikely that service providers give access to their clients to check the services’ code [19]. The second issue is that service clients do not own services; instead,clients can just invoke them, which make it very difficult to create a controlled environment for testing and identifying defects [18]. Testing from the client prospective is more complex for WS because while in web applications the application itself is the front end that is given to the client to use,WS are meant to be integrated with other systems. This puts the integrators at the position of the client. Integrators, could be

automated tools that are configured to automatically integrate services (this is one of the goals SOA based systems want to achieve), however, at the current state, we still find that WS are integrated manually. In a realistic scenario, service integrators identifythe potential services, compare the business needs of their system with the offered services, verifythat servicesdo what they are supposed to do, then formulate aService Level Agreement (SLA) [2] with the providerbefore starting using the service. In this scenario,integrators need a way to verify how services function. This could beachieved simply by reading the specification, however, most of the times, they need to invoke the services to make sure that they work.This leads us to ask questions about how services could be verified. One way that has been proposed in the literature and used in the industry is through testing [1]. Testing gives service integrators the possibility to invoke servicesand validate the expected results [1]. A major limitation of the existing testing techniques is that they do not take advantage of the openness of the Internet[21]; most of the techniques consider WS testing from the prospective of a single user [11]. Web services are software components that are publically accessible on the Internet [16][17]; once a service is deployed communitiesare built around these services (the clients using those services) just like web applications. In this paper, we want to explore the testing advantage we could have by getting these communities involved in the testing process. Mainly, we are interested in seeing how these communities could help building test suites. Traditional tools that are used in web applications include forums and blogs. However, web applications developers started integrating more support by including reputation systems[6]. However, they do not provide testing capabilities because web applications are meant to be used by the clients who have no time norknowledge for testing the applications. SinceWS are meant to be used by service integrators, we assume that they have the minimum knowledge about testing, so we provide them with a simple specification to write their test suites, which are open to the community and can be shared among them. Additionally, the test suites can be rated by the different members of the community, to help to identify the most useful scenarios used by each WS.In this paper we are presenting the architecture and technologies we have selected to build the

framework. However, we are still working on the experimental part, which will have a more valuable contribution in terms of demonstrating the benefit of using an open approach to test WS. The remaining of the paper is organized as follows: After discussing the related work in the next section, we present the approach followed to realize the proposed framework. The implementation of the framework is discussed in Section 4. Then a motivating scenario is described in Section 5. Finally, Section 6 draws the conclusions and presents future work. II.

RELATED WORK

There are mainlytwo research areas connected to our work. The first one is extending existing infrastructures to add more testing capabilities for WS. The second one is test cases generation. The idea of extending the existing specifications or infrastructures to support testing for WS and SOA based systems has attracted many researchers. In fact, it has been the case of many technologies that as time goes new extension are added to include new features. Since the beginning of WS testing different approaches have been proposed, some of them were the results of extending existing methods[9]. One of the first methods that has been proposed and has attracted many is extending a Universal Description Discovery and Integration (UDDI) server. The first attempt to extend UDDI to include mechanisms to help the testing process was in [7]. The authors propose to enhance the UDDI server by adding two steps: one before registering the service check-in and one before invoking the service check-out. The check-in process consists of executing a set of test cases before allowing a service to be registered and only in case the tests pass successfully, the service can be registered. The check-out consists of executing the same set of test cases but this time by the client to verify if a service satisfies the client's needs before using it. Our approach differs since the test cases are generated automatically rather than being supplied by the service provider, which is the case of [7]. Other approaches have used to same idea. [9][10] have built and extend the same idea by storing other information in the registry server such as using graph transformation systems to test individual services [8]. Our approach has some similarities with the mentioned studies since we also use the registry server as a repository. However, the type of information we store is different. We use the registry to collect generated test cases, execute them, and store their results as certificates. Another benefit of our framework is that we take the clients’ votes and use them as a reputation score for the services, which represents the service quality from the clients’ point of view. The second related area is how to generate test cases and test suites. Most of the existing techniques rely entirely on the specifications provided by the WS providers to generate test cases. However, since the service providers do not know the contexts in which the services will be used, they tend to keep a more genetic specification. Examples of studies generating test cases from a published specification are [4] [5]. Our approach differsbecause we relayon real users to

submit test scenarios, users who are going to use the service. This creates more realistic test suites as well as it adds the oracles without the need for more expressive specifications that are still not in use by today’s web services. III.

APPROACH

Our work on WS testing has started by trying to understand how the existing testing techniques are used to test WS. We performed a literature review on the widely used techniques and we found that there is a common pattern that beenused by many techniques such as [3] [4]. Figure 1 shows this pattern.

Figure 1 WS testing pattern reported in the litereture

There are mainly five steps that most of the techniques have in common.  Data extraction. Extracting all necessary information from the used specification such as: operations parameters, data types, operations dependencies;  Test data generation. Generating test data based on the information we extracted. Additionally, data could be provided manually by the tester;  Test cases generation. It is a straight forward task consisting of generating Web Services calls;  Test cases execution. Executing the service calls is also a straight forward task;  Results analysis. Validating the responses coming back from the service. It is one of the hardest task to automate and, in many approaches, it has been done manually or using some heuristics [15].

Starting from this pattern, we have developed a tool to generate and execute test cases automatically. We have used the Web Services Description Language (WSDL) as the main specification to generate test cases [20]. The tool architecture is depicted in Figure 2. The tool supported different techniques such as Boundary values [12], mutation testing [13], and testing services with persistency data [14].

handles users’ feedbacks and rank the test suites according to the feedbacks scores. 3) The user interface, which helps users create test suites, brows existing ones, view results and submit feedbacks.

Figure 2Overview of the tool for generating test cases

However, while experimenting with the tool, we realized that it was limited in many aspects. The first limitation was that the number of test cases generated increased very fast due to all the combinations of the input parameters most of the techniques use [4]. The second limitation was that we did not have ways to generate test oracles automatically. The only oracles we were able to use were whether the test cases generate a runtime exception or not, however, there was no way to find logical errors in the targeted services. Additionally, since the tool was a stand-alone application, if the tool was used by two people who want to test the same service, the same test cases will be executed twice. To address this concerns, we have followed a different approach. The new approach helps solving the issues we mentioned above by making testing a collaborative activity among service integrators, but on the other hand the level of automation decreases. However, since our goal is to generate more robust test cases, automation does not influence our approach at this stage. The new framework is basically a server for managing test suites and their corresponding data such as execution traces and ratings. The test suite server receives test suites submitted by services integrators, execute them and store the execution traces. There are mainly three blocks that make the test suite server. 1) The test suites management, which is responsible of storing and executing test cases, then collecting the execution traces. 2) Thefeedback management, which

Figure 3 Test suite server overview

Figure 3 presents the general overview of the framework. The server is composed of a repository to store test suites and all their related information, and it provides a verity of features to the service integrators: Data repository: The test suite server acts as a repository for all the information concerning test suites. This information includes test suites, test execution traces and test suites ratings. The collected information can be requested by any member of the community; Test Cases Manager: All the test suites submitted are converted to services’ calls then are executed against the specified service. This feature hides all the complexity from service integrators and let them focus on their integration task; Feedback Manager: Every time a user rates a test suite to be useful, the overall rating of that test suite increases, rising the priority of that test suite. This metrics is used to order the test suites when they are presented to the new users; User front end: to communicate with the test suite server, a web interface is used as a front end in which users can insert their test suite in the form of an XML specification, which is discussed later.

A. The test suite server The main reasonability of the test suite server is to hide all the complexity of generating service calls, executing them and storing the traces from the service integrator. The service integrator receives only the report about the overall results of the test suites execution and feedbacks from the other members of the community. The test cases management is performed in six consecutive steps as follows:  Storing Test suites.It consists of storing the submitted test suite in our database.  Data extraction.Extracting all necessary information from the WSDL specification submitted in the test suite such as: operations parameters, data types;  Test data generation. Generating test data based on the information provided in the test suite;  Test cases generation.It is a straight forward task consisting of generating Web Services calls;  Test cases execution.Executing the service calls is also a straight forward task;  Results validation.The response messages are validated against the expected values defined by the test suite. B. Test Suites Specification The whole process starts when a service integrator decides to test a specific WS. To do that, a test suite needs to be built to capture the test scenarios. We have developed an XML specification to define test suites. We have taken into account three aspects to include in the specification: 1. Simplicity: defining a simple specification to build test suites; 2. Dependencies: allow adding dependencies between the different operations, this is useful when a sequence of operations need to be called as part of the same scenario. 3. Oracles: give the possibility to define test oracles to validate the returned responses. The XML specification we came up with looks as in Figure 4. The test suite is composed of a set of test cases. Each test case targets one specific operation. The test case can have two optional attributes depends and config. If a test case a uses the depends attribute by defining that it depends on test case b, it means that before executing the test case a, b needs to be executed. Each test case can have at least one configuration, which means a combination of input values and expected values. Each configuration is assigned an id. If it is the case that we want to have a dependency only to certain configuration, the config attribute can be used. In the current specification only one configuration can be chosen. If the config attribute is not defined, all the configurations will be taken into account. For example, if we have two operations login and orderProduct, and each one of them has two configurations. If orderProduct depends on login and does not define the config attribute, we will have four test cases.

C. Feedback Collection Feedback collection is an important component in our architecture because it helps to understand which are the test suites that are more useful to the service integrators. This way, test suites can be prioritized based on the users feedbacks.We have used a simple feedback model. The “yes it is useful” model, in which user can say if the test suite was useful to them, meaning that probably they have the same needs that the test suite addresses. This strategy is used initially to start the experiment, however, other feedback models will be added as well. The Reputation Manager works as follows: 1. The service integrator browses the existing test suites; 2. At any point the service integrator could request the server to execute any of the existing test suites, or look as the history of execution of a particular test suite; 3. Then service integrators could rate a test suite if the results or the scenario is relevant to their own need. 4. Once a test suite is rated, the server recalculate the rank of test suites and for the next user to browse the test suites a new list with different order is presented.

Figure 4 Test suite specification

IV.

IMPLEMENTATION

To implement the test suite server, we have used some existing components to perform different tasks. The server is

a J2EE application running on top of Tomcat container. We have used WSDL4J1, which is set of libraries that allow the manipulation of WSDL documents and SoapUI 2 library to generate SOAP service calls and execute them. A. Testing For the testing block of the framework, we have two components. The first component manages the test cases generated automatically and the second component manages test cases generated from the users’ test suite specification. To implement the Testing components we have used three existing libraries: TestGen4J, WSDL4J, and SoapUI library. TestGen4J3 is a collection of open-source tools and libraries that automatically generates unit test cases. WSDL4J instead is set of libraries that allow the manipulation of WSDL document. SoapUI library is a set of components for testing WS. 1) Automatic test cases We have used part of TestGen4J to generate test data that is used for WS testing. In particular, the class TestDatageneration takes a method signature and generates test data based on a configuration file. The configuration file assigns the values that each data type could assume. For example, in case of boundary values test data an Integer data type could be configured to have the following values (2147483646, 2147483647, 2147483649, 0, -2147483646, 2147483647, -2147483649). The WSDL4J is used to manipulate WSDL documents and extract all the types of information that is needed to generate test data and test cases. SoapUI for now is simply used for services invocation and capturing the response messages. However, more functionality will be added since SoapUI provides much more functionality including WS mocking and load testing. The three components interact with each other in the following way:  The WSDL is parsed and the needed information is extracted (e.g., methods names, parameter types).  Test data is generated  Test cases are generated using the test data  Tests are executed and the results are reported 2) Test suite test cases An existing parser to parse the test suite XML specification and extract the information needed to generate service calls. WSDL4J has been used to extract information from the WSDL to help the user know the data types needed by the service as well as the operation names. Once we have these information, SoapUI, was used to generate SOAP service calls and execute them. Every time a call is executed, the result value is compared with the expected value defined by the test suite. It is important to mention that in our current

1 2 3

WSDL4J: http://sourceforge.net/projects/wsdl4j/ SoapUI: http://www.soapui.org/ TestGen4J: http://sourceforge.net/projects/spike-test-gen/

implementation only services with simple data types as parameters and return values are supported, however, we are working on supporting complex types as well. All the execution data is stored in a relational database. In the current implementation users can view the execution trace of every service, however, we are planning to give the possibility to download test traces as data sets, which could be used to perform some data mining on them. B. Feedbacks The feedback mechanism is implemented as a simple “yes it is useful to me” feedback. Every time users find a test suite that is suited for their need, they can rate it that it was useful to them. A simple JSP page handles the user ratings and store the received rates into a relational database. These ratings are used then to calculate the rank of every test suite. C. User interface In the current implementation, the user interface is a set of JSP pages that provide a simple interface for the users to perform several operations. Our goal is to build a tool that helps the users to construct test suites automatically. However, for experimentation purposes we are using JSP pages in which users can write their XML test suite specification manually. The user interface gives the possibility to browse the existing test suites V.

SCENARIOS

To give a better understanding on how the framework could be used, in this section we will illustrate the different concepts using a real scenario. The presented scenario covers the creation of test cases and all the other activities derived after that. By looking at the existing test suites, service integrators could learn the different patterns of using the WS of interest, such as what the sequences of operations calls. A. Server under test In this example, we will use a service called Bookshop to illustrate the functionality of the framework. The Bookshop service provides operations to handle ordering books online. It provides a WSDL composed of 5 operations defined as follows: 1. OrderBook(bookId): to order a specific book by providing the book ID. The operation returns a NIL if the book is not available; 2. CheckAvailable(bookId): check the availability of a book by providing its ID. True is retuned if the book is available, otherwise False is returned; 3. Login(username, password): authenticate to the service, The service keeps a session for each user logged in; 4. Logoff: closes the user session; 5. Payment(CreditCardInfo): takes the creditcard information to perform the payment. If the payment is authorized an error message (NOT AUTHORIZED PYMENT) is returned.

If the user tries to call any of the operations before login in an error message (NOT LOGGED ON) is returned. B. Scenario Our scenario covers the creation of a test suite and explaining how other service integrators can make use of it. In this scenario,the service integrator wants to test the operations in different sequences. The test suite may look as the following:

to them. In case they have different requirements that match partially this test suite, they can extend this one by adding their specific requirements.

TestCase 1: Login (testuser, testpass) CheckAvailable(100) OrderBook(100) TestCase 2: Login (nonexistinguser, testpass) CheckAvailable(100) OrderBook(100) Figure 6 Service calls generated from the test suite

For example, if a new user wants to include credit card payment in the test suite, this can be done by creating a new test suite which extends the existing test suite and add the a new test case. The results might look like the following in Figure 8. By doing this, we do not only eliminate replicating test suites, but also we help the service integrators to reuse exiting suites as well as learn from each other.

TestCase 1:

Figure 5 Original test suite

Login (testuser, testpass) PASS CheckAvailable(100) PASS OrderBook(100) PASS TestCase 1: PASS

In this scenario, the service integrator wants to test a particular sequence of operations with particular set of parameters. The first thing to notice is that none of the techniques that we have tried to automatically generate test cases could perform this. Additionally, this sequence of operations gives a real instance of a service call. By submitting this test suite scenario to the test suite server, the server generates two test cases each one of them is composed of three service calls (Figure 6). Then server executes the generated test cases and captures the traces of the execution. Then a report is presented to the user (Figure 7).The second test case did not pass because since the login was wrong none of the other operations could be accomplished. The login passed because the user expected to have a false return value. When other users see this test suite, if they have similar requirements, they can simply rate this test suite to be useful

TestCase 2: Login (nonexistinguser, testpass) PASS CheckAvailable(100) FAIL expected value False actual value NIL OrderBook(100) FAIL expected value False actual value NIL TestCase 2: FAIL Figure 7 Test execution results

VI.

CONCLUSION AND FUTURE WORK

The contribution of our work is to overcome the limitations of the WSDL and to make use of the user communities to strengthen the test cases generation process by adding real scenarios not only the automatic ones which most of the time could not represent what the final user needs. By combining automatically generated tests with tests provided by the users we could increase the level of testability. Additionally, we intend to perform a large scare experiment in which we will use the framework on public services and release the datasets of our traces. This is helpful to use for data mining. Because data sets related to WS testing are difficult to find.

[3]

[4] [5]

[6] [7]

[8]

[9] [10]

[11]

[12]

[13]

[14]

[15]

[16]

[17]

[18]

[19]

Figure 8 Updated test suite

[20]

REFERENCES [21] [1] [2]

Baresi, L. andDi Nitto, E., 2007. "Test and Analysis of Web Services". Springer-Verlag New York, Inc., Secaucus, NJ, USA. Sahai, A.; Machiraju, V.; Sayal, M.; Moorsel, A. P. A. v. & Casati, F., 2002, "Automated SLA Monitoring for Web Services", DSOM

'02: Proceedings of the 13th IFIP/IEEE International Workshop on Distributed Systems: Operations and Management, 28-41. Bartolini, C.; Bertolino, A.; Marchetti, E. & Polini, A., 2008, "Towards Automated WSDL-Based Testing of Web Services", ICSOC '08: Proceedings of the 6th International Conference on Service-Oriented Computing, 524-529. Offutt, J.; Xu, W., 2004,"Generating test cases for web services using data perturbation", SIGSOFT Softw. Eng. Notes 29 : 1-10. Xu, W. Offutt, J. & Luo, J., 2005, "Testing Web services by XML perturbation", Proc. 16th IEEE International Symposium on Software Reliability Engineering ISSRE 2005, 10pp.-266. Xu, Z.; Martin, P.; Powley, W. & Zulkernine, F., "ReputationEnhanced QoS-based Web Services Discovery", ,july 2007, 249 -256. Tsai, W. T.; Paul, R.; Cao, Z.; Yu, L. & Saimi, A., 2003, "Verification of Web services using an enhanced UDDI server", Proc. Eighth International Workshop on Object-Oriented Real-Time Dependable Systems (WORDS 2003),15-17 Jan. 2003, 131-138. Heckel, R., Mariani, L.: 2005, "Automatic conformance testing of web services". In: Cerioli, M. (ed.) FASE 2005. LNCS, vol. 3442, pp. 34–48. Springer, Heidelberg. Bertolino, A. & Polini, A., 2005, "The audition framework for testing Web services interoperability", 134 – 142. Tsai, W. T.; Wei, X.; Chen, Y.; Xiao, B.; Paul, R. & Huang, H., 2005, "Developing and assuring trustworthy Web services", Proc. Autonomous Decentralized Systems ISADS 2005,4-8, 43-50. Canfora, G.; and Di Penta, D.,2006, "Testing services and servicecentric systems: Challenges and opportunities". IT Professional, 8(2):10–17. Reid, S.C.; , 1997, "An empirical analysis of equivalence partitioning, boundary value analysis and random testing," Software Metrics Symposium, 1997. Proceedings., Fourth International , vol., no., pp.64-73, 5-7. Almeida, L.F.J.D.; and Vergilio,S.R., 2006, "Exploring Perturbation Based Testing for Web Services,"Proceedings of the IEEE International Conference on Web Services, IEEE Computer Society, pp. 717-726. Sinha, A.; and Paradkar, A., 2006. "Model-based functional conformance testing of web services operating on persistent data". In Proceedings of the 2006 workshop on Testing, analysis, and verification of web services and applications (TAV-WEB '06). Martin, E.; Basu, S.; and Xie,T., 2007, "Automated Testing and Response Analysis of Web Services,"Web Services, IEEE International Conference on, Los Alamitos, CA, USA: IEEE Computer Society, pp. 647-654. Predonzani P., Sillitti A., Vernazza T., “Components and Data-Flow Applied to the Integration of Web Services”, The 27th Annual Conference of the IEEE Industrial Electronics Society (IECON’01), Denver, CO, USA, 29 November - 2 December 2001. Sillitti A., Vernazza T., Succi G., “Service Oriented Programming: a New Paradigm of Software Reuse”, 7thInternational Conference on Software Reuse (ICSR-7), Austin, TX, USA, 15 - 19 April 2002. Janes, A.; Scotto, M.; Pedrycz, W.; Russo, B.; Stefanovic, M.;Succi, G., 2006,"Identification of defect-prone classes in telecommunication software systems using design metrics". Inf. Sci. 176(24): 3711-3734. Pedrycz, W.; Succi, G.; Musílek, P.; Bai, X.,2001, "Using selforganizing maps to analyze object-oriented software measures". Journal of Systems and Software 59(1): 65-82. Damiani, D; El Ioini, N.; Sillitti, A.;Succi, G., 2009. "WSCertificate". In Proceedings of the 2009 Congress on Services - I (SERVICES '09). Manzalini, A., 2008. "Tomorrow's Open Internet for Telco and Web Federations," International Conference onComplex, Intelligent and Software Intensive Systems, 2008. CISIS 2008. , vol., no., pp.567-572, 4-7 March 2008.