Architecture-Based Testing of Service-Oriented ...

Architecture-Based Testing of Service-Oriented Applications in Distributed Systems Changsup Keum, Sungwon Kang, Myungchul Kim Department of Information and Communications Engineering Korea Advanced Institute of Science and Technology 291 Daehak-ro, Yuseong-gu, Daejeon, 305-701, Republic of Korea {cskeum, sungwon.kang, mck}@kaist.ac.kr

Abstract Context: Testing distributed service-oriented applications (SOA) is more challenging than testing monolithic applications since these applications have complex interactions between participant services. Test engineers can observe test results only through a front service that handles request messages sent by test engineers. Message exchanges between participant services are hidden behind the front service and cannot be easily observed or controlled through the front service. For this reason, testing SOAs suffer from limited observability and controllability problem. Objective: This paper proposes a new test method that is architecture-based and exploits interaction architecture of a SOA. The proposed test method alleviates the limited observability and controllability problem by employing test architecture, thereby facilitating test execution and analysis through monitoring and controlling message exchanges. Method: Our proposed method derives an interaction architecture from the specification of a SOA. Test architectures can be designed from the derived interaction architecture by extending it with additional test elements. At the same time, architecture-neutral test scenarios are automatically generated from the test model that is constructed from the specification. Our method combines test architecture information with the test scenarios to obtain architecture-enabled test scenarios under the selected test architectures. Finally, architectureenabled test execution and analysis are conducted in the real network environment. Results: The efficacy of the proposed method is demonstrated with an industrial case study, which shows that it is practical and effective for testing SOAs. Even though our method increases an additional test generation effort owing to test architecture, it is counterbalanced by higher fault detection rate and faster fault locating time. Conclusion: The main benefit of our approach is that using test architecture it enhances testability of SOA by increasing observability and controllability through monitoring and controlling message exchanges. Our architecture-based test method enables test engineers to detect faults efficiently and also reduce fault locating time significantly. Keywords:

Service-Oriented

Application

Testing,

Service

Composition,

Test

Architecture,

Architecture-Based Testing

1

1. Introduction Service-oriented applications (SOAs) consist of related web services offering business functionality through well-defined interfaces specified in Web Services Description Language (WSDL) [1]. A service can either be atomic or composite. An atomic service is usually implemented in the traditional programming languages such as Java and Python and a composite service is typically composed by using composition languages such as Business Process Execution Language (BPEL) [2]. Service composition is a widely used technique that enables fast development of distributed SOAs. There are two complementary styles of service composition: orchestration and choreography [3]. The orchestration focuses on describing a sequence of process actions to be carried out by a composite service from the perspective of the service. In contrast, the choreography captures a flow of interactions among all involved services from the global perspective and exposes the externally visible behaviors of all collaborating services by means of service choreography specification. As the SOA development paradigm gains industry-wide acceptance and usage, demand on rigorous testing approaches for the paradigm has increased as well. Many formal approaches for SOA testing that have been proposed until now can be classified into two categories: the conformance testing approach and the white-box testing approach. Research in [4,5,6] adopted protocol conformance testing techniques for rigorous SOA testing and augmented services under test with various formal behavior models for automatic conformance test generation. Research in [7,8] applied white-box testing techniques to SOA testing and used data-flow or controlflow testing techniques for service composition testing. Despite these efforts, testing of distributed SOAs still remains a challenging problem. In contrast with the monolithic application running on a single system, a SOA executing on distributed systems exhibits interaction architecture that reflects the communication relationship among participant services. However, the previous research overlooked the effect of interaction architecture on the testing of SOA, which is induced by the characteristics of service composition. Without test architecture, test engineers for SOAs can observe test results only through a front service that handles request messages sent by test engineers. Message exchanges between participant services are hidden behind the front service and cannot be effectively observed or controlled. For this reason, the previous approaches provided only limited observability and controllability.

2

To address this problem, we propose an architecture-based test method that exploits interaction architecture for testing of a SOA under test (AUT). Our method consists of three main phases: test architecture design, test generation, and test execution. In the architecture design phase, an interaction architecture of a SOA is derived by analyzing message exchanges between participant services described in the composition specification of a SOA. Test architectures are then designed from the derived interaction architecture by adding test elements to it along the communication channels of the SOA. In the test generation phase, architecture-neutral test paths are generated from a test model that is constructed from the composition specification of the SOA. A set of architecture-enabled test scenarios are then generated by combining test architecture information with architecture-neutral test paths under the designed test architectures. In the test execution phase, test architecture configurations for the designed test architectures are established before test execution, and then the generated architecture-enabled test scenarios are applied to the testing of AUT. By executing the architecture-enabled test scenarios and analyzing the test results, a test engineer can check whether the behavior of the AUT conforms to the composition specification of it. The proposed test method alleviates the limited observability and controllability problem by using test architecture and thereby facilitates test execution and analysis through monitoring and controlling message exchanges along the communication links. To demonstrate of the efficacy of our approach, we applied it to an industrial case study and the results of the experiment indicate that our method helps test engineers detect faults and substantially reduces fault locating time. To our best knowledge, our approach is the first one that attempts to base SOA testing on the interaction architecture of SOA. The rest of this paper is organized as follows: In section 2, the problem domain and its major challenges are described; In section 3, our architecture-based test method are introduced with an example; In section 4, our method is evaluated through an industrial experimental case study; In section 5, related works are discussed; and, finally, section 6 concludes the paper.

2. Motivation A distributed SOA typically has complex message exchanges among participant services, composite or partner services, to accomplish its business goal. Participant services are independently deployed and

3

managed by different organizations in distributed networks, and are often operated by third parties and business partners. Fig.1 illustrates the problem domain of customer relationship management (CRM). When test engineers try to test a SOA in this environment, two problems can be encountered: 

Limited Observability. In SOA testing, test engineer can only observe test results through the test interface of a composite service that communicates with the test engineer. Such a service is called a front service because it is the first service that receives input messages from a test engineer and then initiates interactions among participant services for handling the input messages. After a sequence of interactions among participant services is finished, the front service sends output messages to the test engineer. Message exchanges between participant services hidden behind the front service cannot be observed by the test engineer during the testing although the interactions among hidden participant services may lead to serious errors when they interwork with each other. It is very difficult for test engineers to assess accurately the correctness of SOA by just observing the output message from the front service.

Test Engineer

CRM Composite Service

CS1

Partner Service

(Front service)

P11

P12

Organization A

CS2

P21

CS3 P22

P31

P32 Organization C

Organization B

Service-oriented application = a group of related web services

Fig. 1. The CRM problem domain. 

Limited Controllability. For acquiring accurate test results, the test engineer should validate the expected behavior of AUT by exercising all possible interactions among participant services with respect to the specification of the AUT. Some interactions are hard to execute since test engineer

4

has no control over the participant services. For example, interactions that contain service fault message exchanges cannot be controlled by just sending a sequence of input messages through the front service. Such exceptional interactions can be controlled by sending messages with invalid input values to the hidden participant services or intercepting messages targeted to the hidden participant services to modify their contents. Since test interfaces of the participant services except for the front service are invisible to the test engineer, it is nontrivial to execute such interactions among participant services. Compared to service orchestration, service choreography model suffers more from limited observability and controllability since multiple composite services and their partner services can be intertwined in the interactions of SOA. The service-oriented CRM application shown in Fig. 1 shows such a case. It has three composite services each with its hidden partner services that are operated by different organizations in distributed networks. In this application, the front composite service is visible to the test engineer, but the other two composite services and all partner services are invisible. With our architecture-based test method, the limited observability and controllability problem associated with functional testing of SOAs in distributed systems can be mitigated.

3. Architecture-Based Test Method In this section, we describe the overview of our architecture-based test method and each step of it with an example application. Our proposed method consists of three main phases as shown in Fig. 2. The first phase, Test Architecture Design, takes the service composition specification of the AUT as an input, and derives an interaction architecture by analyzing message exchanges between participant services in the specification. A service composition specification generally specifies all message exchanges between participant services involved in a SOA. A service composition specification is usually specified in BPEL, Web Services Choreography Description Language (WS-CDL) [9], and message sequence charts. One or more test architectures can be designed from the derived interaction architecture by extending it with additional test elements to monitor and control service collaboration. The second phase, Test Generation, generates a test model from the composition specification and set of

5

architecture-enabled test scenarios (TS) to check functional correctness of the run-time behavior of the AUT in distributed systems. Once a test model is constructed that captures flows of interactions among participant services in distributed systems, TSs that are test architecture independent are derived from the test model. The test architecture design in the first phase and test model construction in the second phase may be performed in parallel. Then our test generation method combines test architecture information with the TSs to obtain a set of architecture-enabled TSs that can trigger all possible interactions among participating services under the selected test architectures. In the third phase, Test Execution, the generated architecture-enabled TSs are applied to the testing of the AUT and finally a test report is generated. Test architecture configurations for realizing the selected test architectures should be established in the real network environment before test execution. By executing the architecture-enabled TSs and analyzing the test results, test engineer can check whether the behavior of the AUT conforms to its specification. Test architecture supports test execution and analysis by allowing interactions among hidden participant services to be monitored and controlled. In the following subsections, each step of our architecture-based test method in Fig. 2 is explained in detail one by one.

Service Composition Specification

Start 1. Interaction Architecture Derivation

Test Architecture Design Interaction Architecture

2. Test Architecture Design Test Generation 3. Test Model Construction Test Architecture

Test Model

4. Architecture-Enabled Test Scenarios Generation Architecture-Enabled Test Scenarios

Test Execution 5. Architecture-Enabled Test Execution and Analysis

Input

Activity

Data

Workflow Dataflow

Output

Test Report

Stop

Fig. 2. Overview of our architecture-based test method.

6

3.1

Interactions Architecture Derivation

Service interaction architecture of a SOA captures all participant services and communication relationship among them and can be derived by analyzing the message exchanges between services in a given service composition specification. When a group of services interact to achieve a system level functionality, there are two basic interaction architecture styles, i.e. the orchestration architecture style and the choreography architecture style, depending on whether the service composition style is orchestration or choreography. In this paper, we assume that the number of participant services in AUT is three for simplicity of presentation although there can be any number of participating services. In the interaction architecture style illustrated in Fig. 3(a), a controller service has the special role of handling interaction between its partner services. In Fig. 3(a), Controller Service A has two channels and can communicate with Partner Service 1 and Partner Service 2. Via one channel exactly two entities can communicate with each other. In this interaction style, Partner Service 1 and Partner Service 2 can interact with each other only through Controller Service A. If a SOA has the orchestration interaction style, a test engineer can view only the interface of the controller service.

Service-oriented application

Service-oriented application Peer Service A

Controller Service A

Partner Service 1

Partner Service 2 Communication channel

(a) Orchestration interaction style.

Peer Service C

Peer Service B

Communication channel

(b) Choreography interaction style.

Fig. 3. Two architecture styles of service interaction. In the choreography interaction style illustrated in Fig. 3(b), each participant service of the SOA is called a peer service. A peer service can be an atomic service or a composite service that contains its partner services. In

7

general, the choreography interaction style consists of multiple services that are independently operated in distributed networks. If a SOA has the choreogrpahy interaction style shown in Fig. 3(b), a test engineer can view only the interfaces of controller services depending on the interaction architectures. To illustrate our method, we use as an example a simple e-business application composed of one controller service and two partner services. The two partner services are (1) a payment service, which authorizes and captures payment information, and (2) a delivery service, which arranges the delivery of the requested item with contact information. The controller service is a shop service that handles a customer’s request and responds to it by collaborating with partner services: Payment and Delivery. The customer interfaces only with the shop service. A BPEL speficiation consists of the three elements: partnerLinks, variables and activities. The element partnerLinks is used to define the relationship bewteen participant services. It has two role attributes: myRole that is a role of the controller service in interaction and partnerRole that is a role of the partner in interaction. The element variables declares data variables for each message in interaction. The behavior of process is defined by a sequence of activities such as sequence, receive, invoke, and reply to capture interaction with its partners. Fig. 4 shows a part of BPEL specification of the e-business application. By anlayzing the BPEL specification in Fig.4, the interaction architecture of the e-buinsess application in Fig. 5 is derived. Topology information specified in partnerLinks and message exchages information specified in activites are exploited to derive the interaction architecture. The interaction architecture has the same orchestration style shown in Fig. 3(a) because the BPEL specification describes service interaction between a controller and two partner services. In Fig. 5, the Shop service communicates with the partenr services, Payment and Delivery, through two communication channels A and B.

8

…… …… ……

Fig. 4. A part of BPEL specification of e-business application.

e-business

Shop A

Payment

B

Delivery

Fig. 5. Interaction architecture of e-business application.

9

3.2

Test Architectures Design

Test architecture consists of test elements involved in testing and their relationships. A test architecture for distributed SOAs is described using the following concepts: • application under test (AUT), which is the executable implementation of the distributed SOA to be tested • system under test (SUT), which is the service system on which the AUT runs • tester, which is the test element that initiates a test case and also accesses the results of a test run • controller service under test (CSUT) • partner service under test (PSUT) • point of control and observation (PCO), which is a test element that observes as well as intercepts, inserts and modifies interactions • point of observation (PO), which is a test element that only observes interactions • message link (ML), which is a means for exchanging possible messages between service under tests • (optional) test coordination procedure, which coordinates the activities between different tester elements • (optional) service stub, which is a local dummy service that emulates functionality of the service to be tested In this paper, we propose two types of test architectures that correspond, respectively, to the orchestration and choreography interaction architecture styles of SOAs. Test architectures can be obtained from the service interaction architecture styles by adding test elements such as tester, POs, and PCOs to monitor and control service collaboration. Test architecture and interaction architecture are different in that test architecture captures a physical view of AUT whereas interaction architecture shows a logical view of AUT. Fig. 6(a) and (b), respectively, illustrate test architectures for an orchestration architecture and an choreography architecture, which are obtained from the orchestration interaction architecture style and the choreography interaction architecture in Fig. 3(a) and (b). In Fig. 6(a), there are one tester Tester, one CSUT TestedController A, and two PSUTs, TestedPartner 1 and TestedPartner 2. TestedController A communicates asynchronously with TestedPartner 1 and TestedPartner 2. Message exchanges between CSUT and PSUT are monitored by PO and intercepted and possibly modified by PCO. There are no direct message exchanges between PSUTs. Tester communicates only with TestedController A asynchronously. PSUTs are invisible to the

10

test engineer unless PO or PCO is inserted in the MLs between CSUT and PSUT. Fig 6(b) has two testers, two CSUTs, and four PSUTs. Each CSUT communicates with its PSUTs via their respective MLs. As the test architecture has the choreography interaction architecture style, there is no central controller in this test architecture and there can be message exchanges between CSUTs. AUT

Tester

ML

SUT

PSUT

CSUT

PO or PCO

AUT

SUT 1 Tested Controller A

Tester

Test System

SUT 2

SUT 3

Tested Partner 1

Tested Partner 2

(a) A test architecture for orchestration. AUT Tester A

SUT 4

SUT 1

Tested Controller B

Tested Controller A Tester B

Test System

SUT 2

SUT 3

Tested Partner 1

Tested Partner 2

WUT B 5 SUT WUT B (PAYMENT) Tested (PAYMENT)

Partner 3

SUT 6 Tested Partner 4

(b) A test architecture for choreography. Fig. 6. Test architectures for service-oriented applications.

In order to observe and control message exchanges along message links between participant services, test architectures of AUT are derived from the interaction architecture of the AUT by augmenting it with various test elements. There may be various test architectures resulting from the deployment topology of test elements, i.e. PCOs and POs. Generally, when designing test architectures, we can control the expected test coverage with POs and PCOs. For execution of test cases, we need to consider that a PCO is an expensive test element. In contrast with a PO that increases the observability by monitoring message exchanges along a message link, a PCO enhances

11

controllability over hidden services, by intercepting, inserting and modifying message exchanges. By adding more PCOs, the controllability over participant services can be increased.

e-business

SUT 1 Shop

A

Tester

SUT 2 Test System

Payment

B

SUT 3 Delivery

Fig. 7. Test architecture for e-business application. A generic test architecture is a template test architecture that can be instantiated to specific test architectures by adding PCOs or POs to its message links. For example, a generic test architecture of the ebusiness application in Fig. 7 is obtained from the interaction architecture of e-business application in Fig. 5. In this case, we may have four test architectures for two message links A and B: (PO1, PO2), (PCO1, PO2), (PO1, PCO2), (PCO1, PCO2). Among the four test architectures, the one with two PCOs has the highest test coverage for the controllability over the partner and delivery service. Let’s call the test architecture with two POs (PO1, PO2) TA1 and the test architecture with two PCOs (PCO1, PCO2) TA2.

3.3

Test Model Construction

For test model construction, we use the extended control flow graph (ECFG) [10] to describe a test model that captures flow of interactions among participant services. An ECFG consists of CFGs (Control Flow Graphs) of participant services and a representation of the communication flows between them. In the ECFG test model, each node corresponds to an input or an output event of a participant service; a solid edge denotes control flow between nodes; and a dotted edge denotes communication flow. An ECFG test model is constructed from a specification of an AUT. In a test model, it is assumed that there are a fixed number of participant services and each participant has a respective control flow. The ECFG test model is

12

constructed from the specifications described in service composition languages such as BPEL and WSCDL. If G is a test model represented as an ECFG, the functional correctness of an AUT with respect to G can be determined by comparing the input and output messages when G is traversed. More specifically, basic activities in BPEL such as receive, reply, and invoke can be transformed into equivalent input or output messaging events in an ECFG test model. Structured activities such as sequence and pick in BPEL are transformed into control and communication flow of input and output events in ECFG. Fig.8 shows the ECFG test model of the e-business application that is constructed from the BPEL specification shown in Fig. 4. As an example of transformation, the first activity of the sequence element in Fig. 4, which receives a message using the variable ItemOrder via the PartnerLink Shop, is transformed into the input event S1 in the Fig. 8.

Fig. 8. The test model of e-business application.

3.4

Architecture-Enabled Test Scenarios Generation

In our architecture-enabled test generation, we consider test architectures as well as test scnearios from

13

the test model. To do so, a set of architecture-neutral test paths that represent sequences of interactions among participant services are automatically generated by applying the coverage criterion all-edges to the ECFG test model. The all-edges coverage is selected to be the basic coverage criterion. After a set of test paths has been generated, the designed test architectures should be selected according to expected test coverage with POs and PCOs. A test architecture can be used for generating a set of architecture-enabled test paths that are sequences of input or output events based on the selected test architectures. For generating architecture-enabled test paths, we propose a test generation algorithm that extends the Chinese Postman (CP) algorithm [11], which combines test architecture information with architecture-neutral test paths to obtain the architecture-enabled test paths by adding test elements to them. More precisely, a transition-tour with the CP algorithm generates a set of minimum-length architecture-neutral test paths that satisfy the all-edges coverage criterion. Next, the test paths that are already included in the existing ones are removed to prune unnecessary test paths. A set of architecture-enabled test paths under the chosen test architecture can be generated by augmenting the set of the architecture-neutral with POs and PCOs of the chosen test architecture in the position of message exchange between participant services. Multiple test architectures can be used for generating sets of the architecture-enabled test paths. Finally, architectureenabled test scenarios under the selected test architecture are generated, which correspond to the architecture-enabled test paths. For scalability of our method, there are several ways to reduce the number of TSs in our method. For example, it can be done by choosing other coverage criteria such as all-nodes instead of all-edges or by using a limited number of POs and PCOs. For example, Table 1 shows a partial result of architecture-enabled test scenarios of e-business application using our method. An architecture-neutral test path NP1 is generated from the test model in Fig. 8. The two test architectures: TA1 and TA2 defined at the end of section 3.2 are then selected. Architecture-enabled test path AP1 is generated by adding two POs of the TA1 to the architecture-neutral path NP1. More specially, (PO1) between S5 and P1 in the AP1 is inserted because the message link A between the Shop service and the Payment service is configured to PO1 in the TA1. Similarly, architecture-enabled test path AP2 is also generated by adding two PCOs of TA2 to the architecture-neutral path NP1. Architecture-enabled test scenario AS1 that corresponds to AP1 is generated by matching actual messages of WSDL of the AUT corresponds to the AP1

14

under the selected test architecture. A BPEL variable is always connected to a message from a WSDL description of partner services. In the architecture-enabled test scenarios, the format tselm?msg represents that the test element (tselm) receives the message (msg) and the format tstelm!msg represents that the test element (tselm) sends the message (msg). For example, Tester!ItemOrder means that the Tester in the TA1 sends ItemOrder message defined in Fig. 4 to the Shop service and PO1?PaymentRq means that the PO1 in the TA1 receives PaymentRq defined in Fig. 4 from the Shop service. In this example, two PCOs of the TA2 are utilized for controlling hidden participant services such as Payment and Delivery behind the front service Shop. The outputs can be observed from the PCOs or POs as well as from the front service. By using the architectureenabled TSs with POs or PCOs, the TSs can control and monitor all possible interactions among hidden participant services. Table 1 A partial result of architecture-enabled test scenarios of e-business application. ArchitectureNeutral Test Path NP1: S1,S2,S 4,S5,P1,P2,S 6,S8,S10,S11, D1,D2,S12,S 14,S1

Test Architectures

Architecture-Enabled Test Path

Architecture-Enabled Test Scenarios

Test Architect AP1: S1,S2,S4,S5,(PO1),P1,P2, AS1: Tester!ItemOrder, ure: TA1 (PO1),S6,S8,S10,S11,(PO2), Tester?ConfirmOrder,Tester!PaymentInfo, D1,D2,(PO2),S12,S14,S1 PO1?PaymentRq,PO1?PaymentRp, Tester?PaySuccess, Tester!DeliveryInfo, PO2?DeliveryRq, PO2?DeliveryRp, Tester?DeliveryStarted Test AP2: S1,S2,S4,S5,(PCO1),P1, AS2: Tester!ItemOrder, Architecture: P2,(PCO1),S6,S8,S10,S11, Tester?ConfirmOrder, Tester!PaymentInfo, TA2 (PCO2),D1,D2,(PCO2),S12, PCO1?PaymentRq, PCO1!PaymentRq’, S14,S1 PCO1?PaymentRp, PCO1!PaymentRp’, Tester?PaySuccess, Tester!DeliveryInfo, PCO2?DeliveryRq, PCO2!DeliveryRq’, PCO2?DeliveryRp, PCO2!DeliveryRp’, Tester?DeliveryStarted

…

3.5

Architecture-Enabled Test Execution and Analysis

For architecture-enabled test execution and analysis, the POs or PCOs of a given test architecture configuration should be deployed by using a SOAP monitoring tool in the real network environment before test execution. By adding a PO or PCO to each message link between participant services of AUT, Simple Object Access Protocol (SOAP) message exchanges between them can be sent to the test element, the PO or PCO, so that it can capture and forward them to their final destination services. For example, PCO1 and PCO2 are added to capture and modify SOAP message exchanges between a controller service and its two partner services, and

15

PO is added to monitor SOAP messages exchanges between two controller services. The general concept and structure of test architecture using POs and PCOs are illustrated in [12,13,14]. A PO was implemented as a first in first out (FIFO) queue for input messages and a PCO as two FIFO queues consisting of an input queue and an output queue. To implement a test architecture configuration using such POs and PCOs, we have chosen an open source SOAP monitoring tool, Membrane SOAP Monitor [15], owing to its capabilities for capturing, analyzing, and manipulating SOAP messages 1 . Fig. 9 illustrates a snapshot of architecture-enabled test execution and analysis by using the two PCOs and one PO created by Membrane SOAP Monitor. For a test architecture configuration, routing rule for each PO or PCO can be created by specifying the source host and the listening port for incoming messages and the target host and port to which the PO or PCO should send the messages. Then the endpoint addresses of the target services must be changed to the address of PO or PCO created by Membrane SOAP Monitor for capturing the messages. Test interfaces are generated from the WSDL specifications of the composite services of AUT. Tester sends SOAP messages through their test interfaces. TSs obtained from our test generation algorithm are abstract in the sense that they do not have actual test data. Each TS should be actualized with real values for test execution. Finally, the resulting test cases can be applied for testing of the AUT under the given test architecture. To realize architecture-enabled test execution and analysis in an integrated way, we have developed a prototype SOA testing tool based on open source, soapUI [16]. A snapshot of the tool is shown in Fig.10. To conduct architecture-enabled test execution and analysis, the test architecture configuration of a given test architecture must be loaded in Membrane SOAP Monitor, which makes it possible to realize the POs and PCOs by monitoring and controlling message exchanges under the given test architecture of AUT.

1

An example of POs and PCOs created by Membrane SOAP Monitor is available at http://www.dropbox.com/s/ak50r9tkm13v8y3/Example_PCOPO.xml?m

16

Fig. 9. A snapshot of architecture-enabled test execution and analysis.

Window for Test Results

Launch Membrane SOAP Monitor

Window for Test Interfaces

Fig. 10. A snapshot of a prototype SOA testing tool.

17

4. Evaluation In this section, our architecture-based testing method is applied to an industrial SOA, the CRM application that is described in section 2. We chose this application as the subject of the case study because it is a real world choreography application containing relatively complex message exchanges among participant services.

4.1

Experiment Setup

The CRM application consists of three composite web services: Customer Care, Bridge Call, and Authorization and five atomic partner web services. Some partner services, i.e. Profile Manager and Stock Quote, are non-standard IT services and the others, i.e. Third Party Call, Call Notification, and Audio Call are Parlay-X [17] Telecom web services. To create faulty versions of the CRM, we invited experienced SOA developers and had them seed faults in single-fault versions of each participant service. In total, 70 faults were injected in the eight participant services. The outputs from test case execution of the original program were used as test oracles. Details of participant services in the CRM application are described in Table 2. We used a choreography specification described in WS-CDL of the CRM application to derive test architectures and a test model. Since the CRM has the choreography interaction architecture style, the test architectures of the CRM can be obtained by applying the design guidelines for the test architecture for choreography introduced in section 3.2. We used full test coverage with POs and PCOs. It means that PO or PCO are added to every possible message link of the AUT. The obtained generic test architecture for the CRM is illustrated in Fig. 11, from this generic test architecture, we derived two alternative test architectures for choreography, TA1 and TA2. TA1 consists of three testers and eight POs, and TA2 consists of three testers and eight PCOs. Our test method generated a set of TSs for each of TA1 and TA2. Table 2 Participant services in the CRM application. Service Test

under

Role

Customer Care

Controller

Bridge Call

Controller

Authorization

Controller

Purpose Provide customer care functions by coordinating Controller services Bridge Call and Authorization Control a bridge call by coordinating partner services TPC and AC Provide a authorization function by coordinating partner services AC and CN

Lang.

LOC

# of Seeded Faults

BPEL

1115

16

BPEL

1150

12

BPEL

1065

9

18

Stock Quote

Partner

Profile Manager

Partner

Third Party Call (TPC)

Partner

Audio Call (AC)

Partner

Call Notification (CN)

Partner

Retrieve stock quotes Retrieve contacts for customers and brokers Initiate a third party call between a broker and his client Provide multimedia message delivery functions Provide functions for collecting digits during media interaction to treat a call

Java

238

5

Java

305

7

Java

2515

9

Java

5137

8

Java

4177

4

Test System Tester A

Tester B

PO or PCO

Tester C

SUT 1

SUT 4

SUT 7

Customer Care

Bridge Call

Authorization

SUT 2 Stock Quote

SUT 3 Profile Manager

WUT B SUT WUT B5 (PAYMENT) (PAYMENT) Third Party

Call

SUT 6

SUT 8

Audio Call

Call Notification

Fig. 11. Test Architecture of the AUT.

The setup of the experiment consists of the web service environment using Java runtime environment 1.6, Eclipse BPEL Designer [18], Apache ODE 1.3.5 [19], and Apache Tomcat 6.0 [20]. We executed the experiment on three Samsung DM-Z600 servers that run Intel Core i5-2400 CPU at 3.10 GHz with 8GB RAM. For comparison, four test methods were selected. They are the current practice, the transition tour without test architecture using the CP algorithm [11], our method with test architecture TA1, and our method with test architecture TA2. The current practice is a manual test generation method based on the composition specification that is used by the service test team. The team consists of expert testers with an average of six years of professional experience in SOA testing.

4.2

Data Analysis

For data analysis, we measured the test generation time, test execution time, fault detection rate and fault

19

locating time. The results are summarized in Table 3. The fault detection rate of a method is defined by X/Y, where X is the number of faults detected by the method and Y is the total number of faults. Our method with TA1 and TA2 resulted in great savings of fault locating time due to the use of POs and PCOs. In the case of the current practice and the transition tour without test architecture, it is quite difficult to find out which participant service contains faults because they only observe the test results from the front service of the AUT. For example, suppose a situation in the experiment, the front service (i.e. Customer Care service) received EndStockTrade message from a test engineer, which is an input-only SOAP request message for releasing related system resources in the involved participant services, then the front service did not trigger relevant message interactions to participant services (e.g. TerminateSession message from Customer Care service to Bridge Call service). This case shows a kind of the message exchange fault because it violates expected workflow sequence specified the choreography specification. Using the transition-tour without test architecture, it is impossible for a test engineer to detect that type of fault because there is no way to get the information for the workflow violation behind the front service. Our method required more time for test generation due to the time spent on test architecture design and test model derivation. It also took much time on test execution because it executed more test cases and data manipulation time within test architectures. These data shows the trade-off between effectiveness and efforts when our method is compared to other methods. However, the additional efforts are negligible compared with the benefits gained from our method, which are higher fault detection rate and faster fault locating time. Table 3 Measurement Results.

Measure Test Generation Effort Number of executable test cases Test Execution Effort Average Fault Locating Time per Fault Fault Detection Rate

Current Practice

Transition-Tour without Test Architecture

Our Method with TA1

Our Method with TA2

17.7 hours

15.8 hours

19.5 hours

19.5 hours

14

22

22

32

22.9 hours

33.8 hours

37.0 hours

44.2 hours

22.6 minutes

27 minutes hours

2.3 minutes

2.7 minutes

42.9 % (30/70)

58.6 % (41/70)

72.9 % (51/70)

94.3 % (66/70)

20

We also analyzed the fault-detection rates of the test methods in terms of fault categories. The 70 faults can be partitioned to the five categories: APIs, service logic, data manipulation, interface, and message exchange. The results in Table 4 show the relationship between fault detection rate and the types of faults for each test method. Our method shows much higher fault-detection rates than the other methods regardless of fault categories. Especially for interface faults, which means faults in WSDL of the controller services, our method with TA1 and TA2 show the best fault detection ability, with a fault detection rate of 100% in the experiment, because the POs of TA1 or the PCOs of TA2 increase observability by monitoring the message exchanges along every possible message link. Our method can easily detect the interface fault by checking whether or not the POs or PCOs receive response messages from the partner services. If interface faults occur, a request message to the partner services is shown in the related PO or PCO but the response message from the partner service is not shown in related PO or PCO. Our method with TA2, which consists of eight PCOs, detects more faults than our method with TA1, consisting of eight POs, in the API Usage category (100 vs. 57.1). Owing to the controlled error generation capability of the PCOs, our method with TA2 executed exceptional test cases more thoroughly than our method with TA1, because the latter provides only observation capability through the POs. Table 4 Fault detection rates by fault categories. Fault Category API Usage Data Manipulation Message Exchange Interface Service Logic

Current Practice 35.7 % (5/14) 36.8 % (7/19) 54.5 % (6/11) 33.3 % (3/9) 52.9 % (9/17)

Transition-Tour without Test Architecture 50 % (7/14) 47.4 % (9/19) 63.6 % (7/11) 77.8 % (7/9) 64.7 % (11/17)

Our Method with TA1 57.1 % (8/14) 63.2 % (12/19) 81.8% (9/11) 100 % (9/9) 76.5 % (13/17)

Our Method with TA2 100 % (14/14) 94.7 % (18/19) 100 % (11/11) 100 % (9/9) 82.4 % (14/17)

Fig. 12 shows accumulated fault detection rates as the number of faults is increased. We can observe that our method with TA1 or TA2 outperforms the other two methods in fault detection by 15 ~ 45 percent. To thoroughly analyze the effects of test architectures, we further measured the fault detection rates by increasing the number of PCOs and POs. We selected six alternative test architectures (i.e., the number of POs or PCOs is two, five or eight) and used test architectures with only POs or only PCOs in this experiment. Fig. 13 shows the result of this analysis in a bar chart. It shows the relationship between fault detection rate and the number of POs and PCOs. The horizontal axis of the graph is divided into six alternative test architectures. As expected, more faults have been detected under test architectures as more

21

POs or PCOs are used. We thus recommend the test architectures that achieve full test coverage with POs or PCOs, with the result that the recommended number of POs is eight and the recommended number of PCOs is also eight because the number of available message links of the AUT is eight in this experiment. A major factor on deciding the numbers of POs and PCOs is the trade-off between the expected fault detection rate and the additional test execution effort.

Fig. 12. Comparisons of accumulated fault detection rates.

Fig. 13. Fault detection rates under various test architectures.

22

Consequently, our method with TA2 detected faults more effectively than the other methods including our method with TA1 because it executed more test cases using eight PCOs introduced by TA2. As already discussed in section 3.2, a PCO is a more powerful test element than a PO because it has the additional capability of intercepting and modifying the SOAP messages in message links under investigation. Our method with TA2 thus provides higher testability that comes from controlled error generation capability of the PCOs for executing exceptional test cases. On the other hand, the other method with no PCOs can only input and observe SOAP messages through the front service. Even though a PCO is a powerful test element with dynamic modification capability, we should be careful for using it in the experiment. For example, a test engineer may violate the preconditions of the message protocol when he modifies contents of messages in transit by using PCOs. Therefore, the test engineer should check whether the message modifications using PCOs conform to the preconditions specified in the protocol specification or not.

4.3

Threats to Validity and Other Discussions

The primary threat to the external validity for our experiment involves the representativeness of our subject SOA programs and injected faults. We only used a limited number of SOA programs and certain types of faults in the experiment. However, our experiment deals with more typical industrial SOA programs than the one in [8]. They used a downsized version of University Enterprise Resource Planning application adapted from the original version developed in Java as a subject application. We used a subject SOA that consists of a WS-CDL specification, three BPEL programs with their WSDLs, and five Java programs with their WSDLs. In our experiment, we are mainly concerned with five categories of faults: API usage faults, data manipulation faults, message exchange faults, interface faults, and service logic faults. Other faults such as QoS unsatisfied faults [21] were not studied in the experiment. The primary threat to the internal validity for this experiment is the possible faults in the implementation of our SOA testing tool and the open source tool, the Membrane SOAP monitor, which is used to implement POs and PCOs. We controlled this threat by extensively testing our tool and Membrane SOAP Monitor, and verifying their results against a smaller SOA program for which we can determine correct results with confidence.

23

We expected that our method would not find any faults other than the injected ones since the original version of our subject SOA is a real application used in industry. However, our method found a few noninjected faults during the experiment. We fixed these faults and then reapplied the relevant test cases and obtained the experimental results. However, existence of the faults other than the injected ones may affect the experimental results. Developers of our subject SOA had a difficulty in testing hidden interactions that cannot be detected at the front service after initial development. Especially for, they suffered from a critical failure that is related to resource consumption, but they did not know the cause of the failure. The reason for the failure stemmed from missed delivery of relevant messages from the front service to hidden participant services, which triggers the release of related system resources in the participant services. It was detected by our method test architectures and fixed before deployment.

5. Related Works We group the research related to our work into four areas: test architecture, conformance testing for atomic services, service composition testing, and interoperability testing for communication protocols. First, as a work on test architectures Walter et al. [12] presented a generic test architecture for conformance and interoperability testing of distributed systems. Ulrich et al. [23] proposed two test architectures for distributed and concurrent systems using a test controller and distributed testers. Existing research on test architecture, however, has focused on generic test architectures for distributed systems that do not fully reflect interaction architectures of SOA. We extended generic test architectures with respect to the interaction styles of AUTs, which allows architecture-based test generation and execution in a novel way. Second, as research on service conformance testing for atomic services, Bertolino et al. [24] proposed a conformance testing framework that augments the WSDL description with a UML2.0 protocol state machine (PSM) diagram. The PSM was then translated into a symbolic transition system, to which an existing formal testing theory and tools can be readily applied for conformance testing. Keum et al. [4] suggested an extended finite state machine (EFSM) specification instead of a simple WSDL for

24

conformance test generation of Web services. Li and Chou [25] proposed an abstract guarded finite state machine as a Web service behavior specification that covers data and control flows of a Web service for conformance testing. However, researches on service conformance testing have only focused on the testing of atomic services running on a single system instead of distributed systems. Third, as a most recent service composition testing techniques, Mei et al. [8, 26] addressed the data dependency problems arising in an XML document using an XPath expression. However, they just applied data-flow testing techniques to SOA specified in BPEL and WS-CDL without considering interactions architectures of SOA. Yan et al. [7] proposed a method for generating test cases for BPEL using a control flow technique. Endo et al. [27] proposed service orchestration testing strategies based on structural coverage criteria for parallel programs. Bentakouk et al. [28] proposed a formal test framework for service orchestration applications specified in BPEL. They applied a protocol conformance testing technique based using symbolic transition system (STS) to service orchestration testing. Cao et al. [29] proposed a framework that automatically generates and executes tests for conformance testing of service orchestration specified in BPEL. They applied a protocol conformance testing technique using timed extended finite state machine (TEFSM). The main limitation of these existing research works for service composition testing is that they overlook the impact of test architectures in test generation and test execution. Bai et al. [30] proposed a framework to enable the dynamic reconfigurable testing of SOAs. However, this approach focuses on a dynamic configurable test management tool based a test broker architecture, which is good only when services under test are frequently changed at runtime. In contrast, our approach exploits test elements such as POs and PCOs to facilitate test execution and analysis through monitoring and controlling the hidden message interactions of SOAs. Fourth, interoperability testing approaches of communication protocols has been conducted in [22,31,32]. These interoperability methods compose formal specifications all-at-once into one global model for system behavior, using reachability analysis, and then generate test cases for the model similar to conformance testing. In our method, the idea of generating test cases from a global system model and testing it as a whole with them are influenced by these approaches. However, these approaches focus on testing for communication domain and compose a global system model from partial specifications whereas

25

our approach focuses testing for SOA domain and constructs a global system model directly from service composition specifications such as BPEL and WS-CDL.

6. Concluding Remarks Testing SOAs is much more challenging than testing monolithic applications since SOAs have complex interactions between participant services in distributed networks. Our approach enhances testability of SOAs by introducing to the testing of SOAs the notion of test architecture that enables the monitoring and controlling of message exchanges between hidden services through POs and PCOs of test architecture. An application of our approach to a real industrial application showed improved fault detection performance by more than 20 percent and required only one-tenth of the fault locating time that other existing methods needed. For future work, we plan to extend our approach with new test models (e.g., global extended finite state machine) and test generation techniques under data flow coverage criteria as well as control flow coverage criteria to investigate effects of various coverage criteria of test models.

References [1] Web Services Description Language (WSDL) 2.0, (accessed July 2012). [2]

Web

Service

Business

Process

Execution

Language

(BPEL)

2.0,

(accessed July 2012). [3] C. Peltz, Web Services Orchestration and Choreography, Computer 36 (10) (2003) 46-52. [4] C. Keum, S. Kang, I. Ko, J. Baek, Y. Choi, Generating Test Cases for Web Services Using Extended Finite State Machine, Testing of Software and Communication Systems, LNCS, vol. 3964, 2006, pp. 103-117. [5] L. Li, W. Chou, An Abstract GFSM Model for Optimal and Incremental Conformance Testing of Web Services, in: ICWS’09: IEEE International Conference on Web Services, 2009, pp.205-212. [6] L. Bentakouk, P. Poizat, F. Zaidi, A Formal Framework for Service Orchestration Testing Based on Symbolic Transition Systems, Testing of Software and Communication Systems, LNCS, vol. 5826, 2009, pp.16-32.

26

[7] J. Yan, Z. Lee, W. Sun, J. Zhang, BPEL4WS Unit Testing: Test Case Generation using a Concurrent Path Analysis Approach, in: ISSRE'06: International Symposium on Software Reliability Engineering, 2006. pp. 75-84. [8] L. Mei, W.K. Chan, T.H. Tse, Data Flow Testing of Service Choreography, in: ESEC-FSE’09: European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, 2009. pp.151-160. [9] Web Services Choreography Description Language (WS-CDL) 1.0, (July 2012). [10] S. R. S. Souza, S. R. Vergilio, P. S. L. Souza, A. S. Simao, A. C. Hausen, Structural Testing Criteria for Message-Passing Parallel Programs, Concurrency and Computation: Practice and Experience 20 (16) (2008) 1893–1916. [11] H. Thimbleby, The directed Chinese Postman Problem, Software Practice and Experience 33 (11) (2003) 1081–1096. [12] T. Walter, I. Schieferdecker, J. Grabowski, Test architectures for distributed systems: State of the art and beyond, in: IWTCS’98: International Workshop on Testing of Communicating Systems, 1998, pp.149-174. [13] ISO/IEC, Information Technology - Open Systems Interconnection - Conformance Testing Methodology and Framework-Parts 1: General concepts, ISO International Standard 9646-1, 1995. [14] ISO/IEC, Information Technology - Open Systems Interconnection - Conformance Testing Methodology and Framework-Parts 2: Abstract Test Suite specification, ISO International Standard 9646-2, 1995. [15] Membrane SOAP Monitor, (accessed May 2012). [16] soapUI, (accessed July 2012). [17] Parlay X 3.0, (accessed July 2012). [18] Eclipse BPEL Designer, (accessed July 2012). [19] Apache ODE, (accessed July 2012). [20] Apache Tomcat, (accessed July 2012). [21] H. La, S. Kim, Static and Dynamic Adaptations for Service-Based Systems, Information and Software Technology 53 (2011) 1275-1296. [22] S. Kang, J. Shin, M. Kim, Interoperability Test Suite Derivation for Communication Protocols, Computer Networks 32 (3) (2000) 347-364. [23] A. Ulrich, H. Konig, Architectures for testing distributed systems, in: TestCom’99: International Conference on Testing of Communication System, 1999, pp. 93-108. [24] A. Bertolino, L. Fantzen, A. Polini, J. Tretmans, Audition of Web Services for Testing Conformance to Open Specified Protocols, Architecting Systems, LNCS, vol. 3938, 2006, pp. 1-25. [25] L. Li, W. Chou, An Abstract GFSM Model for Optimal and Incremental Conformance Testing of Web Services, in: ICWS’09: IEEE International Conference on Web Services, 2009, pp.205-212.

27

[26] L. Mei, W.K. Chan, T.H. Tse, Data Flow Testing of Service-Oriented Workflow Applications, in: ICSE ’08: IEEE International Conference on Web Services, 2008, pp.371-380. [27] A.T. Endo, A. S. Simao, S. R. S. Souza, P. S. L. Souza, Web Services Composition Testing: a Strategy Based on Structural Testing of Parallel Program,

in: TAIC-PART’08: The Testing: Academic and

Industrial Conference - Practice and Research Techniques , 2008, pp.3-12. [28] L. Bentakouk, P. Poizat, F. Zaidi, A formal framework for service orchestration testing based on symbolic transition systems, Testcom/FATES’09, LNCS, vol. 5826, 2009, pp.16-32. [29] T. Cao, P.Felix, R. Castanet, I. Berrada, Online Testing Framework for Web Services, in: ICST’10: IEEE International Conference on Software Testing, Verification and Validation, pp.363-372. [30] X. Bai, D. Xu, G. Dai, W. Tsai, Y. Chen, Dynamic Reconfigurable Testing of Service-Oriented Architecture, in: COMSAC’07: Annual International Computer Software and Applications Conference, 2007, pp.368-378. [31] S. Seol, M. Kim, S. Kang, J. Ryu, Fully Automated Interoperability Test suite Derivation for Communication Protocols, Computer Networks 43 (6) (2003) 735-759. [32] S. Seol, M. Kim, S. T. Chanson, S. Kang, Interoperability Test Generation and Minimization of Communication Protocols Based on the Multiple Stimuli Principle, IEEE Journal on Selected Areas in Communications 22 (10) (2004) 2062-2074.

28

Appendix: Details of the Injected Faults Used in the Case Study Table 5 Categories of the seeded faults Category

Illustrations

# of Seeded Faults

Message Exchange

Interaction workflow sequence violation

11

Service Logic

Incorrect business logic, malfunction

17

Interface

Incorrect partner Endpoint address faults in WSDL

9

API Usage

Incorrect input values, policy exceptions

14

Data manipulation

Incorrect input/output messages in XPath expressions or assignments

19

Table 6 Distributions of faults in the subject programs Category API Usage Data manipulation Message Exchange Interface Service Logic Total

Customer Care 2

Bridge Call 1

Authoriz ation 4

Stock Quote 1

Profile Manager 2

Third Party Call (TPC) 1

Audio Call (AC) 3

Call Notification (CN) N/A

4

5

N/A

2

1

2

2

3

1

N/A

2

1

2

2

2

1

4

3

2

N/A

N/A

N/A

N/A

5

3

1

1

2

4

1

N/A

16

12

9

5

7

9

8

4

29