WEB SERVICES TESTING CHALLENGES AND APPROACHES Sana Azzam CIS department, IT faculty Yarmouk University Irbid, Jordan
[email protected]
Mohammed Naji Al-Kabi CIS department, IT faculty Yarmouk University Irbid, Jord
[email protected]
The (SOAP) is used to exchange XML-based messages over HTTP or HTTPs for identifying the WS not through GUI interface. The user requests a service from a specific Web Service using Application Programming Interface (API), then the Web Service sends the request to the user in formal XML form [1][2][3][4]. In the WS, the client communicates with a service in the server through a program. Testing Web Services is a significant problem that should be studied carefully; the testing has to be extensive and comprehensive to all important levels (unit, component, and system level). Some recent studies show that the error rate of new software programs has the range from 3 to 6 errors per 1000 statement [8]. "Web Services" (as a term) is an important topic in the field of software testing; therefore, different tools have been developed and designed to enhance the use of Web Services. In this research, we will test a number of tools. These tools are: SoapUI, PushToTest, and WebInject. The previous mentioned tools are used primarily for collecting, creating, and testing Web Services. The major question of this research is: "Which is the best tool for testing Web Services?" This paper determines the efficiency of using each tool (such as "SoapUI") with a group of several available and created Web Services. The proposed process is as following: • Understanding the tools that have been selected for the comparison. • Applying the created and collected Web Services on each selected tool.
Abstract— The web is evolving and expanding continuously and its services are getting more and more complex, public and usable. However, one of the major benefits and challenges in web services is for those services to be offered in a very useful, flexible, effective and secure way. In this paper, we will investigate web services, show their generic structure and show examples and challenges related to testing web services. Keywords- Text mining, plagiarism; documents similarity, and string search. Keywords- Software testing, web services, e-transactions , e-services, service security, performance, reliability.
I.
INTRODUCTION
Web Services technology is one of the most significant Web technologies in these days. Moreover, it is considered as an important topic in the field of software testing. Therefore, different tools have been developed and designed to use different types of Web Services. This has motivated the Web experts to propose different methods and different tools to test new Web Services efficiently. In this study, we make a comparison between three important tools for testing Web Services. These tools are: SoapUI, PushToTest, and WebInject. We can define WS as a service oriented architecture or a software system which designed to provide an interoperable application to application interaction through network, where the systems interact with applications using Simple Object Access Protocol (SOAP).
© ICCIT 2012
Izzat Alsmadi CIS department, IT faculty Yarmouk University Irbid, Jordan
[email protected]
291
• Comparing the selected tools based on some factors (like testing coverage).
This tool adopts the methodology which is used in many reputed companies. The methodology consists of four steps: planning, functional test, load test, and result analysis. PushToTest can determine the performance of Web Services, and report the broken ones. Also, it is able to recommend some solutions to the problems of performance [12]. • WebInject Tool
The results of this paper will answer the main question and summarize the most efficient tools for testing different Web Services? II.
BACKGROUND
Web Services provide many means of interoperation between different software applications, and have the ability to work on many frameworks and platforms [4]. The architecture of WS consists of logical evolution of component architecture, and object oriented analysis and design. The architecture of WS is designed to be able to achieve three main purposes. First, providing a conceptual model and context, and clarifying the relationship among them. Second, describing a number of characteristics of common Web Services. Third, supporting the interoperability with older applications [5] [6] [7]. Many tools have been implemented for testing Web Services, and few approaches have been proposed to study the quality of these tools. This study is based on three open source tools. These tools are considered as effective software solutions for testing Web Services. Next subsections describe briefly the three selected tools. • SoapUI Tool
This tool is used to test Web applications and services. It can report the testing results in real time, and monitor applications efficiently. Furthermore, the tool supports a set of multiple cases, and has the ability to analyze these cases in reasonable time. Practically, the tool is written in Perl, and works with the platforms which have Perl interpreter. The architecture of WebInject tool includes: WebInject Engine and Graphical User Interface (GUI), where the test cases are written in XML files and the results are shown in HTML and XML files [13]. III. LITERATURE REVIEW Several researches have been conducted to improve testing of Web Services. This chapter describes the related work to the domain of testing Web Services. We will first present some techniques that have been proposed for testing Web Services. Ashok Kumar et al. have proposed a model called Automated Regression Suite model. The proposed model includes parsing of WSDL file and generating SOAP to represent how to automate the testing of any Web Service with any environment by making a comparison between responses of Golden and SOAP. Moreover, the number of methods that passed or failed is identified through generated response report [9]. Sneed et al. has developed a WSDL test for testing Web Services. This technique is based on generating a request and adjusting it based on a pre-condition. The proposed technique dispatches the requests and capture responses [8]. Siblini et al. has proposed a technique relied on mutation analysis for testing Web services. This technique includes applying mutation operators to WSDL document in order to generate a mutated Web Services interface. The technique is proposed to test Web services through discovering of errors in WSDL
This tool is a Java based open source tool. It can work under any platform provided with Java Virtual Machine (JVM). The tool is implemented mainly to test Web Services such as SOAP, REST, HTTP, JMS and other based services. Although SoapUI concentrates on the functionality, it is also consider performance, interoperability, and regression testing [11]. • PushToTest Tool One of the objectives of this open source tool is to support the reusability and sharing between people who are involved in software development through providing a robust testing environment. PushToTest primarily implemented for testing Service Oriented Architecture (SOA) Ajax, Web applications, Web Services, and many other applications.
292
interface and Web Services as well. Nine mutation operators, mutation groups (switch, special, occurrence) and WSDL document are used in the proposed technique [2]. Automatic Test Generation from GUI Applications for Testing Web Services is another approach that has been proposed by Conroy et al. This approach includes proposing a novel generic approach (smart) to generate test case from reference legacy GAP depends on extraction of data from GUI, and then applying this test in Web services to reduce the time consuming [10].
2. Evaluation and Comparison Comparing between different testing tools of Web Services is a complex task, the testing criteria are not the same in all the selected tools, which means that one tool may have the ability to test the Interoperability (for example), while another one does not test this criteria. Furthermore, some criteria may be affected by factors which are not related to testing tools or Web Services. For example, the performance of internet connection may affect the loading test of Web Services and therefore cause a negative impact in the comparison between tools. To compare selected testing tools, some factors are chosen as a base of the comparison process. Table 2 shows comparison factors.
IV. GOALS AND APPROACHES To use the selected tools to test the collected Web Services, we have to apply these Web Services in the tools. In other words, collected Web Services have to be tested by each tool. Obviously, each testing tool has its own method to test the Web Services. This includes importing the Web Services into the tool, dealing with tool’s options which involves in testing process, and finally, showing the results of the analysis process in the supported format. Actually, this means that we have to understand all the required process for applying Web Services and get the results’ report from each tool deeply.
Table 2: Comparison Factors
Factor Ease of Use Testing Coverage
Details Is the tool user friendly or not? How many criteria the tool can test?
Functionality
What is the testing tool response time?
Performance
Dealing withloading in testing Web Services?
Throughput
How much Web Services the tool can handle in a period of time?
1.Results’ Collecting and Analyzing Testing tools generates its results based on some testing criteria. These criteria include functionality, Performance (loading), Interoperability, and many other criteria. Test of performance has many types. No doubt that discussing these types is helpful to clarify how to test the performance of a specific Web Service. Table 1 shows different types of performance testing.
Our method to make a comparison between testing tools includes the following steps: • Testing each Web Service from the collected Web Services using the three selected tools, taken into consideration the testing criteria. • Analyzing the testing result of each tool based on the comparison factors. • Reporting results between the testing tools.
Table 1: Types of Performance Testing
Type Baseline Testing Load Testing Stress Testing Soak Testing Scalability Testing
Definition Normal modes performance testing.
The results of comparison will help us to specify the more efficient testing tools. Note that it is possible that the tool will be the best in testing one criterion and the worst in testing another one.
High load performance testing. Exceptional loads testing. Long period testing. Large volumes testing.
3.Discussing the Results The tools which used to test the Web Services are varying in the time which they need to react. In other
293
words, some tools take less time than others to test the Web Services. In the evaluation of the three selected tools, response time of tools for each Web Services is calculated. Table 3 shows the response time of each tool for testing the Web Services. Figure 1 shows the average response time of each tool for all Web Services. The results show that the SoapUI testing tool is the fastest tool in responding to and testing of the Web Services. The second fastest tool is PushToTest, where WebInject is the slowest tool.
Weather Forecast
PushToTest
Response Time in (ms)
519.88
600 400 200
PushToTest
SoapUI
Testing Web Services Tools
Table 4 shows the performance of each tool for testing the Web Services. Figure 2 shows the average performance of each tool for all Web Services. The results show that the SoapUI testing tool has better performance than PushToTest and WebInject, while PushToTest is better than WebInject in the performance.
SoapUI
832
Global Weather
1198
943
428
Send Fax
1264
1412
755
Currency Convertor
1603
692
776
Country Details
1869
1364
606
Bible Web Service
1401
1542
373
Text to Braille
958
1526
631
Weight Unit con.
977
601
369
Com. Unit con.
981
635
619
V. Credit Card
929
394
639
Statictic
932
895
804
Stock Quote
953
742
682
Fedach
1329
683
Send SMS World
1272
V. Email Address
966 1029.14
813.761
800
Figure 1: Average Response Time.
1237
Overall average
1029.14
WebInject
Response time for each Tool in (ms) WebInject
1000
0
Table 3: Response Time for the Web Services
Web Service
1200
1120
Table 4: Tools Performance Web Service Weather Forecast Global Weather
Performance (Loading Test) in (ms) WebInject 1271.8
860
SoapUI 399.98
1040
447.81
Send Fax
1215.8
1286
1459.35
565
Currency Convertor
1119.6
0960
1,382.46
842
478
Country Details
1028.6
1534
6
732
596
Bible Web Ser.
1544.6
1260
379.46
813.761
519
4.The Performance Comparison Factor The performance (loading testing) of tools is a critical factor in the comparison process. The conducted tests by this study include studying the normal load of each of the three tools under consideration, where several requests are sent per minutes, and determine how the tool resistant the loading.
Text to Braille
0977.8
850
650
Weight Unit C.
0951.2
700
367.69
Com. Unit Con.
1152.4
800
940
V. Credit Card
1066.8
360
353.09
Statistic
1082.4
553
377.72
Stock Quote
0884.8
462
459.62
Fedach
1028.4
1046
1,635.55
Send SMS
1388
930
337.71
V. Email add.
1164
560
321.58
730.047
556.751
Overall average
294
1115
PushToTest
983.70
6. The Ease of Use Comparison Factor
1200
Performance in(ms)
1000
983.704762
800
The study of the testing tools shows that the SoapUI tool is the most users friendly tool among the selected tools. The reason for that is the excellent graphical user interface which the tool support for testing the Web Services. The PushToTest is somehow can be used with a little effort, but it is still more complex than SoapUI. The last tool which is WebInject is the most complex tool to use, it requires a lot of effort because it needs a configuration process as well as it does not support graphical user interface, and push the user to use a command line.
730.047619 556.7519048
600 400 200 0 WebInject
PushToTest
SoapUI
Testing Web Services Tools
Figure 2. Average Performance of each Tool. 5.The Throughput Comparison Factor Throughput can be defined the as the number of requests per seconds. High number of requests with low load leads to high performance. This research tries to study the throughput of each tool depending on the selected Web Services, but only the SoapUI and PushToTest tools support this kind of testing. Table 5 shows the throughput of SoapUI and PushToTest tools for testing the Web Services. SoapUI has better throughput than the two other tools. Table 5: The Throughput of SoapUI.
Web Service
SoapUI
PushToTest
4.13
1.6
Global Weather
4.02
0.86
Send Fax
0.84
0.81
0.82
1.26
6.8
0.78
Bible Web Service
4.21
0.88
Text to Braille
6.04
1.68
Weight Unit Convertor
4.4
5.5
Computer Unit Convertor
1.32
4.9
Country Details
Validate Credit Card
4.49
4.4
Statistic
4.42
6.5
Stock Quote
4.09
4.8
Fedach
2.04
0.82
Send SMS World
4.54
3.9
Validate Email Address
4.64
5.1
Average
The studying process of the three tools shows that the SoapUI and PushToTest can provide multiple cases in the same testing criteria more than WebInject tool. For example, for the loading test side, SoapUI and PushToTest can test the Web Service in multiple cases: Baseline Testing, Load Testing, Stress Testing, Soak Testing, and Scalability Testing. The SoapUI has a big advantage over the other two tools since it supports high level of security. Clearly, the results illustrates that the three testing tools have the following order depending on the total factors from the best to the worst: SoapUI, PushToTest, and WebInject. SoapUI outperforms PushToTest and WebInject in all the comparison factors, and with a huge difference. SoapUI has an excellent response time, performance, and throughput. It is easy to use and has a group of multiple cases in testing the same criteria. PushToTest is a good testing tool; even it could not outperform SoapUI in any factor. The PushToTest testing tool has got good values in response time and performance. WebInject has the lowest values in the comparison factors. This tool does not have the ability to test the Web Services in the same quality with the other tools. Moreover, it is complex and difficult to use.
Throughput of SoapUI and PushToTest Tool (sec)
USA Weather Forecast
Currency Convertor
7. The Testing Coverage Comparison Factor
4.169
V.
CONCLUSION AND FUTURE WORK
In this paper, several selected web services are evaluated based on several known important quality factors. Examples of those quality factors used in evaluating web services include: performance,
3.9947
295
reliability, security, throughput, etc. The obtained results from the evaluation process show that the SoapUI outperforms PushToTest, and WebInject in all the comparison factors. In the same way, PushToTest outperforms WebInject in the compared factors. In general, the project suggests using SoapUI to test different Web Services because of its good performance and high quality test of Web Services.
[11]
[12]
"Web
Service
(WS)",
(visited
in
[13]
11-october-2010),
Siblini, R.; Mansour, N. (2005), "Testing Web Services",
aiccsa, pp.135-vii, ACS/IEEE 2005 International Conference on Computer Systems and Applications . [3]
Martin, E.; Basu, S.; and Xie, T. (2006), "Automated
Robustness Testing of Web Services", In Proceedings of the 4th International Workshop on SOA And Web Services Best Practices . [4]
Bertolino, A.; Polini, A. (2005), “The Audition Framework
for Testing Web Services Interoperability" in Proceedings of the 2005 31st EUROMICRO Conference on Software Engineering and Advanced Applications . [5]
"What's
Web
Service",
(visited
in
21-october-2010),
http://www.w3.org/TR/ws-arch/#gengag.
[6] "Web Service architecture",(visited in 22-october-2010), https://www.ibm.com/developerworks/Webservices/library/w-ovr/. [7]
Judith, M. "Web Service architectures", Published by Tect,29
South LaSalle St. Suite 520 Chicago Illinois 60603 USA. [8]
Sneed, H, M.; Huang, S.( 2006), " WSDLTest – A Tool for
Testing Web Services", Eighth IEEE International Symposium on Web Site Evolution . [9]
Kumar, A, S.; Kumar, P, G.; and Dhawan, A.(2009),
“Automated
Regression
Suite
for
Testing
Web
Services"
International Conference on Advances in Recent Technologies in Communication and Computing, pp.590-592 . [10]
,
(visited
in
14-November),
"PushToTest tool" , (visited in 16-November), http://
"WebInject"
(visited
http://www.WebInject.org/.
http://en.wikipedia.org/wiki/Web_service. [2]
tool"
www.PushToTest.com.
References [1]
"SoapUI
http://www.SoapUI.org.
Conroy, K, M.; Grechanik, M.; Hellige, M.; Liongosari, E,
S.; and Xie, Q.( 2007), " Automatic Test Generation From GUI Applications For Testing Web Services", Software Maintenance 2007,ICSM 2007,IEEE International Conference on 2-5 Oct 2007, pp.345-354 .
296
in
30-october-2010)
,
