Comparative Evaluation of the Maintainability of ...

Comparative Evaluation of the Maintainability of RESTful and SOAP-WSDL Web Services Valério Brusamolin Ricardo Ramos de Oliveira, Robson Vinícius Vieira Sanchez, Júlio Cezar Estrella and Renata Pontin de Mattos Fortes 11o Centro de Telemática Universidade de São Paulo Rua 31 de março, Pinheirinho, s/n - CEP: 81150-900 Instituto de Ciências Matemáticas e de Computação Curitiba - PR, Brazil Avenida Trabalhador São-Carlense, 400, Centro - CEP: 13566-590 {brusamolin}@gmail.com São Carlos - SP, Brazil {ricardoramos, rsanchez, jcezar, renata}@icmc.usp.br

Abstract—Web services not only are one of the most promising technologies in terms of the availability of network services but also solve the problem of integrating heterogeneous applications on the web. Because of their increasing popularity, maintainability has become an important issue as it helps to reduce maintenance costs and improve software quality. However, from the perspective of evolution and maintenance, there are many issues to be examined. This paper describes a controlled experiment to compare RESTful and SOAP-WSDL web services in terms of specific modifiability sub-characteristics and time spent on web services maintenance. The findings indicate that RESTful web services are more maintainable on the serverside, while SOAP-WSDL web services are more maintainable on the client-side. Studies based on controlled experiments are promising and may help reduce the maintenance costs of web services as well as improve the quality of software-oriented services. Keywords-Software Maintainability, Controlled Experiment, Web Services, RESTful, SOAP-WSDL.

I. I NTRODUCTION Web services are an emerging Service Oriented Architecture (SOA) technology. Their great popularity is due to the adoption of open standards and protocols such as HTTP and XML to solve the problem of integrating heterogeneous systems applications present in technologies such as CORBA, DCOM and RMI [23]. Interoperability enables systems written in different programming languages, developed by different vendors and using different operating systems to communicate [15]. To provide interoperability between heterogeneous technologies and promote loose coupling between the service consumer (client) and service are the main goals of SOA [7]. However, many different styles of web services can be used to integrate business applications. The choice of web service is an important architectural decision and influences the requirements and properties of the integrated system [16]. In addition, software engineering is primarily concerned with large-scale development of software and has therefore given rise to new techniques and methods to support the development of reliable software that can easily be maintained, is flexible and has low development costs. Evolution, which refers to the maintenance and improvement of software systems throughout

their life, is inevitable during the software development cycle. As software evolves, any changes made to it must be carefully managed in order to control activities related to, for example, project modifications, bug corrections and environmental adaptations [26]. Since web services are software, they are inherently vulnerable and subject to constant change and therefore offer a new challenge for software engineering. From the perspective of evolution and maintenance, there are many issues that need to be examined, including questions related to the evolution and maintenance of processes; products and functions involved in processes and organizational changes needed for the adoption of web services [11]. The evolution and maintenance of web services can be very expensive for many software organizations, which must retain highly skilled engineers and make the necessary organizational changes. Failure of a web service can substantially affect an organization’s productivity. Currently, web services technology lacks maintenance support, and organizations could in future experience serious problems with the maintenance of web services [13]. In 1990, Humphrey and Henry [9] conducted a controlled experiment comparing the maintainability of two functionally equivalent systems in order to investigate the claim that systems developed with object-oriented languages are more easily maintained than those programmed with procedural languages. In 2007, Kumar et al. [2] studied the impact of maintainability on Aspect-Oriented (AO) and Object-Oriented (OO) systems and concluded that the former are more maintainable than the latter. They suggested that their technique should be used to compare the maintainability of different AO systems. In [17] Perepletchikov et al. conducted a controlled experiment to assess the impact of coupling on the maintainability of serviceoriented software. Furthermore, since SOAP and RESTful are different variations of a communication protocol, to some extent, this is like comparing implementations of an algorithm written in two different programming languages. The evaluation of maintainability is important as it helps develop mechanisms for producing high-quality software. The aim of this paper is to evaluate the maintainability of RESTful web services and SOAP-WSDL. Our findings indicate that

RESTful web services are more maintainable on the serverside, while SOAP-WSDL web services are more maintainable on the client-side. Studies using controlled experiments have yielded promising results and may help reduce the maintenance costs of web services, thus improving overall software quality. Section II presents the main concepts that were fundamental to the development of this work, the concepts of web services, Experimental Software Engineering and Maintainability of Web Services. Section IV describes the experiment conducted in this work and in Section III the major works of literature that discuss solutions to reduce costs and improve the maintainability of the software developed are presented. In Section VI describes the results obtained during the research developed. Finally, in Section IV-C evaluation of the hypothesis is statistically tested and finally the conclusions are discussed relating to the work performed. II. T HEORETICAL F OUNDATION This section describes the main concepts that underlie the proposal developed during this research work. A. Web Services Concepts Before defining a web service is important to note that a web service is composed of two structures: the service [20] and service description [21]. The service is a software module installed on a computer platform with network access and offered by the "service provider". A service exists to be used by a consumer, acting also as a client of another service. The description of the service (DS) contains the details of the interface and the implementation of a service, including data types, operations, information binding and network location. You can also include metadata and categorization information to facilitate the activities of discovery and use by service consumers. The DS can be published in a register of service to make their services known in a given context. The most accepted definition by the Information Technology community is the World Wide Web Consortium (W3C) that define web services are software systems designed to support interactions machine-to-machine interoperable by network providing an interface described in a format processable by machines (specifically WSDL). Other systems interact with the web service in a way prescribed by its description using SOAP messages, usually transmitted via HTTP with an XML serialization in conjunction with other web standard [20].

software engineering, which were partially extracted from [25] and [27]. In experimental software engineering, a controlled experiment performed in the laboratory can be in vitro, where all variables are controlled, or in vivo, under natural conditions. The aim of a controlled experiment is to manipulate one or a few variables while keeping the others fixed and measuring the effect on the results [25]. Conducting experiments requires not only that large quantities of information be handled, but also that the researchers responsible for the experiment have a knowledge of the vocabulary used. The following are the principal terms used in experimental software engineering as specified in [25]. The variables in an experiment are the information that is manipulated by the researchers. They can be classified into two groups depending on how they are obtained. Independent variables relate to the input to the experiment and correspond to the causes that may affect the outcome of the experiment. They are also called factors, and the value of the variable is called the treatment. The variables that exhibit the effect caused by the factors in the experiment are called dependent variables and relate to the output. The value of the dependent variables is called the result and is obtained after running the experiment. The relationship between the variables is detailed in Figure 1. Initially, the hypotheses are formulated in an attempt to predict the different behaviors of the experiment in the form of a cause-and-effect relationship. When the experiment is carried out, the values of the independent variables produce the values of the dependent variables (results). These results are then compared with those anticipated when the experiment was prepared.

Fig. 1.

Presentation of the concepts in an experiment [27]

B. Experimental Software Engineering To meet the requirements of industry and academia, different methods, techniques, processes and tools are regularly proposed [3]. However, very often little evidence is provided to support these proposed solutions, i.e., there are no details of the practical results of using these solutions or of the improvements they offer over previously studied technologies [4]. Technological studies in the field of experimental software engineering should be based on experiments so that the proposed solutions can be validated and shared effectively [25]. This section introduces the main concepts related to experimental

Experimentation is a systematic process consisting of phases and sub-products generated at the end of each phase. It begins with the definition of the experiment, extending through the planning, operation, analysis and interpretation to the presentation and packaging [27]. These phases make up the experimental process illustrated in Figure 2. In the planning phase, the design of the experiment is developed in greater detail and risk assessment is included. This phase defines the context, hypotheses, variables, participants, experimental design and instrumentation and evaluates the va-

Fig. 2.

The experimental process and associated phases [27]

lidity of the experimental design. Three activities make up the operation phase: preparation, implementation and validation of the data. In the preparation phase, the material to be used in the experiment is organized, the participants are informed about the purpose of the experiment and their consent to take part in it is obtained. During the execution phase, the experiment is carried out, the tasks scheduled in the planning phase are executed and the data are collected. Data validation is required to check if the data have been collected properly. The analysis and interpretation phases use the data collected as input so that these can be analyzed and interpreted with the help of statistical analysis. C. Maintainability of Web Services There are two kinds of metrics in the context of software development: direct metrics, i.e., attributes such as stress, size and cost; and indirect metrics, or derivatives, which can be obtained from other metrics such as complexity, reliability and ease of maintenance. Some direct and indirect metrics are shown in Table I. TABLE I C ATEGORIES OF M ETRICS DIRECT MEASUREMENTS Cost Effort Code Execution Speed Memory Errors

INDIRECT MEASUREMENTS Functionality Quality Complexity Efficiency Reliability Maintainability

Quality assurance is a major concern of the software development industry, as currently most companies in the market use this process to manage their business, their products and their customer relationships and therefore need greater reliability and quality. Several measures of quality assurance are critical to the success of any software application, among them the measurement of software. This assists decision-making by providing quantitative data and making it possible to determine what aspects of the product meet, or fail to meet, the specified quality standards; to evaluate the benefits of new softwareengineering methods and tools; to understand and improve the production process; to evaluate return on investment; and to

manage projects within the practical limitations imposed by the company’s situation. Although the maintenance and evolution of software is relatively expensive, there has been little empirical research in this area. Nonetheless, the impact of maintenance on software costs is clearly identifiable. In 1993 Eastwood [5] reported that 75% of information systems budgets in large corporations was accounted for by maintenance costs. However, for Erlikh [6] these costs are increasing as in 2000 he found that information systems operations and maintenance costs accounted for between 85% and 90% of total software costs. ISO 9126 [10] defines maintainability as the facility of the software product to be modified. Modifications may include corrections, improvements or adaptations of software to cater for changes in the environment (e.g., changes in the requirements or functional specifications). The standard also specifies many software metrics. It should be noted that all the ISO/IEC metrics are on a ratio scale and are therefore suitable for parametric statistical analysis, as used in this study. The metric chosen to evaluate the maintainability of RESTful and SOAP-WSDL approaches is Modification Complexity (MC), which is equal to Sum (T)/N, where T = time spent on each work shift and N = number of changes. (MC) is always greater than zero. The closer (MC) is to 0, the better [17]. III. R ELATED W ORK In the 1970s Thayer [24] described one of the first methodologies for assessing the maintainability of software. However, by 1993 most organizations were still assessing software maintainability manually with the aid of experienced software engineering managers. Zhuo et al. [28] explored ways in which software maintainability can be calculated more accurately and efficiently using automated techniques. From 1996 great progress was made by the Air Force Operational Test and Evaluation Center (AFOTEC) in the development of methodologies for assessing software maintainability [1]. The AFOTEC assessment method identified key desirable features of a piece of software whose presence or absence was essential for software maintenance to be successful. The idea of using software complexity measures to predict maintenance effort and/or compare projects in terms of maintenance is not new. Several measures of the design complexity of OO software for use in determining the maintainability of code have been published. Lavazza et al. [14] made a quantitative assessment of the impact of requirements changes in the maintenance phase and a quantitative estimate of the costs of these changes. Their approach involves measurement of the process and product models in order to characterize software artifacts quantitatively and so obtain a basis for sophisticated analysis. In [22], software product and process measures are used to predict the stability of a software maintenance process in terms of the reliability of the software and the risk of deploying it. The approach is used in the NASA space shuttle flight software.

In 1990, Humphrey and Henry [9] conducted a controlled experiment comparing the maintainability of two functionally equivalent systems in order to explore the claim that systems developed with object-oriented languages are more easily maintained than those programmed with procedural languages. In 2007, Kumar et al. [2] studied the impact of maintainability on Service Oriented Aspect Systems (AO) and ObjectOriented (OO) and concluded that the AO systems are more maintainable than the OO systems. In future, this technique can be used to compare the retention of different AO systems. In [17] the authors conducted a controlled experiment to assess the impact of coupling Oriented Software Maintenance Service. Kajko-Mattsson [12] identified issues that are unique to the evolution and maintenance of web services. Their results revealed differences between the management systems of conventional software systems and web services as a result of loose coupling, flexibility and reusability of services [8]. New software engineering methods for software development, evolution and maintenance are therefore needed, and existing software engineering methodologies should be reviewed to identify any changes that may be required. IV. D EFINITION AND S TUDY P LANNING In the definition stage, the experiment is expressed in terms of problems and goals. The main issues that led to the experiment are described, and based on these, hypotheses are formulated and the scope of the experiment is defined and planned [25]. A. Objectives To evaluate the maintainability of RESTful and SOAPWSDL web services using the criteria “time spent on maintenance” and “modifiability of web services” and answer the following questions: • Which type of web service is more maintainable on the client-side? • Which type of web service is more maintainable on the server-side? • Which of the two web services evaluated provides the lowest-cost maintenance and development for software companies developing web services? We must understand that when we have an experience like this we are not evaluating the models or even technology itself, but essentially, how the tools are suitable for both the complexity of the problem and the experience of the programmer. We can not assess the effect of the complexity of the application, adequacy of tools and experience of the programmer. In these circumstances, the main research question is to reach conclusions about which approach SOAP-WSDL or RESTful, it is more maintainable?

assessed. We defined the context, hypotheses, variables, participants, experimental design and instrumentation and assessed the validity of the study. After the planning the experimental study was prepared so that it was ready to be carried out. C. Hypotheses The study sought to identify those characteristics that had been little studied in the context of service-oriented software maintainability. The hypotheses were not intended to be definitive but to provide evidence that could be used in further studies in the same field. Based on this information, the following questions that the study was expected to answer were defined, with the corresponding null and alternative hypotheses: Are RESTful web services more maintainable than SOAPWSDL web services on the server or web services provider side? To evaluate the maintainability of RESTful and SOAPWSDL web services, the following hypotheses were defined: Hypothesis 1 Null Hypothesis(H0 ): There is no difference in maintainability between RESTful and SOAP-WSDL web services on the server or web service provider side. H0 : Maintainability(RESTful) WSDL)

=

Maintainability(SOAP-

Alternative Hypothesis(H1 ): There is a difference in maintainability between RESTful and SOAP-WSDL web services on the server or web services provider side. H1 : Maintainability(RESTful) WSDL)

6=


In the same way that the maintainability of web services on the server or web services provider side were questioned, so the maintainability of the same services on the client-side can also be examined. The second question was therefore formulated as follows: Are RESTful web services more maintainable than SOAPWSDL web services on the client or web services consumer side? Hypothesis 2 Null Hypothesis(H0 ): There is no difference in maintainability between RESTful and SOAP-WSDL web services on the client or web services consumer side. H0 : Maintainability(RESTful) WSDL)

=


B. Planning

Alternative Hypothesis(H1 ): There is a difference in maintainability between RESTful and SOAP-WSDL web services on the client or web services consumer side.

In this stage the experiment was designed in greater detail, and the risks and threats to the experiment’s validity were

H1 : Maintainability(RESTful) WSDL)

6=


RESTful web services are believed to be more maintainable on both the supplier and consumer side of the web service. The correctness of this belief was tested empirically here.

D. Selection of the Individuals To investigate the impact of the independent variables described in Section IV-F and begin the experiment, a Java web services course was provided for students at the University of São Paulo who were studying Design Patterns in Web Development, for which the prerequisites are Engineering Software and Web Programming courses. To take part in the course, students were required to have a basic knowledge of Java for the web, XML and client-server architecture. Ten students (the subjects) were chosen in the selection process and divided at random into two groups (A and B).

E. Experimental Tasks According to Pressman [19], software maintenance can be divided into three categories: 1) Corrective maintenance: maintenance of a defect. 2) Adaptive maintenance: adaptation to a new environment. This term can also refer to the adaptation of software to new requirements. 3) Evolutionary maintenance: the improvement of the software system through the implementation of new features. In some situations, however, this may refer to retaining the functionality of the system while improving its structure and performance. The first day of the course served as an introduction to the experiment, and the experiment itself was conducted on the second and third days of the course. On the second day, each subject or student was required to perform two tasks for RESTful and SOAP-WSDL web services. Group A modified the RESTful web services on the server while students in group B modified the SOAP-WSDL web services on the server. Two tasks were performed by each subject, and each task was performed once for each of the two web services. The tasks, which were performed in the same order by all the subjects, were very similar in nature and were not selected to exhibit any particular attribute. At the end of the first task all participants filled out a form with the start time and end time of the task before beginning the next task. The server-side tasks allocated to each group on the second day of the web services course are described below:

Group A

Task 1 - Implement the web service getBestStudent() as a resource on the copy of the RESTful server (Maintenance Evolutionary): /* Method that returns the best student discipline */ public Student getBestStudent(Integer idDiscipline) throws DAOException { /* Implementation of the Method */ } Task 2 - Convert all ten web services on the copy of the RESTful server to SOAP-WSDL (Adaptive Maintenance). Group B Task 1 - Implement the web service getBestStudent() as a resource on the copy of the SOAP-WSDL server (Evolutionary Maintenance): /* Method that returns the best student discipline */ public Student getBestStudent(Integer idDiscipline) throws DAOException { /* Implementation of the Method */ } Task 2 Convert all ten web services on the copy of the SOAP-WSDL server to RESTful (Adaptive Maintenance).

The client-side tasks assigned to each group on the third day of the web services course were as follows: Group A Task 1 - Develop a customer SOAP-WSDL client from the RESTful client (Adaptive Maintenance). Group B Task 1 - Develop a RESTful client from the SOAP-WSDL client (Adaptive Maintenance). The evaluation of the maintainability in evolutionary terms of RESTful and SOAP-WSDL clients will be the subject of further studies. F. Variables Experiment Independent Variables: Travassos [25] defines independent variables as those that can be manipulated or controlled in the process of experimentation. The following variables were identified in this experiment: 1 - Programming Language 2 - Server or RESTful web services provider 3 - Client or consumer of RESTful web services 4 - Server or SOAP-WSDL web services provider

5 - Client or consumer of SOAP-WSDL web services 6 - Information on the subject: experience and training 7 - Maintenance tasks Dependent Variables: The variables observed when the independent variables are manipulated, i.e., variables that express the effect in the cause-and-effect relationship defined in the experiment [25]. The dependent variables identified in this study are: 1 - The difference between the maintainability of RESTful web services and SOAP-WSDL web services on the serverside. 2 - The difference between the maintainability of RESTful web services and SOAP-WSDL web services on the clientside. G. Description of Instrumentation Experiment As previously explained, the main objective of the experiment was to compare the maintainability of RESTful and SOAP-WSDL web services. As this required that a suitable architecture be developed, the first stage of the experiment involved the development of web services providers and consumers using the Java programming language. Java was chosen because it is widely accepted and used in large companies. During the development activity, an open source wiki mediawiki1 has been installed in a server in the laboratory at the University of Sao Paulo - Brazil, to assist in documentation of RESTful web services and SOAP-WSDL. It was observed that, for the context of the experiment, the use of collaborative wiki tool on the server2 was sufficient to meet the needs of the documentation of web services. H. Validity One factor that could compromise the validity of this experiment is the complexity of the programs and the business area they are used in. To mitigate this factor, a system controlled by serviceoriented software was developed specifically for this study to support the investigation of the experimental hypotheses presented in Section IV-C. The system was based on a web application to support the University Environment Information System Information (SInAU), which was loosely inspired by the common rules and procedures of academic management. Note that the area of application chosen (educational management) had the advantage of being easily understood by participants, thus ensuring that system requirements could be easily interpreted. V. P REPARATING AND C ARRYING O UT T HE E XPERIMENT After defining the study and completing the planning, the next step is to carry out the experiment. This is a very delicate stage as it involves the human factor. Data collection must be performed in such a way that it does not have a significant effect on the process being studied. Preliminary validation of the experimental data is also carried out in this stage. 1 http://www.mediawiki.org/wiki/MediaWiki 2 http://garapa.intermidia.icmc.usp.br/mediawiki/

A. The Environment Maintenance The maintenance environment for the experiment was prepared using the following artifacts: • Virtual machine installed and configured in the laboratory; • The documentation for and specification of each web service analyzed; and • RESTful and SOAP-WSDL source code on the supplier and consumer side. The benefits afforded by the use of a virtual machine include: 1) portability of the tools and operating system as well as the entire system configuration; 2) the ability to replicate and generate backups in a practical, relatively rapid manner; 3) the ability to evolve tool settings without compromising a configuration that is already stable; 4) the possibility of interrupting a task and continuing it on different dates while retaining the same operating system state and the same state of the programs being used; and 5) the persistence of the results generated internally by the copy in use [18]. Another very important factor should be mentioned that was critical to the success of the experiment: the fact that the use of a virtual machine enabled students to use the web container with administrator privileges. The environment used here had the following characteristics: • Linux OS distribution Ubuntu version 11.10, host for a version management system through an SVN Server repository version 1.6.123 • Operating System Windows XP used in the virtual machines and in the web services course to perform the maintenance tasks. 4 • Apache Tomcat 7 Web Container 5 • Netbeans IDE version 7.0.1 6 • MySQL Driver JDBC version 5.1.13 7 8 9 • Specifications JAX-RS , JAX-WS and JPA 2.0 10 11 • Reference Implementation Jersey , METRO and Eclipse Link12 13 • Java SE Development Kit - JDK 7 14 • Netbeans Simple Code Metrics Plugin to obtain the metrics direct from the source code. 3 http://subversion.apache.org/ 4 http://tomcat.apache.org/download-70.cgi 5 Integrated

Development Environment

6 http://dev.mysql.com/doc/refman/5.1/en/connector-j-reference-configura\

-tion-properties.html 7 http://java.sun.com/developer/technicalArticles/WebServices/jax-rs/index. html 8 http://jax-ws.java.net/ 9 http://jcp.org/aboutJava/communityprocess/final/jsr317/index.html 10 http://jersey.java.net/ 11 http://metro.java.net/ 12 http://www.eclipse.org/org/press-release/20080317_Eclipselink.php 13 http://www.oracle.com/technetwork/java/javase/downloads/index.html 14 http://plugins.netbeans.org/plugin/9494/simple-code-metrics

B. Description of RESTful and SOAP-WSDL Web Services The programs developed for the experiment were implemented in Java. Figure 3 shows the number of lines of code (LOC) in each RESTful and SOAP-WSDL client and server program and gives an idea of the complexity of the programs.

Fig. 3. No. of lines of Code used in each project (Server/client and RESTful/SOAP-WSDL) in the experiment

The average cyclomatic complexity of each client and the server RESTful and SOAP-WSDL web service is shown in Figure 4.

Fig. 4. Average Cyclomatic Complexity each Provider and Customer RESTful and SOAP-WSDL used in the experiment

C. Architecture of RESTful Web services and SOAP-WSDL The architecture developed for the experiment used design patterns: Data Access Object and Model View Controller design patterns in the client application and a two-tier architecture in the server application?to ensure the lowest possible coupling and ease the maintenance task for students as the maintenance of web services was the subject of interest here. Figure 5 shows the architecture of the RESTful and SOAPWSDL web services clients and servers. It can be seen that the View and Controller layers are identical for both the RESTful and SOAP-WSDL consumers, which differ only in the Model layer. Therefore, the architecture on the client-side consists of independent applications. On the server-side there is only one application, which provides both a RESTful service as its complement and the equivalent SOAP-WSDL. However, the business component, which relies on the DAO, uses the simpler interface provided by the DAO for its clients. The DAO completely hides the implementation details of data persistence from its customers. The interface exposed by the DAO to clients does not change

Fig. 5.

Architecture used in the experiment

when the implementation of the code is modified, as this characteristic allows the DAO to adapt to different storage schemes without affecting its clients or business components. Essentially, the DAO acts as an adapter between the component and the data source. In addition, the DAO allows the repository to be implemented within the application server domain. It should be remembered that by using the domain, this will be referenced by the interfaces or contracts, and the DAO will be fully encapsulated, pluggable and completely replaceable as dependency injection reduces the coupling between business model and database. It should also be noted that the server architecture for the RESTful and SOAP-WSDL web services described in this section was only used on the third day of the course. As, the two servers for each web service are part of the same application in order to avoid replicating the code for the View and Controller layers, the only different layer is the Model layer. The architecture used here enabled a single application server to serve both the RESTful and SOAP-WSDL clients in real time. Hence, on the second day of the course each group received a copy with independent projects using the RESTful and SOAP-WSDL servers. D. Carrying out the Experiment On the first day of the course, the web services data were collected for each student using a questionnaire with the aid of the questionnaire tool in Google Docs15 as shown in Figure 6. The second day’s tasks were performed locally on the virtual machine with a copy of the RESTful server (identifiers SR1, SR2,SR3, SR4 and SR5) for members of Group A and a copy of the SOAP-WSDL server (identifiers SS1, SS2, SS3, SS4 15 http://www.google.com/drive/apps.html

Fig. 6.

1st day of the web services course

and SS5) for each member of Group B. The activities on the second day of the course were performed locally, as illustrated in Figure 7.

Fig. 8.

3rd day of the web services course

VI. A NALYSIS OF R ESULTS AND E VALUATION OF THE H YPOTHESES Fig. 7.

2nd day of the web services course

In addition, we created an Apache Subversion (SVN) repository server for each copy of the RESTful and SOAP-WSDL servers, and after each task had been completed the students were required to commit the changes they had made for subsequent analysis of the source code. On the third day of the course, after all the tasks on the server-side had been completed, each team was given only one task as the idea was to reproduce a real scenario in the laboratory. Team A was given the task of implementing a RESTful server application that could communicate with a RESTful client in real time. The scenario was replicated for team B, but this time the web services clients used a SOAPWSDL web service. As shown in Figure 8, the architecture developed for the experiment allowed a single application server (identified as SR if it was a RESTful server or SS if it was a SOAP-WSDL server) to serve RESTful clients or consumers (identifiers CR1, CR2, CR3, CR4 and CR5) as well as SOAP-WSDL clients (identifiers CS1, CS2, CS3, CS4 and CS5) in a distributed manner using SOAP and HTTP protocols. Both web services applications communicate using XML files and HTTP-GET for RESTful client-server applications and HTTP-POST for applications developed in SOAP-WSDL. Likewise, on the third day of the course an SVN repository for each copy of the RESTful and SOAP-WSDL clients was created. After completion of each task the students were required to commit the modified source code for subsequent analysis.

This section presents the results of the comparison of the maintainability of RESTful and SOAP-WSDL approaches to web services. First, the results are shown for each web service so that the data can be compared with the same program implemented using the other web services approach. The steps required to evaluate the experimental hypotheses, the statistical methods used and the results for the evaluation of each case specified in Section IV-C are also described. A. Results on the Server-Side To collect the data we used the Simple Code Metrics16 plugin for Netbeans IDE17 version 6.1 and an SVN repository server version 1.6.12. The results for the different activities were recorded on a form by the students at the end of each task. To better visualize the behavior of the sample, given the number of elements required for each web services approach, Tables II and III with the results of Tasks 1 and 2 for the second day of the web services course were generated. The start time is represented by “ST”, the end time by “ET” and the total time by “TT”. “No. mod” is the number of modifications performed on the source code, and “MC” is the modification complexity. Analysis of the data with the parametric Student t-test for matched pairs revealed a significant change (p < 0.05) in all groups tested, as shown in Table IV. Therefore, the alternative experimental hypothesis H1 was accepted and it 16 http://plugins.netbeans.org/plugin/9494/simple-code-metrics 17 Integrated

Development Environment

TABLE II DATA FOR TASK 1 AND 2 ON THE REST FUL S ERVER ID rest01 rest02 rest03 rest04 rest05

Task 1 2 1 2 1 2 1 2 1 2

ST 19:05 21:00 19:10 21:00 19:08 21:00 19:15 21:10 19:07 21:00

ET 20:50 22:20 21:00 22:12 20:56 22:18 21:10 22:17 20:59 22:15

TT(min) 105 80 110 72 108 78 115 77 112 75

No. mod. 104 1010 98 998 101 978 107 987 108 996

TABLE IV T -T EST F IRST S ERVER -S IDE TASK

MC 1,009 0,079 1,122 0,072 1,069 0,079 1,074 0,078 1,037 0,075

Information

soap02 soap03 soap04 soap05

Task 1 2 1 2 1 2 1 2 1 2

ST 19:10 21:30 19:15 21:40 19:07 21:30 19:12 21:30 19:17 21:38

ET 21:23 22:58 21:38 23:00 21:27 22:53 21:27 22:55 21:38 23:00

TT(min) 133 88 143 80 140 83 135 85 141 82

No. mod. 81 1001 80 960 82 974 79 954 83 978

MC 1,641 0,087 1,787 0,083 1,707 0,085 1,708 0,089 1,698 0,083

was concluded that RESTful web services are more maintainable in terms of evolutionarily maintenance than SOAPWSDL web services on the server-side. The outcome of the adaptive maintenance tasks will be detailed after analysis of the maintainability of the evolutionary maintenance task. The F-test yielded a result of p = 0.709 for task 1 and p = 0.676 for task 2, i.e., the variances of both samples were practically equal. After the f -test, the f -test was performed for both sets of data (one set for each group), as shown in Table V. This shows the result of the t-test for Task 2 performed on the server-side. The result of the t-test for matched pairs from the two groups indicates a significant variation (p < 0.05) in all the groups tested, as shown in Table IV. The alternative hypothesis H1 was therefore accepted, and it was concluded that, as measured by the modifiability metric MC, RESTful web services are more maintainable in terms of adaptive maintenance than SOAP-WSDL web services. B. Results on the Client-Side The Table VI shows the data obtained through the evaluation of the web service client-side RESTful: The Table VII shows the results of the data obtained through the evaluation of the web services client-side SOAP-WSDL: To analyze client-side maintainability, the f -test was used to compare the variances of the two data sets. The resulting p value (p = 0.259) indicated that the variances of the two data sets were equal. The results of the parametric Student t-test are shown in Table VIII. These were analyzed using the same approach as in the previous analysis. These results shows that the use of the WSDL file to describe a service interface helps to abstract the communication protocol and data serialization, as well as the implementation platform of the service (operating system and programming language) [16]. Therefore, SOAP engines and WSDL tools effectively encapsulate the complexity of the application developer and

Value -21,543 8 < 0.01 1,062 1,708

Sample standard deviation of group 1: Sample standard deviation of group 2: Pooled standard deviation:

0,042 0,051 0,047

Alternative Hypothesis: Unlike Confidence Interval

0 95%

Lower Limit Upper Limit

-0,715 -0,577

T

TABLE III DATA FOR TASK 1 AND 2 ON THE SOAP-WSDL S ERVER ID soap01

Degrees of Freedom P -valor Average in group 1: Average in group 2:

TABLE V T -T EST S ECOND S ERVER -S IDE TASK


information

Value -4,973 8 < 0.01 0,076 0,085

Sample standard deviation of group 1: Sample standard deviation of group 2: Pooled standard deviation:

0,003 0,002 0,002


0 95%


-0,013 -0,004

T

integrator. Pautasso [16] emphasizes that it is not necessary to study the specifications of a particular piece of software to be able to develop interoperable services if it is assumed that the runtime and the tools selected maintain the basic WSI18 profile. An important feature is that customers can be generated from WSDL contracts automatically. The Netbeans IDE tool allows code for both METRO API and JAX-RS to be generated, but it is important to note that the WSDL description of SOAP-WSDL web services made maintenance on the client-side easier with the automatic generation of code by means of stubs19 . This automatic code generation helped students reduce the number of errors in their maintenance tasks and streamlined the manual task of maintaining the source code. The results of the parametric Student t-test for matched data pairs from the two sets of samples revealed a significant variation (p < 0.05) in all the groups examined. The alternative hypothesis H1 was therefore accepted and it was concluded that SOAP-WSDL web services are more maintainable in terms of adaptive maintenance than RESTful web services on the client-side. Consequently, we conclude that current WS-* tools favor the transformation of existing software components in the web services consumer [16]. VII. C ONCLUSION This paper presented the results of a controlled experiment examining the relationship between the maintainability of RESTful and SOAP-WSDL web services as measured by the 18 WS-I.

Web Services Interoperability. http://www.ws-i.org. components responsible for part of the interaction between clients and services through SOAP messages are called stubs. 19 The

TABLE VI DATA FOR TASK 1 ON THE REST FUL C LIENT ID rest01 rest02 rest03 rest04 rest05

ST 19:50 19:42 19:40 19:45 20:00

ET 21:50 21:53 21:50 22:35 23:25

TT(min) 120 131 130 170 205

No. mod. 341 394 503 408 426

MC 0,351 0,332 0,258 0,416 0,481

TABLE VII DATA FOR TASK 1 ON THE SOAP-WSDL C LIENT ID soap01 soap02 soap03 soap04 soap05

ST 19:38 20:08 19:38 19:40 19:40

ET 22:21 23:17 22:51 21:00 22:00

TT(min) 163 189 193 80 140

No. mod. 1432 1416 981 1140 1177

MC 0,113 0,133 0,196 0,070 0,118

indicators proposed in ISO-9126 [10] focusing on specific software maintenance modifiability sub-characteristics. Descriptive statistical analysis was used. Our experimental results indicate that RESTful web services are more maintainable on the server-side than SOAP-WSDL web services, which are more maintainable on the client-side. However, with the controlled experiment is not possible to reach overall conclusions about the technology of SOAP-WSDL and RESTful web services. For this, a more detailed study should be carried out with different types of applications and with greater complexity, and other metrics and more experienced programmers. Therefore, the present results indicate that companies interested in providing web services over the internet can reduce the associated maintenance costs by using RESTful web services. Conversely, if the company’s goal is to consume a web service over the internet, SOAP-WSDL web services offer lower maintenance costs. R EFERENCES [1] AFOTEC. Software maintainability-evaluation guide, 1996. AFOTEC Pamphlet 800-2. [2] R. K. Avadhesh Kumar and P. S. Grover. An evaluation of maintainability of aspect-oriented systems: a practical approach. International Journal of Computer Science and Security, 1(3), 2007. [3] V. R. Basili. The experience factory and its relationship to other improvement paradigms. In Proceedings of the 4th European Software Engineering Conference on Software Engineering, ESEC ’93, pages 68– 83, London, UK, UK, 1993. Springer-Verlag. [4] V. R. Basili, L. C. Briand, and W. L. Melo. A validation of objectoriented design metrics as quality indicators. IEEE Trans. Softw. Eng., 22(10):751–761, Oct. 1996. [5] A. Eastwood. Firm fires shots at legacy systems. Computing Canada, 19(2):17, 1993. [6] L. Erlikh. Leveraging legacy system dollars for e-business. IT Professional, 2(3):17–23, May 2000. [7] J. Estrella, A. Endo, R. Toyohara, R. Santana, M. Santana, and S. Bruschi. A performance evaluation study for web services attachments. In Web Services, 2009. ICWS 2009. IEEE International Conference on, pages 799 –806, july 2009. [8] J. Estrella, R. Toyohara, B. Kuehne, T. Tavares, R. Santana, M. Santana, and S. Bruschi. A performance evaluation for a qos-aware service oriented architecture. In Services (SERVICES-1), 2010 6th World Congress on, pages 260 –267, july 2010. [9] S. Henry and M. Humphrey. A controlled experiment to evaluate maintainability of object-oriented software. In Software Maintenance, 1990., Proceedings., Conference on, pages 258 –265, Nov. 1990. [10] ISO. ISO/IEC 9126-1:2001, Software engineering – Product quality – Part 1: Quality model. Technical report, International Organization for Standardization, 2001.

TABLE VIII T -T EST OF A C LIENT- SIDE TASK


Information

Value 5,608 8 < 0.01 0,368 0,126

Standard Deviation sample Group 1: Deviation standard sample Group 2: Pooled standard deviation:

0,084 0,045 0,068


0 95%


0,142 0,340

T

[11] M. Kajko-Mattsson. Evolution and maintenance of web service applications. In Proceedings of the 20th IEEE International Conference on Software Maintenance, pages 492–493, Washington, DC, USA, 2004. IEEE Computer Society. [12] M. Kajko-Mattsson. Future of evolution and maintenance in the world of integrated web service systems. In Integrated Design and Process Technology (IDPT-2005). Society for Design and Process Science, 2005. [13] M. Kajko-Mattsson and M. Tepczynski. A framework for the evolution and maintenance of web services. In Proceedings of the 21st IEEE International Conference on Software Maintenance, pages 665–668, Washington, DC, USA, 2005. IEEE Computer Society. [14] G. V. L. Lavazza. Enhancing requirements and change management. In In Proceedings Fourth International Conference on Requirements Engineering (ICRE ), pages 106–15, 2000. [15] J. Meng, S. Mei, and Z. Yan. Restful web services: A solution for distributed data integration. In Computational Intelligence and Software Engineering, 2009. CiSE 2009. International Conference on, pages 1 –4, 2009. [16] C. Pautasso, O. Zimmermann, and F. Leymann. Restful web services vs. "big"’ web services: making the right architectural decision. In Proceeding of the 17th international conference on World Wide Web, WWW ’08, pages 805–814, New York, NY, USA, 2008. ACM. [17] M. Perepletchikov and C. Ryan. A controlled experiment for evaluating the impact of coupling on the maintainability of service-oriented software. IEEE Trans. Softw. Eng., 37:449–465, July 2011. [18] M. P. Prado. A study of characterization and evaluation criteria structural test between procedural and OO paradigms. Dissertation, ICMC/USP, São Carlos, SP - Brazil, 2009. [19] R. Pressman. Software engineering: a practitioner’s approach. Mcgraw Hill: higher Education. McGraw-Hill, 2005. [20] Retrieved on W3C. Web services architecture. Last access: 10/02/2011, 2004. [21] Retrieved on W3C. Web services description language (wsdl). Last access: 15/02/2011, 2007. [22] N. F. Schneidewind. Measuring and evaluating maintenance process using reliability, risk, and test metrics. IEEE Trans. Softw. Eng., 25:769– 781, November 1999. [23] M. Stal. Web services: beyond component-based computing. Commun. ACM, 45:71–76, October 2002. [24] P. Thayer. Software maintainability evaluation methodology. Air Force Test and Evaluation Cen. Rep., June 1978. [25] G. Travassos. Introduction to Experimental Software Engineering. Technical Report RT-ES 590/02, COPPE/UFRJ, 2002. [26] F. P. Y. Wei-Chung Hu, Chia Hung Kao and H. C. Jiau. Vesta: A view-based software quality assessmentmodel for software evolution management. Proceedings of the 22nd International Conference on Software Engineering and Knowledge Engineering, pages p.345–348, 2010. [27] C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén. Experimentation in software engineering: an introduction. Kluwer Academic Publishers, Norwell, MA, USA, 2000. [28] F. Zhuo, B. Lowther, P. Oman, and J. Hagemeister. Constructing and testing software maintainability assessment models. In Software Metrics Symposium, 1993. Proceedings., First International, pages 61 –70, May 1993.