AbstractâThe performance of web services depends to a great extent on the ... data marshalling and unmarshalling solutions do offer the best performance for ...
2011 IEEE World Congress on Services
A Performance Comparison of Web Service Object Marshalling and Unmarshalling Solutions Tommi Aihkisalo, Tuomas Paaso VTT Technical Research Centre of Finland, Kaitovayla 1, Oulu FI-90571, Finland Abstract—The performance of web services depends to a great extent on the efficiency of data transfer and on a number of factors related to it. Such factors are the network’s possible latency, the efficiency of the used data marshalling and unmarshalling scheme, and the resulting length of the chosen message wire format. Several object marshalling and unmarshalling schemes aimed for data transfer in the web service domain were tested. The chosen schemes relied on XML, JSON and other binary alternatives. The practical marshalling and unmarshalling performance was measured and calculated. The results were analysed as regards pure effective marshalling and unmarshalling throughput and also concerning the resulting overhead affecting the efficiency of network transmission. The results show not only the implications on the performance of the chosen marshalling scheme but also the effect of serialization method and format. It was also clearly demonstrated that JSON is not always the best choice over the traditional XML when aiming at a high performance web services.
and also that of the reverse process of unmarshalling them back to memory objects. The data object types for the tests contained text or binary payload content, but also an array of other objects. The text and binary payloads’ size varied from 512 bytes up to 4 Mbytes. The resulting serialized size was observed and used for calculating a practical marshalling and unmarshalling throughput. Finally, a comparison of the throughput results in relation to the resulting overhead was done. The resulting marshalled length depends on the overhead and ability to compress the marshalled data which then affects directly to the second performance factor of the network transfer delay. High marshalling throughput on the server and client with less overhead and compact marshalling results help achieving higher overall web service performance. We have encountered an ambiguous use of terms and denominations in related research and literature regarding the actual meaning of the marshalling and serialization, e.g. in [2]. In this paper marshalling considered a process of transforming the data of an object into such a format, any mutually agreed format, that can be transmitted and subsequently reconstructed as a copy of the object. The format here can be anything from an XML serialized format to a pass-by-reference type of arrangement. In general, marshalling is a wider concept and a superordinate grouping word. Marshalling can be done by serializing and the reverse process is called unmarshalling. Respectively, serialization is a process of transforming an object’s data to any serial byte array, depending on the selected format. The format can vary from a simple raw byte array to an elaborate XML representation. Serialization is a serial data type format oriented marshalling scheme. The reverse process is called deserialization. The practical approaches for the web services domain, such as SOAP requests and responses, are higher level memory objects which are then transmitted into the serialized format as XML. Another example is the REST [3] architectural style, where resource representations, serving the purpose of request or response, are typically serialized to XML or JSON [4] for transmission purposes. The format of marshalled objects has a great impact on the latency experienced in transferring them between the web service entities. A short and compact representation format is fast and resource sparing to transmit especially in slow networks. However, a short and compact representation format is not very straightforward at the end points. Forming and reading a compact and less verbose representation may be a slow procedure, which causes longer overall response delays. The general understanding has been
I. I NTRODUCTION The web services’ performance is largely dependent on three related factors. One of the key factors has to do with the processing capability of the object marshalling and unmarshalling tasks in the server and those of the client end points. Another key factor is the latency of marshalled object transfer, which depends on the verbosity and the resulting length of the used representation format. The last of the key factors is the performance of the underlying service or business logic, which is usually stabilized and solidified. This paper studies and compares the performance of web service object marshalling and unmarshalling technologies. The study was conducted to find out which of the tested data marshalling and unmarshalling solutions do offer the best performance for the web services. The goal is to review the available solutions for choosing a suitable one for a well performing, most likely a RESTful, web service solution capable of handling varying object graphs with a enclosed binary or text payload data. The studied technologies comprise JAXB, Java Architecture for XML Binding, standard [1] based solutions utilizing different types of serializers, various other less standardized marshalling and unmarshalling schemes, and also some specific binary format based technologies. These technologies were studied in terms of their practical performance by executing a series of tests measuring the marshalling and unmarshalling process delays and calculating the achieved throughput. The measurements made for this paper include the time spent on marshalling three differently structured data objects to a serialization format chosen by each marshalling scheme, 978-0-7695-4461-8/11 $26.00 © 2011 IEEE DOI 10.1109/SERVICES.2011.61
122
platform. The other tested solutions were Hessian and Hessian Compressed [18], originating from the binary web service protocol of the same name, XStream’s [14], Simple XML’s [19], JiBX’s [20] and Castor XML [21] technologies and their respective binding schemes. This paper gives a review of the measurement parameters and structures, and presents the results and their analysis in the following sections.
that the JSON format performs better than XML in the situations presented in this paper. In the marshalling process of the tested solutions, serialization is done to a selected format according to the binding rules of each solution. The performance of the complete marshalling process is deemed to depend mostly on the serializer used. Since XML is generally not considered a very well performing format, a significant amount of work has been done with several different approaches to improve the parsing performance of XML. Advanced XML parsing solutions for the web service domain, such as presented in [5], [6], [7] and [8], are available as research prototypes. Some commonly available marshalling and unmarshalling solutions incorporate some of these technologies, e.g. lazy or deterred and pull parsing, as used in WoodStox [9]. The binding itself may introduce specific schema based solutions, in which compilation time binding is used for improving performance. JSON is generally considered as an advanced alternative for the XML. According to [10] and [11], one may also expect JacksonJSON and its binary variant JacksonSMILE, [12], to provide highperformance variants. As our results demonstrate, not all the achieved performance or the lack of it can be attributed to the utilized parser as the used marshalling control mechanisms also have an influence on the results. Data binding is the key feature in the marshalling process, and thus it constitutes one of the key differences between the different solutions. The tested solutions typically support binding in precompiled or on-the-fly manner, where the binding is generated directly before the object processing, and typically by registering the classes with the marshaller during the instantiation. The registration method usually works with a supported object class annotation method or simply using a Java reflection mechanism. Kryo [13] and Xstreams [14], for example, are used for gathering class structure information for marshalling. At the same time, some other solutions, like Google ProtoBuffer [15], rely on additional classes that perform the binding in compilation time. This requires making use of Protobuffer’s non-standard binding definition files, which are used to create the required classes. The compile time binding solutions provide benefits in the overall processing times by shortening the marshaller instantiation times. The binding delay apparent in runtime binding during the marshaller and unmarshaller setup was excluded to put all the tested solutions on the same starting line. 7 out of the total of 18 measured marshalling and unmarshalling schemes were JAXB based. JAXB provides a coherent and standard interface for simple object graph serialization, supporting both compile and runtime binding. The tests executed with JAXB solutions utilized runtime binding. JAXB was attached to several different serialization and parsing mechanisms when available. The serialization formats with JAXB tested for this study varied from XML, binary XML to JSON and to various proprietary binary formats such as FastInfoset [16]. A highly interesting combination was that of JAXB and Apache Axiom [17], an XML object model and parser used on the Axis2 SOAP web service
II. M EASUREMENTS All the tested schemes and their features are listed in Table I. The binding methods are categorized into runtime or compilation time binding classes. These are further classified into methods using standard JAXB, proprietary annotation, and Java reflection registration. Proprietary class annotation is an annotation method using proprietary vocabulary and markings. Reflection registration uses Java’s reflection mechanism to discover the full object graph. Serialization format, for its part, indicates the used default format with the technology and serializer the parser connected to it. The binary encoding method employs the technology’s default method for encoding, including the object’s binary content. Solutions like JiBX and Simple use a highly verbose method mapping each byte as an integer value in separate elements or as an array, which is, generally, very costly in every sense. Most of the solutions rely on the almost defacto Base64, except for the solutions using the pure binary code. A. Measurement Methods The measurements were conducted with a special tool enabling high accuracy timing of the operation delays with a resolution of 1 nano seconds. delayTool [22], the tool used for the purpose, was instrumented in the test execution block. The measurement points were separated so that they covered the complete marshalling and unmarshalling processes, excluding the initiation and setup operations. The initiation and setup operations differ for each solution and their inclusion would not reveal much about the achieved practical throughput when skewing the results. The measurements were repeated for the three different types of objects 500 times per each chancing attribute, i.e. data block or array size, to generate more stabilized results. Further, to minimize the erroneous results and to stabilize everything between each measurement, a pause of 1 second was held. Figure 1 presents the timed processes. The objects, for which marshalling and unmarshalling was timed, were divided into three categories, type A, B and C. This classification was set up to find out the impact of the different characteristics and structures of the marshalled objects. Type A, contained a field text data of varying length, with the following range: 512B, 1kB, 2kB, 4kB, 16kB, 32kB, 64kB, 128kB, 256kB, 512kB, 1MB, 2MB and 4MB. Type B contained the same lengths of binary data, and type C comprised an array containing 1, 2, 4, 8, 16, 32, 64, 128, 256 or 512 objects of similar structure. Each type also had 4 fields of text with 10 bytes of text content each. Type A will produce a serialized format with one field up to 4Mb, while type B
123
Method JAXB FastInfoset Google ProtoBuf JAXB WoodStox JAXB JacksonJSON Kryo JAXB XML (RI) Xstream DOM Xstream StAX Xstream XPP Xstream JSON Simple XML v2.4 Hessian Hessian Compr JAXB JacksonSMILE JiBX StAX Castor JiBX XPP JAXB AXIOM
Binding runtime, JAXB compilation time, proprietary runtime, JAXB runtime, JAXB Registration, runtime, Java reflection runtime, JAXB runtime, Java reflection runtime, Java reflection runtime, Java reflection runtime, Java reflection runtime, proprietary annotation runtime, java reflection runtime, Java reflection runtime, JAXB compilation time, Java reflection compilation time, proprietary compilation time, Java reflection runtime, JAXB
Format FastInfoset (bin. XML) Proprietary binary XML JSON Proprietary binary XML XML XML XML JSON XML Proprietary binary Compressed proprietary binary Binary JSON XML XML XML XML
Serializer/Parser SUN FastInfoset Google Protobuffer Woodstox 4.0 StAX Jackson 1.6 Kryo Woodstox DOM Woodstox XPP3 Jettison Woodstox Hessian Hessian Jackson Smile Woodstox StAX Xerces SAX XPP3 Woodstox
Binary enc. Base64 Proprietary Base64 Base64 Proprietary Base64 Base64 Base64 Base64 Base64 byte integer array Proprietary Compressed proprietary Proprietary Byte values Base64 Byte values Base64
TABLE I T ECHNICAL DETAILS OF THE TESTED MARSHALLING SOLUTIONS
Object
Marshaller
Unmarshaller
Serializer
Unserializer
Marshalled object
Fig. 1.
Measurement arrangement and measurement points.
produces serializations with binary data. Type C produces a lengthy object structure and large serialization results due to the contained array. Figure 2 presents the structures of the object types A, B and C. Most of the tested solutions were JAXB based, with which we were able to use exactly the same class definitions for the objects. JAXB can be used either with schema based precompiled classes or with the annotation mechanism provided. The tests for this paper were conducted using an annotation method where available and possible. However, solutions like Simple required the use of their own annotation vocabulary. The tests were executed on a standard Java version 1.6.0 on the Ubuntu Linux with kernel version 2.6.35 without any extra runtime parameters. The tests were deployed on a laptop using a 2.53 GHz Intel Core 2 Duo two core processor with 2 gigabytes of RAM. The measurement system was developed to utilize internal plug-and-play architecture, which enabled keeping changes minimal and allowed easy changes one from marshalling solution to another. Lazy
Fig. 2.
Structure of the object types A, B and C.
or deterred parsing, which was encountered especially on some XML parsers, like Woodstox, was prevented by accessing each field of the object individually and thus forcing the parser to go through the complete document. Without this method, the parsers utilizing deterred parsing would gain unfair benefit over the others. The minor delay sourcing from this operation, which affects all the tested solutions, will burden each of them equally, which is why the results remain coherent and comparable. Additionally, the length of the marshalled result was observed for the further overhead analysis presented in this paper.
124
B. Calculations
Marshalling
Effective throughput, which is one of the most interesting parts of the results, had to be separately calculated on the basis of the only measured value, i.e. the delay of the marshalling solutions’ operations. The effective throughput was calculated for both serialization and unserialization phases. The calculations were carried out on the basis of the effective length of the data observed. For serialization, the effective length comprises only the length of the contained data alone, excluding the length of the data field names or other overhead. For parsing, the effective length is the length of the complete resulting serialized object. The effective marshalling throughput was calculated as a quotient of the complete effective data length in bytes and of the measured time spent in seconds on serializing it. The effective unmarshalling throughput T P is the effective length of the input serialization result in bytes divided by the measured time in seconds spent on unmarshalling it as in (1), where L is the effective length and T the measured time in seconds. T P = L/T (1)
Effective Throughput [MB/sec]
100.00 80.00 60.00 40.00 20.00
Fig. 3. object.
JAXB JacksonJSON
Xstream XPP
Xstream JSON
Xstream StAX
Xstream DOM
Castor
Simple XML
JiBX XPP
JiBX StAX
JAXB Axiom
JAXB XML
JAXB WoodStox
JAXB FastInfoset
KRYO
JAXB JacksonSMILE
Hessian
Hessian Compr
Google ProtoBuffer
0.00
Effective marshalling and unmarshalling throughput with type A
highest marshalling throughput of 90 MB/sec is achieved by JAXB FastInfoset while the unmarshalling is modest 30 MB/sec. Other good performers utilizing binary formats are Kryo, JAXB JacksonSMILE. The Hessian Compressed is slowed down by its compression phase, which produces a short object serialization but has a negative effect on the resulting throughput. The solutions utilizing XML serialization format, JAXB XML reference implementation had the highest throughput of 60 MB/sec in marshalling with the JAXB WoodStox that reached the rate of 54 MB/sec with JAXB Axiom not far behind with 51 MB/sec. Others barely attained the level of 40 MB/sec marshalling and for the unmarshalling only JAXB Axiom reached higher than 39 MB/sec throughput, others in the XML group were at or well below it. Among the solutions using the pure text JSON format, only JAXB JacksonJSON was able to achieve a comparable throughput of higher than 35 MB/sec in marshalling and 20 MB/sec in unmarshalling. The Xstream JSON was even slightly slower than the variations using XML with it is throughput close to 12 MB/sec in marshalling and unmarshalling.
The overhead is calculated differently for each object type. For object type A, the overhead is expressed as overhead bytes because the data length in these measurements would mean diminution of it when the data size increases. Overheadb is calculated as a subtraction of the observed serialization length and payload length like in (2), where Lserialized is the length of the serialization result and Lpayload is the length of the payload data including the lenght of the data in the object’s other fields. For object types B and C, the overhead, Overhead% , is a relational percentage of the overhead length over the observed serialization length as in (3). Overheadb = Lserialized − Lpayload
Unmarshalling
120.00
(2)
Overhead% = ((Lserialized − Lpayload )/Lserialized ) ∗ 100% (3) III. R ESULTS Due to the massive amounts of data and restricted space available, the results had to be further averaged to a single value on serialization and unserialization over the range of the contained data length and with respect to the array size for object type C. The single effective throughput values with object types A and B are an average value of the results of the 13 measurements with data sizes of 512B to 4MB. For type C, the single throughput value is a mean for the 1 to 512 array sizes.
B. Object Type B The Figure 4 presents the effective throughput results with the object type B in marshalling and unmarshalling. Unlike the type A, the object type B contains a binary data element. The marshalling solutions based on the binary serialization formats supposedly have a great benefit over the text formats as the results show. In the group of binary format based marshallers, the Google ProtoBufer was able to reach a throughput rate of 180 MB/sec marshalling and 120 MB/sec in unmarshalling, which was also the highest results of the all results within the scope of this study. Hessian reached a good level of 90 MB/sec in marshalling and less than 30 MB/sec in unmarshalling. For the rest in the binary format group, only JAXB FastInfoset with JAXB JacksonSMILE was able to reach a marshalling throughput of over 34 MB/sec while unmarshalling
A. Object Type A The Figure 3 presents the effective throughput results with the object type A in marshalling and unmarshalling. The results show generally how most of the solutions are able to achieve a higher marshalling and unmarshalling throughput. The most notable exception with a high marginal is Google ProtoBuffer. It reaches unmarshalling throughput of little bit below 50 MB/sec and almost 100 MB/sec marshalling. The
125
Marshalling
Unmarshalling 18.00
180.00
16.00
160.00
14.00
Effective Throughput [MB/sec]
140.00 120.00 100.00 80.00 60.00 40.00 20.00 0.00
10.00 8.00 6.00 4.00 2.00
Effective marshalling and unmarshalling throughput with type B
Fig. 5. object.
remained around 23 MB/sec and 34 MB/sec for JacksonSMILE. Both Hessian Compressed and surprisingly, also Kryo, were unable to reach over 20 MB/sec in any case. Kryo is way below 6 MB/sec. JAXB XML, JAXB WoodStox and JAXB Axiom as the only text format format solutions were able to attain a rate higher than 28 MB/sec in marshalling. Further, they reached slightly over 20 MB/sec in unmarshalling. From the rest, only JAXB JacksonJSON breaks the level of 20MB/sec both in marshalling and in unmarshalling. The others are well below this either marshalling or unmarshalling.
JAXB JacksonJSON
Xstream XPP
Xstream JSON
Xstream StAX
Xstream DOM
Castor
Simple XML
JiBX XPP
JiBX StAX
JAXB Axiom
JAXB XML
JAXB WoodStox
JAXB FastInfoset
KRYO
JAXB JacksonSMILE
Hessian
Google ProtoBuffer
JAXB JacksonJSON
Xstream XPP
Xstream JSON
Xstream StAX
Xstream DOM
Castor
Simple XML
JiBX XPP
JiBX StAX
JAXB Axiom
JAXB XML
JAXB WoodStox
JAXB FastInfoset
KRYO
JAXB JacksonSMILE
Hessian
Hessian Compr
0.00
Google ProtoBuffer
Fig. 4. object.
Unmarshalling
12.00
Hessian Compr
Effective Throughput [MB/sec]
Marshalling 200.00
Effective marshalling and unmarshalling throughput with type C
and its imminent effect to the transmission duration between the client and server. The serialized length is influenced upon by the overhead resulting from the used serialization format. XML formats tend to be more verbose than JSON, while the binary formats offer the most compact variant. For the further analysis the results, the combined throughput versus overhead, are presented in Figure 6, Figure 7 and 8. In these figures, the best performing solutions, i.e. those with the least overhead and the highest throughput, are located closest to tpe top left corner and the weakest closest to the right corner. It must be noted that Hessian Compressed reaches a negative overhead due to the date compression and it has been marked as 0 in the results regarding the object types A and C. The results in terms of the overhead with the object type A are presented in the Figure 6, where the overhead is expressed as the overhead bytes. The amount of overhead bytes will remain equal over the range of the used sizes of the data element from 512 bytes up to 4MB, which makes the analysis easier while as there is no need to use averaging. The results clearly show that the best solution here is the Google ProtoBuffer with JAXB FastInfoset as the second best option. An equally low overhead value with Google’s solution was reached by Kryo, showing a slightly lower throughput. The weakest performing solutions in both senses are the family of Xstream solutions, Simple XML and Castor. The JSON solutions reach almost as good an overhead as the binary versions, while showing a serious lack of performance in throughput rates. While there are serializing solutions that are clearly faster than Google ProtoBuffer, in terms of its parsing throughput Google ProtoBuffer is in a class of its own. The results show how Google’s solution is the fastest in unmarshalling, while JAXB with FastInfoset is the fastest in marshalling. However, due to the reciprocal incompatibility of the format, it is impossible to use the two solutions together. JAXB Axiom, using standard XML, is a good general performer in both ways, but it is a little slower performer than JAXB JacksonSMILE with its binary
C. Object Type C The Figure 5 presents the effective throughput results with the object type C in marshalling and unmarshalling. This object type containing an array of similar objects resulting in a long internal chain of elements, produced following results. All the binary solutions, except JAXB JacksonSMILE, reached rates varying from 1 to 4 MB/sec in marshalling throughput. Below this rate was JAXB JacksonSMILE attaining barely 0.5 MB/sec both in marshalling and in unmarshalling. Kryo and Hessian were the fastest in this category before Google ProtoBuffer. The text based solutions did stand out here with this object type that produces lengthy and complex marshalling results. However, solutions utilizing XML managed better than any of the JSON solutions. The JiBX XPP proved to be the fastest of all reaching 17 MB/sec in unmarshalling and almost 6 MB/sec in marshalling. Closest to it, JAXB Axiom was able to reach the rate of 4 MB/sec and 8 MB/sec in marshalling and in unmarshalling respectively. The JSON solutions were disappointing and barely reaching 1 to 2 MB/sec. IV. A NALYSIS The simple effective throughput is not always the most important factor in many cases. Another key factor especially evident in the web service domain is the serialized length
126
160
350
Combined Throughput [MB/sec]
140 120
Combined Throughput [MB/sec]
Google ProtoBuf
JAXB FastInfoset KRYO
100
JAXB JacksonSMILE JAXB Axiom
80
JAXB XML JAXB WoodStox
JiBX StAX JiBX XPP
60
Hessian Castor
JAXB JacksonJSON
40
Xstream DOM Xstream XPP Xstream JSON
20
Xstream StAX
Simple XML
300 250 200 150
Hessian
100 50
0 0%
0 50
100
150
200
Overhead [Bytes]
Fig. 6. object.
JAXB {FastInfoset, Axiom, XML, WoodStox, JacksonJSON} Castor Hessian Compr Xstream{JSON, XPP, KRYO Xstream DOM StAX}
JAXB JacksonSMILE
Hessian Compr
0
Google ProtoBuf
20 %
40 %
JiBX StAX Simple XML
60 %
JiBX XPP
80 %
100 %
Overhead [%]
Effective marshalling and unmarshalling throughput with type A
Fig. 7. object.
Effective marshalling and unmarshalling throughput with type B
in both marshalling and unmarshalling. The only restricting feature of Google ProtoBuffer has to do with compilation time binding. That of course depends on the application scenario itself. Hessian shows very high marshalling rates but a surprisingly low unmarshalling throughput. JAXB FastInfoset and JAXB JacksonSMILE, which are both based on the JAXB standard interface, proved to be fine candidates, with the former producing twice as high rates of marshalling throughput as the latter, while showing equal performance in unmarshalling.
JSON format, or the purely binary based Kryo. The other tested solutions are average or substantially below the average. Only JAXB with Jackson JSON is able to reach an average level equal to that of Hessian. When looking for a standard based solution and binary XML is not considered a drawback, JAXB FastInfoset seems a good overall solution for object type A. If pure text payload is desired, JAXB Axiom offers a good high performance solution with a good unmarshalling performance. In terms of serialization formats, JSON turned out to be a disappointment in every sense. The Figure 7 illustrates the results for the object type B with a binary content payload of the length of 512b to 4MB. All of the solutions using Base64 as the binary to text encoding method reach relatively low throughput rates, and also almost the same overhead. The solutions producing standard XML, binary XML and JSON with JAXB interface, JAXB FastInfoset, JAXB Axiom, JAXB XML, JAXB WoodStox and JAXB JacksonJSON produce roughly the same overhead and combined throughput, 25% and 50MB/sec respectively. Only JiBX and Simple XML are still weaker in their performance due to their way of inserting every single byte value as an integer in separate elements in the XML presentation, which is why they are not even capable of reaching a throughput of 50 MB/sec. While binary methods like Kryo, JAXB JacksonSMILE and Hessian, produce only low levels of overhead as expected, they do show flaws in the throughput unlike Google Protobuffer, which shows a very high combined throughput of over 300 MB/sec and very low overhead. The most apparent conclusion we can make right away is that type B objects with binary content elements favour the solutions that use binary as their serialization format. We were, however, surprised at how the purely binary based Kryo lacks in performance with objects of this type, while reaching reasonable and good levels with other types. Google ProtoBuffer is clearly the first choice as regards performance
Figure 8 shows the results for the object type C. With this object type, the two solutions standing out of the set are JiBX XPP, which has the highest combined throughput but quite high overhead, and Kryo, which shows third highest throughput and the lowest overhead. JiBX with an StAX serializer and parser does not show very good results. All the XML based solutions generate an overhead around 75% or more while JSON solutions produce only a slightly weaker result of 70%, but still lacking in the throughput performance. JAXB Axiom is able to reach the second highest throughput of 11MB/sec. Here the Google ProtoBuffer has slowed down to a level of 5 MB/sec level and JAXB JacksonSMILE to 2 MB/sec. The substantially highest throughput of over 7 MB/sec in this category was reached by the JiBX XPP solution, which in effect, is not a very good result. The other good pure text XML candidate is JAXB Axiom. Even Google ProtoBuffer, with its good performance with object types A and B, falls under 5 MB/sec. The JSON solutions are, once more, lagging far behind the general average level. V. D ISCUSSION It is apparent that a simple comparison of the practical effective throughput might be misleading in the web services domain. In this domain, the transfer in the network plays a big role. Reducing the length of the transferred data improves the overall performance. The factor affecting to the length is the used object marshalling format. We can clearly see that some
127
to a mobile terminal with restricted network bandwidth and computing resources. Another advantage for the mobile terminals and their restricted resources is the general lightness of the JSON resulting possibly a higher overall throughput that includes the network transmission. The binary alternative, JacksonSMILE, however, was able to perform better. With the FastInfoset, Hessian Compressed was the only one utilizing some data compression. This resulted with the objects containing text payload or otherwise lengthy strucuted data a negative overhead. Very low overhead was reached also with the binary payload content. However, the compression and decompression delays processing resulting in a low throughput in marshalling and in unmarshalling. XStream is a bad performer in all cases with any of its available serialization formats so there is no real justification to use it at all. In comparison to other solutions using the same serializer the results lack far behind. E.g. Xstream StAX using Woodstox XML parser falls far behind JAXB WoodStox solution indicating that the marshalling and unmarshalling layer is suboptimal in Xstream solutions.
25
Combined Throughput [MB/sec]
JiBX XPP
20
15 JAXB Axiom
10 KRYO
JiBX StAX
Hessian
5
JAXB WoodStox
Hessian Compr
0 0%
Google ProtoBuf JAXB FastInfoset JAXB JacksonSMILE
JAXB XML JAXB JacksonJSON Xstream JSON
10 %
20 %
30 %
40 %
50 %
60 %
Castor
70 %
Xstream {DOM, XPP, StAX}
Simple XML
80 %
90 %
Overhead [%]
Fig. 8. object.
Effective marshalling and unmarshalling throughput with type C
formats are more favorable to certain kind of object graph structures. The binary formats favor heavy binary and text payloads d XML and JSON text favor lenghty array structures. The Google ProtoBuffer is the best overall performer with other object types, if generally high throughput is sought, but falls far behind in throughput when the length of the structured data increases in the form of the array in the object graph. Other downside is the usage of the proprietary binding scheme. The Xstream family of the solutions likewise JiBX, Castor and Simple XML are the worst performers in throughput and overhead wise with only exception of the substantially high throughput of JiBX XPP with the complex object types. JiBX’s compilation time binding with the usage of the Java reflection mechanism is very convenient. JAXB based solutions do offer convenience of the standardized interfaces and various binding methods of which only runtime binding of POJO was used for this study. JAXB with FastInfoset format reduces the overhead by implementing a binary XML but also thanks to its method of avoiding redundancy in the repeating elements. Of the JAXB based solutions, the conventional text XML using JAXB Axiom was the highest performing solution if binary XML of FastInfoset is out of question. The low throughput results in all tests of the JacksonJSON and the usage of the JAXB binding scheme with the other parsers indicate a throughput problem in the Jackson JSON implementation, not in the JAXB itself. However, the analysis of the results in relation to the resulting marshalled data overhead revealed more information. As the simple throughput results show the text JSON does not achieve very good results and not even in relation to the overhead with any of the tested object types. Currently, even the fastest JSON solutions do not outperform the best XML solutions, yet alone the binary format based solutions, in a simple throughput comparison. Anyway, the slightly smaller length of the JSON presentation may be very beneficial if the serialization is being transmitted
VI. S UMMARY We measured and calculated the throughput of the several object marshalling solutions typically used in the web service domain for three different types of objects. We also observed and calculated the associated resulting overhead in the marshalled objects. The results were presented and compared first as simple marshalling and unmarshalling throughputs and then in relation to the resulting overhead. The JSON based marshalling solutions seem to be seriously lacking in throughput performance, which undermines the benefit of these solutions gained in terms of a slightly better overhead accumulation. However, JAXB JacksonSMILE with its binary JSON presentation proved to be at least an average performer. The other binary methods proved to be generally good performers with Google’s solution standing out in most of the test cases. JAXB Axiom, which is one of the pure text based solutions, shows good general performance, i.e. quite a high throughput with an equal level of overhead as any other of the XML solutions. However, it must be noted that Google’s solution has not been standardized unlike the FastInfoset format. If standardization does not play any major role in the solution and human readability is not needed, Google offers a good choice as regards the format but also considering the available JAXB standard API. Another reasonably good choice is offered by JAXB Axiom, contributing also to the SOAP performance on the Axis2 platform. JSON based solutions proved to be an overall disappointment. ACKNOWLEDGMENT The authors would like to thank Pekka Paakkonen (VTT) for providing delayTool for the measurements and ITEA2CAM4HOME and CELTIC-MOTSWAN projects for the funding.
128
R EFERENCES [1] J. Fialli and S. Vajjhala, “Java architecture for XML binding (JAXB) 2.0,” Java Specification Request (JSR) 222, October 2005. [2] V. Ryan, S. Seligman, and R. Lee, “Schema for representing Java(tm) objects in an LDAP directory,” United States, 1999. [3] R. T. Fielding, “Architectural Styles and the Design of Network-based Software Architectures,” 2000. [Online]. Available: http://www.ics.uci.edu/ fielding/pubs/dissertation/top.htm [4] D. Crockford, “The application/json Media Type for JavaScript Object Notation (JSON),” jul 2006. [Online]. Available: http://www.ietf.org/rfc/rfc4627.txt [5] W. Zhang and R. van Engelen, “High-performance XML parsing and validation with permutation phrase grammar parsers,” in Web Services, 2008. ICWS ’08. IEEE International Conference on, 2008, pp. 286 –294. [6] T. Takase and K. Tajima, “Lazy XML parsing/serialization based on literal and DOM hybrid representation,” in Web Services, 2008. ICWS ’08. IEEE International Conference on, 2008, pp. 295 –303. [7] R. A. Van Engelen and K. A. Gallivan, “The gSOAP toolkit for web services and peer-to-peer computing networks,” in Proceedings of the 2nd IEEE/ACM International Symposium on Cluster Computing and the Grid, ser. CCGRID ’02. Washington, DC, USA: IEEE Computer Society, 2002, pp. 128–. [Online]. Available: http://portal.acm.org/citation.cfm?id=872748.873202 [8] W. Zhang and R. van Engelen, “A table-driven streaming XML parsing methodology for high-performance web services,” sep. 2006, pp. 197 –204. [9] “Woodstox - home.” [Online]. Available: http://woodstox.codehaus.org/ [10] “Japex report: Test suite for json drivers.” [Online]. Available: http://www.cowtowncoder.com/perf/json-bind-2009-09-23/index.html [11] “Benchmarking - thrift-protobuf-compare - project hosting on Google code.” [Online]. Available: http://code.google.com/p/thrift-protobufcompare/wiki/Benchmarking [12] “Jackson JSON processor - home,” id: 1. [Online]. Available: http://jackson.codehaus.org/ [13] “Kryo - project hosting on Google code.” [Online]. Available: http://code.google.com/p/kryo/ [14] “Xstream about xstream.” [Online]. Available: http://xstream.codehaus.org/ [15] “Developer guide - protocol buffers - Google code.” [Online]. Available: http://code.google.com/apis/protocolbuffers/docs/overview.html [16] ISO, “Generic applications of asn.1: Fast infoset,” ISO/IEC 24824-1, Tech. Rep., 5 2005. [17] “Axiom - apache axiom - the xml object model.” [18] “Hessian binary web service protocol,” id: 1. [Online]. Available: http://hessian.caucho.com/ [19] “Simple 2.4.1.” [Online]. Available: http://simple.sourceforge.net/ [20] “JiBX: Binding XML to java code.” [Online]. Available: http://jibx.sourceforge.net/ [21] “The Castor project.” [Online]. Available: http://www.castor.org/ [22] P. Paakkonen, J. Prokkola, and A. Lattunen, “Instrumentation-based tool for latency measurements,” in International Conference on Performance Engineering ’11. ACM, March 14-16, 2011 2011.
129
