A Comparative Evaluation of EJB Implementation Methods Andreas Stylianou Software Engineering Division Amdocs Dev. Ltd., Cyprus
[email protected]
Giovanna Ferrari School of Computing Science Newcastle University, UK
[email protected]
Abstract As E-businesses are becoming ubiquitous, enhancing the performance and scalability of ebusiness systems has become an increasingly important topic of investigation. As Vitruvius (70-25 BC) put it succinctly ‘function follows form’, the ability of a system to perform well and scale easily is influenced by how the system itself is formed or implemented. A common approach to implement ebusiness systems is to make use of off-the-shelf enterprise middleware systems, such as a J2EEcompliant application server. Such middleware systems handle several, often complex, issues and thus simplify application development. They however allow developers the freedom not to use particular forms of support they offer and build their own mechanisms instead. This flexibility gives rise to many implementation methods. The work reported here evaluates these methods for Response Time and Throughput under various environments related to both client side (external to the system) and application execution (internal). To this end, one of the most widespread technologies used by the industry, the Enterprise Java Beans (EJB), is chosen; we have considered six commonly used implementation methods for an e-auction application and five different clientside and execution environments. The resulting study, which involves 78 experimental runs, identifies the strengths and the weaknesses of each implementation method under 13 different scenarios. It thus offers reliable guidelines for developers and valuable insights to researchers.
1. Introduction Success of an e-business depends on the system providing the best performance in the environment it has been deployed. The latter in turn depends on how the system itself has been structured and implemented [14]. E-business systems can be built in many ways.
Paul Ezhilchelvan School of Computing Science Newcastle University, UK
[email protected]
They can be built almost ‘from scratch’ (e.g., using Java Servlets); alternatively, and more commonly, by using off-the-shelf enterprise middleware systems, such as a J2EE (Java 2 Platform Enterprise Edition) compliant application server [18]. These middleware systems, commonly referred to as application servers, offer support for handling several, often complex, issues related to transactions, database (DB) interactions, concurrency control, clustering etc., which need to be managed for the deployment of many ebusiness applications. They do not however constrain an application developer into using all, or even only, the infrastructure support they have to offer. For example, a developer can choose to achieve data persistence in an application specific manner rather than use the mechanism native to the application server. This flexibility gives rise to a variety of ways in which an application can be implemented using an application server. The objective of this paper is to investigate comprehensively the impact these implementation options have on Response Time (RT) and Throughput (TH) of an implemented system. To make this objective achievable and the underlying task tractable, we keep the context of our investigation quite focused, and meaningful at the same time, as pointed out by the related works below. Brebner et al. [4] evaluate and compare the performance of application servers from six leading vendors. They show that the servers can exhibit significant differences in performance which become more pronounced as the application is scaled to run on several server nodes within a cluster. Similar conclusions have also been drawn in [17]. The investigation reported in this paper is therefore concerned only with a J2EE-compliant application server, viz. JBoss Server [6]. The choice of JBoss server is driven mainly by the fact that it is opensource and is considered to be the most widely used amongst the open-source servers. Enterprise Java Bean (EJB) is the core of J2EE programming model. EJBs implement the business logic of the enterprise [11] and “provide a level of
indirection, allowing the application logic to execute on different machines” [1]. An enterprise application can be implemented using EJBs mainly in five different ways (which will be described shortly). Common to all these implementation methods are configuration parameters defining the maximum number of application server threads and the DB connections made available to running of the application. According to [17], implementations need to be configured differently for optimal performance and that optimal configuration will depend on the nature of applications. For example, a manufacturer application is observed to require more DB connections to perform better, whereas an auction application can do with fewer ones; also, the latter respond well if fewer application server threads are concurrently serving client requests and hence competing for resources. In this paper, we will focus on an e-auction application, a representative example of many e-business applications; furthermore, we will fix the configuration parameters to the default values of the JBoss server. Analyzing the impact of varying these parameters on system performance is a nontrivial task and we refer the reader to [7], [8] and our earlier work [9]. The choice of JBoss server and of the application is also motivated by an earnest attempt to address the seemingly contradicting opinions found in the literature. The authors of [2] consider J2EE servers from two different vendors and investigate the server performance for several implementation methods. They conclude that the implementation using Java Servlets generally scales better than the one using Session Beans, which in turn scales far better than the implementation with Entity Beans and Container Management Persistence. On the contrary, the authors of [3] state that the conclusions of [2] are misleading and that all implementation methods can perform equally well provided that the configuration parameters are tuned properly; they however put a caveat, as we noted earlier, that it is not straightforward to predict the effects of varying configuration parameters on performance. The purpose of our investigation is to cross-evaluate various EJB implementation methods under default settings and expose the trade-offs available to the application service provider who is assumed not to possess the means to predict the optimal settings for configuration parameters. This work is therefore orthogonal to those [7], [8], [9] which dynamically evaluate the optimal settings for any given implementation method even while the application may be subject to (unpredictably) varying loads. The paper is organized as follows. Section 2 introduces the experimental set-up and the sub-systems
we have built exclusively for evaluation. Section 3 details the series of experiments we have conducted over different scenarios, the performance measured and the observations derived. An extended analysis of the application implementation methods employed is provided in Section 4. Section 5 concludes the paper.
2. Experimental set-up The experimental set-up, as shown in Fig. 1, is composed of (i) the target application system (RUBiS), (ii) the client-base, and (iii) the deployer system.
Fig. 1: Experimental set-up components
2.1. The target system RUBiS (Rice University Bidding System) [10] is the e-auction application system subject to comparative evaluation of implementation methods. It defines several interactions that can be performed from the client’s Web browser, such as browsing for items, bidding, buying or selling items and leaving comments. A majority of these interactions require data to be generated dynamically, involving reading and/or writing data to database (DB). In our study, RUBiS is implemented using six different methods, of which five use, in varying degrees, the infrastructure support or containers offered by the underlying JBoss server. These methods are succinctly described below. Stateless Session Beans: They are business processes (here, e-auction processes) that implement business logic, business rules, algorithms and workflow; each bean represents the unit of work to be performed in response to a given client call. While being stateless, the contents of instance variables are not preserved between successive method calls. Entity Beans with Container Managed Persistence (CMP): An entity bean represents a business object in persistent store. It is uniquely associated with a primary key and hence can be located and accessed directly by clients. In CMP, entity beans rely on containers to manage data persistence and therefore a vast majority of the DB queries are generated by the EJB containers themselves. Entity Beans with Bean Managed Persistence (BMP): The developer produces the code to
manipulate the DB and essentially to perform all the work done by the container in the CMP implementation method. Considering BMP implementation method allows us to evaluate the relative cost of the container's persistence service. Entity Beans with Session Façade Beans: Session façade uses stateless session beans to reduce the number of business objects that are exposed directly to the client over the network. Calls between façade and entity beans are local to the EJB server and are usually optimised to reduce the overhead of multiple network calls. This implementation method obviously involves the use of a larger number of beans and thus stresses the component pooling of the container. Furthermore, containers are used for managing persistence. EJB Local Interfaces: This method involves using entity beans with only local interfaces and session façade beans with remote interfaces. Due to this arrangement, interactions between session and entity beans are efficiently implemented bypassing the communication layers. Servlets only: Java Servlets are not part of the EJB technology. Implementation based on them has been considered here to assess the merits of using the EJBs themselves. Servlets allow Java code to be run on the Web server and HTML pages to be sent to a browser. Using the features of Java, they completely replace any functionality provided by the JBoss server. In particular, they manipulate the DB (using use the JDBC interface) and use the Java built-in synchronisation mechanisms to prevent deadlocks when accessing shared DB objects.
2.2. Client base The client-base emulates clients accessing the target system using their Web browsers. Moreover, it allows emulation to be configurable so that system performance can be studied under various settings. Using the client base, the evaluator can vary the following aspects of clients’ interactions with the target system: • The nature of interactions concerning the DB access which can be read-only, write-only or readwrite; • The sequence of interactions generated by a client can be fixed or be randomly composed; • The ‘thinking time’ – the amount of time a client takes between successive interactions; and, • The number of concurrent clients interacting with the target system; In addition, the evaluator can also vary settings concerning the interactions between RUBiS and DB
(irrespective of the implementation method used for RUBiS). They are: DB Updates: Changes to object states can be committed to the DB in a transactional (ensuring atomicity) or non-transactional manner. The former requires running an explicit commit protocol, commonly the 2-phase commit protocol. Cache Commit options: These options relate to policies for managing consistency of cached data in the context of transactions [17]. In option A, bean instances have exclusive access to object states in persistent storage. Consequently, it is not necessary to synchronise the object states (in cache) with those in the DB at the beginning of a new transaction. This option corresponds to pessimistic locking. Option B is optimistic and forbids bean instances to have exclusive access to object states in persistent storage; thus the cached object states at the end of a transaction must be verified for consistency before starting another transaction. That is, the object states cached at the beginning of the transaction must match those in the DB. Option B is the default one for JBoss servers. Note that the differences between these two options are meaningful only when DB updates are transactional.
2.3. Deployer It resides between the target system and the clientbase (see Fig. 1). Its role is to realise the evaluator’s settings governing the RUBiS-DB interactions, initiate the experimental runs and to monitor RUBIS RT and TH during these runs. During an experimental run, RUBiS beans are appropriately initiated, the corresponding methods invoked and the measurements taken. The run terminates when all the clients’ interactions have completed their execution. Once done, the deployer returns the RT and TH measurements to the client-base in a meaningful format. The software environment of our experimental setup consists of: Apache Web server [15], JBoss application server [16], both running on J2SDK 1.4.1; the JBoss container implements the EJB 2.1-6 [11] specification; the DB server was mySQL. The experiments were conducted on an Intel Pentium4 CPU 2.8GHz processor with 2GB physical memory, running Suse Linux 10.0 2.6.13-15-smp, i386 operating system. The deployer employed Apache JMeter [12] for performance measurements and output generation
3. Experimental process The experimental process consists of five experiments each of which focus on one specific aspect
of Client-RUBiS or RUBiS-DB interactions. The aim is to demonstrate how the six different implementation methods are influenced when that specific aspect alone is changed. The first two columns of Table 1 indicate the experiment numbers with the corresponding aspects. Table 1: Experimental scenario Exp
Focus
1
DB Access
2
3
4 5
Scenario 1
Parameter Read-only op (R)
2
Write-only op (W)
3
Read-write op (R/W)
4
Min TT (1ms)
5
TT = Sess T. (300ms)
6
TT > Sess T. (1000ms)
7
Large (100 clients)
8
Small (1 clients)
9
Medium (10 clients)
DB updates
10
Tx updates
11
Non-tx updates
Cache
12
Tx C-Commit A
13
Tx C-Commit B
Think Time (TT) No of clients
Note that a given aspect can be changed in multiple ways. Hence each experiment itself considers multiple scenarios and the aspect under focus is set in one particular way (parameter) in each scenario. The last two columns of Table 1 depict the scenarios and the chosen parameters. Experiment 1 considers three scenarios in which DB accesses mandated by client interactions result in read-only, write-only and readwrite operations. Performance measured in these scenarios will indicate how a given implementation method performs as the DB access pattern is varied. If an implementation method performs exceptionally well in scenario 1, then it can be concluded that it is best suited to e-business applications that involve mostly read-only access pattern. Experiment 2 focuses on user think time, set to 1, 300, and 1000 ms in scenarios 4, 5, and 6 respectively. The chosen values also reflect three distinct cases in which think time is less than, equal to, and larger than the session time – the maximum time allowed before a connection to an interacting client is closed by the JBoss server. Thus, if the think time is larger than 300 ms, the later interaction will require a new connection (or session) to be set-up; otherwise, the existing connection will be re-used, saving the connection setup time. Moreover, if the think time is 50 ms, then a
client can interact a maximum of 7 times without having to establish a new session. Note also that if the think time is equally likely to be between 1 ms and 1000 ms, then the approximate average performance for a given implementation method will be the average over the performance measured in these three scenarios for that method. Similarly, in Experiment 3, if the number of concurrent clients accessing the application is equally likely to be between 1 and 100, then the approximate average performance for a given implementation method will be the average of the performance measured for that method in scenarios 7, 8 and 9. Experiment 4 determines the effect the DB Updates has on performance by considering transactional and non-transactional update scenarios. Finally, the effect of caching options is analysed in scenarios 12 and 13 of Experiment 5. Thus, overall 78 experimental runs will be carried out covering 13 scenarios for 6 implementation methods. The main frames of reference used during the experimental process are server RT and TH. The former is measured in milliseconds (ms) as the time taken for the server to receive and complete an HTTP request from the emulating client. TH can be described as “the requests per minute the server handled” [12]. In addition to measuring TH (as req/s), we also measure the data handled by the server as (KB/s). These two metrics, req/s and KB/s, indicate the extent to which the implementation methods stress the server in maintaining a given TH value. If a method stresses the server less (by recording low KB/s), then there is additional capacity within the server for increasing the number of server threads which is likely to result in an increase in req/s; on the other hand, if the server is already stressed for a given req/s, then increasing the number of server threads is unlikely to result in a proportional increase beyond that req/s. Next, we present the results obtained from each experiment separately, followed by a discussion of each scenario which includes an analysis of the observed performance.
3.1. Experiment 1 – DB access Experiment 1 consists of three different scenarios with the only difference between them being the way the DB is accessed; all other parameters are identical. A client interaction gives rise to: In scenario 1: Read only DB operations (R): Browse items and View information about users/items/bids In scenario 2: Write only DB operations (W): Register items and Register users In scenario 3: Read-Write DB operations (R/W): Browse items, Buy, Place bids, Register items,
Response time (ms)
7000 6000 5000
Entity Beans with BMP Entity Beans with CMP EJB Local Interfaces
4000
Servlets only
3000 2000
Entity Beans with Session Facade
Stateless Session Beans
1000 0 Read-only
Write-only
Readwrite
Fig. 2: DB access: variation in RT It is interesting to discover that Stateless Session Beans and Entity Beans with Session Façade Beans do not follow the expected pattern: their RT is smaller in write-only and read-write operations. This is mainly due to the fact the given implementation methods do
not retain their states between operations, i.e. they do have to store information between method calls from the clients. The implication is that the connection needs to be re-established after each method call, which naturally adds to the RT. Consequently, a read operation takes two DB accesses where as the first four methods may incur no DB access due to caching. When it comes to writing, the last two methods write directly on to the DB whilst the first four access and then update DB objects which takes significantly longer. Other interesting observations are: Entity Beans with BMP take much longer to write to the DB than every other method. This can be attributed to efficient implementation of JBoss container services which are not made use of in BMP. Also, Entity Beans with Session Façade Beans perform fastest when operations involve both read and write; this is one reason, this is the default implementation method in EJB 3.0 standard. 60 50
Requests/sec
Register users, Comment and View information about users/items/bids. In this experiment the sequence of interactions is fixed throughout the experiment; this means that clients invoke the operations in the order indicated above for each scenario, i.e., in scenario 3, ‘Browse items’ is followed by ‘Buy’, which in turn is followed by ‘Place bids’ and so on. When the sequence parameter is stated as random, clients will invoke operations chosen randomly from the list of interactions indicated for a given scenario. The parameters that remain fixed throughout the experiment are: • Transactional (False), i.e. non-transactional • Number of clients (10) • Session time (300 ms) • Thinking time (50 ms) • Sequence (Fixed) • Commit option (B) The main purpose of this experiment is to determine the variations in performance due to the way the DB is accessed. A basic concurrency control policy is followed whereby concurrent threads are allowed for read and write locks but with exclusive write access. Observations: Read-only operations (see Fig. 2) are expected to have significantly lower RT than writeonly and read-write operations, as they do not lock the DB object they interact with and thus allow concurrent access from other clients at the same time. In the other two cases (write-only and read-write operations) the RT is much higher due to the fact that once the shared resource (DB) is accessed by a particular client, then another client wanting to access the same resource must wait until the lock is released, and this naturally increases the server's RT. Reason for this is the fact that clients are persistent, in the sense that they do not give up until they successfully complete the interaction they are intended to.
40
Read-only
30
Write-only Read-write
20 10 0 Entity BMP
Entity CMP
Local Interfaces
S e rvlets only
Stateless Session
Session Facad e
Fig. 3: DB access: variation in TH The highest TH is observed (see Fig. 3) for readonly operations for all implementation methods whilst write-only operations provide the lowest TH for all methods except for the local interfaces implementation, in which read-write operations are the lowest in TH. In relation to the average TH for each implementation method (i.e. a combination of read-only, write-only and read-write operations), the observations are that TH ranges at similar levels for all six implementation methods, the lowest being BMP (37.83 req/s), and the highest being EJB Local Interfaces (47.83 req/s). On average, read-only operations achieves the highest TH with 50.02 req/s, read-write operations slightly under-achieves with 42.43 req/s whereas writeonly operations has the lowest TH of 35.43 req/s.
3.2. Experiment 2 – Thinking time Experiment 2 focuses on client thinking time and the effect it has on TH and server RT. Thinking time between interactions is the rate at which requests are sent to the application server (see subsection 2.2). This implies that the shorter the thinking time, the greater is
Response time (ms)
17500 15000 12500 10000 7500 5000 2500 0
Entity Beans with BMP Entity Beans with CMP EJB Local Interfaces Servlets only Stateless Session Beans Entity Beans with Session Facade
Min thinking time (1ms)
Thinking time equal to session time (300
Thinking time greater than ses-
Fig. 4: Thinking time: variation in RT It is obvious from Fig. 4 that all implementation methods behave very similarly when the thinking time is much greater than the session time. The RT for each method is relatively high as well as TH (see Fig. 5); there is very small variation amongst the six implementations. It is therefore safe to assume that if the rate at which requests are sent to the application server is much greater than the session time itself, then all different application implementation methods are expected to provide similar TH and RT. The results are somewhat similar in the other two cases (1 ms and 300 ms) where a slightly greater deviation is observed between the six methods. EJB
Local Interfaces perform better than any other method in the minimum thinking time scenario, and the same implementation is second best in the 300 ms scenario. The average TH (see Fig. 5) is again almost identical for all methods, with the lowest being Entity Beans with BMP (27.00 req/s) and the highest being Entity Beans with Session Façade Beans (32.03 req/s). 50.00
Requests/sec
the load on the application server. We devise three scenarios to realise how thinking time affects performance. Scenarios 4, 5 and 6 comprise the second experiment: scenario 4 executes the emulation with the minimal thinking time of 1 ms; scenario 5 's thinking time is the same as the session time, that is 300 ms; and finally scenario 6 incorporates a thinking time much greater than the session time, that is 1000 ms. The idea behind these three scenarios is to test the application by emulating the two extreme cases (in terms of thinking time) with respect to the session time, which are a minimum value and a value much greater than the session time. We enforce the experiment with an intermediary value for thinking time, which is equal to the session time (300 ms). The access pattern followed is listed below and is identical for all three scenarios: The list of interactions during a client session is: Browse items, Buy, Place bids, Register items, Register users, Comment, and View information about users/items/bids. Fixed parameters: • DB access (R/W) • Transactional (False) • Number of clients (1) • Session Time (300 ms) • Sequence (Random) • Commit option (B) Observations: RT with minimum thinking times (scenario 4) is considerable lower for all cases in comparison with the other two scenarios (5 and 6).
40.00
Min thinking time (1ms) Thinking time equal to session time (300 ms) Thinking time greater than ses sion time (1000 ms)
30.00 20.00 10.00 0.00 En - En - Loc- S e rtity tity al In- vlets B M P C M P teronly face
State less Session
Faca de Bean s
Fig. 5: Thinking time: variation in TH On average, the highest TH is 32.03 req/s for the Entity Beans with Session Façade Beans, and the lowest is just 27 req/s for the Entity Beans with BMP. Roughly speaking, the TH for the minimum thinking time is approximately twice as high as the TH for the maximum thinking time, for five out of six implementations. Overall, the worst performance is given by the implementations running with the highest thinking time. They achieved 15.60 req/s. This is mainly due to the clients' compulsory wait between interactions, which naturally adds to the RT, hence the poor TH. The other two scenarios record similar results, with the implementations running under a thinking time equal to the session recording 35.85 req/s, a slightly worse result than the 39.93 req/s that is recorded when the thinking time is at the lowest. Moreover we noticed that the servlets implementation seems to be more efficient in terms of the amount of data handled and therefore the server can be stressed further (by increasing the number of threads) with little risk of encountering resource saturation.
3.3. Experiment 3 – No. of concurrent clients In this experiment we are dealing with the number of simulated clients and how this affects in various ways the underlying emulation and the overall performance. We incorporate three scenarios for this purpose with 100 clients on scenario 7 (high concurrency), 1 client on scenario 8 (no concurrency), 10 clients on scenario 9 (medium concurrency), and the following access pattern: The list of interactions during a client session: Browse items, Buy, Place bids, Register items,
5.00
Requests/sec
4.00
Entity Beans with BMP Entity Beans with CMP
3.00
The fourth experiment focuses on transactional versus non-transactional DB updates. Scenario 10 implements transactional updates, whereas scenario 11 implements non-transactional updates. The access pattern involves read-write operations: 50.00 40.00
Requests/sec
Register users, Comment, View information about users/items/bids Fixed parameters: • DB Access (R/W) • Transactional (False) • Session Time (300) • Thinking Time (100) • Sequence (Fixed) • Commit option (B)
EJB Local Interfaces Servlets only Stateless Session Beans Entity Beans with Session Facade
10.00 0.00
Servlets only
Transactional
Stateless Session Beans Entity Beans with Session Facade
1.00
Non-transactional
Fig. 8: Transactional DB updates: TH
0.00 10 clients (x 500 threads)
1 clients (x 500 threads)
Fig. 6: Varying no. of clients: average TH Observations: As Fig. 6 shows, there is a vast difference in TH as the number of concurrent clients varies; this is mainly due to the fact it is far easier and faster for the server to complete a single client's request rather than 10 or 100 clients simultaneously. 250000 Entity Beans with BMP
200000
Entity Beans with CMP EJB Local Interfaces
150000
Servlets only Stateless Session Beans
100000
Entity Beans with Session Facade
50000 0 100 cli ents (x 500 threads)
1 clients (x 500 threads)
10 clients (x 500 threads)
Fig. 7: Varying no. of clients: average RT When it comes to RT, Servlets only respond swiftly when the concurrent clients are at the maximum (see Fig. 7), while the Entity Beans with Session Façade Beans is the slowest. However, the latter responds the best when the number of concurrent clients drops to 10. A probable reason could be that this method requires more beans to pursue, hence its RT deteriorate rapidly as the level of concurrency is increased. The reasonably good TH and swift response of Servlets only approach when the number of concurrent clients is high, are probably the reasons for big commercial sites like ebay.com [13] to take this approach (instead of relying on the infrastructure support of application servers).
3.4. Experiment 4 - Transactions
The list of interactions during a client session: Browse items, Buy, Place bids, Register items, Register users, Comment, View information about users/items/bids. Fixed parameters: • DB Access (R/W) • Number of clients (10) • Session Time (300 ms) • Thinking Time (50 ms) • Sequence (Random) • Commit option (B) Observations: It is easy to note that nontransactional updates actually cause the underlying implementations to perform better compared to transactional updates. Fig. 8 illustrates this, where the best TH for transactional updates (15.30 req/s – Stateless Session Beans) is actually lower than the worst one for non-transactional updates (21.30 req/s – Entity Beans with Session Façade Beans). On average, non-transactional updates perform approximately three times better than transactional updates (34.07 compared to 12.43 req/s). 50.00 40.00
Requests/sec
100 clients (x 500 threads)
Average Response Time (ms)
Entity Beans with CMP
20.00
EJB Local Interfaces
2.00
Entity Beans with BMP
30.00
Transactional Non-transactional
30.00 20.00 10.00 0.00 Entity Entity Local BMP CMP Interfaces
S e r- Statevlets less only Session
Facad e Beans
Fig. 9: Transactional Vs non-transactional Fig. 9 shows clearly that transactional updates slow down performance in all six implementations. In today's Internet, most operations in commercial websites are non-transactional, as they are faster and can handle more requests from clients. Transactional
operations are mostly used in secure applications such as online banking, e-purchases etc.
3.5. Experiment 5 - Caching Experiment 5 focuses caching commit options in scenarios 12 and 13 where option A and option B are considered respectively. These two scenarios repeat the runs for scenarios 1, 2 and 3, with the only exception of DB updates being transactional, and take the average over the three runs. Fixed parameters: • DB Access (R/W) • Transactional (True) • Number of clients (10) • Session Time (300 ms) • Thinking Time (50 ms) • Sequence (Fixed)
perform far worse while all other implementation methods almost equally well with their TH values being around Avg_C; a developer could choose any of the latter methods that performs well with respect to other aspect(s) of interest. The row-wise average, Avg_R, suggests how each implementation method performs relative to the aspects focused. For example, Entity Beans with BMP method performs well when DB access is varied but fails to deliver if the number of concurrent clients can vary. Table 2: Overall TH results Exp.
Requests/sec
30.00
31.17
Local EJB 47.83 30.90 19.13 27.55 52.63
35.61
Servlets
42.67 30.77 17.07 18.85 53.83
32.64
Stat. SB
41.17 31.70 16.10 26.30 50.23
33.10
Ses. F. EB 46.13 32.03 14.20 18.00 54.03
32.88
Fig. 10: Caching: variance in TH Observations: The chart in Fig. 10 shows the TH achieved for each implementation method individually. In all cases, cache-commit A achieves a higher TH than B, since the container does not need to synchronise and verify the instances of objects from the persistent state at the beginning of transactions, thus saving valuable time, resulting in higher TH.
4. Performance analysis and implications
42.63 30.46 16.64 23.25 49.83
The big picture that emerges from Table 2 can be regarded to be similar to the one drawn in [3], if one regards the variation in TH to be too small to be significant: all implementation methods perform very similarly. Nevertheless, it is a fact that certain methods are more suited than others in given scenarios. The choice of an implementation method does matter and it should be something developers consider before building an application. If we take a closer look at the numbers in Table 2, the percentage difference between the highest and the lowest achieved TH is approximately 19%, a figure that makes considerable difference on large-scale projects.
Average Response Time (ms)
Facad e Beans
Avg_R
40.13 30.37 17.80 24.00 43.53
Avg_C S e r- Statevlets less only Session
5
CMP EB
10.00 Entity Entity Local BMP CMP Interfaces
4
29.98
20.00 0.00
3
37.83 27.00 15.53 24.80 44.73
50.00 Cache-commit B Cache-commit A
2
BMP EB
60.00 40.00
1
6000 5000
This section provides an extended report on each application implementation method and determines the best overall implementation method, if there is one, and which implementation methods can be considered to be more suitable than others in a given setting. We have enough information at this stage to be able to derive such conclusions. Table 2 presents the overall TH for all implementation methods arranged as per each aspect focused in the five experiments. The column-wise average, Avg_C, allows the developer to evaluate the relative goodness of implementation methods with respect to one aspect. For example, when thinking time alone is varied with all other things being identical, the Avg_C = 30.46 req/s and Entity Beans with CMP
4000 3000 2000 1000 0 Entity Beans with BMP
Entity Beans with CMP
EJB Local Interfaces
S e rvlets only
Stateless Session
Fig. 11: Overall RT results
Entity Beans with Ses-
Let’s consider the overall illustration of the average RT as recorded from all five scenarios, shown in Fig. 11. As already mentioned, EJB Local Interfaces have the best TH and lowest RT with 4229.37 ms. CMP Entity Beans seem to have the highest RT, 5465.92 ms, and is the least recommended method in most cases
during the experiments. And even in this case the developer should choose the implementation method that performs best in the scenario closer to the usage pattern of the application What follows are detailed statements on each of the six application implementation methods and a discussion of the overall results and of our research outcome, which can be used as guideline by a developer in choosing the best implementation method for a J2EE application. Entity Beans with BMP: achieve the lowest TH out of all implementation methods with 29.98 req/s. It represents the worst method in three out of the five experiments we have conducted: 1, 2 and 3 that focus on DB access, the rate at which requests are sent to the server and the number of simulated clients respectively. This is mainly due to the fact that although the EJB container notifies the entity bean when it is safe to access the DB, it provides no other help, thus DB access is heavily dependent on the developer, hence the low TH. This method performs best on the fourth experiment, the one focused on the DB transactions, where it achieves the third best TH, behind EJB with Local Interfaces and Stateless Session Beans. Entity Beans with CMP: perform moderately in almost all experiments; they perform worst on the experiment on caching. Here, since the container knows how the persistence of the instance of a bean and its relationship map to the DB, it can automatically take care of any data manipulation required with respect to entities in the DB. Thus, the performance of this implementation method is not as bad when it came to DB access and updates. EJB with Local Interfaces: achieve the highest overall TH with 35.61 req/s. This implementation method perform better than any other implementation method in 3 out of the 5 conducted experiments (1, 3 and 4), which respectively observe the DB access, the number of emulated clients and DB update. Due to the local interfaces' special design (they use containermanaged persistence and have a remote interface that is exposed to the servlets, and a home interface that is used by the session façade beans), they can bypass the communication layers during interactions between session and entity beans, and this feature help them to achieve a high enough TH when it came to accessing the DB, updating the DB and coping with multiple clients. This is never the worst implementation method. It performs badly on the second experiment, where it achieves the third best TH, behind Stateless Session Beans and Entity Beans with Session Façade Beans. Therefore, we can assume that in cases where the rate at which requests are sent to the application server,
EJB with Local Interfaces, although not the worst possible implementation, are not recommended. Java Servlets: although not part of EJB, have the fourth best TH amongst all. They achieve their best performance in experiment 5, which was focused on caching. Since servlets allow running of Java code on the server, after the initial load the servlet engine uses local calls only, thus the high performance in the caching experiment. A relatively low TH is achieved in experiments 1 and 4 (DB access and updates), this is no surprise as there are interprocess communications taking place for each request, since the Java virtual machine is a separate process from the web server, which affects performance to a considerable extent. Stateless Session Beans: have the third best TH overall, and although they do not achieve top TH in any of the conducted experiments, they are consistently providing a more-than-average performance. They have the second best performance in the second experiment, where the rate at which requests are sent to the server is tested, and this relies heavily on the fact that Stateless Session Beans have no state associated with them, which means that they allow concurrent access to the beans, thus being able to provide a high number of requests to the server. RT is not as low however, since the content of instance variables in session beans is not guaranteed to be preserved across method calls. Entity Beans with Session Façade Beans: achieve the second highest TH with 32.88 req/s, only 2.73 req/s behind EJB with Local Interfaces. Entity Beans with Session Façade Beans have the highest TH in two cases: experiment 2 (thinking time) and experiment 5 (caching). Nevertheless they achieve the worst lowest TH in other two cases: experiment 3 (number of clients) and experiment 4 (DB updates). This implementation method has the best TH in the scenarios where caching and thinking time matter is justified by the fact that the method uses Stateless Session Beans as a façade to abstract the entity components from the business logic. This gives the pattern all the advantages provided by Stateless Session Beans. On the other hand, it involves a larger number of beans leading to low TH in experiment 3.
5. Concluding remarks We have evaluated six different implementation methods by conducting performance analyses ranging over seven different scenarios, and thereby identified which method performs best under a given scenario. We achieved this by building a client-browser emulator and considering an e-auction application. Since business applications are structured identically, our
findings presented here will be applicable to other ebusiness applications as well. The performance evaluation results have been presented and analyzed, with respect to the application implementation method they were obtained from, and the emulating scenario they were executed under. The strengths and weaknesses of each implementation method have been identified according to the results obtained, and how each method performed under different scenarios. We have found out that, although one implementation method can be considered most appropriate for a given scenario, at the same time it should be avoided in another scenario. Moreover, while there are connecting links between the various conducted experiments, their results are best interpreted in isolation, by focusing on the simulation attribute tuned throughout the experiment. Variance in results depends heavily on that attribute, which implies that the results can act as a reliable guideline if the underlying application focuses on the same property. A simple addition to the existing system would be the implementation of a tool which receives the performance results from the deployer, carries out an analysis and, based on that analysis, automatically suggests the most suitable implementation method on a given scenario. The tool could be an extension of the client-base application, useful for evaluating a single implementation method based on its TH and RT in comparison with the other methods.
[4] P. Brebner, S. Chen, I. Gorton, J. Gosper, L. Hu, A. Liu, D. Palmer, and S. Ran. “Evaluating j2ee application servers”. Technical report, CSIRO Middleware Technology Evaluation Series, 2002.
Acknowledgements
[11] EJB Technology, http://java.sun.com/products/ejb/
This work has been supported in parts by the UK EPSRC funded Platform Grant, the EC funded Network of Excellence ReSIST and by the School of CS, University of Newcastle, UK.
[12] JMeter, http://jakarta.apache.org/jmeter/
6. References [1] E. Cecchet, A. Chanda, S. Elnikety, J. Marguerite, W. Zwaenepoel, “A Comparison of Software Architectures for E-business Applications”. Technical Report TR02-389, Rice University, 2001. [2] E. Cecchet, J. Marguerite, W. Zwaenepoel, “Performance and Scalability of EJB Applications”. In Proceedings of the 17th ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications, Washington, 2002. [3] J. Spacco, W. Pugh, “RUBiS Revisited: Why J2EE Benchmarking is Hard”. In Companion To the 19th Annual ACM SIGPLAN Conference on Object-Oriented Programming Systems, Languages, and Applications, Vancouver, 2004
[5] M. Raghavachari, D. Reimer, and R. D. Johnson, “The deployer’s problem: Configuring application servers for performance and reliability”. In ICSE ’03: Proceedings of the 25th International Conference on Software Engineering, Washington DC, 2003 [6] M. Fleury, F. Reverbel, “The JBoss Extensible Server”. In proceedings of the ACM/IFIP/USENIX International Middleware Conference, Springer LNCS 2672, Rio de Janeiro, 2003. [7] D. Menasce, D. Barbara and R. Dodge, “Preserving QoS of E-commerce sites through self-tuning: a performance model approach”. In Procs. 3rd ACM Conf. on Electronic Commerce, 2002. [8] Y. Liu, A. Fekete and I. Gorton, "Predicting The Performance of Middleware-based Applications at the Design Level". ACM Proceedings of the Fourth International Workshop on Software and Performance (WOSP), 2004 [9] G. Ferrari, P. Ezhilchelvan and I. Mitrani, "Performance Modeling and Evaluation of E-Business Systems". In Proceedings of the 39th Annual Simulation Symposium, IEEE Computer Society, Alabama, 2006. [10] RUBiS, http://rubis.objectweb.org
[13] eBay, http://www.ebay.com/ [14]http://www.theserverside.com/tt/articles/article.tss?l=Dis tCompute [15] Apache Web Server http://www.apache.org/ [16] JBoss Application Server, http://labs.jboss.com/ [17]CMPCaching,http://wiki.jboss.org/wiki/Wiki.jsp?page= CMPCaching [18] J2EE Architecture, http://java.sun.com/j2ee/
