1
Performance Prediction of Component-based Applications Shiping Chena , Yan Liub , Ian Gortonc
∗
and Anna Liud
a
CSIRO Mathematical and Information Sciences, Australia
b
School of Information Technologies, University of Sydney, Australia
c
Pacific Northwest National Laboratory, Richland, WA 99352,USA
d
Microsoft, Australia
One of the major problems in building large-scale enterprise systems is anticipating the performance of the eventual solution before it has been built. The fundamental software engineering problem becomes more difficult when the systems are built on component technology. This paper investigates the feasibility of providing a practical solution to this problem. An empirical approach is proposed to determine the performance characteristics of component-based applications by benchmarking and profiling. Based on observation, a model is constructed to act as a performance predictor for a class of applications based on the specific component technology. The performance model derived from empirical measures is necessary to make the problem tractable and the results relevant. A case study applies the performance model to an application prototype implemented by two component infrastructures: CORBA and J2EE.
1. Introduction One of the key challenges in building enterprise scale distributed systems is performance prediction [15]. These systems are susceptible to large volumes of transactions and concurrent users at peak loads. They are expected to provide acceptable response times to users, and scale to handle bursts of traffic in a predictable manner [13]. Most of these modern systems are based on component-based technologies. While component-based technologies provide an infrastructure solution for enterprise applications, they do not help in understanding the eventual performance of an application before it has been built. In fact, component technologies make the problem even more difficult, as each component technology may have a different infrastructure and implementation, and thus can exhibit different performance characteristics. As a result, software architects are currently forced to build complex prototypes to give some assurance of the suitability of the software and hardware in∗ Corresponding
Author. Email:
[email protected]
frastructures that the application uses. Although well-designed prototypes can provide certain levels of performance assurance, they are expensive and time-consuming to build. On the other hand, considerable work on software performance modelling has been done in academic community. However, most existing models are either too complex to use [17], or do not take the underlying component infrastructure into account [21], making them impractical for performance prediction of component-based applications in industrial software engineering projects. Recognizing these issues, in this paper we investigate the feasibility of a practical solution to this problem. An empirical approach is proposed to determine the performance characteristics of component-based applications. A benchmark is used to exercise components and measure their behaviour. Deeper observations on the internal component behaviour can be derived from these empirical results. This leads to the construction of a performance model that describes the generic performance behaviours for a class of componentbased applications. The model reflects the in-
2 teractions between key performances factors in applications, especially the infrastructure components, and can be used to predict the performance of the component-based applications hosted by the common technologies such as CORBA, J2EE and .NET. The paper presents the design of the test suite and the derivation of a performance model. The predictive capability of the model is then demonstrated using a test application implemented using two different component infrastructures, CORBA and J2EE. 2. Performance Issues based Applications
of
Component-
Component-based technologies strive to make it easier to build distributed component systems by providing extensive sets of services and a supporting run-time infrastructure. These services are an abstraction that provides a set of common operations required by applications, typically including object location, security, transaction management, integration and resource management. A run-time infrastructure, known as a container, is responsible for managing the lifecycle and execution of application components and proving interfaces to the services used by the application components.
…...
Application Components
…... Resources Pool
Naming Transaction Security
d
Middleware Services Application Server / Component Framework Operation System
… Database 1
Database 2
Legacy Application
ERP
Figure 1. Anatomy of a component-based middleware applicaion
As shown in Figure 1, application level components are based on components supplied by the middleware infrastructure and hence are tightly coupled with each other at run-time. The general QoS (Quality of Service) of application components depends on the underlying components provided by the middleware. This tight coupling of application components and component infrastructure significantly increases the complexity of predicting the effects of various architectural trade-offs. In particular, it is no longer possible to execute the application components independently and measure their performance [21]. Nor is it possible to manually inspect the code in an attempt to analyse performance, as the infrastructure source code is unlikely to be available [4]. Open standard component infrastructures, such as CORBA and J2EE, actually exacerbate this problem. With these technologies, many vendors develop their own products, which are implemented differently. As a result, they will perform very differently [1,6]. Figure 2 (a) illustrates the results of executing an identical application against six different component infrastructures based on the Enterprise JavaBean (EJB) standard [7]. All the tests are executed on the same hardware, operating system and database. Each product is configured and tuned to achieve optimized performance. Figure 2 (a) shows the application throughput achieved in terms of tps (transactions per-second) for a range of 100 to 1000 clients. The six EJB application servers perform significantly differently, both in terms of the peak throughput achieved and scalability in handling increasing client loads. These differences become even more significant as the same test case is scaled out to run on more application server machines in an attempt to increase overall application throughput [12]. Even for the same product, different component architectures perform differently. Figure 2 (b) compares the throughput of two EJB application architectures for a particular EJB implementation. The tests implement identical functionalities running on the same EJB container and test platform, but using different application component architectures. The session bean only (SB) and session bean as a facade to an entity
900 800 700 600 500 400 300 200 100 0
700 T hro ug h pu t (t ps)
Th r ou g hp u t ( tps)
3
100 EJB Server -1 EJB Server -4
300 500 No. of clients EJB Server -2 EJB Server -5
1000
600 500 400 300 200 100 0 100
(a) Performance of 6 EJB appServers
300
500
1000
No. of clients
EJB Server -3 EJB Server -6
SB
SB+EB
(b) Performance of 2 EJB architectures
Figure 2. Experimental results from CSIRO MTE Project
bean (SB+EB) are two common design patterns for EJB components. Interestingly, the throughput achieved by the SB+EB architecture is less than 50% of the throughput achieved by the SB architecture in most cases.
3.1. Test Suite Design Performance analysis and prediction must be performed within the context of a particular component technology. Therefore, it is essential to carry out some level of baseline benchmarking to understand the performance profile of a given component technology. Seltzer and Zhang [20,23] cogently discuss the inadequacy of industrial standard benchmarking. The authors claim that no industry standard benchmark can accurately represent the behavior of all different types of applications, and thus the results tend to be misleading. For example, the major J2EE benchmark, ECperf, includes relatively complex business logic in the application components, and is heavily reliant upon the performance of the external database system used in the tests. This makes it difficult to distinguish application server specific performance issues from those occurring in the database. Kounev and Buchmann [10] have discovered that a performance bottleneck of ECperf is related to benchmark specification itself
Naming
Client Requests
3. Capturing Performance Characteristics of Component Technology
and its corresponding implementation.
Transaction
Request Processing
Security
Component Pool (Cache)
Persistence
Database Connection Pool
Component Container Component Interface { val read (Key ClientID) void write (Key ClientID) }
DataBase Key Val 0 n 1 m ... ... N k
Figure 3. Benchmark test suite design
In order to obtain a performance profile of solely the underlying component technology and minimize the effects of application behaviour, a simplified application has been designed as our basic benchmark. The benchmark application abstracts the typical transaction operations such as reading and writing data from the database and exercises the key elements of a component infrastructure. A diagram of the test suite, the application component interfaces and the database table structure are shown in Figure 3. The benchmark has several important characteristics that make
4
The eventual aim is to develop models of the underlying middleware infrastructure, such that it is possible to predict behavior of the applications build on that infrastructure [8]. 3.2. Benchmarking and Profiling As an example of the effectiveness of the benchmark application, Figure 4 shows the normalized execution time breakdown of an EJB application configured with different numbers of threads in each test. The test platform comprised three quad-processor Windows 2000 machines, one for the client, one for the application server, and one for the database. A 100 Mbps network connected the machines. A Java profiling tool was used to obtain the performance metrics, such as the execution time of each method and the number of invocations. Because the EJB server under test is a black box, we can only roughly estimate the method execution time in subsystems such as the transaction manager, the database connection pool and the container, etc. This is achieved by aggregating the execution time of all the methods in a subsystem. To efficiently support large numbers of simultaneous accesses from clients, multi-threaded and/or replicated servers must be used to increase the processing capacity of an application. A multi-threaded server creates a pool of threads, which handle requests concurrently. For example, IONA’s Orbix2000 and BEA’s WebLogic Server use a configuration parameter named Thread-
40 35 30 25 20 15 10 5
20 threads
30 threads
transaction m ananger
security
socket processing
thr ead pooling
naming
container
third party
15 threads
db connection pooling
java.lang.*
0 bean m ethod (edu.usyd.*)
• A single externally visible application component that has read and write methods in its interface. The read() method reads the value field from the database, given the key. The write() method increments the value field in the database associated with the key.
45
com munication
• For simplicity, only one database table is defined. The table contains two fields, key and value. Each client is randomly allocated a key, which is used in reads and writes.
50 Nomalized service tim e (% )
it an appropriate test application for examining and exposing the QoS and behaviour characteristics of a component infrastructure. They are:
50 threads
Figure 4. EJB application server execution time breakdown
Count to set the maximum number of threads in the pool of the application servers. The results in Figure 4 can be explained as follows. As the number of application threads available in the container increases, more processing capacity is available to read requests from the sockets and simultaneously process the requests. This reduces the queue length to access the server process and consequently the amount of thread pooling and socket processing decreases. However, increasing the thread count also increases the contention for the application resources and database connections. It can be seen that the execution time for communication, database connection pooling, container, security and transaction manager is increasing as the number of threads increases from 20 to 50. Overall, varying the amount of internal concurrency in the component container has effects on application performance, as shown clearly in Figure 5. We discuss three key influences on the application performance as follows: • Client request load: In Figure 5 (a), as more concurrent client requests arrive at an application server, more contention is incurred for server threads to process these requests. Consequently the request queue in the container will grow as requests wait for
5 1500
SB SB+EB
2500 2000
T h ro ug h pu t (tp s )
T h roug hp u t (t ps )
3000
1500 1000 500 0 0
500
1000
1500
2000
800 600 100 clients 400 clients 800 clients 1600 clients
400 200 0 0
20
0 50 100 No. of threads
14000 12000 10000 8000 6000 4000 2000 0
150
100 clients 400 clients 800 clients 1600 clients
2
40 60 80 100 120 140 No. of threads
(c) Throughput vs. thread counts
100 rows in DB 1000 rows in DB
(b) Two database contention levels R es po ns e tim e (m s )
T hro ug hp u t (t ps )
1000
500
0
No. of clients
(a) Throughput vs. workloads
1000
8 32 64 No. of threads
128
(d) Response time vs. thread counts
Figure 5. Observations on application server performance
service. From 100 to 1600 clients, the additional load being placed on the container by the increasing client request load causes decline in the overall throughput for the two architectures. • Database contention: To maintain data integrity, a transaction has to issue a lock on a table or a record when updating the data within it. The lock protects the data from being accessed by other transactions until the working transaction is committed or aborted. The more transactions that simultaneously use shared data, then the higher the contention overheads that occur. To explore the effects of database contention, the benchmark test case can be configured to: – Limit the number of rows available in the database table to a specific maximum value; – Force each client request to randomly choose a database row to access in a transaction within the range available.
To explore the effects of database contention, tests varied the number of rows in the database from 100 to 1000 and left all the other parameters unchanged. When the number of rows in database increases, the probability of database contention decreases, and thus the overall performance improves. This effect is clearly shown in Figure 5 (b). • Container threading: The throughput and average response time are shown in Figure 5 (c) and (d) for various workloads respectively. The convex shape of the throughput and concave shape of response time indicates that there exists an optimal value of threads in the container that can result in an optimal performance. 3.3. Observations The analysis of the benchmark results leads to the observation that contention for communication sockets, database connections and other container resources as well as CPU cycles and mem-
6 ory all contribute to the overall overhead in an application. The level of threading in the container is a key factor in determining the overall contention level in the application. This observation has been empirically validated in both J2EE technologies (as addressed above) and CORBA technologies [3]. This means care must be taken to determine the degree of concurrency in an application. An optimal degree of concurrency can exist such that the overall performance is optimal for a specific client load. 4. Performance Modelling Based on the above observations, a cost model was developed to estimate the overall overhead (performance) of a component container. For simplicity, we consider only one container, which creates a set of threads to serve client requests concurrently. Figure 6 illustrates the general process of handling concurrent requests in a container.
Notice that in most Internet systems, the interactive client code does not call the business logic layer directly. Instead, it is performed by ’proxy’ client components, somehow launched by the web server (e.g. JSPs/ASPs, servlets, etc). When the host of the proxy clients, namely the web server engine, is saturated, the application server components receive a constant series of requests, with the negligible elapsed time between sequential requests. Therefore the workload of the application server is the number of requests received from these proxy clients. Denoting the number of requests as x and the number of concurrent server threads as y, we define the overall overhead in an application server as follows: T = ax +
bx + cy y
(1)
where: • T is the overall overhead in time; • a is the overhead ratio per concurrent request to access the container;
Current Requests From Clients
Thread Thread Queue
Thread
• b is the average processing time for a single thread. With multiple threads, we define the average processing time under workload x is bx y ; • c is the contention overhead on the back-end resources, such as database.
Figure 6. Concurrency vs. overhead
The overall overhead in a container results from three sources. First, overhead occurs when all concurrent requests compete for service by the container process. The overhead includes network bandwidth and socket handling. Second, request overhead includes both the service time for the request by the container and the waiting time in the queue. Third, overhead arises from concurrent access to resources managers such as databases. We assume the overall overhead is a summation of the three resources.
To make T independent from specific testing execution times and transaction types, we define the contention overhead T as the Average Response Time (ART) for all transactions, namely t T = W N , where Wt is the wall-time of a test run and N is the total number of transactions completed in the test. In addition, we assume that the number of database connections be equal to or more than y. This is a common configuration in application server deployments. Theoretically, overall performance is application and platform dependent. Also, many factors may affect performance, such as application complexity, client access patterns, machine capacity, network traffic, and so on. While our
7 model focuses on two key performance factors, namely workload x and degree of concurrency in a container y, all the other factors are absorbed in the model parameters a,b,c. This requires that the testing prototype and environment be close to the one to be used when applying the model. Even with this limitation, the model can be used for performance prediction, middleware configuration and capacity planning for a class of applications hosted by different component-based middleware infrastructures. 5. Case Study In order to assess the predictive capabilities of the cost model, the StockOnline test application [7] was implemented using two common middleware technologies, CORBA and EJB. For simplicity, a single application server is considered in this case study. [12] shows that considerably higher performance can be achieved by applying the optimal configuration derived from the single server model to multiple servers in a cluster. StockOnline is a simulation of an online stockbroking system. It enables subscribers to buy and sell stock, inquire about the up-to-date prices of particular stocks, and get a holding statement detailing the stocks they currently own. Two versions of StockOnline were implemented using CORBA and EJB technologies respectively. Table 1 lists the detailed configurations of the StockOnline implementations. We have developed a multithreaded benchmark client that spawns a specified number of client threads to drive StockOnline according to the scenarios specified in Table 1. Each thread submits a service request and waits for a response. Once a transaction is complete and the results are returned from the server, each simulated client immediately starts the next selected transaction. The think time between individual client requests is zero to ensure that the workload of the application server is constant during the measurement period. 5.1. Experiments and Modelling A set of tests was run against the application with the number of threads in the container held
constant. Each test set varied the numbers of clients. Every client thread executed 10 iterations of the specified transaction mix for StockOnline. The transaction mix represents the concept of one complete business cycle at the client side. It comprises 43 individual transactions with 81% readonly transactions and 19% read-write transactions. Thus the total number of transactions for each test client is 430. The contention overheads are then derived from the wall-times measured at t the client-side by using T = W N , where Wt is the wall-time in milliseconds and N = 430. The experiment was repeated for different numbers of server threads, namely y = 2, 4, 8, 16, 32, 64, 128. The values of a, b, c are solved and listed in Table 2.1
a b c
CORBA 6.65 ms/client 5.42 (ms/client)·thread 5.24 ms/thread
EJB 1.04 ms/client 8.11 (ms/client)·thread 2.01 ms/thread
Table 2 StockOnline coefficients of the cost model
Next, the cost model with these three parameters was used to estimate contention overheads, and the CORBA and EJB results were compared with the corresponding experimental results as depicted in Figure 7 and 8, respectively. While the contention overheads increase as the number of concurrent clients increase, the contention overheads change in a parabola shape as the number of threads increases monotonically. The model closely predicts the behaviour of the application under various workload and thread counts, except the scenario of 100 clients running with 128 server threads. In this case, as the number of container threads exceeds the simultaneous client request load, the number of requests waiting in the application server queue is 1 The aim of using StockOnline in this paper is to verify the performance model rather than evaluate the performance of the two middleware. The results should not be used to compare either the two technologies or the two products.
8
AppServer Hardware
Client Database OS AppServer
Software
†
Database Interface to DB Transaction†
CORBA Dell: 2 GenuineIntel CPUs 450MHz,512MB RAM Dell: 2 GenuineIntel CPUs 450MHz,512MB RAM Dell: 2 GenuineIntel CPUs 450MHz,512MB RAM MS W2K Server SP2 IONA Oribx 2000 V 1.0 Oracle 8.1.5 OCI DB managed
EJB Dell: 4 Pentium III Xeon CPUs 700MHz,2GB RAM Dell: 4 Pentium III Xeon CPUs 700MHz, 2GB RAM Dell: 8 Pentium III Xeon CPUs 700MHz, 2GB RAM MS W2K Server SP2 BEA WebLogic AppServer V 6.1 + Sun JDK V.1.3.1 01 Oracle 8.1.7 Oracle thin JDBC driver Container managed
The StockOnline database is centralized and non-xa transactions are used
Table 1 The Hardware/Software configurations for StockOnline implementations very small or non-existent. The additional container threads remain idle, and the actual contention at the database is less than the value y in the model. Therefore the model overestimates the response time when y is much greater than the actual threads associated with an active database connection. This analysis verifies the assumption that too high a degree of concurrency in a container degrades the application’s overall performance. Higher contention at the database restricts an application server from using too many concurrent threads or processes.
nent characteristics within a specific application under a specific platform. This result can be further explained as follows:
5.2. Application Of The Cost Model 5.2.1. Predicting the optimal concurrency Predicting the optimal container concurrency for the application is equivalent to minimizing the overall overhead T in Equation (1). The optimal number of threads is derived from: bx ∂T =− 2 +c=0 (2) ∂y y
• More concurrent clients result in higher contention in the container. The degree of the concurrency is proportional to the square root of the number of concurrent clients.
Then we get the optimal value for the number of server threads y ∗ : r bx (3) y∗ = c where x is the given number of clients; b and c are the model parameters reflecting the compo-
• The network contention for an application has no direct effect on the degree of the concurrency within the application container. This means that we do not need to consider the network contention outside of the container when determining the degree of concurrency.
Based on the above parameters, we can derive the optimal number of container threads by applying Equation (1). A set of theoretical optimal numbers of server threads for different numbers of concurrent clients is listed in Table 3. According to the model, the minimal response time under a particular workload occurs at the point of the optimal number of server threads. The errors in predicted response time over the best experimental results in the tests are given in Table 3.
600 Cost model
500
Experimental
400 300 200 100 0 2
4
8
16
32
64
128
R es po ns e t im e (m s )
R es p o nse tim e (m s )
9
1200 1000
Cost model
800
Experimental
600 400 200 0 2
4
(a) 100 clients
16
32
64
128
(b) 200 clients
2500
5000
2000
Cost model Experimental
1500 1000 500 0 2
4
8
16
32
64
128
R e s p o nse t im e (m s )
R e spo n se tim e (m s )
8
No. of threads
No. of threads
Cost model Experimental
4000 3000 2000 1000 0 2
4
8
16
32
64
128
No. of threads
No. of threads
(c) 400 clients
(d) 800 clients
Figure 7. StockOnline EJB implementation: experimental results vs. cost model
CORBA
EJB
Optimal y∗ 10 14 20 29 20 28 40 56
ART T (y ∗ ) (ms) 771 1481 2873 5621 145 306 573 988
1400
Error 5.5% 3.5% 2.1% 3.0% 27% 12% 0.1% 9%
Table 3 Minimal ART: modelled vs. experimental
5.2.2. Predicting application performance Once the thread configuration is fixed, we can further predict the client response time under various workloads with Equation (1). For example, Figure 9 compares measured response times with those predicted from the cost model, with the thread pool size in the container set at 20. The standard deviation is again within 20%.
Re s p o ns e t im e (m s )
Impl.
Client x 100 200 400 800 100 200 400 800
Cost model
1200
Experimental
1000 800 600 400 200 0 0
200
400
600
800
1000
No. of clients
Figure 9. StockOnline (EJB) response time under various workloads
5.2.3. Estimate the client workload for specific response time The meaning of the contention overhead T in Equation (1) is exactly the same as the client side response time. Therefore we can represent the client request load x as a function of response
600
R e s p on se t im e (m s )
R es p o nse tim e (m s )
10
Cost model
500
Experimental
400 300 200 100 0
2
4
8
16
32
64
2600
Cost model
2400
Experimental
2200 2000 1800 1600 1400 1200 1
128
2
4
No. of threads
5500 Experimental
4000 3500 3000 2500 1
2
4
8
16
32
32
64
128
11000
Cost model
4500
16
(b) 200 clients
64
128
No. of threads
R e s po n se tim e (m s )
R es p o ns e t im e (m s )
(a) 100 clients 5000
8
No. of threads
10000
Cost model Experimental
9000 8000 7000 6000 5000 1
2
4
8
16
32
64
128
No. of threads
(c) 400 clients
(d) 800 clients
Figure 8. StockOnline CORBA implementation: experimental results vs. cost model
time T and thread configuration y. This makes it possible to predict the number of clients that can be serviced under a specified response time threshold by the application configuration. For example, assume the required client side response time is 2 seconds, T = 2000ms, and the configuration of the container thread count is 20. Using Equation (4), it is possible to estimate the capacity of the system. For this example, x = 1352 means the system can support a client request load up to 1352 and provide the required average response time. If more workload arrives at the application, other scaling strategies should be considered, such as updating the hardware, and clustering multiple servers (scale out). x = F (T, y) =
T y − cy 2 ay + b
(4)
6. Related Work This analysis highlights difficulties and limitations in performance prediction of component-
based applications using existing approaches intractable. For example, Smith and Williams proposed the Software Performance Engineering method [21]. Developers must generate models to represent the application under development. The models require detailed performance parameters of resource demand and utilization for the purpose of performance prediction. However, with component technologies, the automatic services, diversity of implementations from vendors, and the tight coupling of application components and the container create a highly complex run time environment. This makes it hard, if not impossible, for the required models to be constructed and the performance parameters to be derived. Other work such as MVA [17] and LQNS [5,16, 18,22] can be used to model distributed systems. However, their models are algorithm-oriented and thus no closed-form analytic expressions are provided to help either performance predictions, or performance tuning for the component-based ap-
11 plications. On the other hand , these models also rely on understanding the internals of the technology being modelled, which is not practical with black box component technologies. A framework for understanding performance issues in EJB based applications was proposed by Mos and Murphy [14]. Monitoring tools extract performance data from a system during run time and UML models enriched with performance profiles are automatically generated. The performance prediction is then based on simulation of the UML performance profile. This framework has the assumption that the running version of the system exists. Gunther proposed a super-serial model similar to ours, although he regards his model as a scalability function [9]. In this model three parameters are defined, namely user load, contention and coherency. While the model can be used to predict performance (scalability in his notion), it does not show where contention occurs, and thus provides no guidance for server configuration. 7. Conclusion This paper has described the inherent difficulties of predicting the performance of N-tier enterprise applications built using component technologies. The key problem revolves around the diversity and complexity of the component-based middleware products used to build large-scale distributed applications. In practice, these problems force software architects to experimentally discover application configurations that provide the desired levels of performance. The approach explained in this paper explicitly recognizes these problems, and provides a potential solution based on empirical testing and mathematical modelling. A test suite is used to discover the performance characteristics of a specific component technology, and this leads to the derivation of a performance model. The model describes the generic behaviour of application server components running on component-based middleware technologies. The parameter values in the model are necessarily different for different component technologies and products, as these products have unique performance and be-
havioural characteristics. The values must therefore be discovered through empirical testing. To this end, a set of test cases are defined and executed for each different component-based middleware product. The results of these tests make it possible to solve the models for each product, so that performance prediction for a class of component-based applications on a specific platform becomes possible.
Acknowledgements We wish to thank the middleware vendors for providing their CORBA and J2EE products for this work. We also appreciate the referees for their useful comments and suggestions to revise and improve this paper.
REFERENCES 1. Cecchet,E., Marguerite, J.,Zwaenepoel, W., 2002. Performance and Scalability of EJB Applications, Performance and scalability of EJB applications, ACM SIGPLAN Notices, vol.37, no.2, 2002, 246–261 2. Chen, S.,Gorton, I., 2002. A Predictive Performance Model to Evaluate the Contention Cost in Application Servers, Proc. of 9th Asia-Pacific Software Engineering Conference, p.435-440, Australia. 3. Chen, S., Gorton, I., Liu, A., Liu, Y., 2002. Performance Prediction of COTS Component-based Enterprise Applications, Proc. of 5th ICSE Workshop on Component-Based Software Engineering: Benchmarks for Predictable Assembly. www.sei.cmu.edu/pacc/CBSE5/liu-cbse529.pdf 4. Dilley, J., Fredrich, R., Jin, T., Rolia, J., 1996. Measurement Tools and Modeling Techniques for Evaluating Web Server Performance, HP Lab Technical Report. www.hpl.hp.com/techreports/96/HPL96-161.html 5. Franks, G., Majumdar, S., Neilson, J.E., Petriu, D.C., Rolia, J., Woodside, C.M., 1996. Performance Analysis of Distributed Server
12
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
Systems, Proc. of 6th International Conference on Software Quality, p.15-26, Canada. Gorton, I., Liu, A., 2002. Software Component Quality Assessment in Practice: Successes and Practical Impediments, Proc. of 24th International Conference on Software Engineering(ICSE 2002), p.555-558, USA. Gorton, I. et al., 2000-2002. Middleware Technology Evaluation Report. CSIRO Technical Report. www.cmis.csiro.au/adsat/mte reports.html Grundy, J.C., Cai, Y., And Liu, A., 2001. Generation of Distributed System Test-beds From High-level Software Architecture Descriptions, Proc. of 16th IEEE International Conference on Automated Software Engineering (ASE 2001), p.26-29, USA. Gunther, N., 2001. Evaluating Scalability Parameters: Fitting End. www.teamquest.com/html/gunther/fitting.shtml Kounev, S. D., Buchmann, A P., 2002. Performance Issues in E-Business Systems, International Conference on Advances in Infrastructure for eBusiness, e-Education, e-Science, and e-Medicine on the Internet, SSGRR 2002w. www.ssgrr.it/ssgrr2002w/papers/1.pdf Litoiu, M., Rolia, J., Serazzi, G., 2000. Designing Process Replication and Threading Policies: A Quantitative Approach. IEEE Transactions on Software Engineering, vol.26, no.12, 2000, 1168-1178. Liu, Y., Gorton, I., Liu, A. Chen, S., 2002. Evaluate The Scalability of Enterprise JavaBeans Technology, Proc. of 9th Asia-Pacific Software Engineering Conference, p.74-83,Australia. Menasc`, D.A., Almeida, V.A.F., 2000. Scaling for E-Business: Technologies, Models, Performance, and Capacity Planning. Prentice-Hall, Upper Saddle River, New Jersey, p. 96. Mos, A., Murphy, J., 2002. Performance Management in Component-Oriented Systems Using a Model Driven Architecture Approach, Proc. of 6th IEEE International Enterprise Distributed Object Computing Conference, p.227, Switzerland. O’neill, T., Leaney, J., Martyn, P., 2000.
16.
17.
18.
19.
20.
21.
22.
23.
Architecture-Based Performance Analysis of the COLLINS Class Submarine Open System Extension (COSE) Concept Demonstrator (CD). Proc. of 7th IEEE International Conference and Workshop on the Engineering of Computer Based Systems, p.26-35, Scotland. Petriu, D. C., Wang, X., 2000. From UML Description of High-level Software Architecture to LQN Performance Models, Applications of Graph Transformations with Industrial Relevance,Lecture Notes in Computer Science 1779, 2000, 47-62. Reiser, M., Lavenberg, S., 1980. Mean-value Analysis of Closed Multi-Chain Queuing Network, Journal of ACM, vol. 27, no. 2, 1980, 313-322. Rolia, J. A., Sevik, K.C., 1995. The Method of Layers,IEEE Transactions on Software Engineering, vol. 21, no. 8. 1995, 689-700. Schmidt, D.C., 1998. Evaluating Architectures for Multi-threaded CORBA Object Request Brokers, Communications of the ACM Special Issue on CORBA, vol. 41, no. 10,1998, 54-60. Seltzer, M., Krinsky, D., Smith, K., Zhang, X., 1999. The Case for Application-Specific Benchmarking. Proc. of the 1999 Workshop on Hot Topics in Operating Systems, p.102, USA. Williams, L.G., Smith, C.U., 1998. Performance Engineering Evaluation of Software Architectures, Proc. of 1st International Workshop on Software and Performance, p.164-177, USA. Woodside, C.M., Neilson, J.E., Petriu, D.C., Majumdar, S., 1995. The Stochastic Rendezvous Network Model for Performa of Synchronous Client-Server-Like Distributed Software, IEEE Transactions on Computers, vol. 44, no. 1, January, 1995, 20-34. Zhang, X., Seltzer, M., 2000. HBench: Java: An Application-Specific Benchmarking Framework for Java Virtual Machines. Proc. of ACM Java Grande 2000 Conference, p.6270, USA.
