Performance Comparison of Web-based Database Access Gabriele KOTSIS Department for Telecooperation, Johannes Kepler University Linz, A-4040, Austria Email:
[email protected] and Lukas TAFERNER Abteilung für Wirtschaftsinformatik, Wirtschaftsuniversität Wien Vienna, A-1090, Austria Email:
[email protected]
Keywords: n-tier-architecture, php, java, xsp, web benchmarking, performance of e-commerce services, case study.
databases, prepare and layout the information as desired and present it to the user. Especially web-servers of big companies such as Microsoft, Hewlett Packard, but also search engines such as Yahoo.com and Google.com have to serve a couple of million users per day, causing heavy load on the servers. The consequences for a website of delivering poor performance or being 'down', even only for a couple of minutes can be ruinous. The total loss caused by denial of service in 2000 amounted to almost 8.3 million US$ [Att00]. Considering these figures, which include only reported incidences, and adding the loss that cannot be measured (loss of customers confidence, loss of brand) make it worth to consider strategies to prevent those scenarios. In this paper we will have a closer look at the factors influencing web server performance and the options a Webmaster has to find the most suitable solution for a web project. First, we will explain the principles and concepts of web server performance evaluation, and introduce the components that are responsible for the performance of a web application. In the next section, the case study used in the evaluation will be briefly presented. The major part of the paper is dedicated to the description and interpretation of the performance tests. We conclude with a summary of results and some general insights that could be derived from the case study.
1.
2.
ABSTRACT In the recent decade the internet has gained importance as a medium for information and entertainment. Many ‘old economy’ enterprises discovered new possibilities to reduce e.g. advertisement costs taking advantage of this new medium. A lot of ‘new economy’ enterprises were founded in order to base their business model solely on advantages the Internet offers. In such an environment where turnover is generated by the users visiting a web site or purchasing goods or services using a web site, performance and uptime are crucial factors for being successful. As recent events have shown, a downtime due to hard / software failure or Denial of Service attacks causes the loss of transactions and turnover during the downtime and may inflict a considerable damage to a company’s public image. In this paper we demonstrate a systematic approach for evaluating the performance of a web server. We will apply this approach in the analysis of an existing Web Site, serving mainly requests involving data base transactions. The objective of the case study is to identify the most performing technique for web-based data base access.
INTRODUCTION
The World Wide Web (www), which originally was nothing more but a system allowing researchers to link static documents together by hyperlinks, turned out to be an interesting place for all types of companies to advertise their products on one hand and to interact with potential customers on the other hand. Hypertext Markup Language is due to its static nature not able to fulfil those needs. Powerful server-sided applications were needed in order to add interactivity to static web sites. The first well known evolved in the early 1990ies, the Common Gateway Interface (CGI), later on other powerful applications like PHP, ASP, JAVA and Cold Fusion where developed [1]. These applications provide the technology needed for E-Commerce, supporting interaction with database systems, presentation of content dynamically and responsiveness to users inputs. These applications are often very resource intensive, they have to retrieve information from
BASICS IN WEB PERFORMANCE EVALUATION
A typical (web) performance evaluation life cycle [2] starts with the definition of the system under study and the goals of the study along with a specification of the performance metrics to be considered. In this work, we will focus on multi-tier architectures, which are typical for nowadays web-based e-commerce sites. This architecture can be represented as shown in Figure 1. The main components to be considered in the performance evaluation are the users´ client, the network connecting clients and servers (typically the internet), the web server, the web application server, the DB management system as well as the network linking the web server, the web application server and the DB system if those system components do not reside on the same machine. The first performance metrics of interest is response time, which is defined as the time between the
Figure 1 submission of a request from a user and the completion of the response (i.e. the instance in time when the client has received the response). This measure is a critical factor in the evaluation of web services. It is known from empirical studies, that if response time exceeds 8 seconds, users tend to terminate the session and will leave the site (eight-second rule, [3]). Such a bound set upon a performance metric is called “service level agreement”. The second important criterion is throughput, which is defined as the number of requests the system can complete per unit time. A typical question in web service capacity planning is to identify the maximum throughput that the system can achieve under a given service level. Both measures are related to the “speed” at which requests can be completed, either seen from the users´ point of view (response time) or from the system point of view (throughput). Other criteria include scalability, which is defined as the systems ability to handle an increase in the load, and availability, which is defined as the ratio of time the system was up serving requests to the total time (up and down times).
3.
CASE STUDY
The application that is taken to test the speed, scalability and reliability of different components of dynamic websites is the homepage of the Vienna City Marathon. This marathon is carried out in three different heats starting at the same time: the marathon (the traditional 'Marathon' - distance has to be mastered), the spring run (15,4 km) and the
marathon relay where the marathon distance is done in teams. The purpose of the website is to offer real-time results online via HTTP and WAP. While those services are offered free of charge to the customers, the architecture of the site is also typical for commercial services and therefore serves as a representative test-bed. The reason for choosing this site is, that the system can be observed under real, heavy load, namely at the day of the race, where several thousand database requests are submitted from all over the world. In this paper only the HTTP access is considered because it is the most important part of the application in terms of performance, the number of requests via the wireless application protocol (WAP) is not significant. In the following, we will discuss the different components of this test site briefly. Database The database for the marathon project consists of three tables corresponding to the three heats mentioned before. These tables contain similar information, namely the result records for the respective participants. One significant difference is the size of the tables: marathon contains about 10000 records (runners), whereas relay and spring amount to about 4500 entries. The different table sizes will allow us to study the effect of the table size on system performance. The database used in the project was a MySQL database server [4]. As only simple queries (selects involving no joins and only rarely update statements) are expected, this database delivers sufficient performance, there was no need to use a more powerful commercial product such as an oracle
database (performance tests were performed to verify this conjecture, the performance was mainly dominated by the amount of memory available at the physical machine, not by the type of DBMS used). The DBMS is installed on a standalone machine running SUSE Linux, it is important to provide enough RAM to keep the databases in the cached memory to increase speed. Furthermore, MySQL has to be compiled for heavy duty and with specification of the cache memory. To provide equal conditions to all web server applications, this MySQL configuration will be used in all three tests. There may be differences in connection speed however, depending on the quality of ODBC and JDBC drivers available for the platforms tested. Web server Application The objective of this paper is to test the performance of different web server applications, much attention has to be given to developing the server sided scripts and servlets. Three very different technologies were chosen to provide an overview and to detect weaknesses and advantages of different solutions. JAVA servlets [5] are a fairly new technology, they provide a way to enhance the functionality of a web server. The advantage of servlet technology results from the fact that the (JAVA) source code is already compiled at runtime. This implies the advantage of 'Write Once, Run Anywhere', i.e. the byte code can be transferred across operating systems and web servers. To run servlets in a web server environment a servlet container is needed, which acts like a HTTP server and invokes the servlet when requested. In this work, the Open Source Tomcat Servlet Engine was used, which is considered to be the most mature platform. The servlets were developed within J2EE (Java 2 Enterprise Edition). Cocoon/XSP [6] is a pure Java publishing framework that relies on new W3C technologies (such as DOM, XML, and XSL) to provide dynamic web content. It offers a different way of working, allowing content, logic and style to be separated out into different XML files, and uses XSL transformation capabilities to merge them. Cocoon provides an environment that allows a server side embedded script language to be executed known as XSP (Extensible Server Pages). The server is able to contact a DBMS via JDBC and can retrieve data which are stored in an XML document. This document can be cached by the server and is sent to the clients Web Browser as plain HTML. The actual implementation for this study was built on a cocoon 2.0 platform using the same Tomcat installation as mentioned above in the section on Java/Servlets. PHP [7] is probably the most popular server sided scripting language. Basically one has two options for how to install PHP on a Apache web server: as a standalone executable that is executed on demand or as a compiled module of the web server that is loaded at start-up. For the purpose of benchmarking PHP it is better to compile mod_php (4.0.6) into the Apache web server (1.3.22) because it is said to be
more stable and reliable [7]. PHP has a built in MySQL support, no additional driver or interface is needed. In contrast to JAVA and XML applications, PHP is not a compiled language but is compiled at run-time. We shall see in the performance tests to which extent this behavior affects performance. Network Finally we have to consider the importance of bandwidth as a factor in evaluating web server applications. Bandwidth is usually measured in bps (Bits per second) that can be transported at a given time over a TCP/IP connection. The average size of a request can be estimated to be 170 Bytes. The average size of a response amounts to approximately 4450 Bytes (both figures were derived from measurement experiments). In total, one transaction requires therefore approximately 4620 Bytes to be transferred. This number has to be compared to the bandwidth available during the test. All benchmarks were carried out on a 100mbps network. Due to overlay effects that occur in any network, the technical possible throughput is decreased by 20 per cent, i.e. 80mbps (equivalent to 10 MB per second) could effectively be used. The result of this simple calculation shows that more than 2000 transactions can be handled per second. If this number is exceeded, than the network will become a bottleneck. Note that the network can in fact become a significant performance bottleneck considering the capacity offered by the Internet.
4.
PERFORMANCE TESTS
User behavior To standardize the test and invoke the same balance on all three implementations of the Vienna Marathon Website it was necessary to have a close look at the log files of previous competitions, where server load was considerable. After an analysis the average user behaviour could be reduced to six different types of information a user would request: (1) Search for the first name of a runner in the standard interface (e.g. All runners whose name is 'Michael') (2) When the person desired is found in the resulting list, a search for the start number of this person is requested (e.g. Startnumber = 1201) (3) In the third step the visitor will have a closer look at the results of the runner selected and proceed to the page with detailed results, where average speed and results are displayed. (4) Next request will be submitted using the detailed search form looking for all runners that are from a specific country (e.g. Nation = AUT) (5) Another point of concentrated interest is to have a look at the first couple of positions in a heat (e.g. Position < 100) (6) Finally, the visitor may be interested in the fastest runners and may request a list of all runners, that completed the first half-marathon
Response XSP
200000
Response JAVA
150000
Response PHP
100000 50000
300
280
240
260
220
200
160
180
140
100
120
60
80
40
20
0
1
responsetime (ms)
Responsetime (XSP, JAVA, PHP)
concurrent sessions Figure 2
Sequential Tests In the first test environment the objective was to test the web server applications in a sequential test, i.e. requesting the same query 100 times after each other, to see how the three applications compare to each other. In addition, the size of the database was varied to get insight about the amount of time that is consumed by requesting data and processing data. The web server (Apache 1.3.22, Tomcat 4.0.1, Cocoon 2.0.1) and the DBMS (Mysql 3.23.47) were installed on the same system (Intel Pentium II, 399 Mhz Processor, 256 MB RAM) running SUSE Linux 7.3. The requests were submitted from another machine acting as the client. To avoid caching effects, the search string in the first user request (search for the name of a runner) is fetched at random from a pool of different strings to represent a more realistic load. Response Time The results of the first sequential tests were not quite surprising: Java was supposed to be the fastest in terms of response time, PHP second and XSP last. Detailed results can be seen in Figure 2. The graph shows the sum of response times of all tests conducted in comparison to each other. As expected, the Java application behaved best under stress, performance did not significantly decrease with increasing size of the Database. PHP, which had a good start, however, increased its total response time faster than database growth, which is an indicator of poor scalability. Drill Down Tests To gather more information about the behavior with even more load, several drill-down tests were conducted to identify the system capacity, i.e. to find out how much load the three applications can handle and how they perform under an increasing amount of requests. The benchmarks were carried out on the same computers (i.e. identical hard- and software) as described in the chapter sequential
tests. Scalability In a first series of tests, httperf [8] was
Com parison Responsetim e (all tests) 200,0000
Sum Responsetime (n= 100)
faster than a specific time (e.g. 1.HM < 1:30) Between the requests an average user think-time (i.e. the time between displaying the requested results and the issuing of a new request,) was set to two seconds.
175,0000 150,0000 125,0000 Php
100,0000
Java Xsp
75,0000 50,0000 25,0000 0,0000 500
1000
5000
10000
15000
DB Size
Figure 3 used to benchmark the system and to evaluate and compare the response times of the three different application server techniques. Figure 3 shows the average response time measured during each httperf drill down benchmark, comparing all three technologies to each other. As predicted, JAVA servlets show by far the best performance relative to the other technologies, PHP comes second place with almost double response time (granularity: the number of concurrent sessions was increased by 10. The benchmarks were carried out without breaks between the single tests in order not to provide relief to the server, i.e. starting a standalone test with e.g. 200 concurrent sessions would lead to another result as shown here). Worst performance could be monitored during the XSP tests, at a level of about 150 concurrent sessions cocoon crashed and left the server in an uncontrollable state. After the test was repeated several times with unchanged result, I decided to leave out XSP for further tests due to its uncompetitive nature concerning performance under heavy load. Availability In a second series of tests, http_load [9] was used, which runs multiple http fetches in parallel, to test the throughput of a web server. However unlike most such test clients, it runs in a
Throughput http_load
14000 12000 fetches
10000 8000
php (slow )
6000
php (fast) php3db
4000
java (slow )
2000
java (fast) java 3db
1000
900
800
700
600
500
400
300
200
100
1
0 concurrent requests
Figure 4
The benchmark using http_load was carried out several times on every hardware setup. This test was used to test the availability in counting the number of observed errors when serving requests. Again, Java showed best performance (see , where (slow) means the results of the benchmarks using the slow mysql machine, (fast) indicates the usage of the strong hardware for the database machine and 3db represents the results of three hardware databases). The Java servlet implementation does not produce any errors (and seems to be independent of database speed) at all up to approximately 700 requests per second, reaching a level of requests per second where the application becomes unavailable (a restart of the server is necessary). PHP however, starts producing errors from the very beginning of the test, the error rate increases with requests per second and seems to be depending on the speed at which database result tables are served. Throughput The most interesting result is the total throughput that can be produced by all three
applications compared to each other. These results are shown in. It is obvious that java servlets gain most advantage using one fast database machine, the handling of three machines containing one slow compute decrease performance considerably. Php produces a lot of errors as soon as performance is critical due to high requests per second. To sum it up, php’s performs by far not as good as java does, this effect is strengthened if the web server undergoes heavy load. Real Life Test After identifying the Java Servlets implementation as the best performing method, we decided to use this technology for the data base queries for the Vienna City Marathon 2002. An array of server was used to cope with the expected load on the day of the race and on the days after. The architecture used is shown in Figure 7. The main load was supposed to be taken by the two machines gretchen and zerberus which were the best machines in terms of hardware. The machine called snoopy was the mysql master for four mysql slaves (i.e. UPDATE, INSERT and DELETE on the master causes the same action on all slaves) and carried out the updates coming from time measurement and organized the import to all other databases. Hits on all m achines 700000
612248
600000
427633
500000
Hits
single process, so it doesn't bog down the client machine. The results obtained were similar to the results obtained with httperf. In addition, several benchmark sequences were carried out with different hardware sets. The machine which is running tomcat and the apache – php module was kept the same, the main aim was to find out, to what extent the database performance would influence the total response time. Three different hardware setups were tested: - The web server has to fetch database result tables from one mysql machine with weak hardware (Pentium II 200 Mhz, 256 Mb RAM). - The web server has to fetch database result tables from one mysql machine with strong hardware (Pentium IV 2000 Mhz, 512 Mb RAM). - The web server has to fetch database result tables from three different machines (Pentium II 200 Mhz, 256 Mb RAM; Pentium III 500 Mhz, 256 Mb RAM; Pentium IV 2000 Mhz, 512 Mb RAM) and manage connectivity to all three mysql machines.
400000 300000
136395
200000 100000
63474 30734
0 May 26
May 27
May 28
May 29
May 30
Figure 5 Figure 5 summarizes the total number of database requests served under real load. Response time was within the range of seconds for the average requests, but sometimes the service level of 8 seconds could not be met despite of the pool of servers in cases of heavy load (requests arriving in large bulks when popular runners are finishing). Nevertheless, from a
performance point of view the Vienna City Marathon was completed successfully.
5. CONCLUSIONS As we have seen in the benchmarks it is vital to make a good choice which web server application to use. This depends heavily on the circumstances and desired properties of the web server application. The chart in Figure 6 provides an overview about which web server application can be used in terms of load it can manage.
Light 200 req/sec
-
-
X < 15 sec
-
X < 8 sec
most professional tool available but on the other side difficult to learn and to deploy. The results of this case study can be generalized to other application domains as well. The assumption in this case study was a specific pattern of the load submitted to the server. The database was mainly serving select statements of different complexity. There was no significant load from updates or delete statements. This is a typical load pattern for information systems, where the purpose of the underlying database is mainly to store information which is basically entered once for frequent subsequent retrievals. It is in general not suitable for e-commerce applications like electronic shopping malls, where users would typically purchase goods and the purchase data must be entered into the database. But it might be a representative model for the information retrieval phase of the business transaction.
6.
REFERENCES
[1] Krause, Jörg: PHP, Webserver Programmierung unter Windows und Linux, 2000, Carl Hanser Verlag, Munich. [2] Jain, Raj: The Art of Computer Systems Performance Analysis, 1991, John Wiley and Sons Inc. [3] Menasce, D., Almeide, V.: Capacity Planning for Web Services, Prentice Hall PTR, 2002 [4] MySQL Database Server, Product Description, http://www.mysql.com/ [5] Callaway, Dustin R.: Inside Servlets, Second Edition, 2001, Addison Wesley Verlag, Munich [6] Cocoon Developers Homepage, http://xml.apache.org/cocoon/ [7] PHP Documentation and Download, http://www.php.net/download [8] HTTPERF Benchmarking Tool, http://www.hpl.hp.com/personal/David_Mosb erger/httperf.html [9] HTTP_LOAD Benchmarking Tool, http://www.allan.org/load/http_load/
X > 8 sec
Figure 6 From the results of the benchmark series the following conclusions could be drawn: XSP is in an early state of development to be competitive. Furthermore documentation was virtually non existent and the implementation process (separation of content, style and management) is too complex for small projects. PHP showed acceptable performance during the benchmarks, could, however, not beat JAVA. Its advantages are a comprehensive language and easy deployment and configuration of the server platform. A PHP-based solution can be recommended for small-sized enterprises expecting low to moderate load, where the major concern is low development and maintenance costs. JAVA servlets performed best compared to the other technologies, in the sequential tests as well as during the drill down benchmarks. It is by far the
JavaScript Clientside LoadBalancing snoopy(Mysql master)
titan (tomcat - weak)
zerberus (tomcat - strong)
gretchen (tomcat - strong)
michelangelo (tomcat - weak)
paris (mysql - weak)
goedel (mysql - strong)
snoopy (mysql - strong)
aea51 (mysql - weak)
Figure 7
