A performance evaluation of distributed database ... - Semantic Scholar

CONCURRENCY AND COMPUTATION: PRACTICE AND EXPERIENCE Concurrency Computat.: Pract. Exper. (2012) Published online in Wiley Online Library (wileyonlinelibrary.com). DOI: 10.1002/cpe.2891

A performance evaluation of distributed database architectures Shiping Chen1,*,†, Alex Ng2 and Paul Greenfield3 1

CSIRO ICT Centre, Australia University of Ballarat, Australia 3 CSIRO CMIS, Australia

2

SUMMARY The globally integrated contemporary business environment has prompted new challenges to database architectures in order to enable organizations to improve database applications performance, scalability, reliability and data privacy in adapting to the evolving nature of business. Although a number of distributed database architectures are available for choice, there is a lack of an in-depth understanding of the performance characteristics of these database architectures in a comparison way. In this paper, we report a performance study of three typical (centralized, partitioned and replicated) database architectures. We used the TPC-C as the evaluation benchmark to simulate a contemporary business environment, and a commercially available database management system that supports the three architectures. We compared the performance of the partitioned and replicated architectures against the centralized database, which results in some interesting observations and practical experience. The findings and the practice presented in this paper provide useful information and experience for the enterprise architects and database administrators in determining the appropriate database architecture in moving from centralized to distributed environments. Copyright © 2012 John Wiley & Sons, Ltd. Received 1 May 2011; Revised 26 December 2011; Accepted 17 March 2012 KEY WORDS:

distributed database architecture; database partition; database replication; database benchmarking

1. INTRODUCTION Codd [1] proposed the foundation for relational database management system (RDBMS) in 1970, highlighting the use of relational view of data to provide a means of describing data with its natural structure, which formed the solid basis for treating derivability, redundancy and consistency of relations. Since then, many commercial and open source relational database systems were developed, such as Oracle, DB2 and MySQL. Today, RDBMS has become an essential component in most contemporary enterprise systems. With the advancement of the Web and the business extending globally, enterprise systems are facing challenges to support ever-increasing clients and business partners, globally data sharing and messaging and 24
7 availability. From the viewpoint of database systems, these challenges imply the back-end database systems: (a) must scale to support increasing loads; (b) must support different distributed configurations to facilitate an enterprise’s global operations; and (c) must be robust and reliable for various technical failures and natural disasters. To meet these new challenges in the Internet-age business environments for performance, scalability and resiliency, these commercial and open-source relational database systems have offered enhanced features to meet the new demands by incorporating a variety of new features and configurable distributed database architectures. For example, IBM’s DB2 pureScale provides cluster solution for non-mainframe platforms up to 128 database servers with a fault-tolerant architecture and shared-disk storage to achieve automatic *Correspondence to: Shiping Chen, PO Box 76, Epping, NSW 1710, Australia. † E-mail: [email protected] Copyright © 2012 John Wiley & Sons, Ltd.

S. CHEN, A. NG AND P. GREENFIELD

load balancing. Other suppliers such as the Oracle Database 11g provides a set of rich choices in partitioning such as reference partitioning, interval partitioning, partitioning virtual columns and extended composite partitioning [2]. Even the open-source MySQL server provides range, list, hash and key types of partitioning strategies as well as the replication strategy. However, we observed that little work has been carried out to understand how different distributed database architectures perform in different commercial environments. The potential benefits provided by the different distributed database architectures are not well understood. Enterprise architects may face difficult decisions to make well-informed justification of under what scenario that a particular centralized, partitioned or replicated architecture should best fit the requirement. Therefore, this research sets the objective to study the performance characteristics exhibited in different database scalability and virtualization strategies, which includes (a) centralized, (b) partitioned and (c) replicated database architectures, using an industry standard benchmark that can reflect a typical commercial workload environment. The rest of this paper is organized as follows: Section 2 presents an overview of current distributed database architectures. Section 3 discusses our evaluation methodology and experimental designs. Section 4 presents the results of our evaluation and detailed discussions of our findings. Lastly, the related work and conclusions are provided in Sections 5 and 6, respectively.

2. OVERVIEW OF CURRENT DISTRIBUTED DATABASE ARCHITECTURES Historically, DBMS has been focusing on the capturing and retrieving of structured data through the use of schemas. Recently, there are sophisticated data types arising over the Web for DBMS to manage text, multimedia, spatial and temporal data. The need for scalable database services for Web applications demands the database server being replicated in different locations or to deploy a parallel and/or distributed architecture for the back-end database server. Distributed database systems and shared-nothing parallel database systems are increasingly being deployed to support the scalability and performance demands of end users. A distributed database is designed with portions of the data being dispersed among a number of computer systems. A distributed database has a directory that identifies the physical location of each data element in the database. Distributed database architectures typically make use of multiple computers with the objective to process the database requests in a way that is most effective on the basis of performance, operational, economic and geographic considerations. A distributed design can reflect an organization’s structure or function. Data can be stored locally and under control. Distributing data across multiple autonomous sites confines the effects of a computer breakdown to its point of occurrence. The collection of data need not be limited by a computer’s size and processing power. Three categories of distribution strategies have been developed for parallel database systems according to the criteria used in reducing costs incurred in resource utilization [3]. If the objective is to reduce data transmission over the network, then the policy could be based on network traffic [4]. If the goal is to balance the amount of data, or disk I/O access frequency, the strategies are based on size [12] and disk access frequency [9]. In this section, we are going to discuss some of the popular database architectures. 2.1. Centralized database architecture In centralized database architecture (CA), data processing support is provided by one or a cluster of computers, generally large computers, located in a central data processing facility. All data are stored at the central facility and are controlled by and accessible by the centralized database servers to serve many units in the organization. Centralized database is typically housed in a central computer facility. It is often used with a vertical data processing organization. The CA requires that all applications communicate via the centralized database. Figure 1 shows a typical enterprise system architecture where multiple web applications store and share business data using a central database. This has the advantage that any application can now reliably communicate with other applications through the database and all data relevant to each application are now held in the Copyright © 2012 John Wiley & Sons, Ltd.

Concurrency Computat.: Pract. Exper. (2012) DOI: 10.1002/cpe

A CENTRALIZED DATABASE ARCHITECTURE SHARED BY MULTIPLE WEB APPLICATIONS

Web Application 1

Database Web Application n

Figure 1. A set of web system applications are using a centralized database.

shared database. By making the database central to all business applications, the benefits of integrated database reliability and reduced redundant maintenance can be realized. Traditional enterprise DBMS applications perform data analysis within their organizational domains using a common schema to be ready for subsequent querying in a data warehouse. However, CA is weak at storing many-to-many relationships and faces the limitation of limited number of operational applications within an organizational boundary. The Internet business-to-business (B2B) commercial applications have raised the demand for ad hoc real-time on-the-fly information integration across different organizations. According to Marlin [5], commercial companies have shown trend to shift their computing infrastructures from monolithic servers to small server-based systems, which performs complex risk calculations to support the derivatives-trading business. 2.2. Partitioned database architecture The partitioned database architecture (PA) configures the database as distinct and non-overlapping segments that are dispersed among multiple computer systems. Typically, there is no duplication of data among the segments of a partitioned database. The aim of this approach is to disperses the load and eliminate a single point of failure. However, this approach can be counterproductive if the typical database request will involve data from multiple partitions. There are two strategies of partitioning: horizontal partitioning and vertical partitioning. It is often the case that an organization’s computing function will include both horizontal and vertical partitioning for different architectural considerations, which will be explained in the following sections. 2.2.1. Horizontally partitioned database architecture. One of the most effective ways to scale out data is to partition it among several databases so that each database server can handle a portion of the data. Although this is an obvious way to scale out data, not all data can be effectively partitioned, and if it can be partitioned, the way it is partitioned has a major effect on performance. Horizontally partitioned architecture (HPA) assumes data processing is distributed among a number of computers that have a peer relationship. Horizontal partitioning divides a table into multiple tables with each table containing the same number of columns but fewer rows. Databases in a horizontal configuration normally operate autonomously, although in some cases this configuration is used for load leveling. In many cases, horizontal partitioning reflects organizational decentralization. Figure 2 shows an illustration to partition a product database by warehouse ID range horizontally. This might make sense if products are most often accessed by warehouse ID instead of by product

Warehouse ID

Product ID

…...

Stock

1

P-0001

…...

1000

1

P-0002

1

…...

2

P-0001

…...

2

P-0003

…...

700

2

…...

…...

…...

…...

Database-A …...

DB-A

500 …… 2000

DB-B

Figure 2. Horizontally partitioned architecture. Copyright © 2012 John Wiley & Sons, Ltd.

Concurrency Computat.: Pract. Exper. (2012) DOI: 10.1002/cpe

S. CHEN, A. NG AND P. GREENFIELD

ID, but if there are many joins with the product table, this scheme would also require distributed joins. The only way to solve the join issue would be to partition the product database by warehouse ID number, so that, for a given warehouse, the products in that warehouse are readily available. In addition to load balancing for performance, HPA is a good database architectural choice for enterprises to easily manage large database tables, in which the data records belong to different business units (e.g. branches, shops etc.) 2.2.2. Vertically partitioned database architecture. Vertically partitioned architecture (VPA) assumes data processing is distributed in a hierarchical fashion. Vertical partitioning divides a table into multiple tables that contain fewer columns. There are two types of vertical partitioning: (a) normalization, which is the standard process of reducing redundant columns from a primary table using secondary tables that are linked by primary key and foreign key relationships; and (b) row splitting, which divides the original table vertically into tables with fewer columns. As an illustration of the vertical partitioning using the row splitting technique, let us examine how a product database might be partitioned as shown in Figure 3. One design option is to partition products based on what are available in stock and sale—stock level in one database, sale figures in another database and so on. Each logical row in a split table matches the same logical row in the other tables as identified by a unique key column that is identical in all of the partitioned tables. Aside from the obvious issues, such as what happens to a product that requires both stock and sales, this scheme may not perform well if the majority of queries join orders to products, because this join would have to check all the product databases for matching products. On the other hand, this type of partitioning will yield good performance for applications that require frequent localized access to stock or sale figures only. This distribution may reflect organizational structure, or it may simply be most appropriate for the application. The benefit is processing load can be assigned to the level of hierarchy at which it is most cost effective. Such an arrangement combines some of the best features of both centralized and distributed architectures. In addition, VPA also offers a good architecture to protect the privacy of the data in the partitioned database, where different business units/staff can only view and update a particular ‘party/view’ of the whole table according to their roles. 2.3. Replicated database architecture Database replication is widely used to improve both fault tolerance and DBMS performance. Replicated database architecture (RA) has all or part of the database copied at two or more computers. The key advantage of data replication is that it provides backup and recovery from both network and server failure. Because the database content is replicated and thus needs to consider the issue of how to efficiently maintain the consistency of several copies of the database stored on different machines. Real-time replication is often used in transactional systems, such as order entry, where all copies of data must be synchronized immediately. Updates involve the use of techniques such as the two-phase commit, which attempts to enforce consistencies in multiple databases by adding a confirmation phase to each update. Two-phase commit protocol [6] is one of the main techniques used to provide a consistent view in a distributed replicated database system, which

Product ID

Sale Data

Stock Data

P-0001

10000

3000

P-0002

1200

500

P-0003

1300

700

…...

…...

…...

DB-A

DB-B

DB-C

Figure 3. Vertically partitioned architecture. Copyright © 2012 John Wiley & Sons, Ltd.

Concurrency Computat.: Pract. Exper. (2012) DOI: 10.1002/cpe

A CENTRALIZED DATABASE ARCHITECTURE SHARED BY MULTIPLE WEB APPLICATIONS

requires substantial communication cost on each transaction and may require the full connectivity of all replicas to recover from some fault scenarios. Three-phase commit protocol [24] improves some of the problems of two-phase-commit protocols by adding an extra communication round. Although not common in real-time applications, non-real-time replication may be used. In this case, backups occur in batches, with small amounts of lag time. Deferred replication involves bulk transfer of a large number of changes at infrequent intervals, such as once or twice a day. This approach minimizes network resource requirements, but does not provide current data. In theory, data that are updated very frequently is often difficult to replicate very efficiently, because the overhead of replicating updates limits the scalability of the replicated copies [7]. According to Filip [8], DBMS such as Oracle uses two main techniques to implement a database replication process: (1) basic non-bidirectional snapshot replication which is implemented using materialized view or snapshot and can make only data copies (not code); and (2) multi-master replication which is bidirectional so that the data copy will be updated simultaneously. Technique based on materialized view or snapshot is easier to set up, but its functional potential is more limited than the more powerful multi-master replication technique. Peer-to-peer (P2P) replication uses the replication topology that all databases are at the same level (called nodes) with each database acting as a publisher and a subscriber in the replication process. Transactions originating at one node will be replicated to all other nodes. It is obvious that P2P replication has the benefits that (a) queries can be spread across multiple nodes to enable load balancing. This enables performance to remain consistent as reads increase; (b) system failure of one of the nodes will not affect the other nodes as application layer can redirect the writes for that node to another node; and (c) overall system availability can be maintained with each node can be taken offline and added back to the system without affecting the availability of the application. Amir and Tutu [9] proposed using a database replication technique over partitionable networks utilizing the enhanced group communication protocol called extended virtual synchrony to avoid the need for end-to-end acknowledgment per action. There are techniques using non-diverse redundancy by combining transactional concurrency control and group communication primitives in order to guarantee data consistency among replicas to provide both fault-tolerance and good performance [10]. The delivery of transactions can make use of the reliable multicast Read Once-Write All Available protocol [11] to guarantee one-copy serializability.

3. EVALUATION METHODOLOGY The performance of a distributed database system can be viewed from different aspects: (1) the connectivity aspect refers to the ability of components in the system to exchange data. In a vertically partitioned distributed database system, components of the system generally need links only to components above and below them in the hierarchical structure. In a horizontally partitioned distributed database system, it may be necessary to allow data exchange between any two systems; (2) the availability aspect refers to the percentage of time that a particular function or application is available for users. Depending upon the application, availability may be merely desirable or it may be essential; and (3) the performance of the communications aspect reveals the nature of the distributed database system and the applications that it supports. For highly interactive systems, response time is very critical. On the other hand, for systems used to move many data around, throughput is important. In this section, we will explain our experimental design and the environment we used to carry out our evaluations. 3.1. TPC-C benchmark We used our own implementation of the industry standard benchmark for online database processing: TPC-C to evaluate the performance of different database architectures: (a) CA; (b) PA; (c) two nodes peer-to-peer replicated architecture (2P2P); and (d) three nodes peer-to-peer replicated architecture (3P2P). The TPC-C benchmark provides a mechanism for performance comparison of three different database architectures using the same hardware configurations and operating systems. Copyright © 2012 John Wiley & Sons, Ltd.

Concurrency Computat.: Pract. Exper. (2012) DOI: 10.1002/cpe

S. CHEN, A. NG AND P. GREENFIELD

TPC-C is one of the industry-standard benchmark for online transaction processing. Although there are other benchmark standards available, we choose TPC-C on the basis of the following observations: • TPC-C is widely established and being used by the database community for many years. Both the database and software architecture communities can readily understand the performance figures produced from this exercise. • The nature of TPC-C is distributed, which fits nicely in the distributed requirements of this evaluation exercise. • Sample test codes that targets the TPC-C benchmark platform are readily available from existing software libraries, which shortened the development effort for this exercise and minimize the effect of newly written programs instability issues. TPC-E [12] is a later version. However, when we look into the specifications of TPC-C and TPC-E, we recognize TPC-C is for manufacturers and TPC-E is for broker. Although TPC-C application has stronger requirements for widely distributed database due to the nature of widely distributed warehouses (even globally), a broker (TPC-E) usually locates single country/capital city. TPC-E benchmark focus is the central database that executes transactions related to a firm’s customer accounts. On the other hand, TPC-C is more mature with mature codes and widely recognized by industry. Therefore, in our studies, we choose TPC-C instead of TPC-E for our benchmark exercise. 3.2. Experimental design Buch [13] pointed out that the TPC-C schema can be easily distributed using warehouse ID as the partitioning key and a majority of data accesses can be made local. It is because the TPC-C schema consists of nine tables and seven out of these nine tables have warehouse ID as part of their primary key. In our design of the physical location of different tables for the PA, we have put this in our consideration as such we have evenly spread the tables into different physical machines. Typically, the most costly operation in terms of database performance is disk I/O. Nowadays, many database servers offer dynamic caching capabilities to reduce the amount of physical disk access that is required to retrieve and modify data, and the query processing engine has read-ahead features to anticipate the data pages that are required for a given execution plan and preemptively read them into the cache. In our design, we have ensured that available performance enhancement features available in our test environment are turned on. To build an effective federation of database servers, there are points to be considered: (1) partition the individual tables in the original database so that most related data is placed together on a member server; (2) devise data routing rules that can be incorporated in the business services tier, so that applications can send each query to the member server that stores most of the data required by the statement. The most important goal is to minimize distributed processing in such a system. It is important to collect related data on the same member server and then route each query to a member server that contains most, if not all, of the data required to process the statement; (3) create member servers where each server has the horizontally partitioned data for one or more regions. If applications can identify the region currently referenced in the user’s input, the application can submit any generated query statement to the member server containing the data for that region. The only SQL statements that will generate distributed queries are those that reference data from multiple regions. We chose SQL Server 2005 as our evaluation platform because it supports partitioning of tables and indexes, which enables us to control the physical placement of data by assigning partitions from the same table or index to multiple file groups on separate physical storage devices. Also, it enables us to perform optimizations to the query processing engine for parallelize access to partitioned data. The specified measure of throughput is the number of transactions completed per minute (under the specified mix of transaction types). Our measurements were more detailed than those required by the TPC-C standard. We recorded the response times of the individual SQL statements and transactions executed by the DBMSs used in each node. Table I listed a summary of all the transactions that will be performed in the benchmark process for different database architectures. 3.2.1. Overview of test environment. Our test environment consists of a test manager, three client machines and one to three server machines to reflect the configuration requirement of different Copyright © 2012 John Wiley & Sons, Ltd.

Concurrency Computat.: Pract. Exper. (2012) DOI: 10.1002/cpe

A CENTRALIZED DATABASE ARCHITECTURE SHARED BY MULTIPLE WEB APPLICATIONS

Table I. Summary of TPC-C transaction profiles. Transaction

Query weighing Database access Frequency of execution

New order

Moderate

Read and Write

High

Payment

Light

Read and Write

High

Order status Delivery

Moderate Light

Read Only Read and Write

Stock level

Heavy

Read Only

Low Low (with deferred execution) Low

Response time requirements Stringent response time requirement to satisfy on-line users Stringent response time requirement to satisfy on-line users Low response time requirement Must complete within a relaxed response time requirement Relaxed response time requirement

database architectures. The three client machines are with equivalent hardware configuration setup. The specification of each machine is detailed in Table II. We used three server machines with equivalent hardware configuration setup for the three servers of different architectures to eliminate any extra variables. Microsoft SQL Server 2005 is installed in each server machine to support CA, 2P2P, 3P2P and PA. Their specifications are detailed in Table III. We used a dedicated machine to act as the test manager. The test manager’s role is to provide us with the ability to configure and manage the execution of the benchmarking on the database servers. The test manager also allows for the simultaneous execution of multiple instances of the client simulator application on multiple client machines, via the use of Microsoft message queuing (MSMQ). The test manager provides a graphical interface with an output window on the left, and a control panel on the right. The specification of test manager is listed in Table IV. Table II. Hardware and software specification for database clients. Machine

Clients A, B and C

CPU Memory Hard disk drive Motherboard Network interface Operating system Installed software

Pentium III 866 MHz 384 MB SEAGATE ST320414A 20 GB GIGABYTE GA-6VXC7-4X 3Com EtherLink XL 10/100 PCI Microsoft Windows XP Pro SP2 Microsoft Visual Studio 2005

Table III. Hardware and software specification for database servers. Machine

Servers A, B and C

CPU Memory Hard disk drive Motherboard Network interface Operating system Installed software

Pentium III 866 MHz 384 MB SEAGATE ST320414A 20 GB GIGABYTE GA-6VXC7-4X 3Com EtherLink XL 10/100 PCI Microsoft Windows XP Pro SP2 Microsoft SQL Server 2005 SP2

Table IV. Hardware and software specification for test manager. Machine CPU Memory Hard disk drive Motherboard Network interface Operating system Installed software

Copyright © 2012 John Wiley & Sons, Ltd.

Test manager Pentium 4 2.4 GHz 760 MB SEAGATE ST340014A 40 GB MSI MS-6526 Intel PRO/100 VE Microsoft Windows XP Pro SP2 Microsoft Visual Studio 2005 Concurrency Computat.: Pract. Exper. (2012) DOI: 10.1002/cpe

S. CHEN, A. NG AND P. GREENFIELD

We used an Ethernet 8 port hub SURECOM EP-508X-TM that provides a fast link at 100 Mbps. We used Netperf (command: netperft-a4.exe) by transferring 8 Mb of data for 10 s and to measure the throughputs between the test manager and other remote machines. It was recorded that the minimum throughput is at 86.43 Mbps. Figure 4 explains the data flow of our test environment. The test manager performs simultaneous execution of multiple instances of the client simulator application on multiple Client machines via the use of MSMQ. Each client simulator PC has a private message queue called TPCCClientRunner. Once the message queue has been successfully opened in a client

Figure 4. TPC-C benchmark dataflow. Copyright © 2012 John Wiley & Sons, Ltd.

Concurrency Computat.: Pract. Exper. (2012) DOI: 10.1002/cpe

A CENTRALIZED DATABASE ARCHITECTURE SHARED BY MULTIPLE WEB APPLICATIONS

simulator PC, the test manager will then send a message that encapsulates a serialized client config object, which stores the default warehouse ID, default SQL server connection string, a client ID and also the benchmark time in minutes. Once all initializing messages have been sent to the client simulator PCs, the test manager will then open up its local message queue, called TPCCClientManager, and wait for a replying message from each of the client simulator PCs. The reply message from the client simulators will contain a serialized test data object, which contains the list of all transactions that ran during the benchmark for all simulated clients on a client simulator instance. The test manager will then create a benchmark report based on all transactions that had been run and will write a benchmark report onto the disk. 3.2.2. Test manager design. While the test manager waits for the client simulators to complete their transaction execution, the test manager will periodically checkpoint the SQL databases. Checkpoints are a way to tell the SQL server to commit any changes to disk, rather than keeping them in virtual memory and this is specified by TPC-C to be necessary to provide data durability to the application. The checkpoint interval is set before benchmarking begins and is executed via the CHECKPOINT TSQL command. The test manager will checkpoint the databases involved in the benchmarking and will include all three SQL databases, except for CA benchmarking, which will only require checkpoints for server A. The client simulator works in similar ways to a server on the client side, in that it waits for incoming MSMQ messages containing client config objects. The client config object will specify • • • • •

how many simulated client terminals the client simulator should create the SQL server connection the client terminal should connect to the home warehouse ID and district ID benchmark duration database architectural type

At the completion of a test run, the test manager will receive a test data object message from each of the three client simulators. It will extract the transaction array lists from each of the test data objects and merge the transactions of each type into another set of five transaction array lists. Each array list contains all transactions of each type that had been run in the benchmark process. The test manager will pass the five array lists into a benchmark report object, along with the benchmark duration, the total time taken to run the benchmark and the number of checkpoints taken and the checkpoint interval. For each of the five transaction types, the benchmark report object will generate the following: • • • • • • • • •

90th percentile response time average response time maximum response time minimum keying time average keying time maximum keying time minimum thinking time average thinking time maximum thinking time

The maximum throughput of the application is calculated as the number of transactions per minute (using tpmc as its unit) by taking the number of transactions run in total, divided by the benchmark duration in minutes. A screenshot of the report is given in Figure 5. 3.2.3. Database client (test driver) design. Once the configuration has been parsed, the client simulator delegates client creation and management to the client runner class, which creates a thread for the specified number of clients. For each thread, client objects that represent client terminals are created with configuration settings passed in during instantiation. Each thread is then fired off and each client object will begin running transactions and recording response and latency times. The execution of the client begins by firstly obtaining a connection to the COM+ server by creating a TPCCComServer object. This COM object will remain alive and will run in the COM+ server over the duration of the benchmark execution, handling and processing all transactional requests. Copyright © 2012 John Wiley & Sons, Ltd.

Concurrency Computat.: Pract. Exper. (2012) DOI: 10.1002/cpe

S. CHEN, A. NG AND P. GREENFIELD

Figure 5. Screenshot of benchmark report.

Selection of a transaction is random, with each of the five different transactions having different chances of being selected. Following the TPC-C specifications, the total mix of transactions run need to be in the following percentages: • • • •

stock level—4% minimum delivery—4% minimum order status—4% minimum payment—43% minimum

The remaining percentages would be taken up by new order transactions. Once a transaction type is selected, the client creates a SetInputData in order to generate random data for the transaction, encapsulated in an InputData type object customized for that transaction type (See Section on TPC-C Benchmark). Before execution of a transaction, the thread the client runs in will undergo sleep for a length of time, in order to simulate data entry by the client. This time is known as keying time. TPC-C specifications has the following requirements for keying time: • • • • •

new order—18 s minimum payment—3 s minimum order status—2 s minimum delivery—2 s minimum stock-level—2 s minimum

Once keying time has passed, the transaction will then be called via the TPCCComServer object, passing in the InputData object as a parameter, as well as the SQL connection string to be used. The connection string is specified for the reasons of (1) to specify which SQL server is the home server for the given client and (2) to specify the database to connect to. After the transaction is successful and committed, a result string is returned. Otherwise, an exception will be thrown by the transaction, and the transaction will be rerun if there is a deadlock issue. All other transactions will be logged, the transaction marked as failed, and the next transaction will run. Timestamps will be taken just before and after the transaction is completed so as to obtain the response time of the transaction. Copyright © 2012 John Wiley & Sons, Ltd.

Concurrency Computat.: Pract. Exper. (2012) DOI: 10.1002/cpe

A CENTRALIZED DATABASE ARCHITECTURE SHARED BY MULTIPLE WEB APPLICATIONS

A second thread sleep will occur before the completion of the transaction run in order to simulate a client’s post transaction thinking time. The thinking time is calculated by the following formula, as specified in TPC-C: Tt ¼ logðrÞ  ų where log Tt r ų

natural log (base e) think time random number uniformly distributed between 0 and 1 mean think time

The mean think time for each transaction is • • • • •

new order—12 s payment—12 s order status—10 s delivery—5 s stock level—5 s

Before the next transaction is chosen, the keying, response and wait times, as well as the transaction type, and a timestamp of the transaction start time will be stored in a transaction object, which is then placed into an array list for the particular transaction type. The clients will stop executing transactions when the benchmark duration has elapsed. The client runner will generate a test data, which will be sent back to the test manager via MSMQ. The test data object contains five array lists, which is a concatenation of the five transaction array lists from each of the clients that ran the benchmark. The client runner will now open up the TPCCClientManager message queue remotely to the test manager, and send the test data object as a serialized object in a message, completing the client simulator’s job in the benchmark process. 3.2.4. Centralized architecture test environment. The test environment for CA consists of three client machines simulating three simultaneous TCP-C clients sending requests to one server machine as depicted in Figure 6, which consists of a CA database server A and three database clients client A, client B and client C being monitored by the test manager. 3.2.5. Partitioned architecture test environment. The test environment depicted in Figure 7 for PA consists of three client machines generating TPC-C requests to three server machines, which are partitioned using the warehouse ID as the potential partitioning column into three partitions according to the partition view of warehouse ID = 1, 2 and 3. By doing so, a collection of data belonging to a warehouse can be located in one server and redirecting most of the queries to the local server. Table V shows the percentage of transaction on each partitioned server. The distribution transaction coordinator in Microsoft SQL server is enabled to manage the committing of transactions when there are several different data sources involved. To enable partitioning of a table into a different number of tables, distributed partitioned view (DPV) provided by SQL server is used. The table is partitioned by declaring a CHECK constraint according to the partitioning column. The original table is replaced with several smaller member tables residing on different machines. There were a few changes that had to be made to the TPC-C benchmark specification. Because the DPVs can only maintain referential integrity within the local database, the foreign keys FK_HISTORY_REF_CUSTOMER in table HISTORY and FK_ORDER_LINE_REF_STOCK in table ORDER_LINE had to be removed. These particular keys will otherwise be violated whenever a reference to a foreign customer on a foreign server cannot be made on the local database. There is a mandatory requirement in executing a distributed transaction. The SET XACT_ABORT must be set to ON, as this will specify whether SQL Server automatically rolls back the current transaction when a transact SQL statement raises a run-time error. 3.2.6. Replicated architecture test environment. The performance of RA will depend on the performance of the replicas deployed and on the performance characteristics of the middleware Copyright © 2012 John Wiley & Sons, Ltd.

Concurrency Computat.: Pract. Exper. (2012) DOI: 10.1002/cpe

S. CHEN, A. NG AND P. GREENFIELD

USER TEST_MANAGER

CLIENT_A

CLIENT_B

CLIENT_C

Legend

SQL Server

Simulation of multiple clients

SERVER_A Instruction Flow

Figure 6. Test configuration for centralized architecture.

itself. Figures 8 and 9 depict the test environment for 2P2P and 3P2P. The 2P2P configuration uses two clients and two servers, whereas the 3P2P configuration uses three clients and three servers. One of the problems identified when deploying the replication of TPC-C database was the HISTORY table does not contain a primary key. We have modified the HISTORY table to contain an additional column named H_ID. This was the only modification to TPC-C’s database schema in this research and we believe this modification will not incur significant impact on the performance of the P2P replication architecture. The sample database size used in our test is 252 MB. It took 45 min 31 s to populate a threewarehouse database using CA with the cardinality listed in Table VI. For PA, the population of database was run on each server using a designed database population program. The average of duration in populating the database was recorded 16 min 23 s. The average size of the database was 93 184 MB. 3.2.7. Performance metrics. We used the following metrics to study the performance of different database architectures: • Network Throughput: It measures the amount of data per time unit that can be transferred over a physical link. It is measured in kilobits per second (kbps), Megabits per second (Mbps) or Gigabits per second (Gbps). Network throughput is important in distributed system because the rate of data transfer will set the limit of data that can be transferred over the network. • Maximum Qualified Throughput (MQTh): It is a performance metric that resembles the number of orders processed per minute, in the unit tpmc. The measurement takes other business transactions into account, which enter new orders, query the status of existing orders, deliver outstanding orders, enter payments from customers and monitor warehouse stock levels. • Response Time: We recorded the response times of the individual SQL statements and transactions executed by the DBMSs used in each node. Copyright © 2012 John Wiley & Sons, Ltd.

Concurrency Computat.: Pract. Exper. (2012) DOI: 10.1002/cpe

A CENTRALIZED DATABASE ARCHITECTURE SHARED BY MULTIPLE WEB APPLICATIONS

USER TEST_MANAGER

CLIENT_A

CLIENT_B

CLIENT_C

View

View

WAREHOUSE ID = 3

WAREHOUSE ID = 1

SERVER_A

SERVER_C

Legend

SQL Server Redirected Transactions Simulation of multiple (20+) clients

SERVER_B View

Instruction Flow

WAREHOUSE ID = 2 View

Distributed Partitioned View

Figure 7. Test configuration for partitioned architecture. Table V. Responsibility of each partitioned server. Server

Percentage of transactions

A B C

90% of W_ID = 1 and 10% of W_ID = (2,3) 90% of W_ID = 2 and 10% of W_ID = (1,3) 90% of W_ID = 3 and 10% of W_ID = (1,2)

4. EVALUATION RESULT AND DISCUSSION We followed the strategies recommended by the database industry to detect a possible bottleneck by monitoring the big four counters: memory, processor, disks logs and network counters. Three tests were run with PerfMon on and off to study the impact of PerfMon and the resulting tpmc throughputs show slight variation from 1352 to 1456. Furthermore, to ensure all the servers are running at optimal speed during all our tests, we monitored the servers CPU load and the results shown in Figure 10 confirm that the servers are using average 40% CPU load over the core test time. Hence, we concluded that the use of PerfMon does not increase additional overhead to the system under test. Microsoft MSDN [14] explains that some queries can be much slower thus degrading the overall performance. The following queries can be the bottleneck in terms of query processing: Copyright © 2012 John Wiley & Sons, Ltd.

Concurrency Computat.: Pract. Exper. (2012) DOI: 10.1002/cpe

S. CHEN, A. NG AND P. GREENFIELD

Legend

SQL Server Replicated Data

USER Simulation of multiple clients

TEST_MANAGER Instruction Flow

CLIENT_A

SERVER_A

CLIENT_B

SERVER_B

Figure 8. Test configuration for two nodes P2P replication architecture.

• Queries containing data conversion functions • Queries containing TOP clause in the SELECT statement • Queries using any columns are of the bit, timestamp or unique identifier data types Unfortunately, we also found there are four unavoidable queries that used TOP clause in SELECT statement. The initial benchmark tests that were run on partitioned database yielded poor results in terms of performance. Investigations showed that even processing of local queries through DPVs had a long duration due to improper routing. The client simulator was redesigned so that any local query is directly queried to the corresponding local table. In the event of remote query, the query will as usual be directed to the local DPV, which will then be redirected to the corresponding remote partition’s DPV. Another issue we have encountered is incompatible data type between the database and the application. For example, the data type of c data is actually CHAR. The supplied data type, in contrast, is VARCHAR. According to [25], non-deterministic query like this will cause the optimizer to send the query to all servers. We have to modify the line of codes to circumvent this situation. 4.1. Throughput analysis Instead of reporting on individual client’s performance figures, we took an average between the three clients because we have observed the clients performed similarly during the tests. The result in Table VII shows that CA has reached a maximum throughput of MQTh at 1591 tpmc serving 130 concurrent client requests with the server CPU processing time reaches 97% indicating a saturation condition has reached. The server machine disk time figures showed that the server machine has experienced extreme disk activity and has exhibited unstable behavior when the number of simulated concurrent clients was above 110. On the other hand, the network utilization figures showed no network issue for this architecture which is under 8% network utilization. Copyright © 2012 John Wiley & Sons, Ltd.

Concurrency Computat.: Pract. Exper. (2012) DOI: 10.1002/cpe

A CENTRALIZED DATABASE ARCHITECTURE SHARED BY MULTIPLE WEB APPLICATIONS

USER TEST_MANAGER

CLIENT_A

CLIENT_B

SERVER_A

CLIENT_C

SERVER_B

Legend

SQL Server Replicated Data

SERVER_C Simulation of multiple clients

Instruction Flow

Figure 9. Test configuration for three nodes P2P replication architecture. Table VI. Table cardinality for database with three warehouses. Table Warehouse District Customer History New order Order Order line Item Stock

Cardinality as benchmarked 3 30 90 000 90 000 27 000 90 000 900 000 100 000 300 000

The throughput performance for 2P2P is constantly less than both 3P2P and PA. It is even worse than CA at below 130 simulated clients load per machine but getting improved performance at higher client loads when compared with CA. This is a useful finding for database architects by highlighting the fact that implementing 2P2P does not necessarily produce better performance than CA when the numbers of concurrent clients are low. 2P2P required less amount of time to complete synchronization between the two nodes when compared with 3P2P. In theory, 2P2P should perform better than 3P2P at low client loads but the actual performance throughput figures show the Copyright © 2012 John Wiley & Sons, Ltd.

Concurrency Computat.: Pract. Exper. (2012) DOI: 10.1002/cpe

S. CHEN, A. NG AND P. GREENFIELD

Figure 10. Performance monitor for server C shows the average CPU load over the core test time was 40.178%. Table VII. Performance results for centralized architecture. Simulated clients per client machine

MQTh (tpmc)

Machine 60 70 80 90 100 110 120 130

1032 1145 1254 1464 1490 1589 1510 1591

CPU: processor time

Memory: available Mbytes

Physical disk: disk time

Network interface: utilization

Clients (%)

Server (%)

Clients (%)

Server (%)

Clients (%)

Server (%)

Clients (%)

Server (%)

33.7 34.0 39.5 47.0 45.4 48.1 48.4 50.4

65.9 72.3 83.1 92.0 93.6 97.2 84.9 84.3

45.1 41.8 38.5 37.2 35.8 34.4 31.3 30.6

8.1 8.1 8.1 8.3 8.1 7.8 7.8 7.0

1.1 1.1 1.4 1.5 1.4 1.5 1.3 1.2

290.4 273.7 258.5 276.5 252.1 263.2 812.8 769.3

1.8 1.9 2.2 2.5 2.5 2.7 2.6 2.7

5.0 5.6 6.5 7.2 7.4 7.7 7.5 7.9

contrary. We observed 2P2P servers started to peak at 100% CPU load from 130 simulated clients per client machine. Timeout errors started to appear at 170 simulated clients per client machine. Also, the memory usage for 2P2P shows only 8% was left when processing the TPC-C transactions and performing replications whereas the %disk time figures are well above 400%, which shows signs of high disk utilization. Therefore, we conclude that although 2P2P provides redundancy benefit, it is not a good solution for distributed database to improve throughput performance. Table VIII shows that 3P2P yields overall better throughput performance than CA and 2P2P for different number of simulated clients. Network utilization is not a concern for 2P2P, 3P2P and PA. In regard to throughput performance, we observed a linear relationship for 3P2P as the number of simulated clients increased, the throughput increase linearly. However, it shows signs of saturation at above 190 simulated clients as we observed the servers started to peak at 100% CPU load and timeout errors started to appear at the client machines. The time for the SQL servers to complete synchronization between each other also increased as there are more clients to simulate. In another words, the time to wait for all machines to idle became longer as there are more transactions to replicate between the nodes. It was observed that the servers were still busy for quite a while after the client machines have finished all TPC-C transactions. When we looked at memory utilization figures for 3P2P, we observed the servers were only left with 7% available memory when processing TPC-C transactions and replication among the nodes. This shows a clear concern for database architects to be aware of the need for monitoring memory usage when replicated database Copyright © 2012 John Wiley & Sons, Ltd.

Concurrency Computat.: Pract. Exper. (2012) DOI: 10.1002/cpe

A CENTRALIZED DATABASE ARCHITECTURE SHARED BY MULTIPLE WEB APPLICATIONS

Table VIII. Throughput performance for different architectures. Simulated clients per Centralized architecture Partitioned client machine MQTh (tpmc) architecture 60 70 80 90 100 110 120 130 150 170 190 210

1032 1145 1254 1464 1490 1589 1510 1591 — — — —

— — — 1408 — — — 2052 — 2483 2784 2873

Two nodes P2P replication architecture

Three nodes P2P replication architecture

— — — 775 — 1192 — 1387 1454 1604 — —

— — — 1472 — 1797 — 1939 2284 2526 2577 2698

architecture is deployed. We observed 3P2P server machines suffer from high disk utilizations well above 600% due to lack of main memory resulting high disk usage and paging. The servers started to show sign of increase in disk time when the number of simulated clients is above 150. Our explanation to the fact that 3P2P performs better than 2P2P even at low volume scenarios is that overheads in cross database server machine synchronization when the transaction volume is low, which hinders the performance of 2P2P whereas 3P2P has spare capacity to handle the client requests. One of the future research opportunities in this area is to study the behavior of P2P architectures in relation to the number of replication server machines. Figure 11 shows both PA and 3P2P produce similar throughputs performance at below 170 simulated clients per client machine. When the load is increased to 190 and 210 simulated clients, the throughput performance of 3P2P started to degrade and produced around 6% less than PA. In theory, PA should outperform RA for all test scenarios. However, we observed the throughput performance for PA is worse than 3P2P at low client load but better than 3P2P at high client load as shown in Figure 11. An explanation of this behavior is client loads are served evenly by RA servers and the synchronization overheads have less impact on the RA servers at low client load than overhead for server replication at high client load. Hence, we assert that 3P2P is most suitable for low volume transactions when compared with other architectures. Although with well-defined partitioning scheme in place, the throughput performance of PA is the best when compared with CA and RA. At 210 simulated clients, we observed the PA server machines have not reached the 85%

Figure 11. Throughput comparison between PA and 3P2P RA. Copyright © 2012 John Wiley & Sons, Ltd.

Concurrency Computat.: Pract. Exper. (2012) DOI: 10.1002/cpe

S. CHEN, A. NG AND P. GREENFIELD

CPU load threshold yet as shown in Tables IX–XI. No sign of memory bottleneck was observed, which shows better result than CA and RA. PA suffers high disk utilization level, which has %disk time constantly over 200%. However, when compared with other architectures, PA’s disk utilization levels are lowest. Overall speaking, PA is the choice of architecture in our throughput test results. The optimal number of clients to simulate per client machine was determined to be 90 for CA. As we have three client machines, we used 270 as the number of clients to run formal tests for CA, RA and PA and the resulting throughput performance is presented in Figure 12. PA offered the best performance as expected because the entire load has been spread out to three servers. RA was the slower performer, as it needs to handle the load as well as replicate the data to other nodes. Table XII shows the optimal throughput performance (tpmc) for all architectures under test. PA yields the best optimal performance at 630 concurrent clients with 2667 tpmc whereas 3P2P can handle 540 concurrent clients with 2236 tpmc.

Table IX. Performance results for partitioned architecture. Simulated clients per client machine

MQTh (tpmc)

Machine 90 130 170 190 210

1408 2052 2483 2784 2873

CPU: processor time

Memory: available Mbytes

Physical disk: disk time

Network interface: utilization

Clients (%)

Servers (%)

Clients (%)

Servers (%)

Clients (%)

Servers (%)

Clients (%)

Servers (%)

49.5 60.6 78.4 87.0 91.2

40.1 55.8 66.8 69.1 72.3

44.7 41.2 37.0 31.6 31.6

26.1 25.0 24.0 22.0 22.6

1.7 2.2 1.9 2.1 4.8

238.3 372.7 455.9 527.2 526.9

2.2 3.3 4.0 4.4 4.5

2.6 3.7 4.6 5.1 5.3

Table X. Performance results for two nodes P2P replication architecture. Simulated clients per client machine

MQTh (tpmc)

Machine 90 110 130 150 170

775 1192 1387 1454 1604

CPU: processor time

Memory: available Mbytes

Physical disk: disk time

Network interface: utilization

Clients (%)

Servers (%)

Clients (%)

Servers (%)

Clients (%)

Servers (%)

Clients (%)

Servers (%)

41.6 51.8 64.2 70.5 81.3

56.9 79.9 82.6 84.4 87.2

46.5 41.4 37.6 34.5 32.3

8.5 8.1 8.1 7.9 7.7

1.7 1.7 1.4 1.2 1.4

813.9 774.0 595.9 832.5 511.6

2.0 2.7 3.2 3.3 3.7

2.9 4.0 4.5 4.1 5.0

Table XI. Performance results for three nodes P2P replication architecture. Simulated clients per client machine

MQTh (tpmc)

Machine 90 110 130 150 170 190 210

1472 1797 1939 2284 2526 2577 2698

CPU: processor time

Memory: available Mbytes

Physical disk: disk time

Network interface: utilization

Clients (%)

Servers (%)

Clients (%)

Servers (%)

Clients (%)

Servers (%)

Clients (%)

Servers (%)

37.1 47.2 51.4 65.1 73.1 73.5 79.9

70.1 77.6 77.6 79.3 83.7 85.6 84.8

44.6 40.8 37.2 32.6 30.2 28.9 27.6

6.0 6.1 7.5 7.5 7.6 7.5 7.4

1.2 1.5 1.4 1.3 1.6 1.5 1.4

622.9 643.0 764.3 626.6 815.4 1108.3 1318.2

2.0 2.5 2.6 3.4 3.8 3.8 4.1

3.7 4.1 3.8 4.2 4.9 4.7 5.1

Copyright © 2012 John Wiley & Sons, Ltd.

Concurrency Computat.: Pract. Exper. (2012) DOI: 10.1002/cpe

A CENTRALIZED DATABASE ARCHITECTURE SHARED BY MULTIPLE WEB APPLICATIONS

Figure 12. Maximum qualified throughput (tpmc) of different architectures (270 total clients processed). Table XII. Optimal performance of different architectures. Architecture

Optimal number of clients to simulate per client machine

Total clients processed amongst servers

Maximum qualified throughput (tpmc)

Centralized P2P 2 nodes P2P 3 nodes Partition

90 135 180 210

270 270 540 630

1340 1232 2236 2667

4.2. Response time analysis Apart from measuring the MQTh figures, we have also studied the response times for the following TPC-C transactions: new order, payment, order status, delivery and stock level transactions. Our test result for CA (Table XIII) reveals the majority are new order and payment transactions contributing 87.5% of the total number of transactions. We have determined that 90 clients per client machine is the optimal number for each client machine. Testing with 90 concurrent clients per client machine, the peak throughput for new order is 3500 tpm with response time of 2.2 s; payment is 4000 tpm at response time of 0.4 s; order status is 375 tpm at 0.22 s response time; delivery is 310 tpm at 3.97 s. We have determined the optimal number of clients per client machine is 135 for 2P2P because it produces less number of timeout errors than the higher levels. Testing with 135 simulated clients per client machine (Table XIV), the peak throughput for new order is 3500 tpm with response time of Table XIII. Response time (in seconds) for CA (90 simulated clients per client machine). TPC-C transaction New order Payment Order status Delivery Stock level

Transaction mix (%)

90th percentile

Average

Max

44.24 43.31 4.15 4.08 4.22

5.45 1.01 0.57 7.45 0.53

2.86 0.62 0.35 3.97 0.37

16.91 13.43 17.60 25.60 28.54

Table XIV. Response time for two nodes P2P RA (135 simulated clients per client machine). TPC-C transaction New order Payment Order status Delivery Stock level

Transaction mix (%)

90th percentile

Average

Max

44.15 43.21 4.30 4.05 4.29

8.79 2.01 1.32 9.97 1.14

4.70 1.04 0.60 5.67 0.51

30.76 18.21 7.89 29.09 20.11

Copyright © 2012 John Wiley & Sons, Ltd.

Concurrency Computat.: Pract. Exper. (2012) DOI: 10.1002/cpe

S. CHEN, A. NG AND P. GREENFIELD

4.7 s; payment is 3800 tpm at response time of 1.04 s; order status is 480 tpm at 0.6 s response time; Delivery is 320 tpm at 5.67 s; and stock level is 585 tpm at 0.51 s response time. To compare the response time performance of 3P2P against CA, we used the 90 simulated clients per client machine load to run our tests (Table XV). We found the average response time figures for 3P2P are slower than the corresponding CA figures (3.9 s for 3P2P vs 2.86 s for CA new order transactions, 0.81 s for 3P2P vs 0.62 s for CA payment transactions, 0.42 s for 3P2P vs 0.35 s for CA order status transactions, 4.81 s for 3P2P vs 3.97 s for CA delivery transaction, and 0.53 s for 3P2P vs 0.53 s for CA stock level transactions). As the load increases, we found CA is not capable to handle loads above 135 simulated clients per client machine. 3P2P was able to handle loads at 180 simulated clients per client machine as listed in Table XVI. This result tells us again that RA will not deliver better response time performance than CA at low transaction volume scenario. As we can see in Table XVI, even after the load has doubled to 180 simulated clients per client machine, with the exception for order status transaction (which is 240% worse), the average response time for 3P2P was only getting 60–150% worse than the 90 simulated clients per client machine load scenario. Our studies on the response time figures for PA showed it can handle high transaction volume at 210 simulated clients per client machine with reasonably good result. We performed test on PA at 90 simulated clients per client machine to compare its performance against CA and 3P2P (Table XVII). The result in Figure 13 reveals PA performed best amongst the three architectures for all the five different TPC-C transaction categories with at least 30% better response time than the other two architectures. The exception is order status transaction where PA produces equal response time figure as CA (1.19 s). PA can handle higher volume than 3P2P up to 210 simulated clients per client machine as shown in Table XVIII. Even at this level, the response time produced by PA for new order (5.13 vs 6.71 s) and delivery (5.79 vs 7.99 s) transactions was faster than 3P2P at 180 simulated clients per client machine.

Table XV. Response time for three nodes P2P RA (90 simulated clients per client machine). TPC-C transaction New order Payment Order status Delivery Stock level

Transaction mix (%)

90th percentile

Average

Max

44.07 42.88 4.36 4.37 4.31

6.55 1.22 0.42 6.97 0.81

3.90 0.81 0.345 4.81 0.53

42.10 42.62 15.14 52.93 43.01

Table XVI. Response time for three nodes P2P RA (180 simulated clients per client machine). TPC-C transaction New order Payment Order status Delivery Stock level

Transaction mix (%)

90th percentile

Average

Max

43.97 43.06 4.54 4.28 4.15

12.47 3.49 2.00 13.75 2.10

6.71 1.91 1.19 7.99 1.41

82.67 35.54 94.57 94.95 133.21

Table XVII. Response time (in seconds) for PA (90 clients per client machine). TPC-C transaction New order Payment Order status Delivery Stock level

Transaction mix (%)

90th percentile

Average

Max

44.05 43.57 4.11 4.21 4.07

4.15 1.65 0.17 4.95 0.13

1.90 0.67 0.17 2.46 0.14

8.82 33.63 11.97 9.12 16.70

Copyright © 2012 John Wiley & Sons, Ltd.

Concurrency Computat.: Pract. Exper. (2012) DOI: 10.1002/cpe

A CENTRALIZED DATABASE ARCHITECTURE SHARED BY MULTIPLE WEB APPLICATIONS

Figure 13. Average response time at 90 simulated clients per client machine. Table XVIII. Response time (in seconds) for PA (210 clients per client machine). TPC-C transaction New order Payment Order status Delivery Stock level

Transaction mix (%)

90th percentile

Average

Max

44.33 42.97 4.19 4.20 4.31

7.41 4.94 4.61 7.6 4.44

5.13 2.48 1.55 5.79 1.57

22.91 31.10 10.00 14.54 16.23

We observed the PA servers started to peak at 100% CPU load at 210 simulated clients per client machine. However, no timeout error was recorded. There is no sign of memory bottleneck, which means the servers operate at better memory condition than CA and RA. With the hard disk utilization for PA at 200%, it is the best result compared with other architectures. Therefore, we conclude that PA is the choice of database architecture for both high and load volume database environment.

5. RELATED WORK The performance characteristics of distributed database architectures have been studied in the past, which were emphasized on the performance of different hardware components. Yu and Dan [15] have studied extensively the performance characteristics of hardware-sharing architectures: (a) the shared nothing architecture, where neither disks nor memories are shared; (b) the shared disk architecture, where all disks are accessible from all nodes; and (c) the shared intermediate memory architecture, where a shared intermediate level of memory is introduced. Wisniewski et al. [16] used the Linux Trace Toolkit Next Generation and the Linux Trace Toolkit Viewer to merge the data collected from each software stack layer and organized them in data structures that permit the identification of the producer of these events. The result shows a large number of processes and threads simultaneously active on each processor, which differs from typical scientific computing systems where on each processing element there is a single, or small number of, executing threads. Rahm [17] has studied the performance characteristics of different concurrency coherency control protocols for replicated database architecture. Sousa et al. [18] on the other hand studied the performance characteristics of replicated database based on group communication protocols with interest in the impact of faults in the performance of the system by injecting various fault types in the simulation models. Amir and Tutu [9] focus on the problems of maintaining data consistency between all the replicas of a replicated database system. As for the studies of the performance for Copyright © 2012 John Wiley & Sons, Ltd.

Concurrency Computat.: Pract. Exper. (2012) DOI: 10.1002/cpe

S. CHEN, A. NG AND P. GREENFIELD

partitioned database architecture, Yang and Wright [19] have presented a simple privacy-preserving protocol for learning the parameters of Bayesian network on vertically partitioned databases. Furtado et al. [20] have studied the performance of OLAP database clusters with focus on the adaptive virtual partitioning and hybrid design techniques in physical and virtual partitioning aspects only. In regard to the evaluation of performance for commercially available distributed database systems, Stankovic and Popov [21] implemented the TPC-C benchmark with five types of transactions: new order, payment, order status, delivery and stock level and set the probability of execution of each to evaluate the performance of two diverse replicated DBMS system: InterBase 6.0 and PostgreSQL 7.4.0. Database vendors such as Oracle and IBM have produced benchmark figures for their database products. IBM has reported [22] the replication performance of its database system DB2 in a single zSeries environment. However, in this exercise, IBM did not use TPC-C but its own version of test environment with multiple instances of a capture program to store changes to a changed data table and an apply program to fetch the data from the changed data table, then stores the rows in memory or spill files and applies each change to a target table. Oracle has [23] compared the Oracle database with IBM DB2 UDB to justify its superiority in the database market. Our study differs from other performance evaluation exercises because our focus is not on comparing the performance of different database products but instead the performance characteristics of particularly distributed database architectures.

6. CONCLUSIONS The paper has systematically studied the performance characteristics for four different database architectures: (a) centralized database; (b) partitioned database; and (c) replicated database. The TPC-C industrial standard benchmark was selected to conduct this study. We have found both replication and partitioning architectures do not incur network inefficiency for a reasonably fast network of 100 Mpbs. We confirmed that disk utilization was one of the main concerns hindering the performance of all the four architectures in our study. From performance viewpoint, partitioning architecture should outperform replicated architecture for all test scenarios because partitioning architecture does not require synchronization overheads. Our research confirms that partitioning architecture is the choice of architecture for high volume transaction scenarios, which consistently outperforms all other architectures for all test scenarios. However, our research shows the throughput performance for partitioning architecture is worse than three nodes peer-to-peer architecture at low client load. As for the peer-to-peer replication architecture, we found it is not worth considering two nodes peerto-peer architecture for performance improvement ground especially at low transaction volume because it produced poorer performance than centralized architecture at low volume scenarios due to overheads in server synchronization had used up the system resources of the database servers. However, three nodes peer-to-peer architecture did not suffer the same problem as the two nodes counterpart and yielded better performance than the two nodes peer-to-peer architecture and centralized architecture although not as good as the partitioning architecture. We understand that peer-to-peer replication architecture is required to enhance data reliability with full or partial data replication. Therefore, it is necessary for the database architects to make their own trade-offs between performance and other quality-of-services when choose to apply the results presented in this paper. In our best knowledge, this is the first work on benchmarking and comparing the three typical database (centralized, partitioned and replicated) architectures all together using TPC-C in a common way used in enterprise systems (i.e. standard usage of ODBC/JDBC, rather than extremely turned C/C++ code and very complicated SQL procedures). This evaluation has led to a number of findings and observations. Although some confirms our common beliefs, some are new findings. Both contribute to the literature in different aspects. We provide technical details about the evaluation, from evaluation methodology to testing configurations, from experimental results to analysis, to have demonstrated a complete software performance evaluation process and practices. In addition to performance, this paper also discusses the other benefits and weakness of each database architecture as common knowledge and experience to share with readers, who has little knowledge about database partition and replication, but would like to learn. Copyright © 2012 John Wiley & Sons, Ltd.

Concurrency Computat.: Pract. Exper. (2012) DOI: 10.1002/cpe

A CENTRALIZED DATABASE ARCHITECTURE SHARED BY MULTIPLE WEB APPLICATIONS REFERENCES 1. Codd EF. A relational model of data for large shared data banks. Communications of the ACM 1970; 13(6):377–387. 2. Nanda A. Oracle Database 11g: the top new features for DBAs and developers, (online) Accessed 12 Mar 2010 http://www.oracle.com/technology/pub/articles/oracle-database-11g-top-features/11g-partitioning.html. 3. Ye H, Kerhervé B, Bochmann GV, et al. Towards database scalability through efficient data distribution in e-commerce environments. In Proceedings of Third International Symposium on Electronic Commerce (ISEC’02), 2002. 4. Apers PMG. Data allocation in distributed database systems. In Proceedings of TODS. 1988, 263-292. 5. Marlin S. Wall street firms embrace cutting-edge IT, in information week, 2005. 6. Gray J, Reuter A. Transaction Processing: Concepts and Techniques. The Morgan Kaufmann Series in Data Management Systems. San Mateo (CA), USA: Morgan Kaufmann Publisher Inc., 1993 7. Microsoft, Scaling out SQL server 2005, 2006. 8. Filip I, Vasar C, Robu R. Considerations about an Oracle database multi-master replication. In Proceedings of 5th International Symposium on Applied Computational Intelligence and Informatics, Timişoara, Romania, 2009. 9. Amir Y, Tutu C. From total order to database replication. In Proceedings of the 22nd International Conference on Distributed Computing Systems (ICDCS’02), Vienna, Austria, 2002. 10. Jiménez-Peris R, Patiño-Martínez M, Alonso G. An algorithm for non-intrusive, parallel recovery of replicated data and its correctness. In Proceedings of 21st IEEE International Conference on Reliable Distributed Systems (SRDS 2002). Osaka, 2002. 11. Bernstein PA, Hadzilacos V, Goodman N. Concurrency Control and Recovery in Database Systems. Boston, MA, USA: Addison-Wesley Longman Publishing Co., Inc., 1987. ISBN:0-201-10715-5. 12. TPC-E, (online) Accessed 18 November 2009 http://www.tpc.org/tpce/tpc-e.asp. 13. Buch V. Database architecture: federated vs. clustered, Oracle, February 2002. 14. Microsoft, Peer-to-peer transaction replication, 2007. 15. Yu PS, Dan A. Performance analysis of affinity clustering on transaction processing coupling architecture. IEEE Transactions on Knowledge and Data Engineering 1994; 6(5):764–786. 16. Wisniewski RW, Azimi R, Desnoyers M, et al. Experiences understanding performance in a commercial scale-out environment. In Proceedings of European Conference on Parallel Computing (Euro-Par 2007), 2007. 17. Rahm E. Empirical performance evaluation of concurrency and coherency control protocols for database sharing systems. ACM Transactions on Database Systems 1993; 18(2):333–377. 18. Sousa A, Pereira J, Soares L, et al. Testing the dependability and performance of group communication based database replication protocols. In Proceedings of the 2005 International Conference on Dependable Systems and Networks (DSN’05), Yokohama, Japan, pp. 792–801, 2005. 19. Yang Z, Wright RN. Improved privacy-preserving Bayesian network parameter learning on vertically partitioned data. In Proceedings of the 21st International Conference on Data Engineering (ICDE ’05). Tokyo, Japan, pp.1196, 2005. 20. Furtado C, Lima AAB, Pacitti E, et al. Physical and virtual partitioning in OLAP database clusters. In Proceedings of the 17th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD’05). Rio de Janeiro, Brazil, pp. 143–150, 2005. 21. Stankovic V, Popov P. Improving DBMS performance through diverse redundancy. In Proceedings of 25th IEEE Symposium on Reliable Distributed Systems, 2006. 22. IBM, DB2 data replication for z/OS performance evaluation and analysis, May 2003. 23. Oracle, Technical comparison of Oracle database vs. IBM DB2 UDB: focus on performance, May 2005. 24. Keidar I, Dolev D. Increasing the resilience of atomic commit at no additional cost. In Proceedings of Symposium on Principles of Database Systems. pp. 245–254, 1995. 25. Raab F, Kohler W, Shah A. Overview of the TPC benchmark C: the order-entry benchmark, (online) Accessed 27 October 2009 http://tpc.org/tpcc/detail.asp.

Copyright © 2012 John Wiley & Sons, Ltd.

Concurrency Computat.: Pract. Exper. (2012) DOI: 10.1002/cpe