Database Under Pressure - Testing Performance of ...

Database Under Pressure - Testing Performance of Database Systems Using Universal Multi-Agent Platform Dariusz Mrozek, Bo˙zena Malysiak-Mrozek, Jakub Mikolajczyk, Stanislaw Kozielski

Abstract While testing suspicious transactions that cause performance problems in a production database it is helpful to use a specialized tool that can simulate an increased workload. In the paper, we show the model and architecture of the DBPerfTester@UMAP stress testing application that allows the controlled execution of various SQL scripts from many workstations. It thus provides the possibility to test how the specified database behaves under a large pressure from concurrent transactions. DBPerfTester@UMAP is based on the UMAP multi-agent system, which allows the dynamic scalability and extensibility of the testing system. As an addition, we present sample tests of the chosen database management system while running a long transaction. Key words: databases, performance, stress testing, database workload simulation, database workload replay, multi-agent system

1 Introduction Along with the development of global services provided through the Internet, current relational database management systems (RDBMS) [1] must meet the increasing demands regarding their performance. Database performance itself, however, can be understood in different ways. For example, most of the database developers and programmers will define it as a response time, i.e. interval between submitted query and obtained result. But for some other, the performance can refer to a throughput, i.e. a number of transactions that can be processed in a unit of time [2].

Dariusz Mrozek, Bo˙zena Malysiak-Mrozek, Jakub Mikolajczyk, Stanislaw Kozielski Institute of Informatics, Akademicka 16, 44-100 Gliwice, Poland, e-mail: dariusz.mrozek@ polsl.pl

This is an author's version of the paper. Original version: Gruca A., et al. (Eds.): Man-Machine Interactions 3, Advances in Intelligent Systems and Computing Volume 242, 2014, pp 631-641 The final publication is available at link.springer.com: 10.1007/978-3-319-02309-0_68

1

2

D. Mrozek, B. Malysiak-Mrozek, J. Mikolajczyk, S. Kozielski

There are many factors deciding on database performance. Delaney and Agarwal gave the most common of them in their book [2] with respect to Microsoft SQL Server. However, we can state that most of them are universal and are true with respect to other RDBMSs. These factors are [2]: • • • • • •

application architecture, application design, transactions and isolation levels, SQL code, hardware resources, and DBMS configuration.

We can observe that current databases, those that are situated locally and those in the cloud, often suffer the performance degradation in the face of increased throughput. This degradation is not very difficult to be observed by using tools provided by the concrete RDBMS vendor, but at the same time, it is not so easy to test the performance of a user’s database in terms of the escalating number of transactions. Therefore, we have developed a software application for testing performance of database systems by executing concurrent transactions. The software application uses the Universal MultiAgent Platform (UMAP) [16] and software agents [8] working in parallel and replaying the workload in order to increase a pressure on the chosen RDBMS, while testing its performance in various circumstances. The UMAP is a compliant with FIPA standards (Foundation for Intelligent Physical Agents) [4, 5, 6, 7] multi-agent system (MAS) [3, 17] that provides ready to use solutions for the implementation of software agents. Software agents are autonomous programs that have the ability to take action and communicate with other agents in order to achieve a common goal [17]. In the software application that we have developed this goal is to execute the given set of SQL statements forming a workload. The testing application is universal and can execute any SQL workload. Although our sample tests presented in the paper are focused on one of the mentioned performance factors, i.e. database transactions and isolation levels, we can use our testing application in many other situations. Model and architecture of the testing application is presented in section 3. Sample tests for a chosen RDBMS are shown in section 4. In section 5, we briefly compare our tool to other tools, which are listed in section 2, and give final conclusions on the application and its abilities.

2 Related Works Testing performance of database systems has been carried out for many years. Various approaches and benchmark sets has been designed and deployed. The Transaction Processing Council (TPC) is a corporation that develops

Database Under Pressure

3

standard workloads that can be used to test performance of various types of database systems, including OLTP, OLAP, cloud and Big Data systems [13, 15]. On the other hand, there are also various tools and frameworks created for testing workload on DBMSs, like e.g. Oracle Database Replay and SQL Server Distributed Replay. Oracle Database Replay enables users to replay a full production workload on a test system to assess the overall impact of system changes [9]. Microsoft SQL Server Distributed Replay is a tool used for application compatibility testing, performance testing, or capacity planning. It allows replaying a workload from multiple computers and better simulation of a mission-critical workload [14]. None of the mentioned tools uses multi-agent systems to this purpose. While reviewing scientific articles we found some works of other researchers that are close to our domain by applying multi-agent systems in database management and testing. In [12] author shows the use of the AgentTeam framework for dynamic database management (DDBM) and CourseMan environment for testing different DDBM protocols on heterogeneous DBMSs. In [11] authors present how to test multi-tier database applications by the use of a number of agents. And in [10] the same authors present an agent-based approach for the maintenance of database applications in terms of changes on the schema of the database underlying the application. In this paper we will try to show that applying the multi-agent system in testing performance of DBMSs brings several advantages. However, before we discuss them, we will see the architecture of the testing system and sample tests in the following sections.

3 Model and Architecture of the Testing System A system that simulates an increased workload is one of the tools that can be very useful while testing database performance. For this reason, our testing environment was established using the multi-agent system. Multi-agent systems contain many software agents cooperating together on a common task. In our scenario the task is to examine the performance of SQL workload executed against the chosen database management system. We have constructed a model of the testing system, which has a hierarchical structure, containing a set of agents A. The set A is defined as follows: A = AM ∪ ASE ,

(1)

where: AM is a singleton set containing a Manager Agent, and ASE is a set of Script Execution Agents: ASE = {ASEi |i = 1, ..., N } , where N ∈ N+ is a number of Script Execution Agents.

(2)

4

D. Mrozek, B. Malysiak-Mrozek, J. Mikolajczyk, S. Kozielski

In our testing system, the Manager Agent is responsible for managing and coordinating testing process. The main role of Script Execution Agents is to submit SQL commands to the specified database system and gathering execution indicators (e.g. execution time). The communication between agents is vertical and satisfies the following assumption: (3) ∀Ai ,Aj ∈A,i6=j Ai ≻ Aj =⇒ Aj ∈ AM , where the ≻ operator represents the act of communication. Like in all multi-agent systems, the agents of the testing system use messages in order to communicate to each other. The set of messages M is defined as follows: [ M= Mi , (4) i∈TM

where TM is a set of message types: TM = {S, P, R, ER} ,

(5)

where: • • • •

S messages contain scripts to be executed, P messages contain server connection parameters, R messages contain results of script execution, ER messages represent execution requests. We can also note that: ∀i,j∈TM i 6= j =⇒ Mi ∩ Mj = ∅.

(6)

Architecture of the modeled system is presented in Fig. 1. The superior Manager Agent contains database connection parameters and testing script with SQL commands to be executed in the specified database. The script and connection parameters are provided by a user (tester) through the graphical user interface (GUI) of the Manager Agent. Connection parameters and testing script can be then transferred to subordinate Script Execution Agents by appropriate communication channels (messages of appropriate types P and S). Script Execution Agents will establish the connection with the database server and execute obtained script in parallel, if they obtain the execution request message (ER) from the Manager Agent. They usually execute simultaneously the same script against the database. However, in some cases the script can be provided for each agent individually, if agents should execute completely different workloads. This can be done by a user through the GUI of each of the Script Execution Agents. Script execution is triggered by a user and mediated by the Manager Agent. The Manager Agent then starts measuring the execution time, which lasts until the last Script Execution Agent reports a completion of the script execution. The Manager Agent registers re-

Database Under Pressure

5

Fig. 1 Architecture of the testing application that uses UMAP agents.

ports from Script Execution Agents and counts the number of these reports. When all Script Execution Agents report a completion, the Manager Agent takes the latest completion time and calculates total execution time ∆tE of the whole workload, according to the expression 7: ∆tE = tE − tS ,

(7)

where: tS is a start time measured when the Manager Agent triggers execution of scripts on Script Execution Agents, and tE is an end time measured according to the expression 8: tE = max {tEi } , and i ∈ N,

(8)

where tEi is an end time for execution of the script by the ith agent, N is a number of Script Execution Agents taking part in the testing process. The overhead required for communication between agents is negligible. Messages are usually small in size, and larger messages, e.g. SQL scripts, are sent prior to the measurement of time. Moreover, in all reasonable scenarios when the DBPerfTester@UMAP can be used, the processing time far surpasses the time needed for the communication. Using such an approach, we can appoint many Script Execution Agents, coordinate their actions, and find out how the database system behaves un-

6

D. Mrozek, B. Malysiak-Mrozek, J. Mikolajczyk, S. Kozielski

der a strong pressure from concurrent transactions. The testing application that we have developed on the basis of the model and presented architecture is called DBPerfTester@UMAP (Database Performance Tester at Universal Multi-Agent Platform). Results of the sample tests with the use of the application are presented in the following section.

4 Sample Tests of Chosen Database Management System DBPerfTester@UMAP testing application has a universal purpose and can be used while testing different DBMSs. In the presented case, we used the testing application to examine the performance of chosen database system while executing particular SQL workload on a varying number of Script Execution Agents. The chosen database management system was Microsoft SQL Server 2012 Enterprise Edition hosted on the workstation with two CPUs Intel Xeon E5600 6 core, 16 GB RAM, and Windows Server 2008R2 64-bit operating system. During our work, we executed various SQL scripts with the use of DBPerfTester@UMAP. However, in the presented case we intentionally chose a very long transaction consisting of SQL commands, which perform scanning of entire tables while retrieving data, due to the complex filtering criteria, grouping and sorting operations. This had two goals: (1) we wanted to examine how the DBPerfTester@UMAP behaves itself while running long transactions, and (2) we wanted to test how Microsoft SQL Server 2012 manages locks in various transaction isolation levels and how it affects performance. The SQL script is presented below and it was executed against AdventureWorks2012 database - one of the Microsoft’s sample databases distributed for testing purposes. Transact-SQL Script Used as a Workload for Script Execution Agents BEGIN TRANSACTION SET STATISTICS TIME ON; SELECT pc.Name, p.Name, Color, SUM(h.SubTotal) as Sum_Subtotal, SUM(h.TotalDue) as Sum_TotalDue FROM Production.Product p JOIN Sales.SalesOrderDetail d ON d.ProductID = p.ProductID JOIN Sales.SalesOrderHeader h ON h.SalesOrderID = d.SalesOrderID JOIN Production.ProductSubcategory ps ON ps.ProductSubcategoryID=p.ProductSubcategoryID JOIN Production.ProductCategory pc ON pc.ProductCategoryID = ps.ProductCategoryID WHERE p.Name like ’%Frame%’ GROUP BY pc.Name, p.Name, Color ORDER BY pc.Name, Sum_TotalDue DESC; SELECT h.SalesOrderID, h.OrderDate, h.DueDate, h.TotalDue, p.Name AS Product_Name, d.OrderQty, d.UnitPrice, d.LineTotal FROM Sales.SalesOrderDetail d JOIN Sales.SalesOrderHeader h ON h.SalesOrderID = d.SalesOrderID JOIN Production.Product p ON d.ProductID = p.ProductID WHERE h.TotalDue > 2000 AND h.OrderDate >= DATEADD(yy, -9, GETDATE())

Database Under Pressure

7

AND DATEDIFF(dd, h.OrderDate, h.ShipDate) >2; UPDATE Sales.SalesOrderHeader SET SubTotal = 1.1 * SubTotal FROM Production.Product p JOIN Sales.SalesOrderDetail d ON d.ProductID = p.ProductID JOIN Sales.SalesOrderHeader h ON h.SalesOrderID = d.SalesOrderID WHERE p.Color =’red’; SET STATISTICS TIME OFF; COMMIT TRANSACTION;

The script contains two SELECT statements and one UPDATE statement. The first SELECT statement displays names, colors and categories, together with summary sales values, of these products which names consists of Frame. The second SELECT statement displays details of sales orders for products that have been sold in the last nine years, for which the difference between the order date and ship date exceeded two days, and total sales value on a single invoice exceeded two thousand dollars. The UPDATE statement adds ten percent to the sales order subtotal for products which have a red color. Transaction isolation levels themselves protect (isolate) concurrently executed transactions against threads and anomalies, like lost update, dirty read, unrepeatable read, and phantoms [1], that can occur during data modifications, and decide which of concurrency side-effects are tolerated by these transactions. For better concurrency we should keep the isolation at low level, e.g. Read Committed is usually the default level for most RDBMSs. However, some transactions are sensitive for data changes that can occur before they are completed, which can affect what they currently do. If the nature of data processing does not tolerate unrepeatable reads or phantom rows during a transaction, we have to increase the transaction to the Repeatable Read or Serializable isolation level. Database management systems implement isolation by requesting locks on processed data resources. The higher isolation level is set, the higher consistency is guaranteed, which unfortunately comes at the cost of concurrency. The SQL script presented above was executed in six transaction isolation levels available for Microsoft SQL Server 2012: • • • • • •

Read Uncommitted, Read Committed, Repeatable Read, Serializable, Snapshot, Read Committed Snapshot.

During our tests, the SQL script was executed three times for each of the isolation levels and results were averaged over these three measurements. In Fig. 2 we show the total execution time while executing the SQL workload for varying number of Script Execution Agents. For four out of six isolation levels the execution time is growing very slowly with the increasing number of agents. For all four isolation levels the execution time is similar - average

8

D. Mrozek, B. Malysiak-Mrozek, J. Mikolajczyk, S. Kozielski

time is 15.49s with a standard deviation of 0.29s for one agent, and 39.63s with a standard deviation 0.49s for eight agents. However, for Serializable and Repeatable Read isolation levels we can observe a large increase of the execution time for two, four, six and eight concurrently running agents. Certainly, we expected that by raising the isolation level we reduce the concurrency of transactions, but Fig. 2 shows how significant it can be for eight agents (clients). The question now arises - why there is so large decrease of performance for eight agents executing these complex SQL commands?

Fig. 2 Total execution time of SQL workload for varying number of Script Execution Agents.

We can find the answer for the question, if we study the number of deadlocks that occur while executing the transaction (Fig. 3).

Fig. 3 Number of deadlocks raised by executing SQL workload on varying number of Script Execution Agents.

Database Under Pressure

9

We can see that for Serializable and Repeatable Read isolation levels the number of deadlocks dramatically grows from 2 deadlocks for two agents, 11 deadlocks for four agents, up to 26 deadlocks for six agents, and 50 deadlocks for eight agents. For the other four isolation levels the number of observed deadlocks was 0. It is worth noting that the SQL workload was constructed and executed in such a way that every time the deadlock occurred and one of the transactions were aborted, the aborted transaction was restarted until it was finally completed. The presence of deadlocks in higher isolation levels (i.e. Serializable and Repeatable Read) is nothing unusual for this type of workload, but such a large escalation of deadlocks can be a bit surprising. Due to the complex filtering criteria, and even after creation of indexes covering the SQL statements, the query processor had to scan some of the used tables looking for rows that satisfy given filtering criteria. Shared locks are set up on the scanned rows when reading data, and the moment when they are released depends on the transaction isolation level. We have to remember that above the Read Committed isolation level Shared locks, which are set up on rows under consideration, are held until the end of a transaction. This prevents Exclusive locks during data modification to be placed on processed data, since Shared locks and Exclusive locks are not compatible. Therefore, the UPDATE statement in concurrent transactions has to wait for data resources very long and deadlocks occur. Our experiments gave us also the possibility to test two non-standard isolation levels - Snapshot and Read Committed Snapshot. Both isolation levels are based on row versioning. This improves read concurrency by eliminating locks for the read operation. Therefore, SELECT statements that are present at the beginning of our sample transaction do not lock data during the read operation, which does not block writers. Therefore, readers can access the last consistent value (version) of the row without getting blocked, while other transactions are updating the row at the same moment. This is reflected in short execution times (Fig. 2) and a small number of deadlocks (exactly 0, Fig. 3) even for eight agents.

5 Discussion and Concluding Remarks DBPerfTester@UMAP testing application allows the simulation of increased workload in the chosen database management system by executing various SQL scripts from several workstations. It thus provides the possibility to test how the specified database behaves under a large pressure from concurrent transactions or how it behaves after making other chages on the server side. This is very important, e.g. when we have suspicious transactions that cause performance problems in a production database, or we want to change the

10

D. Mrozek, B. Malysiak-Mrozek, J. Mikolajczyk, S. Kozielski

configuration of the database and check how it will influence the performance of the database system. Comparing the capabilities of DBPerfTester@UMAP to frameworks and tools mentioned in research papers in section 2, we can notice that although they relate to a wide database field, they have a slightly different purpose. While in case of [12] this is a dynamic management of various DBMSs, and in case of [10, 11] this is maintenance of complex database applications, the purpose of DBPerfTester@UMAP is stress testing against various DBMSs and workload simulation with the use of prepared and predefined transaction scripts. In this area, the goal of the DBPerfTester@UMAP is convergent with mentioned commercial tools. Comparing the DBPerfTester@UMAP to commercial tools, like Oracle Database Replay and Microsoft SQL Server Distributed Replay, we can say that it shares similar general architecture, including many workload replay agents (called here as Script Execution Agents). The critical look at DBPerfTester@UMAP reveals that commercial tools provide more functionality in terms of workload capturing, preparation, analysis and reporting. DBPerfTester@UMAP collects only basic statistics of the transaction execution time. Other execution statistics have to be collected by submitting appropriate commands, typical for the tested DBMS (e.g. for displaying statistics for I/O operations), or by using external tools. These tools, however, are usually available and delivered by vendors of database management systems. On the other hand, the core functionality of DBPerfTester@UMAP is very similar to compared commercial tools, and what is more important, oppositely to commercial tools, our tool is not tied to any particular DBMS. Apart from tests shown in section 4, which were just an example how we can exploit our system, the DBPerfTester@UMAP allows also to run tests that will verify the database performance in the following situations: • when migrating a database to work under control of different DBMS, e.g. when thinking of changing a database provider, • when providing changes in database schema, business logic implemented in stored procedures, functions and triggers, • when changing transaction isolation levels, • when changing the SQL code in user’s database application, • when scaling hardware resources, • when making changes in DBMS configuration. DBPerfTester@UMAP is based on the UMAP multi-agent system. This provides at least three advantages. (1) Script Execution Agents can be run on many workstations and we can put a pressure on the specified database. (2) The system is scalable; we can join additional agents at any moment of the testing process. (3) The system is extensible; .NET developers can run their own agents, since agents’ classes are loaded at runtime. Future works will cover further development of the DBPerfTester@UMAP, including improvements of the mechanism to re-establish a database connec-

Database Under Pressure

11

tion after it was broken. We also plan to adapt the testing application in order to test cloud-based data storage systems. Acknowledgements This work was supported by the European Union from the European Social Fund (grant agreement number: UDA-POKL.04.01.01-00-106/09). The UMAP platform used as the runtime environment for DBPerfTester@UMAP is available for free from: http://zti.polsl.pl/dmrozek/umap.htm

References 1. Date, C.: An Introduction to Database Systems, 8th edn. Addison-Wesley (2003) 2. Delaney, K., Agarwal, S., Freedman, C., Talmage, R., Machanic, A.: Inside Microsoft SQL Server(TM) 2005: Query Tuning and Optimization, 1st edn. Microsoft Press (2007) 3. Ferber, J.: Multi-Agent System: An Introduction to Distributed Artificial Intelligence. Harlow: Addison Wesley Longman (1999) 4. FIPA Abstract Architecture Specification, FIPA Standard (Accessed on April 2, 2013): http://www.fipa.org/specs/fipa00001/ 5. FIPA Agent Management Specification, FIPA Standard (Accessed on April 2, 2013): http://www.fipa.org/specs/fipa00023/ 6. FIPA Communicative Act Library Specification, FIPA Standard (Accessed on April 2, 2013): http://www.fipa.org/specs/fipa00037/ 7. FIPA Message Structure Specification, FIPA Standard (Accessed on April 2, 2013): http://www.fipa.org/specs/fipa00061/ 8. Franklin, S., Graesser, A.: Is it an agent, or just a program?: A taxonomy for autonomous agents. LNCS 1193, 21–35 (1996) 9. Galanis, L., et al.: Oracle Database Replay. In: Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, pp. 1159–1170 (2008) 10. Gardikiotis, S., Lazarou, V., Malevris, N.: An agent-based approach for the maintenance of database applications. In: Proceedings of the 5th ACIS International Conference on Software Engineering Research, Management & Applications (SERA 2007), pp. 558–568 (2007) 11. Gardikiotis, S., Lazarou, V., Malevris, N.: Employing agents towards database applications testing. In: Proceedings of the 19th IEEE International Conference on Tools with Artificial Intelligence (ICTAI 2007), pp. 173–180 (2007) 12. Kumova, B.: Dynamic re-configurable transaction management in agentteam. In: Proceedings of the Ninth Euromicro Workshop on Parallel and Distributed Processing, 2001, pp. 258–264 (2001) 13. Nambiar, R., Poess, M., Masland, A., et al.: TPC benchmark roadmap 2012. In: R. Nambiar, M. Poess (eds.) Selected Topics in Performance Evaluation and Benchmarking, LNCS, vol. 7755, pp. 1–20. Springer Berlin Heidelberg (2013) 14. SQL Server Distributed Replay, SQL Server 2012 Books Online, Quick Reference, June 2012 (Accessed on June 7, 2013): http://msdn.microsoft.com/en-us/library/ ff878183.aspx 15. Transaction Processing Performance Council, TPC-C/App/E BENCHMARKTM Standard Specification (Accessed on June 7, 2013): http://www.tpc.org 16. Walig´ ora, I., Malysiak-Mrozek, B., Mrozek, D.: UMAP Universal Multi-Agent Platform. Studia Informatica 31(2A (89)), 85–100 (2010) 17. Wooldridge, M.: An Introduction to Multiagent Systems, 2nd edn. John Wiley and Sons (2009)