Performance Advantages of Running an Oracle® MySQL Replicated ...

An Oracle White Paper March 2014

Performance Advantages of Running an Oracle® MySQL Replicated Database on Oracle Linux with Oracle’s Sun Flash Accelerator F80 PCIe Card

Performance Advantages of Running an Oracle ® MySQL Replicated Database on Oracle Linux with Oracle’s Sun Flash Accelerator F80 PCIe Card

Introduction ....................................................................................... 1 Advantages of Using Flash-based Storage with a MySQL Replicated Database ......................................................................... 2 Overview of Oracle’s Sun Flash Accelerator F80 PCIe Card ............. 2 Configuring Oracle Linux and the MySQL Database for Optimum I/O Performance ................................................................. 3 Hardware/Software Used for Benchmarks ..................................... 3 Configure Oracle’s Sun Flash Accelerator F80 PCIe Card to use as a File System ................................................................. 4 Configuring the MySQL Database for Semi-Synchronous Master/Slave Replication ................................................................... 5 Master Database .................................................................................... 5 Slave Database ...................................................................................... 6 Starting MySQL Replication ................................................................... 7

Benchmark Results ........................................................................... 8 Baseline Results ............................................................................ 8 Implementing Sun Flash Accelerator F80 PCIe Card ..................... 8 Test 1 Results ......................................................................................... 8 Test 2 Results ......................................................................................... 9

Conclusion ........................................................................................ 9 Resources ......................................................................................... 9

Performance Advantages of Running an Oracle ® MySQL Replicated Database on Oracle Linux with Oracle’s Sun Flash Accelerator F80 PCIe Card

Introduction This document demonstrates how to increase MySQL database performance on the master server while simultaneously reducing replication lag time on the slave server by implementing ®

the PCIe flash based Sun Flash Accelerator F80 PCIe card. This guide demonstrates optimization of MySQL master and slave servers running Oracle Linux with Unbreakable Enterprise Kernel. By offering high performance with low latency and minimal CPU burden, the F80 PCIe Card can help maximize transactional I/O performance for applications in an Oracle Linux environment like online transaction processing (OLTP), data warehousing (DW), and data mining. By delivering the I/O performance of hundreds or even thousands of traditional hard disk drives (HDDs), the F80 PCIe Card is designed to improve user satisfaction and productivity by enabling applications that use the MySQL database to meet challenging service-level agreements (SLAs) for response times and throughput. Oracle Linux with Unbreakable Enterprise Kernel (UEK) was chosen for this study because of its upstream support for the latest hardware relevant to modern data center operations. In addition, MySQL database workloads benefit from the platform’s deep integration with the solution stack, optimizations resulting from industry collaborations and enhancements in the UEK. This solution guide provides a basis for planning and deploying PCIe compatible flash storage in a MySQL environment on Oracle Linux with a typical use case scenario.

1

Performance Advantages of Running an Oracle ® MySQL Replicated Database on Oracle Linux with Oracle’s Sun Flash Accelerator F80 PCIe Card

Advantages of Using Flash-based Storage with a MySQL Replicated Database OLTP and data warehouse (DW) applications have demanding requirements for short response times and high throughput, making it difficult for database administrators (DBAs) to maintain these systems as the number of users grows and the amount of data increases. During the application life cycle, performance bottlenecks might originate in one or more areas, including the network, processor, or storage devices, such as HDDs. Correcting a bottleneck in one area might cause a bottleneck in another area. In addition, if a customer wants to take advantage of MySQL’s built-in replication features to horizontally “scale out” their database, replication process performance can become a concern. MySQL offers different methods of replicating a database to provide for high availability, disaster recovery, scalability for high performance, or a combination of any of these. MySQL replication provides the capability for all the slave database(s) to synchronize data to the master database. The default replication operation mode in MySQL is asynchronous, meaning that the master server pushes changes to the slaves once the updates are committed. Replication has thus little overhead in transaction execution latency and the slaves eventually become consistent with the state of the master. In the following use case, the semi-synchronous replication method was used. When deploying semi-synchronous replication, the master server halts the commit of a transaction, similar to synchronous replication, until the master receives at least one acknowledgement from any of the slave servers. Another benefit of using semi-synchronous replication is that the acknowledgement from the slave server only states that the new data has been received and saved to the relay log file; the acknowledgement doesn’t mean that the slave server has actually applied the data to the database and the result is better performance. Flash-based storage provides a new tool for DBAs to help solve their MySQL database performance issues. For example, 5 ms is the typical response time for a small data read from a HDD. Flash-based devices such as the F80 PCIe Card can complete the same reading /dev/sda1 /mountpoint

4. Modify the kernel I/O scheduler. The I/O schedulers in Oracle Linux have new I/O capabilities, including options to modify these settings at boot time. In the tests described in this document, the DEADLINE I/O scheduler was used with the above file system mount options. To invoke the DEADLINE scheduler for the entire system, add this line to the /etc/grub.conf file, and bounce the system. $ kernel /vmlinuz-2.6.39-300.28.1.el6uek.x86_64 ro \ > root=/dev/mapper/vg_nytro-lv_root elevator=deadline

5. An alternative way to invoke the DEADLINE IO scheduler is to modify the /etc/rc.local file with the following command: $ echo "deadline" > /sys/block/sda/queue/scheduler

6. To verify that the DEADLINE scheduler is enabled, issue the following statement as root. $ cat /sys/block/sda/queue/scheduler noop anticipatory [deadline] cfq

7. In addition to changing the I/O scheduler, the file system mount options stated above were invoked, which was added to the /etc/fstab file. This example shows how the /etc/fstab entry invokes these options: /dev/sda1 /osfc ext4 defaults,noatime,nodiratime,max_batch_time=0,nobarrier,discard 1 2

An alternative to invoking these options is to specify them when executing the mount command: $ mount -o noatime,nodiratime,max_batch_time=0,nobarrier,discard \ > /dev/sda1 /osfc

4

Performance Advantages of Running an Oracle ® MySQL Replicated Database on Oracle Linux with Oracle’s Sun Flash Accelerator F80 PCIe Card

8. Performance of the F80 PCIe Card can be improved by increasing Queue Depth (QD) from the default of 128 to 512 or higher, depending on the database workload. Since the latency of the F80 PCIe Card is so small, more I/O operations/commands can be run in parallel on the F80 PCIe Card compared to a HDD. In order to modify queue depth, the nr_requests parameter will need to be modified to the same or larger value of the new queue depth value. Here are examples of modifying both the nr_requests and queue_depth parameters for /dev/sda which was used in our tests: $ echo "512" > /sys/block/sda/queue/nr_requests $ echo "512" > /sys/block/sda/device/queue_depth

9. Further performance benefits can be recognized by telling the operating system to complete an I/O request on the same CPU that initiated it. In some workloads this can provide a nice jump in performance. To enable this feature on the Sun Flash Accelerator F80 PCIe card: echo "2" > /sys/block/sda/queue/rq_affinity

10. Another operating system setting that can provide more performance is setting the rotational setting to 0 which means that the device is non-rotational, it is a flash device: echo 0 > /sys/block/sda/queue/rotational

Note: For these settings to persist across reboots, place all of these commands in the /etc/rc.local file.

Configuring the MySQL Database for Semi-Synchronous Master/Slave Replication This section describes the modifications needed to enable semi-synchronous replication using MySQL. To enable semi-synchronous master/slave replication in MySQL 5.6, the following configuration settings need to be applied. Master Database Install Plugins

The semi-synchronous plugin will need to be installed before starting the master server with the my.cnf changes for replication. From a MySQL prompt: INSTALL PLUGIN rpl_semi_sync_master SONAME ’semisync_master.so’; Configure Database

These are the database settings for the master to enable replication used for this test case: /etc/my.cnf [mysqld] log_bin=/u04/binlog/mysql-bin

5

Performance Advantages of Running an Oracle ® MySQL Replicated Database on Oracle Linux with Oracle’s Sun Flash Accelerator F80 PCIe Card

sync_binlog=1 binlog_format=’MIXED’ server_id = 10 # semi-sync replication rpl_semi_sync_master_enabled=1 datadir = ’/u04/datadir’ max_connections = 600 innodb_flush_log_at_trx_commit=1 innodb_buffer_pool_size = 4G innodb_thread_concurrency=49 slow_query_log=1 slow_query_log_file = "/u04/slow_query_logfile.log" innodb_log_group_home_dir = ’/u04/logs’ innodb_data_home_dir = ’/u04/data’ innodb_file_per_table=1 innodb_flush_log_at_trx_commit=1 innodb_log_files_in_group=4 innodb_log_file_size=2G innodb_thread_concurrency=0 innodb_flush_method = O_DIRECT innodb_write_io_threads=64 innodb_io_capacity=20000 expire-logs-days=1

Slave Database Install Plugins

The semi-synchronous plugin will need to be installed before starting the slave server with the my.cnf changes for replication. From a MySQL prompt: INSTALL PLUGIN rpl_semi_sync_slave SONAME ’semisync_slave.so’;

Configure Database

These are the database settings for the slave to enable replication used for this test case: /etc/my.cnf [mysqld] log_bin=/u04/binlog/mysql-bin binlog_format=’MIXED’ datadir = ’/u04/datadir’

6

Performance Advantages of Running an Oracle ® MySQL Replicated Database on Oracle Linux with Oracle’s Sun Flash Accelerator F80 PCIe Card

server_id = 2 # for semi-sync replication rpl_semi_sync_slave_enabled=1 innodb_flush_log_at_trx_commit=1 innodb_buffer_pool_size = 4G innodb_thread_concurrency=49 slow_query_log_file = "/u04" innodb_log_group_home_dir = ’/u04/logs’ innodb_data_home_dir = ’/u04/data’ innodb_file_per_table=1 innodb_log_files_in_group=4 innodb_log_file_size=2G innodb_flush_method = O_DIRECT innodb_io_capacity=20000 relay_log = ’/u04/logs/mysql-relay-log’ log_slave_updates = 1 skip_slave_start read_only = 1 expire-logs-days=1

Starting MySQL Replication

To start the replication process, start both the master and slave databases then execute the following commands from each database: MySQL Commands to Execute on Master Database

mysql> show master status\G; File: mysql-bin.000064 Position: 120 Binlog_Do_DB: Binlog_Ignore_DB: Executed_Gtid_Set: 1 row in set (0.00 sec) MySQL Commands to Execute on Slave Database

mysql> Reset slave; mysql> change master to master_host=’master_hostname’, master_ user=’repl’, master_password=’mysql’, master_log_file=’mysql-bin.000064’, master_ log_pos=120; mysql> start slave; mysql> start slave IO_THREAD;

7

Performance Advantages of Running an Oracle ® MySQL Replicated Database on Oracle Linux with Oracle’s Sun Flash Accelerator F80 PCIe Card

Benchmark Results Before adding the Sun Flash Accelerator F80 PCIe Card to the test configuration, Oracle Linux with UEK and the MySQL database were configured using techniques described in the above sections. OLTP benchmarks were executed to establish a baseline before and after implementing the Sun Flash Accelerator F80 PCIe Card. Quest® Benchmark Factory® software was used for these benchmarks with transactions per second (TPS) and average response time results documented. The slave metric “Seconds_Behind_Master”, indicating the number of seconds that the slave is behind the master in processing, was captured during these tests as well.

Baseline Results The results from the benchmark before installing the Sun Flash Accelerator F80 PCIe Card on the servers were the following:

 Average TPS: 312  Average Response Time: .32 seconds  Slave Seconds_Behind_Master: 1172 seconds These tests were executed with a configuration that resulted in the best results from HDD right before performance started to drop off due to overloading the HDDs.

Implementing Sun Flash Accelerator F80 PCIe Card After replacing the HDD where the database files are located on both servers with Sun Flash Accelerator F80 PCIe Cards, the following benchmarks were executed:

 Test 1: The baseline user load was determined when the HDD was almost 90% busy. When the F80 PCIe Card was implemented, performance increased dramatically. Instead of using the same load as the all HDD test, the user load of the benchmark was increased. The results show that even when increasing the load, implementing the Oracle F80 PCIe Card was dramatically faster than the all HDD setup.  Test 2: The Slave Seconds Behind Master test using the same configuration as with HDD Test 1 Results

 Average TPS: 996  Average response time: .1 second Note: By installing the F80 PCIe Card in both servers, it was possible to double the user load on the system, while improving average response times using the same 4G buffer size of the database.

8

Performance Advantages of Running an Oracle ® MySQL Replicated Database on Oracle Linux with Oracle’s Sun Flash Accelerator F80 PCIe Card

Test 2 Results



Slave Seconds_Behind_Master: 0

Conclusion In our benchmark testing with a Sun Flash Accelerator F80 PCIe Card, Oracle Linux with UEK and a MySQL database utilizing the master/slave semi-synchronous replication feature, large performance gains were realized and could directly improve enterprise environments using the configuration presented in this guide for workloads that are: 1. I/O bound due to large number of HDDs 2. I/O bound due to large HDD throughput requirements (MB/s) By implementing the Sun Flash Accelerator F80 PCIe Card in an Oracle Linux with UEK and MySQL replicated environment, not only will the F80 PCIe Card increase performance on the master, but it will also reduce the number of seconds that the slave is behind the master in replicating data. In addition, there are benefits from Oracle’s deep integration of the MySQL database solution stack, as well as optimizations resulting from industry collaborations and enhancements in Oracle Linux with the Unbreakable Enterprise Kernel.

Resources  Oracle's Sun Flash Accelerator F80 PCIe Card  Oracle MySQL Database  Oracle Linux

9

Performance Advantages of Running an Oracle ® MySQL Replicated Database on Oracle Linux with Oracle’s Sun Flash Accelerator F80 PCIe Card

10

Performance Advantages of Running an Oracle ® MySQL Replicated Database on Oracle Linux with Oracle’s Sun Flash Accelerator F80 PCIe Card

White Paper: Performance Advantages of

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

Running an Oracle® MySQL Replicated Database on Oracle Linux with Oracle’s Sun

This document is provided for information purposes only, and the contents hereof are subject to change without notice. This

Flash Accelerator F80 PCIe Card

document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in

March 2014

law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any

REV.03

Author: Rick Stehno, LSI Contributing Author: Phillip Goerl, Oracle

liability with respect to this document, and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission.

Oracle Corporation World Headquarters

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

500 Oracle Parkway Redwood Shores, CA 94065

Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and

U.S.A.

are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group. 0114

Worldwide Inquiries: Phone: +1.650.506.7000 Fax: +1.650.506.7200 oracle.com