data migration within cloud-based key-value stores has to face new challenges: effects of VM interference, and the need to trade off between the two ingredients ...
Optimizing Data Migration for Cloud-based Key-Value Stores Xiulei Qin1,2,3, Wenbo Zhang1, Wei Wang1, Jun Wei1, Xin Zhao1,2,3, Tao Huang1,2 1 Institute of Software, Chinese Academy of Sciences 2 The State Key Laboratory of Computer Science 3 University of Chinese Academy of Sciences Beijing, China
{qinxiulei08, zhangwenbo, wangwei, wj, zhaoxin09, tao}@otcaix.iscas.ac.cn commonly used in today’s cloud platforms. Key-value stores such as Gigaspaces XAP, Hazelcast and Couchbase support running on Amazon EC2, which uses Xen virtualization.
ABSTRACT As one database offloading strategy, elastic key-value stores are often introduced to speed up the application performance with dynamic scalability. Since the workload is varied, efficient data migration with minimal impact in service is critical for the issue of elasticity and scalability. However, due to the new virtualization technology, real-time and low-latency requirements, data migration within cloud-based key-value stores has to face new challenges: effects of VM interference, and the need to trade off between the two ingredients of migration cost, namely migration time and performance impact. To fulfill these challenges, in this paper we explore a new approach to optimize the data migration. Explicitly, we build two interference-aware models to predict the migration time and performance impact for each migration action using statistical machine learning, and then create a cost model to strike a balance between the two ingredients. Using the load rebalancing scenario as a case study, we have designed one cost-aware migration algorithm that utilizes the cost model to guide the choice of possible migration actions. Finally, we demonstrate the effectiveness of the approach using Yahoo! Cloud Serving Benchmark (YCSB).
As the load changes during different usage periods, live data migration with minimal cost and no disruption to the service availability is critical for the elasticity. However, due to the new virtualization technology, real-time and low-latency requirements, data migration within cloud-based key-value stores has to face new challenges. First, the VM interference (i.e., cache interference, I/O interference) poses adverse impacts on running applications and is especially severe for I/O intensive ones due to the resource aware CPU scheduling[2,3]. Second, it is necessary to trade off between the two ingredients of migration cost, namely migration time and performance impact1 . Explicitly, if too few network resources allocated to the data migration and migration time is long, it may cause an extended period of suboptimal performance problems. However, if too many network resources are allocated, the data migration completes quickly while the performance may suffer from a big hurt (see Figure 1).
Categories and Subject Descriptors H.3.4 [Information Storage and Retrieval]: Systems and Software, Distributed systems; C.4 [Performance of systems]: Design studies, Modeling techniques, Performance attributes;
General Terms Management, Performance, Design, Experimentation, Algorithms.
Keywords
Figure 1: Two ingredients of migration cost
Data migration, Key-value store, VM interference, Migration cost.
In this paper, we first study the impacts of VM interference on data migration for cloud-based key-value stores, which has received comparatively little attention. To fulfill these challenges, we explore a new approach to optimize the data migration within cloud-based key-value stores. Explicitly, we construct two interference-aware prediction models using statistical machine learning (SML), including migration time and performance impact models. The new factor of VM interference is incorporated. A rectangular area-based cost model is then presented to make tradeoffs between the two ingredients. In order to evaluate the new approach, we use the load rebalancing scenario as a case study and present a cost-aware migration algorithm that utilizes the cost model to guide the choice of possible migration actions. The experiments show that the migration approach could respond well to load imbalance and run data migration with low costs.
1. INTRODUCTION With the growing popularity of cloud computing, applications are more prone to provide highly-responsive, low latency service to a vast number of users. Too often, the database is the primary bottleneck. Additionally, the traditional OLTP applications are evolving into Extreme Transaction Processing (XTP) applications [1]. Supporting this confluence of trends requires new techniques and mechanisms. As a result, elastic Key-Value stores have emerged as a preferred choice. Virtualization techniques are Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CIKM’12, October 29–November 2, 2012, Maui, HI, USA. Copyright 2012 ACM 978-1-4503-1156-4/12/10 …$15.00.
1
2204
Performance impact is defined as the difference between the average response time during and after data migration.
Migration time (normalized)
2. BACKGROUND 2.1 Xen I/O Overview In Xen architecture, domain 0 has access to the hardware via native drivers and performs I/O operations on behalf of the unprivileged guest domains (In this paper, we also call them VMs), acting as a privileged domain. The frontend drivers in the guest domains and a backend driver in domain 0 communicate with each other using shared I/O rings and event channels [4]. In order to enable low overhead I/O, Xen implements a page-flipping mechanism to exchange data between the frontend and backend drivers. Figure 2 shows a typical Xen architecture where CPU scheduling, resource control and processing virtual interrupts, etc are all performed in VMM level.
4 3 2 1 0
vm=2 vm=4 vm=6 vm=8 vm=10 Data_size=50
Data_size=100
Data_size=200
Data_size=300
Figure 3: The migration time (normalized against vm=2)
Migration time (normalized)
5 4
Data_size=50
3
Data_size=100
2 1
Data_size=200
0
Data_size=300
Figure 4: The migration time (normalized against vm=(1,1)) Performance impact (ms)
Figure 2: Xen architecture
2.2 The Key-Value Store The key-value store used in this paper is called ElastiCamel developed based on memcached. The whole system consists of multiple independent servers mainly responsible for data service and executing migration actions, and one configuration server responsible for membership management, routing table maintenance, migration planning and control. We build the request routing mechanism on client side. With respect to the partitioning algorithm, we adopt a variant version of consistent hashing proposed by Amazon [5]. In the implementation, each node is allocated 16-18 partitions.
50 40 30 20 10 0 2
4
6
8
10
12
The number of VMs
Figure 5: Performance impact during data migration The VM interference has a significant impact on data migration time. We observe from Figure 3 that the migration time increases at most from 1.74× to 2.95× (50M), while at least from 1.19× to 1.8× (200M) when varying the number of VMs from 4 to 10 and migration data size from 50M to 300M. In Figure 4, the degree of interference is relatively more severe, the migration time increases at most from 2.02× to 4.02× (50M), while at least from 1.45× to 2.06× (300M). The different levels of interference are mainly due to their different data transmission mechanisms. In Figure 3 the data transmission between VMs is only through a virtual network bridge, while in Figure 4, data is transmitted through a virtual network bridge and the physical NIC which multiple VMs are competing severely for access to. Some previous work (i.e.,[7]) has suggested that performance impact is related with the bandwidth throttle and workload of migration node. However, we find the VM interference still has a strong correlation with the impact value. We observe from Figure 5 that when there are a small number of running VMs and the level of VM interference is low, the migration actions are quite efficient in making good use of the available bandwidth, resulting in a big impact (up to 30.1ms). We also find the interesting effects of VM interference. At first, it helps reduce the impact value little by little. However, when its level reaches a certain point, this factor becomes negative and the impact value begins to increase again (up to 40.2ms).
2.3 Motivation To study the impacts of VM interference on data migration, we use the Yahoo! Cloud Serving Benchmark (YCSB) which is applied to help evaluate the performance of cloud data serving systems [6].YCSB allows users to vary different parameters to emulate a variety of workloads. The parameters include read/write ratio, distribution of requests, thread count and data size. In our experiments, each node corresponds to a separate VM instance. We run each test under a fixed workload pattern (the read/write ratio, request distribution and are set to 95/5 and Zipfian respectively), and vary the number of VMs on the host to indicate different degrees of interference. We focus on understanding its impact on migration time and system performance. Figure 3 shows the migration time measured under the condition that two data migration nodes are co-located on the same host. We vary the number of VMs and data size to be migrated (50M, 100M, 200M and 300M) in x-axis and measure the different migration time which is normalized against that when there are only two running VMs on the host. Figure 4 shows the migration time measured when two data migration nodes are co-located on multiple hosts. We use the tuple (x,y) to denote the number of VMs running on host A and host B respectively. The results are normalized against those when each host has only one running VM. Figure 5 illustrates the performance impact of data migration. Based on these figures, we have the following interesting observations:
2205
make this assumption no longer hold true. Thus we also explore the NLR approach, explicitly a quadratic polynomial regression formally presented as equation (1).
Due to the facts mentioned above, the VM interference has non-ignorable effects on data migration, not only migration time but also performance impact during data migration. This new factor needs to be incorporated into the migration design. Here the data migration problem is divided into three subproblems: how to model the migration time and performance impact per action (migration action) for cloud-based key-value stores? How to strike a right balance between the two ingredients of the cost? How to determine the choice of migration actions when creating a migration plan using the cost model?
m
y= c+ i Pi + i 1
i 1 j i 1
m
ij Pi Pj + i Pi2
(1)
i 1
3.2 Training the Models To build the models, we use YCSB clients to simulate a variety of workload mixes(workload characteristics include read-write ratios 95/5 80/20 50/50, distribution of requests zipf/uniform/hotspot/ latest, thread count and request size ranging from 512B, 1k, 4k to 10k), as well as varying the number of VMs. The bandwidth throttle is controlled in six different intensities ranging from 8M, 20M, 40M, 60M, 80M to 100Mbps. Size of data to be migrated is also adjusted over a range from 1M to 400M. We measure the migration time and impact value on client side with the help of log information and test agent. Recall that two migration nodes may reside on the same (condition 1) or multiple hosts (condition 2), we differentiate these two conditions when constructing models. In order to ensure the models' accuracy, we adopt an iterative training approach [10]. During each iteration, a set of test data is used to measure the prediction accuracy of the model when the training is finished. If the accuracy threshold (It is set to 70% in this work) is not reached, we further subdivide the range of responses and examine whether each sub-range is represented by existing training data. If not, additional training data needs to be collected that aims at creating a more uniform spread in the output. This procedure iterates until the prediction accuracy achieves the desired level.
3.1 Interference-aware Prediction Models TO characterize the underlying VM interference, we choose a set of resource consumption metrics listed in Table 1. Metrics for guest domains include CPU usage, waiting time, execution count and blocked time. Considering all I/O requests of guest domains (VMs) are forwarded to domain 0, which is often regarded as the causes of communication interferences, its resource consumption metrics are also included. All these metrics can be easily collected using the XenMon tool. With respect to modeling, we apply several regression analysis methods that are commonly used for modeling the relationship between controlled input variables and observed response. Bandwidth throttle refers to the maximum amount of bandwidth allocated to data migrations for each node. Model 1: the migration time (ct) is as a function of VMs' resource consumption (v), size of data to be migrated (s) and bandwidth throttle (b) denoted as ct =ft (v, s, b). Model 2: the performance impact (ci) is as a function of VMs' resource consumption (v), workload of migration node (l) and bandwidth throttle (b) denoted as ci =fi (v, l, b).
3.3 The Robustness of Prediction Models
Table 1: The system-level metrics
Domain 0
m
where i ,βij , i are coefficients, p is principal component and c is a constant.
3. DESIGN CONSIDERARIONS
Guest domains
m 1
The accuracy of offline prediction models might degrade once new unmodeled features appear. This degradation has a negative impact on decision-making. In this work, we keep track of the prediction errors online and periodically rebuild the models. To trace the impact value, storage client library collects response time data for "get" and "put" requests (group by node) periodically, and send data to the configuration server at a fixed interval (We use 2 seconds). We collect the impact value of each action on overall system performance. Time taken for each migration action is traced by configuration server.
CPU utilization of the domain The percent of waiting time when the domain is in runnable state The percent of blocked time when the domain is in blocked state The number of execution periods per second for the domain Waiting time for each execution of the domain The number of VM switches per second over the measurement interval in domain 0 Total time blocked over the measurement interval in domain 0 The number of wakeups over the measurement interval in domain 0 The number of page mapping operations over the measurement interval in domain 0
3.4 Migration cost model
With respect to the details on how to build the prediction models, we introduce three approaches, including a stepwise linear regression (SLR) approach, a PCA-based linear regression (PLR) approach and a non linear regression (NLR) approach. The SLR approach aims to fit a multiple linear model using a stepwise technique [8]. It builds a model by successively adding or removing variables one at each time. The PLR approach uses Principal component analysis (PCA) [9] to reduce dimension and then applies the least squares curve fitting to deduce a linear model. Sometimes linear models may not provide a good fit of observed data, i.e., the complex VM communication mechanism, time-varying or skewed access patterns of applications tend to
Figure 6: The rectangular area-based cost model To make trade-offs between the two seemingly contradictory ingredients, we represent the migration cost in a two-dimensional space where x-axis denotes migration time ct while y-axis denotes performance impact ci. The cost value (cost) can be represented by the area of a given rectangle and computed as follows: cost = ct
2206
ci . The value varies as different choices of rectangles (see Figure 6).Consequently, the trade-off problem is converted into a rectangle selection problem. In fact, this rectangular area-based cost model is built based on the idea of integration in mathematics that given a function of response time and an interval of time, the definite integral could be computed.
unit of migration) and monitor the workload on these partitions. Before calling the algorithm, the cluster nodes are divided into two sets based on their normalized load values, MI_set and MO_set, such that MI_set contains all nodes whose load values are less than 1/n while MO_set contains all nodes that meet the opposite condition.
4. CASE STUDY In order to evaluate the new data migration approach, we use the load rebalancing scenario as a case study and build one rebalancing framework based on the Monitor-Analyze-PlanExecute (MAPE) loop model. The imbalance detection module calculates the balance rate, a metric that denotes how load is distributed across the cluster nodes. A rebalancing is invoked when it falls below the predefined threshold. The migration planning module determines a set of migration actions.
Figure 7: The rebalancing framework
4.1 Imbalance Detection Let li be the load value on node i. We firstly normalize each load value using formula (2). Let P={ p1, p2, p3,…, pn} be the normalized load values for cluster nodes (n is the number of storage nodes). According to Shannon's theory, the information entropy may serve as a measure of "mix-up" of a distribution. Thus we build the balance function F based on this concept. The entropy of P is denoted as formula (3). High entropy means P follows a uniform distribution while low entropy means the distribution tends to be varied [11]. n
pi =li / li i 1
(2)
H(P) =
n
pi log pi
As mentioned above, the data migration problem can be generalized to a multi-objective optimization problem, which is NP hard [12]. We seek to solve this problem using a greedy-based approach. Algorithm 1 gives the pseudo-code of the algorithm. It firstly greedily selects one migration node from MO_set and determines which of its partition to be migrated (lines 4-7) and then selects several candidate nodes from MI_set being able to accommodate the selected partition (lines 9-11). To determine the target migration node is a non-trivial task. For each candidate node, it first calculates the bandwidth throttle b allocated to data migration that leads to a minimum cost given the observed values (i.e., v, s and l). The algorithm then computes its minimum cost (denoted as costmin). Finally, it compares each computed minimum cost and chooses a minimum one. Thus one migration action and its target node are determined (denoted as w, lines 12-19). Each selected action is organized in a four-tuple style.
(3)
i 1
Obviously, the maximum of entropy is H(P)max=log n obtained for pi =1/n that corresponds to the most uniform load distribution. In order to present a simple and intuitive interpretation, the balance function that we build ranges in value from zero to one. To meet this requirement, we use the normalized entropy to denote how load distribution tends to be (see formula (4)).
4.3 Running Data Migration
For each node that participates in data migration "out", multiple migration actions initiated by this node are executed in sequence. 1 actions initiated by different nodes are scheduled in F= pi log pi / H P max pi log pi / log n However, parallel. During data migration, the location of data is changing while i 1 i 1 the system has to determine whether user request needs be routed to (4) the new or original server in accordance with the migration state. The routing table also needs to be synchronized across the cluster nodes 4.2 Cost-aware Migration Planning and clients. In order to guarantee the availability of the service, we In order to identify hot data efficiently and deliver a more fine-grained have designed a lightweight n-phase data access protocol based on data migration while reduce the amount of data to be migrated, we piggyback. Due to the space limitation, we do not present more manage data at each node in a small unit called partition (the basic details. n
n
2207
further research should explore whether it has a significant impact on the approach. The work uses the load rebalancing scenario as a case study and aims to eliminate the load imbalance while minimizing the migration costs. However, we do not consider the performance SLAs. The goal of reconfiguration should be optimizing resource allocations while not violating performance SLAs. In fact, based on the cost model, we could compute the optimal bandwidth throttle for each action under the SLA constraint. In the future, we hope to involve the SLAs in migration planning.
5. EVALUATION 5.1 Experimental Setup The experimental testbed consists of four powerful servers each with 16G of memory and two Quad-Core Intel Xeon E5620 processors (2.4GHz per core) running Xenserver 6.0. The store system consists of 30 nodes, each of which corresponds to a separate VM instance and is allocated 500M of memory. We create 15, 10 and 5 VMs running on three host servers respectively. Each VM is allocated one virtual core and 800M of memory.
7. CONCLUSION
5.2 Evaluation of Data Migration
In this paper we explore a new approach to optimize the data migration. We build two interference-aware prediction models to predict the migration time and performance impact using statistical machine learning (SML) and then create a rectangular area-based cost model to make trade-off between the two ingredients. A cost-aware data migration algorithm based on the cost model is proposed. The algorithm delivers a fine-grained control over bandwidth throttle for each migration action.
We compare the performance between our cost-aware (CA) approach and cost oblivious (CO) approach which only considers the imbalance rate when creating a migration plan and then uses a fixed bandwidth throttle (We use 20M, 40M and 60Mbps respectively) for all migration actions. Figure 8 shows the measured response time for the two approaches. We can see that the CA approach usually performs better than the CO approach. This could be explained as follows: Firstly, the CO approach uses a fixed bandwidth throttle while the optimal bandwidth throttle for each migration action can be varied due to the different levels of workload at each node and interference between VMs. Secondly, different possible migration actions have different costs. However, it is not considered in the selection of target nodes in the CO approach. We also compare the throughput for the two approaches shown in Figure 9. As can be seen, the CA approach performs better than its counterparts by a percent of 9-18%.
8. ACKNOWLEDGMENTS The work was supported by the National Natural Science Foundation of China under Grant No. 61173003, 61100068, the National Grand Fundamental Research 973 Program of China under Grant No. 2009CB320704 and the National High-Tech Research and Development Plan of China under Grant No. 2012AA011204.
9. REFERENCES [1] A. K. Nori. Distributed Caching Platforms. In VLDB, pp. 1645-1646, 2010. [2] X. Pu, L. Liu, Y. Mei, S. Sivathanu, Y. Koh, C. Pu. Understanding Performance Interference of I/O Workload in Virtualized Cloud Environments. In CLOUD, pp. 51-58. 2010. [3] Y. Mei, L. Liu, X. Pu, S. Sivathanu, X. Dong. Performance Analysis of Network I/O Workloads in Virtualized Data Centers. IEEE Transactions on Service Computing. 2011. [4] A. Menon, A. L. Cox, W. Zwaenepoel. Optimizing Network Virtualization in Xen. In USENIX, pp.15-28. 2006. [5] D. Hastorun, M. Jampani, G. Kakulapati, A. Pilchin, S. Sivasubramanian, P. Vosshall, and W. Vogels. Dynamo: Amazon’s highly available key-value store. In SOSP, pp. 205-220, 2007. [6] Cooper, B.F., Silberstein, A., Tam, E., Ramakrishnan, R., Sears, R. Benchmarking Cloud Serving Systems with YCSB. In SoCC, pp.143-154. 2010. [7] H. C. Lim, S. Babu, J. S. Chase. Automated control for elastic storage. In ICAC. pp. 1-10. 2010. [8] N. R. Draper and H. Smith. Applied Regression Analysis. 3rd edition. John Wiley and Sons, New York, 1998. [9] R. A. Johnson, D. W. Wichern. Applied multivariate statistical analysis. 6th edition. Pearson Prentice-Hall, Inc., Upper Saddle River, NJ, USA, 2007. [10] Kundu, S., Rangaswami, R., Dutta, K., Zhao, M. Application Performance Modeling in a Virtualized Environment. In HPCA, pp.1-10. 2010. [11] A. W. Moore. Information Gain. http://www.autonlab.org/tutorials/infogain11.pdf. [12] B. Trushkowsky, P. Bodik, A. Fox. The SCADS Director: Scaling a Distributed Storage System Under Stringent Performance Requirements. In FAST, pp.163-176. 2011.
Figure 8: Measured response time for different approaches
Figure 9: Measured throughput for different approaches
6. DISSCUSSION The approach works well in such scenarios that workloads tend to experience cyclic fluctuations. It also works efficiently when meeting new and unexpected workloads and can be used for other cases, e.g., scaling a key-value store system. However, the work still yields to some limitations. In order to reduce the complexity of problem formulation and modeling, we assume all nodes are homogeneous and with the same configuration, and the regression models do not take into account processor and memory heterogeneity. A
2208
