Dynamic Tuning of Online Data Migration Policies in Hierarchical Storage Systems using Reinforcement Learning
David Vengerov
Dynamic Tuning of Online Data Migration Policies in Hierarchical Storage Systems using Reinforcement Learning*
David Vengerov SMLI TR-2006-157
June 2006
Abstract: Multi-tier storage systems are becoming more and more widespread in the industry. In order to minimize the request response time in such systems, the most frequently accessed (“hot”) files should be located in the fastest storage tiers (which are usually smaller and more expensive than the other tiers). Unfortunately, it is impossible to know ahead of time which files are going to be “hot”, especially because the file access patterns change over time. This report presents a solution approach to this problem, where each tier uses Reinforcement Learning (RL) to learn its own cost function that predicts its future request response time, and the files are then migrated between the tiers so as to decrease the sum of costs of the tiers involved. A multi-tier storage system simulator was used to evaluate the migration policies tuned by RL, and such policies were shown to achieve a significant performance improvement over the best hand-crafted policies found for this domain.
*This
material is based upon work supported by DARPA under Contract No. NBCH3039002.
Sun Labs 16 Network Circle Menlo Park, CA 94025
email address:
[email protected]
© 2006 Sun Microsystems, Inc. All rights reserved. The SML Technical Report Series is published by Sun Microsystems Laboratories, of Sun Microsystems, Inc. Printed in U.S.A. Unlimited copying without fee is permitted provided that the copies are not made nor distributed for direct commercial advantage, and credit to the source is given. Otherwise, no part of this work covered by copyright hereon may be reproduced in any form or by any means graphic, electronic, or mechanical, including photocopying, recording, taping, or storage in an information retrieval system, without the prior written permission of the copyright owner. TRADEMARKS Sun, Sun Microsystems, the Sun logo, Java, and Solaris are trademarks or registered trademarks of Sun Microsystems, Inc. in the U.S. and other countries. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the U.S. and other countries. Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc. UNIX is a registered trademark in the United States and other countries, exclusively licensed through X/Open Company, Ltd. For information regarding the SML Technical Report Series, contact Jeanie Treichel, Editor-in-Chief .All technical reports are available online on our website, http://research.sun.com/techrep/.
Dynamic Tuning of Online Data Migration Policies in Hierarchical Storage Systems using Reinforcement Learning∗ David Vengerov Sun Microsystems Laboratories UMPK16-160 16 Network Circle Menlo Park, CA 94025
[email protected]
1
Introduction
The computer industry is reaching a consensus that large scale systems should be self-managing and self-optimizing. Otherwise, too many human experts would be needed to manage them, and their performance would still be inadequate because of constant unpredictable changes in external environment. Most of the work done toward achieving these goals has focused on the self-management objective, which is much easier to achieve. The current industry-standard policy-based management approaches use built-in rules (or local agents) that specify what needs to be done when certain conditions arise. These rules do “something” toward achieving the performance objectives that the system designers had in mind. However, it is very difficult to make claims that such built-in rules actually optimize the system’s behavior. In order to realistically achieve the self-optimizing objective, a utility function should be formed for representing the desired performance objectives, and the system should then automatically take actions that optimize the current utility function. This approach raises the level of abstraction at which humans get involved into system management: instead of specifying what actions the system should take, human administrators will specify/adjust the optimization goals, which the system will automatically try to achieve. In order to implement this approach, however, the administrators need to decide on how to form the utility function for each automated decision-making agent responsible for implementing a certain policy. The optimization objective should ideally account for what is likely to happen in the future, rather than just optimizing the current state of affairs, as is done in classical linear and nonlinear programming optimization. Reinforcement Learning (RL) is an emerging solution approach for making decisions based on statistical estimation and maximization of expected long-term utilities (e.g., [13]). Some researchers have recently demonstrated how RL can be used for learning utility functions in real-world dynamic resource ∗
This material is based upon work supported by DARPA under Contract No. NBCH3039002.
1
Figure 1. Multi-tier Hierarchical Storage System.
allocation problems [2, 17]. Inspired by these works, this paper gives a demonstration of how RL can be used for learning utility functions in the problem of dynamic data migration in hierarchical storage systems. Many important details need to be worked out in order to apply RL to any large-scale real-world problem. The following important issues are addressed in this paper: deciding what intelligent agents the system will contain, choosing the variables to be used as inputs to the utility function of each agent, deciding how the outputs of utility functions should be translated into actual data migration decisions, choosing the computational architecture for approximating utility functions, and specifying the form of the RL algorithm to be used for updating different parameters of the utility approximation architectures.
2
Domain Description
Current enterprise systems store petabytes of data on various storage devices (fast disks, slow disks, tapes) interconnected by storage area networks (SANs). As more data needs to be stored, new storage devices are connected to such networks. The currently available storage devices range widely in price and performance. For example, accessing a file on a new tape drive requires a robot to physically change the tape drives before the new data can be accessed, which can take several seconds (while data on disks can be accessed in milliseconds). For performance reasons, it is desirable to keep the most frequently accessed “hot” data on fast disks while keeping the old “cold” data on the inexpensive slow disks or tapes. Figure 1 shows an example of a multi-tier storage system. Unfortunately, the file access patterns change over time, and the data layout cannot be optimized once and for all. For example, in e-mail servers the file access frequency tends to decrease as files stay longer in the system. The file access patterns in commercial Data Centers and Grids depend greatly on the clients connected to such systems at each point in time. Thus, online data migration has the potential to significantly increase performance of computing systems with hierarchical storage. Recently developed storage technologies (e.g., [9, 14]) allow transparent migration of files between storage tiers without affecting the running applications that are accessing them. However, the data migration policies they use are still based on some simple heuristics and no formal mathematical framework for minimizing the future request response time has been developed. The Aqueduct system [7] uses a control-theoretic framework to dynamically adjust the rate of background data migration to avoid Quality of Service (QoS) violations for the running applications. However, this system assumes that the data migration plan is given to it as an input. The STEPS architecture [18] for life cycle management in 2
distributed file systems also uses feedback control to dynamically adjust the data migration rate. The migrations in that system are triggered by heuristic threshold-based policies such as IF space utilization in PREMIUM POOL is greater than 80%, THEN migrate .tmp files greater than 1MB ordered by size to IDLE POOL until the space utilization of PREMIUM POOL is 60%. A number of online data migration approaches have been proposed in the research community. However, the approaches we are aware of are reactive in nature: they suggest migrating “hot” files that have been accessed more frequently in the recent past to the faster storage tiers while swapping out the “coldest” files. As a representative example, [12] suggests estimating the expected cost of accessing each file based on the file location and on its estimated access frequency. A file is then upgraded to a faster tier of storage (e.g., from disk to a local cache) if the total cost of all files involved in this migration (such as those that need to be dropped from the cache) is reduced. The cost of accessing each file is dynamically estimated using a sliding average of accesses to similar objects in the recent past. While reactive approaches can definitely bring some benefit to multi-tier storage systems, they do not address several important system-level considerations. For example, reactive approaches do not consider the impact of migration overhead on the future response times of other files in the affected storage tiers, even though a constant migration overhead added to an overloaded storage tier can significantly increase its queuing delays. Therefore, the total benefit received by the transferred files needs to be weighed against the future impact of migration overhead in the particular states of the affected storage tiers. The future response times of other files are also affected by changes in the file size distribution in each storage tier that result from file migrations. For example, the expected response times of all files in a storage tier will increase if a very large file is migrated into this tier, since the I/O requests to the original files will have to be queued for a long time while an I/O request for the very large file is served. Therefore, faster storage tiers have a preference for “hot” AND small files, and these two properties need to be traded off against each other on a case-by-case basis depending on the particular files and the states of the storage tiers involved in the migration. This paper presents a methodology for addressing the above tradeoffs by estimating the cost function for each storage tier – its expected future response time as a function of its current state. Given the cost functions for all storage tiers, the true impact of each migration decision can be easily estimated as the total change to the cost functions of the affected storage tiers that will result if the considered migration were to take place.
3
Hierarchical Storage System Model
Consider a multi-tier storage system as shown in Figure 1. Tier i has a limited bandwidth Bi , which is shared between file I/O requests and file transfers between the tiers (data migration). Multiple copies of a single file can be present in the storage system. Read/write/update requests arrive stochastically for each file. We assume that a certain policy already exists for routing new file write requests (a possible heuristic is to write the file to the fastest tier that has enough available space to fit this file). A file update request is routed to all tiers containing this file. A file read request is routed to the tier which has the smallest expected wait time out of the tiers containing this file. Requests at each tier form a queue and wait for their turn to be processed. The processing time for a read/write/update request for file k in tier i is the size of file k divided by Bi . File migrations can either be performed proactively or they can be triggered by I/O requests. The latter approach imposes a smaller load on the system, since the read or the write part of the migration 3
process will need to happen anyway as a part of executing the I/O request. This consideration was found to be very important in our simulations of heavily loaded systems, since a small increase in the load significantly increases the request response time in such systems. Therefore, the RL methodology will be applied in this paper to the I/O-triggered migration policies. When a file update request is received by some tier, the system decides whether this request should be routed instead to the next fastest tier, creating a new copy of the file in that tier and erasing the old copy present in the original tier. This consideration is made only if the file is not already present in the next fastest tier. When a file read request is received, the system first responds to that request and then decides whether this file should be upgraded to the next fastest tier. If an upgrade decision is made and not enough space is available in the faster tier, then some files are migrated out of that tier into the original tier. The migration-induced read/write operations are implemented as file requests that are sent to the corresponding tiers (that is, file migration operations do not take precedence over the regular file IO requests). The decision to upgrade a file increases the queue length of the new tier and may also increase the queue length of the old tier if some large files need to be swapped out of the new tier. However, such a decision may place the overall system into a more desirable state from the point of view of the expected future response time, if the file being upgraded is “hot” enough and the files being swapped out (if any) are “cold” enough. The objective is to design an automated policy for upgrading files so as to minimize the average system response time (possibly divided by the size of each request). The need for an automated migration policy extends to many different implementations of a multi-tier storage system, and some of the mechanisms described above can be easily modified without changing the spirit of the problem.
4
Solution Methodology
4.1
Defining and Using Cost Functions
This section addresses the first three issues (as was mentioned in the Introduction) that need to be resolved when applying RL to a real-world problem: deciding what intelligent agents the system will contain, choosing the variables to be used as inputs to the utility function of each agent, and deciding how the outputs of utility functions should be translated into actual data migration decisions. Since storage tiers generally possess different characteristics and can be dynamically added or removed from a multi-tier storage system, it is impractical to define a single utility function, mapping each possible data distribution among the tiers and properties of all the tiers into the expected future response time of I/O requests. Instead, we propose a scalable approach of viewing each storage tier as a separate decision-making entity (agent), allowing any pair of such agents to decide whether they want to redistribute the files among themselves at any point of time. In order to make data migration decisions, each storage tier (agent) learns its own long-term cost function C(s), which predicts the future average request response time divided by the file size (which will be called average weighted response time, AWRT), starting from a state s for the tier. An adequate state description is very important for learning such a function: the learning time for a given accuracy is exponential in the number of variables used to describe the state (so fewer variables are preferred), but at the same time all major influences on the future cost should be included so as to predict more accurately the future evolution of the agent’s state. Some methods are proposed in [8] for dynamically 4
choosing the most appropriate set of features that an agent should use in making its decisions, trading off compactness vs. completeness. Such investigations, however, are outside of the scope of this work. The state vector s we used for each tier is based on the notion of “temperature” of each file, which approximates its access rate. In order to compute this estimate, the system keeps track of read/update requests for each file and uses these statistics to predict the probability of that file being accessed in the near future. More precisely, whenever a file k is accessed at time t, the system updates an estimate of the request interarrival time τk for this file: τk = ατk + (1 − α)(t − Lk ),
(1)
where Lk is the last access time for file k and α < 1 is a constant that determines how fast the past information should be discounted. The temperature Tk of file k is computed as Tk = 1/τk . The following state variables were then formed: • s1 = average temperature of all files (stored and queued) in the tier. This variable predicts the number of accesses per unit of time the tier will receive in the future. Note that if requests are infrequent and queuing does not occur, AWRT for each request to tier i is 1/Bi . Thus, a higher value of s1 suggests a higher probability of queuing and thus a larger expected cost C(s). • s2 = average weighted temperature of all files (stored and queued) in the tier (temperature multiplied by the file size). This variable predicts the amount of data that will be requested from the tier per unit of time in the future. For a given value of s1 , a higher value of s2 implies that larger files become relatively hotter, which suggests longer expected queuing times for requests to smaller files (many requests for small files need to wait for a long time while a single request for a large file is being processed) and hence a larger expected cost C(s). • s3 = current queuing time in the tier (wait time for a new request to start getting processed). This variable directly correlates with the queuing time for requests that arrive in the near future. The first two variables are likely to evolve gradually in real storage systems with many files per tier, and hence can be computed at regular time intervals that are large enough to allow for a noticeable change in these variables (e.g., a 1% change). Whenever a read/update request is received for file k that is not already in the fastest storage tier, the system decides whether this file should be upgraded to the next fastest tier, as described in Section 3. In order to make such a decision between tiers i and j, each of them computes its long term cost Cnot = C(state s if no migration is to take place) and Cup = C(state s˜ that would result after file k is upgraded). The file is then upgraded if j i i j · s˜j1 < Cnot · si1 + Cnot · sj1 , Cup · s˜i1 + Cup
(2)
where si1 is the average temperature of all files in tier i if no migration is to take place and s˜i1 is the same quantity if the file in question were to be upgraded. The predicted average response time of a tier (cost) is multiplied by the tier temperature (s1 ) in the above migration criterion because AWRT for the multi-tier storage system can be computed as a weighted sum of AWRTs for all tiers, with the weights being the number of requests each tier has served, which is estimated by s1 . Note that once the tier cost functions C(s) are learned, any file reconfiguration decisions among the tiers can be evaluated using the criterion in equation (2). This approach is robust to tiers being added or removed from the system, is more scalable than centralized approaches and does not have a single point of failure inherent in centralized approaches. 5
4.2
Fuzzy Rulebase Representation of Cost Functions
Any parameterized function approximation architecture can be used for representing the tier cost functions C(s) and then tuning them with RL. We will use a fuzzy rulebase (FRB) for this purpose because its structure and parameters can be easily interpreted, as will be explained below. An FRB is a function f that maps an input vector x ∈
