Flexible Resource Allocation for Reliable Virtual Cluster Computing Systems Thomas J. Hacker #∗1 , Kanak Mahadik #∗2 #
Computer and Information Technology, ∗ Discovery Park Cyber Center, + Discovery Park NEEScomm
Purdue University West Lafayette, IN USA 1
[email protected]
ABSTRACT Virtualization and cloud computing technologies now make it possible to create scalable and reliable virtual high performance computing clusters. Integrating these technologies, however, is complicated by fundamental and inherent differences in the way in which these systems allocate resources to computational tasks. Cloud computing systems immediately allocate available resources or deny requests. In contrast, parallel computing systems route all requests through a queue for future resource allocation. This divergence of allocation policies hinders efforts to implement efficient, responsive, and reliable virtual clusters. In this paper, we present a continuum of four scheduling polices along with an analytical resource prediction model for each policy to estimate the level of resources needed to operate an efficient, responsive, and reliable virtual cluster system. We show that it is possible to estimate the size of the virtual cluster system needed to provide a predictable grade of service for a realistic high performance computing workload and estimate the queue wait time for a partial or full resource allocation. Moreover, we show that it is possible to provide a reliable virtual cluster system using a limited pool of spare resources. The models and results we present are useful for cloud computing providers seeking to operate efficient and cost-effective virtual cluster systems.
1.
INTRODUCTION
The availability of reliable virtualization and cloud computing technologies now make it possible to create scalable and reliable virtual high performance computing clusters. By unlinking the traditional tight coupling between the operating system image and the physical computing hardware, a new world of possibilities are opened for creating extremely flexible, configurable, and customized cluster computing systems that can support a wide variety of operating system images and runtime environments. As a rule, cloud computing systems either provide immediate access to requested computational resources or deny (block) a request if resources are unavailable. A request to a cloud computing system
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. SC11 November 12-18, 2011, Seattle, Washington, USA Copyright 2011 ACM 978-1-4503-0771-0/11/11 ...$10.00.
2
[email protected]
for a virtual cluster requires access to two or more computational nodes. This brings up the question: what kind of cloud allocation policies are needed to create on-demand virtual clusters? More specifically, what types of scheduling policies are needed to meet virtual cluster demands in an efficiently operated cloud computing system? In this paper, we explore this question. We present a range of scheduling policies for a cloud computing system, and demonstrate the effects of these polices on the probability of blocking resource requests for virtual clusters. We explore the impact of the size of the provisioned cloud on the probability of blocking requests for each scheduling policy, and present new hybrid scheduling polices for a cloud computing system that can reduce the probability of denying requests while increasing the overall efficient use of the cloud computing system. Finally, we present a model for assessing and improving the reliability of virtual clusters within a cloud computing system.
2.
VIRTUAL CLUSTER SCHEDULING
Supercomputers based on cluster computing architectures using commodity hardware now dominate the Top500 list of supercomputers [1]. Parallel applications running on these systems rely on exclusive access to set of processors to ensure good scaling and performance. Mediating access to processors for parallel applications is based on the use of a scheduling system. Scheduling systems used today include PBS [2], Condor [3], and LoadLeveler [4]. Resource requests wait in a queue managed by the scheduling system when demand exceeds available resources. The drawback of this approach is that the queue can grow and wait times can become very long. Consequently, it is expensive to pre-reserve resources for future use, since these resources may sit idle while jobs that could have used the resources languish in a queue. Cloud computing systems that provide Software-as-a-Service and Infrastructure-as-a-Service (SaaS & IaaS) [5] use a different model for managing access to resources. In contrast to the queueing model used for supercomputing systems, the cloud computing model either immediately provides resources to service a request, or rejects the request. The drawback of this allocation policy is that cloud providers must reserve adequate slack resources to accommodate a future stream of requests. Consequently, resource requests for n nodes (n > 1) for a virtual cluster will be denied, even if (n − 1) nodes are immediately available. This allocation policy of "all-ornothing" is fundamentally different from the queue-based model for parallel applications.
2.1
The need for a hybrid model
Strictly adhering to the traditional supercomputing queueing model or the "all-or-nothing" resource allocation approach of cloud computing has several inherent problems that limit the usefulness of these approaches for a parallel computing workload. In the "all-ornothing" cloud computing approach there is no option to provide a partial allocation of resources to service a request. A parallel application that could use a fraction of the requested allocation as an alternative to a full allocation would benefit from a scheduler that would grant an initial partial allocation with the option of granting access to the remainder later. This quick immediate allocation could provide the resources needed to start a multi-node virtual cluster. The second problem with the "all-or-nothing" cloud scheduling policy is that there is no option to manage a queue for requests that could wait for resources. Finally, there are no good methods to estimate the number of resources that must be kept "slack" waiting for future requests. This is a serious problem for cloud computing providers seeking to minimize costs and maximize profits. There are also problems inherent to the traditional supercomputing scheduling approach. The need to wait in a queue for resources makes it difficult for a cloud computing system to quickly provide resources to web-based science gateway systems (such as the NEEShub [6][7]) when user-initiated tools need "instant on" access to computational resources. Second, users with large resource requests can experience unpredictably long queue wait times, especially when computational resources are limited. Using these scheduling policies independently makes it difficult to quickly create a virtual cluster or run an efficient cloud computing system for virtual clusters. What is needed is a continuum of scheduling and allocation policies that is a blending of supercomputing and cloud computing scheduling policies for parallel applications. This continuum is anchored at one end in the "allor-nothing" cloud scheduling policy, and at the other end by the traditional supercomputing queueing policy. Between these two anchors are two other scheduling options: 1) provide a partial immediate allocation to a request and block the remaining fraction of the request; and 2) provide a partial immediate allocation and queue the remaining fraction of the request. In this blended model, resource requests can specify the type of service they require: immediate full allocation; immediate partial allocation of a fraction of the request, with an option to wait for remaining resources; or a queueing of the entire request. Cloud computing operators seeking to provision a physical and virtual infrastructure for virtual clusters face the problem of estimating the amount of resources needed to satisfy a future workload. In prior work, Hacker [8] developed a technique to estimate the probability of blocking a mixture of multi-node requests offered to a high performance computing system based on historic workload. We extend this work by considering the sizing of virtual clusters and the impact of a range of scheduling policies on the probability of blocking a request and the wait time of a request. Our approach is a blending and harmonization of the traditional HPC scheduling policy with the "all-or-nothing" instant allocation policy used in cloud computing. The need for a mix of resource allocation policies is motivated by an emerging set of parallel applications that can be used through a science gateway (such as the NEEShub and HUBzero [9]), in which quick access to parallel applications is provided as Software-as-a-Service (SaaS) to the community. Our approach is tailored to meet the needs and polices of the resource providers and user community. Our work is motivated by the need for techniques to predict the size of the physical and virtual infrastructure needed to operate an efficient, responsive, re-
liable, and cost-effective cloud computing infrastructure for virtual clusters within an on-demand cloud computing service.
3.
CLUSTER ALLOCATION PROBLEM
A formal description of the problem we address is: given a physical system consisting of N nodes with p processors capable of hosting virtual machines, a historic record of past offered workload, and a queue of requests for M virtual processors running on virtual machines with each request seeking Rp processors, how can we estimate apriori the number of virtual machines and virtual clusters needed to satisfy demand for access to resources with a predictable probability of blocking access to resources? Moreover, if requests can tolerate waiting in a queue for access to resources, what is the probability that the queue wait time will be bounded by some time W? The answers to these problems can be used by cloud resource providers to estimate the amount of resources needed to satisfy an offered stream of resource requests, and thus find a profitable balance between infrastructure costs and revenue.
3.1
Virtual Cluster Allocation Policies
We consider four resource allocation policies. The first policy (Type I) is the usual cloud allocation policy. In this scenario, a request for resources is immediately and fully fulfilled or is completely rejected. In this situation, resource requests are refused due to insufficient free resources available to satisfy requests. At the other extreme is the traditional HPC scheduling policy (Type IV), in which all resource requests go through a queue for complete immediate or delayed resource allocation. If all of the requested resources not available, the entire resource request waits in a queue for resource availability. The Type I and Type IV resource scheduling policies represent two extreme ends of a spectrum of potential resource allocation strategies. An additional set of policies are a blending of these two approaches that provide the option for partial immediate allocation and queueing. We consider four distinct types of scheduling polices.
3.1.1
Type I - All or nothing resource allocation
Type I scheduling is a complete cloud approach: all-or-nothing resource allocation. The most useful characteristic of Type I scheduling is that resource requests can be immediately satisfied when resources are available. Applications that require on-demand SaaS and IaaS systems as well as science gateway systems that provide applications as a service [10] are the primary uses of this type of scheduling approach. The downside of Type I scheduling is that requests for access can be unexpectedly blocked when there are no available resources. This unpredictability makes it difficult for applications to rely on the availability of a cloud service when resources are frequently oversubscribed.
3.1.2
Type II - Partial allocation with blocking
Type II scheduling is a partial cloud approach that provides an option for a complete or partial allocation. A partial allocation occurs when a fraction of the request is immediately satisfied, and the remainder of the request is denied (blocked) if resources are not available. Type II scheduling differs from Type I scheduling in that it provides a partial allocation option to the all-or-nothing allocation policy. The advantage of this scheduling approach is that requests that can use a mixture of immediate and deferred access to resources can gain partial access rather than being completely blocked. Applications that can adapt to a partial resource allocation can make
effective use of this type of scheduling. An example of this type of application is adaptive MapReduce [11], which can scale the number of processors needed during the splitting, mapping, and reduction phase in which the problem is partitioned into subproblems. Other types of applications that can utilize this type of scheduling are adaptive mesh refinement, on-demand Saas/IaaS systems, and applications that run within a science gateway. The drawback of Type II scheduling is that resource requests can be denied when all resources are busy. This drawback makes it difficult to rely on a service when the cloud system is overloaded and resources are not reliably available.
3.1.3
Type III - Partial allocation with waiting
The Type I and Type II scheduling approaches either provide immediate access to resources in response to a request, or deny access when there are no available resources. Applications that could otherwise wait for resources in a queue would not benefit from these types of scheduling systems. Type III scheduling is a partial cloud approach with the option to wait. Requests are granted a complete or partial allocation, queueing the unsatisfied fraction of requests for resource availability if complete allocation request cannot be immediately satisfied. Type III schedulers build on Type I and Type II by providing the option for resource requests to wait in a queue. The Type III scheduling policy has the advantages of Type II scheduling (full or partial resource allocation) with the additional benefit of providing the option to wait for some or all of the requested resources. Type III is a fractional scheduling approach, in which a request specifies an immediate need of X% of resources, and can tolerate waiting for the remaining fraction of resources. An example of a type of application that could effectively use a Type III approach are scientific workflow systems that use a mix of single threaded and parallel executables as constituent parts of the workflow. The ability to allocate additional processors to the workflow as work proceeds and resources become available is another benefit of this type of scheduling. The Type III scheduling policy is very flexible with few drawbacks. The primary downside of this policy is the additional computational complexity involved in managing the queue and scheduling systems, the need for users to specify the fraction of resource immediately needed, and the additional steps needed to use additional processors as they become available.
3.1.4
Type IV - Queue based allocation/scheduling
The other end of the scheduling spectrum is the Type IV scheduler, which is the type used today for high performance computing systems. The advantages of this scheduling approach is that is a well-tested and familiar scheduling model for users and is useful for parallel applications that can wait for access to resources. The downside of this approach, especially for cloud computing systems, is that it is not useful for applications that need consistent "instant on" access, since queue wait times can range from seconds to days.
4.
ESTIMATING RESOURCE NEEDS FOR EACH SCHEDULING TYPE
Cloud resource providers seeking to support a range of scheduling types need to be able to predict the amount of computational resources needed in the cloud computing system to satisfy a stream of resource requests. In this section, we present an analytical model to estimate the number of virtual machines needed in the cloud computing system to satisfy peak demand, to calculate the probability of blocking a resource request, and to estimate the average waiting time in a queue for resources. In this model, a request for n
computational nodes is satisfied by providing access to n uniprocessor virtual machines. Our analysis is based on Erlang stochastic models, which can be used to compute the probability of blocking resource requests during a peak busy period based on the historic offered workload [12].
4.1
Type I Scheduler
In the Type I scheduling model, resource requests are immediately and fully satisfied or are completely blocked. To satisfy an offered workload, cloud resource providers must provision a sufficient number of idle virtual machine resources during the peak busy period or must deny requests and thus lose potential revenue. To estimate the probability of blocking a resource request during a busy period, we use a generalized Erlang loss station model [13] [14] [15]. Consider a virtual cluster (running on a cloud computing service) that can support a total of C virtual machines. Each virtual machine can be assigned independently to service a request. The workload offered to the virtual cluster is a series of resource requests from a community of users from 1 to a maximum of C computational nodes. We assume that the number of processors requested in typical job workloads follow a pattern of powers-of2 observed on supercomputing systems [16]. We can partition the workload into powers-of-2 bins based on the number of processors p requested, specifically bin b = log2 (p) + 1. Each bin contains from 2b to (2b+1 − 1) processors, thus the base number of processors in each bin is: 1, 2, 4, 8, 16, 32, 64, 128, etc. This partitions the system into resource classes, distinguished by the number of computational nodes in each bin, with each partition corresponding to a maximum number of nodes in a bin. The number of servers in each resource class i can be described as a row vector − → b = (b1 , b2 , b3 , . . . , bk )
(1)
where bi represents the virtual cluster nodes in each resource class i with a base number of processors 2i−1 . If the system contains a maximum of C processors, k = log2 (C) for C > 1. At any given point in time, the residency of computational jobs running within each resource class can be described by a row vector − → n , where − → n = (n1 , n2 , n3 , . . . , nk )
(2)
where ni represents the number of jobs using resource class i. There are a total of C virtual cluster nodes running in the cloud computing service. The total number of resource units occupied is the vector dot product − → → j=− n · b j is a random variable that is limited to the total number of processors C available in the system k
− → X → j=− n · b = ni bi 6 C i=1
Resource requests offered to the system seek p virtual cluster nodes or processors. These requests can be partitioned by the number of requested processors into resource class bi where i = log2 (p)+ 1. Requests submitted by users arrive independently following a Poisson process, and thus arrive at a partition independently with an exponentially distributed arrival rate λi . The mean service time to run a computation once a request has been granted is µ1i .
The offered workload to the system for partition class i is thus λi µi where ai is in units of Erlangs, λi is the average rate of job arrivals of class i and µ1i is the mean service time of job of class i. In this system, the computational nodes are completely utilized and shared by the workload [15]. This represents a policy in which → the universe of all possible job allocations − n must satisfy the inequality
then
− → → 06− n · b 6C
(3)
The stochastic distribution of the dot product j over the universe of allocations and resources is described in Kaufman [15] as X
q(j) =
→ − − − {→ n :→ n · b =j}
k i Y an 1 i · n G(C, k) i! i=1
( k ) X ai bi qˆ(j − bi ) C i=1
1 qˆ(j) = j
ai =
(8)
where qˆ(0) = 1, q(j) = q(0)ˆ q (j) and
q(0) =
( C X
)−1 qˆ(j)
(9)
j=0
To compute the probability of blocking of a resource request for class i, which we define as Pbi , let P (·) denote the state distribution with sharing policy Ω [15]. Then X Pbi = P (n) (10) n∈Bi+
for i = 1, . . . , k where
where Bi+ = n ∈ Ω : n+ i 6∈ Ω k i X Y an i n i! i=1
G(C, k) =
Therefore, for a complete sharing policy [14]
P olicies
The distribution of the number of resource units q(·) occupied for a complete sharing policy satisfies the equation k X
X
P bi =
q(n)
(11)
→ − − {n:→ n · b >(C−bi )}
q(n) can be computed using the recurrence relation ai bi q(j − bi ) = jq(j)
(4)
i=1
q(n) =
for j = 0, 1, . . . , C and q(x) = 0 for x < 0 and C X
k X ai bi q(n − bi ) nC i=1
(12)
Since q(j) = 1
j=0
PC
One of the known problems of the Erlang computation is that the computed value in Erlangs can grow unmanageably large and cause computational overflow when the workload is very high. Hubner [17] describes an approach to normalize the offered traffic. Without normalization, the offered traffic Ai is defined as:
C X
q(n) = 1
n=0
we can compute the probability of blocking a resource request for class i using the equation
The normalized offered traffic αi can then be computed ai bi C Modifying Equation 4 using this approach we obtain
(13)
and
λi Ai = bi = ai bi µi
αi =
n=C−bi +1 q(n) PC n=0 q(n)
P bi =
(5) bi −1
P bi =
X
q(C − i)
(14)
i=0 k X ai bi q(j − bi ) = jq(j) C i
(6)
for j = 0, 1, . . . , C and under a complete sharing policy [15] C X
q(j) = 1
j=0
If we define
Equation 14 can be easily computed, and is linear in the number of resource classes k. This computation is much more feasible than − → → the direct computation of all combinations of b and − n , which are exponential in the number of resource classes k. The approach described above can address the case of single node requests satisfied by a uniform pool of resources. In the case of virtual clusters, however, the problem is to estimate the probability of blocking an individual request for R nodes.
4.2 q(j) qˆ(j) = q(0)
(7)
Type II scheduler
Type II scheduling systems can provide a partial allocation of resources in response to a user request. Users willing to accept
a partial allocation provide a first choice allocation request to the scheduling system, which can be completely satisfied if access to all of the requested nodes is granted. If a fraction of the first choice can be satisfied by the system, it will grant a partial allocation, and consequently block access to the remaining fraction of the request with no option of queueing. The model for the Type I scheduler can be extended for a partial allocation by using the probability of blocking the first choice request as a predictor of converting the request into a partial allocation that falls into the resource class for the partial allocation. The offered workload for a resource class i is described by Equation 5 and represents the normalized average number of jobs αi in − → Erlangs during the peak busy period. As a reminder, b is defined as the number of servers in each resource class i, which is described as a row vector − → b = (b1 , b2 , b3 , . . . , bk ) (15) − → If we order the resource vector b in decreasing order by the number of processors in each partition, then the resulting sorted − → resource vector b can be defined as − → c = (c1 , c2 , . . . , ck ) where ci > ci+1 for i ∈ [1..k]. We define an individual resource request from a user with first, second, third, etc. up to mth priority resource requests as − → L = (l1 , l2 , . . . , lm ) where li represents the number of processors requested for the i-th choice, and m represents the maximum number of choices for the request. Each choice li falls into a resource class − → cj ∈ C when cj+1 5 li < cj − → When a resource request for li ∈ L processors is blocked with probability Pbi for a class i, then the resource request will need to be satisfied by the next smallest resource class u > i in which li+1 can fit. The probability that a blocked resource request will seek resources for the next best choice is Pbi . Thus, the average peak workload for the next best choice will be increased by the additional workload from resource requests that are shunted to it. Since the base average peak workload (in terms of the number of requests) for resource class u is αu , the additional number of requests from shunted resource requests from the large resource class will be the fraction of requests that are blocked, which is Pbi αu . The reduced pressure on the larger resource class i is then βi = αi (1 − Pbi )
Qbi = T (n) =
k X β i bi T (n − bi ) n i=1
(17)
for u ∈ [1, m] where Pbi is the probability of blocking a request to class i calculated using the Erlang-B approach described by Equation 14 and αi is computed from Equation 5.
(18)
As in the case of Equation 14, Equation 18 can be easily computed iteratively to calculate the probability of blocking a resource request. We next present the model for the Type IV scheduler, followed by the model for the Type III scheduler that builds on the Type IV model.
4.3
Type IV scheduler
Type IV schedulers are similar to scheduling systems used in HPC systems today, where resource requests that cannot be immediately satisfied must wait in a queue for resources to become available. We can extend the model for the Type I scheduler to predict the probability of immediately blocking a resource request and the average queue wait time. The model derived in this section estimates the blocking probability and queue wait time for an offered workload for a Type IV (all-or-nothing with a queue) scheduling policy. If a job request arrives when all servers are busy, then the request is queued and waits until a server of the appropriate service class becomes available. The job requests follow a Poisson arrival process and exponentially distributed service times. This queueing system is also called the Erlang delay system M/M/n with Poisson arrival process (M), exponential service times (M), n servers and an infinite number of waiting positions. In this case the offered traffic equals the carried traffic as no requests are blocked. For the system we can calculate the probability of a positive waiting time, mean queue lengths, and mean waiting times and carried traffic per resource class. Since resource requests arrive as a Poisson process independent of the state of the system, by the PASTA property (Poisson Arrivals See Time Averages), the probability that an arbitrary arriving job has to wait in the queue is equal to the proportion of time all servers are occupied. For an arbitrary arriving job request we are interested in the probability that a request will be queued, and the average length of time the request will wait in the queue. If we define the waiting time W as a stochastic variable, the probability that a request will be queued is PC = p{W > 0} To compute PC for n servers with offered traffic intensity a, Zeng [19] derived an expression for PC
(16)
and a fraction of the average peak workload of the resource class u, will be shunted down to the next smaller resource class. Since the resource requests arise from a Poisson process, the request can be included [18] as βu = αu (1 − Pbu ) + Pbi αi
Equation 17 can be solved iteratively by starting with the largest resource class size and "flowing down" the resulting second, third, and successive choices to smaller resource classes. The resulting set of adjusted β values from Equation 17 for the resource classes can then be used with Equation 14 to compute the adjusted probability of blocking requests for each resource class i
PC(n,a) =
1+
a 1!
+
a2 2!
+
an n n! n−a an−1 · · · + (n−1)!
+
an n n! n−a
(19)
Statistical equilibrium is only obtained for a < n. Otherwise, the queue will continue to increase towards infinity. The Erlang B formula for a system with offered traffic a Erlangs and n servers is PB(n,a) =
an n!
1+
a 1!
+
a2 2!
+ ··· +
an−1 (n−1)!
+
an n!
(20)
Erlang’s C-formula is similar to Erlang’s B-formula except for the factor n n−a in the last term. We can modify Equation 14 to compute Erlang-C using the relationship for obtaining numerical values of the Erlang C-formula described by Zeng [19]: PC(n,a) =
1−
PB(n,a) a ( n )(1 − PB(n,a) )
The queue length is a random variable F. The probability of having customers in queue at a random point of time is then a p{F > 0} = PC(n,a) n with average queue length a Fn = PC(n,a) (21) n−a The mean waiting time W experienced by all the job requests in the system is an indication of the average time cloud resource requests must wait in a queue for resource, and thus the responsiveness of the system. By Little’s Law, the average queue length is the product of the arrival intensity and the mean waiting time Fn λ using the mean waiting time Wn for queue n, Wn =
a 1 Wn = PC(n,a) . λ n−a Since
The quantities of interest for a Type III scheduler from a cloud computing perspective are the probability of a resource request being queued rather than immediately satisfied, and the average wait time in the queue if the request is queued. The probability of blocking the 1st choice request l1 can be computed using Equation 14. To compute the probability of a next best choice being queued, we can modify the derivation presented for the Type IV scheduler. When the 1st choice request is blocked and a next best choice is considered, the number of jobs shunted is the product of the average number of resource requests for the 1st choice resource class and the probability of blocking a request for that class. The reduced pressure on the larger resource class i is then βi = αi (1 − Pbi )
(23)
and the resulting number of jobs shunted from resource class i and submitted to the next best class u is γu = αu (1 − Pbu ) + Pbi αi
(24)
for u ∈ [1, m] where Pbi is the probability of blocking a request to class i calculated using the Erlang-B equation Equation 14 and αi is computed using Equation 5. The probability of a shunted request queueing in the next best resource class is based on the effects of the aggregate shunted workload on the resource class. The probability of blocking a request (Erlang-B) is then
Vbi = Z(n) =
k X γ i bi Z(n − bi ) n i=1
(25)
We can use this updated probability of blocking requests to compute the probability of a shunted request queueing (Erlang-C)
a=λ·s where s is the mean service time [20]. The mean service time for each resource class i is then s (22) n−a Using Equation 22, we can compute the probability of waiting and the average amount of time a job will need to wait for access to a cloud computing service for each resource partition. Wn = PC(n,a)
4.4
Type III scheduler
Type III schedulers are a variation of Type II schedulers in which the blocked fraction of the request can be directed to wait in a queue for available resources, rather than simply blocked. As in the case of Type II schedulers, a request for resources is comprised of a ordered list of 1st choice, 2nd choice, etc. number of processors. Specifically, we define an individual resource request from a user with first, second, third, etc. up to mth priority resource requests as − → L = (l1 , l2 , . . . , lm ) where li represents the number of processors requested for the i-th choice, and m represents the maximum number of choices for the request. Each choice li falls into a resource class − → cj ∈ C when cj+1 6 li < cj
Vbi (26) 1 − na (1 − Vbi ) The queue length is a random variable G. The probability that the queue length is non-zero is PIV =
a PIV n The average queue waiting time experienced by queued job requests can be computing using Little’s Law. The average queue length is p{G > 0} =
Gn = PIV
a n−a
and thus Gn λ To compute the average queue waiting time for resource class i, we can assess the impact of queueing the blocked portion of requests into a resource class in which blocked portion can fit. Wn =
1 a PIV λ n−a Thus, the probability of queueing a shunted partial resource request and the average waiting time for the resource request can be computed. This will allow a cloud computing provider to estimate the amount of resources they need to provision to provide an infrastructure to support immediate access to fractions of resource requests with queueing for the shunted fraction. Wn =
ASSESSING SCHEDULING POLICIES FOR A LARGE-SCALE HPC SYSTEM
To assess the impact of the scheduling polices, we measured the offered HPC workload from scheduling logs from a large supercomputing system (Steele) at Purdue University, which is a 7,216 core cluster operated by the Rosen Center for Advanced Computing (RCAC) and a component of Purdue’s Community Cluster Program. Steele consists of five logical sub-clusters, each with a different combination of memory and interconnect. Steele-A nodes have 16 GB RAM and Gigabit Ethernet; Steele-B, 16 GB RAM and InfiniBand/Gigabit Ethernet; Steele-C, 32 GB RAM and Gigabit Ethernet; Steele-D, 32 GB RAM and InfiniBand/Gigabit Ethernet; Steele-E, 32 GB RAM and Gigabit Ethernet. Steele nodes run Red Hat Enterprise Linux 5.5 (RHEL5.5) and use Portable Batch System Professional 10.4 (PBSPro 10.4) for resource and job management. Steele also runs jobs for BoilerGrid [21] whenever processor cores in it would otherwise be idle. The logs we analyzed contain 1,355,851 job submissions from 10th April 2008 to 3rd August 2009. The log records contain the following fields: jobid, time at which the job is queued, time at which the job starts execution, time at which the execution completes, the event due to which the job is stopped and the count of the node and the core requested.
5.1
Measuring the offered HPC workload
To use Erlang based stochastic models to predict the probability of blocking a job request, we first needed to determine the peak busy scheduling period for each day. This approach is motivated by the need to ensure that sufficient resources are available to meet demand at all times. The peak period of demand, the busy period, is the time of maximum resource request pressure on the system. If the system can predictably meet demand during this period, the probability of blocking a request will be the worst case blocking behavior of the system. In a communications system, the sliding 60-minute period during which the maximum total traffic load occurs in a given 24-hour period is the busy hour. The traffic during this period is the busy hour traffic. The busy hour traffic is the average, over several days, of the average number of concurrent calls during the same one-hour period each day, where that period is selected to give the highest result or the busy hour traffic values can be separately calculated for each day and the average of these values can be taken. However there is an inherent difficulty in applying the concept of busy hour traffic to scheduler logs for the cluster. While calls in the communication system rarely last for an hour, most of the computational jobs execute for large time spans and can even run for days. Hence in the calculation of the busy hour traffic we consider a window of 4 hour duration. There is another problem in the calculation of busy hour traffic. While jobs that span multiple windows contribute to the traffic in each window, their service times overlap among the windows considered, thus their service time is counted multiple times leading to superficial and large value for the busy hour traffic. To solve this problem we consider the windows as intervals. The busiest interval will then be the one in which there is a maximum point of overlap. We used an interval tree to efficiently find all intervals that overlap with any given interval or point. We constructed the interval tree by designating the endpoints a and b of each interval [a, b] to be the start time and stop time for each job. For each day the four hour window with the maximum overlapping intervals was marked, and the service times for the jobs in this window were recorded. The busy hour traffic was found by taking an average of all the jobs’ service times that contributed to the busy hour.
Table 1: Stochastic workload parameters derived from scheduling logs for the Steele supercomputer. Resource Node Arrival Rate Service Time Erlangs Class Count λ(jobs/h) 1/µ (h) α=λ/µ 1 1 61.36 56.28 3453.60 2 2 15.56 9.17 142.76 3 4 14.73 9.48 139.58 4 8 18.12 44.04 798.00 5 16 4.44 5.01 22.26 6 32 5.47 5.36 29.36 7 64 7.75 15.68 121.56 8 128 3.26 11.30 36.85 9 256 2.95 8.23 24.28 10 512 1.39 2.00 2.78 11 1024 0.28 1.72 0.47 12 2048 .09 .27 .02 13 4096 .10 .44 .04 1
Probability of blocking request
5.
0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 16
32
64
128
256
512
1024
2048
Cloud Computing System Size (number of nodes) 1 node class 32 node class
2 node class 64 node class
4 node class 128 node class
8 node class 256 node class
16 node class 512 node class
Figure 1: Probability of blocking a resource request for a Type I scheduler. To model the workload, we computed the busy hour traffic (in Erlangs) using the average arrival rate and the average service time for each of the resource classes. The resulting resource classes, arrival rate, service time, and computed Erlang values from the scheduling logs are shown in Table 1. To assess the effects of each scheduling policy on the probability of blocking requests, we assessed each scheduling type by varying the total number of virtual machines (nodes) available for use in the cloud computing system. We measured the probability of blocking a resource request for the Type I and Type II schedulers. For Type III and IV schedulers, in which a requests is queued rather than immediately rejected, we measured the probability of queueing a request along with the queue wait time for requests that were queued. The next four sections presents the results of these measurements. In each of these assessments, we used the offered workload derived from the Steele scheduling logs to derive the workload α for each resource class.
5.2
Type I Scheduler
We assessed the impact of system size on the probability of blocking resource requests for each resource class when the system was managed using a Type I scheduling policy. Figure 1 shows the resulting probability of blocking requests. In this figure, the X-axis represents the total number of nodes in the cloud computing system, and the Y-axis represents the probability of blocking a resource request. Each category described in the legend of this figure is the individual resource class. From the
0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 16
32
64
128
256
512
1024
2048
Cloud Computing System Size (number of nodes) 1 node class 32 node class
2 node class 64 node class
4 node class 128 node class
8 node class 256 node class
results shown in this figure, it is clear that as the size of the cloud computing system grows, more resource requests can be satisfied and consequently are not blocked. The probability of blocking for a given cloud computing system size depends on the size of the resource class. Smaller classes are easier to schedule, and larger classes (such as the 256 and 512 node class) are difficult to schedule on a busy system, and are more likely to be blocked. Cloud computing resource providers can use the information contained in Figure 1 as dimensioning curves to determine the number of virtual machines they must provision within the cloud computing system, and the effects of provisioning on blocking requests from different resource classes.
2 1.5 1 0.5 0 0
Type III and IV Schedulers
The Type IV scheduler is the queue based model used for high performance computing systems. The Type III scheduler is similar to the Type IV, but differs in that requests that would otherwise be queued are shunted to smaller resource classes to attempt to provide a partial allocation for the request. For Types III and IV, we assessed the average queue wait time and the probability of queueing a request for the base number of processors 2i−1 is each resource class i. Figure 3 shows the average queue wait time for Type III and IV schedulers for each resource class. This figure shows that there is a slight advantage to using a Type III scheduler, which results in a reduction in the average queue wait time for a request. In our analysis for these types of schedulers, when the total number of free cloud computing nodes were small, requests for the large resource class were frequently queued. Type I and II schedulers allowed us to predict the probability that a request would be immediately serviced of blocked. For the Type IV scheduler, we were in-
1
Type IV
2
3
Type III
4
5
6
7
8
9
10
11
12
Resource Class
Figure 3: Average queue wait time for Type III and Type IV schedulers. 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 01 0.1 0 0
5
10
15
20
25
30
35
Percentage increase in the number of virtual machines above base resource class size Class 1 Class 2 Class 3 Class 4 Class 5 Class 6 Class 7 Class 8 Class 9
Type II Scheduler
Results for the Type II scheduler are shown in Figure 2. The policy for a type II scheduler shunts blocked requests to smaller resource classes to attempt to provide a partial allocation to requests. This results from this scheduling policy are similar to the results for the Type I scheduler. To model the offered workload for this policy, we assumed that every request specified the next smallest resource class as an acceptable next choice. The impact of the ability to shunt requests resulted in a slight slight increase in the probability of blocking requests for a resource class compared with the Type I scheduler. We investigated this, and found that the increase was due to the additional workload arriving at a resource class originating from shunted resource requests from larger resource classes. Shunting allowed the cloud computing system to grant resources to an average of 11.3% more jobs than the Type I policy.
5.4
2.5
16 node class 512 node class
Figure 2: Probability of blocking a resource request for a Type II scheduler.
5.3
Average queue wait time across e queue wait time across source classes (seconds) all resource classes (seconds)
0.8
Probability of Queueing
Probability of blocking request
1 0.9
Figure 4: Effects on increasing the number of processors beyond (2i−1 ) in resource class i on the probability of queueing a request for a Type IV scheduler. terested in understanding how the probability of queueing a request was affected as the number of virtual machines in the resource class were increased. Figure 4 and 5 shows the effects on increasing the number of virtual machine resources above the base bin resource size on the probability of queueing and queue wait time. These figures show that a modest increase - 0 to 10% - in the size of the resource class can have a profound effect on reducing the probability of forcing a request to be queued and the average queue wait time for requests that must be queued.
6.
VIRTUAL CLUSTER RELIABILITY
The results in the previous sections address the question of modeling the availability of a cloud computing service offered a workload. Once an application or user is granted access to a cloud computing service, the next problem that arises is the inherent reliability of the service provided. If one of the nodes fail during the allocated time, the application will lose service, and may potentially terminate. The problem addressed in this section is: given a cloud computing partition of size M available over a time period [0, T ], what is the probability of a failure over that time? Additionally, if a failure occurs, what strategies could be used to recover from the failure and provide the required number of functional nodes for the entire allocation time? To measure the operational mean time to failure of a large cluster in the field, we gathered the kernel message logs from a a large
Queue wait time (seconds)
14 12 10 8 6 4 2 0 0
5
10
15
20
25
30
35
% increase in the number of virtual machines above base resource class size Class 1 Class 6
Class 2 Class 7
Class 3 Class 8
Class 4 Class 9
Class 5 Class 10
Figure 5: Effects on increasing the number of processors beyond (2i−1 ) in resource class i on average queue wait time for a Type IV scheduler. commodity cluster at Purdue University from April 13 to December 9, 2010. We analyzed the logs to find the time and node address of each system as it booted. Similar to prior results by Hacker [22] for BlueGene systems, we found significant temporal and spatial clustering in the data, which we found were due to events (such as system wide reboots) that affected several nodes in the cluster simultaneously. To decluster this information, we used the R cclust cluster analysis package to reduce the number of dependent events. We computed the average time between reboot clusters to be 1258 hours, or 52.4 days. The system partition consisting of a set of virtual machines allocated by the cloud computing system for a virtual cluster can experience one or more failures of a virtual machine during the lifetime of the allocation. A cluster computing allocation can be hosted on N virtual machines on a cloud computing system with a overall capacity of C virtual machines. If we assume that the MTTF of the virtual machine matches the MTTF of the physical node on which the virtual machine runs, we can compute the MTTF of the virtual cluster partition of size N of the C node cloud computing system. Using the MTTF of 1258 hours, the MTTF for the virtual cluster can be calculated. As described in Hacker [23], and Nurmi, Brevik, and Wolski [24, 25, 26], the observed time between failures of computer systems operating in the field do not follow an exponential distribution, but instead follows a Weibull distribution with scale parameter τ and shape parameter β. The probability of the failure of a virtual machine in the time interval [0, ∆t] for the Weibull distribution of scale τ and shape β is [27] Z ∆t β−1 β β t e−(t/τ ) dt F (∆t, β, τ ) = τ τ 0 =
β
1 − e−(∆t/τ ) .
(27)
The probability that a virtual machine node in the cluster computing system will not fail in this time is R (∆t, β, τ )
=
1 − F (∆t, β, τ ) β
= e−(∆t/τ ) .
(28)
The mean of the Weibull distribution is given by τ Γ (1 + 1/β), where Γ is the Euler Gamma function. Assuming that the virtual machine MTTF corresponds to the mean τ is τ =
MTTF . Γ 1 + β1
(29)
Table 2: Reliability of cloud computing partitions Resource Node 1/µ Reliability Class Count (hours) 1 1 56.28 0.912 2 2 9.17 0.958 3 4 9.48 0.915 4 8 44.40 0.546 5 16 5.10 0.808 6 32 5.36 0.638 7 64 15.68 0.120 8 128 11.3 0.038 9 256 8.23 0.006
Assuming that all nodes are identical, i.e., characterized by the same τnode and βVM [28] we find that the reliability of the cluster computed from the reliability characteristics of the M node virtual cluster built from a set of virtual machines V M is Rcluster (∆t) = exp −M
∆t τVM
βVM ! .
(30)
We can compute the Weibull reliability function for an N node virtual cluster by assuming that the reliability function for the virtual cluster also follows a Weibull distribution with scale parameter τcluster and shape parameter βcluster . Using βcluster = βVM = β, we derive 1 τVM N 1/β and the MTTF of the virtual cluster is: τcluster =
(31)
1 τ Γ 1 + VM β N 1/β 1 = MTTFVM (32) N 1/β Using Equation 30 with an estimated Weibull shape parameter β = 0.8 and a mean time to failure M T T FV M = 1258 hours, we can compute the probability of a virtual machine failure within each virtual cluster over a time [0, T ]. Table 2 shows the computed reliability for the resource classes described in previous sections, which consist of partitions of the cloud computing system running as virtual clusters. The average runtime 1/µ for jobs running in the virtual cluster are calculated from the offered workload to the Steele supercomputer described in Table 1. It is clear from this table that as the size of the virtual cluster grows, the reliability of the virtual cluster declines, even with a reduced runtime. This poor reliability is due in large part to a short mean time to failure, which in practice is caused by the combination of hardware failures, software failures, and the need to occasionally reboot the physical hardware nodes for operational activities. To improve the reliability of a virtual cluster partition, the cloud computing system could set aside a shared pool of physical hardware nodes to be used as failover nodes. The failover nodes could be used as a hot-swap node or a target platform for the live migration of virtual machines running a parallel MPI application from a failing node [29] in the event of the predicted failure of a physical node. The reliability model for this approach is a k-out-of-N system [30]. In this system, at least k out of N nodes must remain functional during the time [0, T ] for the entire system to be reliable. If the virtual cluster size is k, and (N − k) failover nodes are availMTTFcluster
=
1
Reliability Over Job Runtime
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Number of Spare Virtual Machines 8.23 Hours
20 Hours
Figure 6: Reliability of 260 node virtual cluster over 8.23 and 20 hour job lifetimes with additional hot spare nodes.
able for use when one of the k virtual machines fails, the resulting reliability of the N node partition can be computed. The reliability function for the virtual cluster is then
R(∆t)k/N =
N −k X i=0
! N RN −i (∆t) (1 − R(∆t))i i
(33)
Using Equation 33, we can compute the probability of a virtual machine failure within an k node cloud computing partition if (N − k) spare nodes are available to take the place of a failed node. We assessed the reliability of a 260 node virtual cluster using the average job arrival rate and runtime of 8.23 hours for a 256 node resource class described in Table 1, and determined the reliability effects of providing from 1 to 20 spare virtual machine nodes. Figure 6 shows the resulting reliability for the virtual cluster system. From this figure, it is clear that adding only a few nodes only 4% additional nodes - had a significant impact on improving the reliability of the virtual cluster. We also assessed the reliability impact of increasing the job runtime from 8.23 to 20 hours, and found that 20 spare virtual machines - around an 8% addition - was sufficient to provide good reliability when more than doubling the job runtime. There are two possible approaches for allocating spare virtual machines to virtual clusters for failover. One approach is to dedicate a partition of a sufficient number of spare virtual machines to achieve a desired level of reliability for each individual virtual cluster, which cannot be accessed or used by other virtual clusters. The advantage of this approach is that the virtual cluster has guaranteed access to spare virtual machine slots in the event of a node failure. The downside, however, is that reserving dedicated spare virtual machine slots for each virtual cluster can result in a large total number of resources. Another approach is to provide a common central pool of spare resources from which any virtual cluster can draw resources in the event of failure.
7.
RELATED WORK
There have been several recent related efforts in the area of cloud computing and virtual cluster scheduling. Sotomayor [31] explored the use of resource leases for advanced reservation leases within the Haizea scheduling component of Open Nebula. The types of leases provided in Haizea include advanced reservation (scheduling in the future, similar to our Type IV), best-effort (immediate scheduling with queueing if necessary, similar to our Type III); and immediate leases (all or nothing scheduling, similar to our Type I). The models
we developed for the scheduling types described previously can be used in conjunction with Haizea to allow sites to predict the resources needed for a mix of the scheduling policies described for Haizea. Another related work is OpenPEX [32] by Venugopal, which uses advanced reservations as the primary method of scheduling resources for VM instances. In the OpenPEX system, users submit a resource allocation request to the scheduler through a web interface, specifying the size, desired future start and end time, the necessary OS template, and the number of VM instances. The OpenPEX approach is similar to our Type IV scheduler, in which resource requests wait in a queue for the future availability of nodes. The models we developed for the Type IV scheduler could be used within OpenPEX to predict the waiting time for resources. Eucalyptus [33] employs an all-or-nothing immediate allocation approach similar to our Type I scheduler. It calculates the number of simultaneous instances of specific characteristics described in the users’ request, and schedules VM execution if the nodes are not busy. The resource allocation uses static greedy and round robin scheduling, and could use our prediction models to determine the size of the cloud computing system needed to provide service for an offered workload. de Assunção [34] assessed naive, shortest queue, weighted queue, selective, aggressive, and conservative deadline job scheduling strategies An aspect of de Assunção’s work that is similar to our work is that the authors assessed the rate of rejected jobs and associated costs with meeting varying workloads and deadline constraints. The derivation described in our paper could be used to predict the rejection rate and costs of the scheduling strategies explored by de Assunção. Other prior and related work focuses on: sizing the number of pool accounts for grid computing systems [35]; Nurmi [36] investigated the Binomial Method Batch-Queue Predictor named to predict job wait times; and Karnak [37] uses Instance Based Learning to predict queue wait times for the TeraGrid.
8.
CONCLUSIONS
The availability of stable cloud computing systems and growing need for operating system customization for parallel applications is driving the development of virtual clusters. The integration of these technologies is complicated by the fundamentally different resource allocation policies inherent to each technology. In this paper, we presented a continuum of four scheduling polices and a resource prediction model that: 1) allow cloud computing providers to estimate the resources needed within a cloud computing system to satisfy a workload and to predict the probability of blocking requests; 2) estimate the queue wait time for requests that can wait for a partial or full allocation; and 3) estimate the size of the pool of spare resources needed to ensure good reliability for virtual clusters. These models can be integrated into existing and new cloud computing scheduling systems, and used by resource providers to operate an efficient and cost-effective cloud computing system for virtual clusters.
9.
ACKNOWLEDGMENTS
The work described in the paper is supported by the National Science Foundation NEES Program CMMI-0927178, NSF CAREER CCF-0952960, and OCI-0749140.
10.
REFERENCES
[1] H. Meuer, E. Strohmaier, J. Dongarra, and H. Simon, “Top500 Supercomputer Sites,” The report can be
downloaded from http://www.top500.org/. [2] R. Henderson, “Job scheduling under the portable batch system,” in Job Scheduling Strategies for Parallel Processing. Springer, 1995, pp. 279–294. [3] M. Litzkow, M. Livny, and M. Mutka, “Condor-a hunter of idle workstations,” in Distributed Computing Systems, 1988., 8th International Conference on. IEEE, 2002, pp. 104–111. [4] S. Kannan, M. Roberts, P. Mayes, D. Brelsford, and J. Skovira, “Workload management with loadleveler,” IBM Redbooks, vol. 2, p. 2, 2001. [5] B. Sotomayor, R. S. Montero, I. M. Llorente, and I. Foster, “Virtual infrastructure management in private and hybrid clouds,” IEEE Internet Computing, vol. 13, pp. 14–22, 2009. [6] R. Eigenmann, T. Hacker, and E. Rathje, “Nees cyberinfrastructure: A foundation for innovative research and education,” in Proceedings of the 9th US / 10th Canadian Conference on Earthquake Engineering, 2010. [7] T. Hacker, R. Eigenmann, S. Bagchi, A. Irfanoglu, S. Pujol, A. Catlin, and E. Rathje, “The neeshub cyberinfrastructure for earthquake engineering,” Computing in Science & Engineering, vol. 13, no. 4, pp. 67–78, 2011. [8] T. Hacker, “Toward a Reliable Cloud Computing Service,” Cloud Computing and Software Services: Theory and Techniques, p. 139, 2010. [9] M. McLennan and R. Kennell, “HUBzero: A Platform for Dissemination and Collaboration in Computational Science and Engineering,” Computing in Science & Engineering, vol. 12, no. 2, pp. 48–53, 2010. [10] N. Wilkins-Diehr, “Special issue: Science gateways Common community interfaces to grid resources,” Concurrency and Computation: Practice and Experience, vol. 19, no. 6, pp. 743–749, 2007. [11] F. Zhang, J. Cao, X. Song, H. Cai, and C. Wu, “AMREF: An Adaptive MapReduce Framework for Real Time Applications,” in 2010 Ninth International Conference on Grid and Cloud Computing. IEEE, 2010, pp. 157–162. [12] T. Hacker and B. Athey, “A methodology for account management in grid computing environments,” in Grid Computing GRID 2001, ser. Lecture Notes in Computer Science, C. Lee, Ed. Springer Berlin / Heidelberg, 2001, vol. 2242, pp. 133–144. [13] H. Kobayashi and B. Mark, System Modeling and Analysis: Foundations of System Performance Evaluation. Pearson Education, 2009. [14] T. Bonald, “Insensitive traffic models for communication networks,” Discrete Event Dynamic Systems, vol. 17, no. 3, pp. 405–421, 2007. [15] J. S. Kaufman, “Blocking in a shared resource environment,” IEEE trans. on commun., vol. COM-29, 10, pp. 1474–1481, 1981. [16] H. Li, D. Groep, and L. Walters, “Workload characteristics of a multi-cluster supercomputer,” in Job Scheduling Strategies for Parallel Processing, D. G. Feitelson, L. Rudolph, and U. Schwiegelshohn, Eds. Springer-Verlag, 2004, pp. 176–193, lect. Notes Comput. Sci. vol. 3277. [17] F. Hubner and P. Tran-Gia, “An analysis of multi-service systems with trunk reservation mechanisms,” 1992. [18] S. M. Ross, Introduction to Probability Models, 4th ed. Academic Press, 1989. [19] G. Zeng, “Two common properties of the erlang-B function, erlang-C function, and Engset blocking function,”
[20] [21]
[22]
[23]
[24]
[25]
[26]
[27]
[28]
[29]
[30] [31]
[32]
[33]
[34]
Mathematical and Computer Modelling, vol. 37, no. 12-13, pp. 1287–1296, 2003. V. Iverson, “Teletraffic engineering and network planning, Technical University of Denmark, Revised January 2007.” P. Smith, T. Hacker, and C. Song, “Implementing an industrial-strength academic cyberinfrastructure at purdue university,” in Parallel and Distributed Processing, 2008. IPDPS 2008. IEEE International Symposium on. IEEE, 2008, pp. 1–7. T. Hacker, F. Romero, and C. Carothers, “An analysis of clustered failures on large supercomputing systems,” Journal of Parallel and Distributed Computing, vol. 69, no. 7, pp. 652–665, 2009. T. J. Hacker and Z. Meglicki, “Using queue structures to improve job reliability,” in Proceedings of the 16th International Symposium on High-Performance Distributed Computing (HPDC-16 2007), 25-29 June 2007, Monterey, California, USA. ACM, 2007, pp. 43–54. D. Nurmi, J. Brevik, and R. Wolski, “Quantifying machine availability in networked and desktop grid systems,” University of California, Santa Barbara, Computer Science, Tech. Rep. ucsb_cs:TR-2003-37, Nov. 2003. J. Brevik, D. Nurmi, and R. Wolski, “Automatic methods for predicting machine availability in desktop grid and peer-to-peer systems,” in CCGRID. IEEE Computer Society, 2004, pp. 190–199. D. Nurmi, J. Brevik, and R. Wolski, “Modeling machine availability in enterprise and wide-area distributed computing environments,” in Euro-Par 2005, Parallel Processing, 11th International Euro-Par Conference, Lisbon, Portugal, August 30 - September 2, 2005, Proceedings, ser. Lecture Notes in Computer Science, vol. 3648. Springer, 2005, pp. 432–441. D. N. P. Murthy, M. Xie, and R. Jiang, Weibull Models. Wiley Series in Probability and Statistics, Wiley-Interscience, 2003. M. Rausand and A. Høyland, System Reliability Theory: Models, Statistical Methods and Applications Second Edition. Wiley-Interscience, 2003. F. Romero and T. J. Hacker, “Live migration of parallel applications with openvz,” in Proceedings of the 2011 IEEE Workshops of International Conference on Advanced Information Networking and Applications, ser. WAINA ’11. Washington, DC, USA: IEEE Computer Society, 2011, pp. 526–531. [Online]. Available: http://dx.doi.org/10.1109/WAINA.2011.156 D. L. Grosh, Primer of Reliability Theory. New York, NY: John Wiley, 1989. B. Sotomayor, R. Montero, I. Llorente, I. Foster, and F. de Informatica, “Capacity leasing in cloud systems using the opennebula engine,” Cloud Computing and Applications, vol. 2008, 2008. S. Venugopal, J. Broberg, and R. Buyya, “OpenPEX: An open provisioning and execution system for virtual machines,” in 17th International Conference on Advanced Computing and Communications (ADCOMŠ09). Citeseer, 2009. D. Nurmi, R. Wolski, C. Grzegorczyk, G. Obertelli, S. Soman, L. Youseff, and D. Zagorodnov, “The eucalyptus open-source cloud-computing system,” in Proceedings of Cloud Computing and Its Applications, 2008. M. De Assunção, A. Di Costanzo, and R. Buyya, “Evaluating the cost-benefit of using cloud computing to
extend the capacity of clusters,” in Proceedings of the 18th ACM international symposium on High performance distributed computing. ACM, 2009, pp. 141–150. [35] T. Hacker and B. Athey, “A methodology for account management in grid computing environments,” Grid ˚ ComputingUGRID 2001, pp. 133–144, 2001. [36] D. Nurmi, A. Mandal, J. Brevik, C. Koelbel, R. Wolski, and K. Kennedy, “Evaluation of a workflow scheduler using integrated performance modelling and batch queue wait time prediction,” in Proceedings of the 2006 ACM/IEEE conference on Supercomputing, ser. SC ’06. New York, NY, USA: ACM, 2006. [Online]. Available: http://doi.acm.org/10.1145/1188455.1188579 [37] W. Smith, “A service for queue prediction and job statistics,” in Gateway Computing Environments Workshop (GCE), 2010, nov. 2010, pp. 1 –8.
