Elasticity Management in Private and Hybrid Clouds

Elasticity Management in Private and Hybrid Clouds Rhodney Simões¥, Carlos Kamienski £ ¥ Universidade Federal de Alagoas (UFAL), Brazil, [email protected] £ Universidade Federal do ABC (UFABC), Brazil, [email protected]

Abstract Cloud computing requires elasticity management for dynamically allocating and releasing resources. Even though the adoption of cloud services has been growing, there is little knowledge available for guiding users when they need to manage elasticity. This paper analyzes elasticity in private and hybrid clouds, using a university lab, PlanetLab and Amazon EC2. Results show that the choice of metrics and thresholds plays a key role in meeting performance levels and controlling costs and that cloudburst can be effectively used for a hybrid cloud but the choice of the type of virtual machine in the provider has a significant impact.

1 Introduction Cloud computing is based on the concept of utility computing, where the amount of resources can dynamically adapt according to workload variations and users pay only what they really use [1]. Elasticity is the key feature for providing on demand and instant resource allocation and deallocation and therefore should be widely well known by the user community. Yet other important features exist - such as selfservice, broad network access, resource pooling and measured service [2] - rapid elasticity is the most innovative service offer of cloud providers. We adopt the NIST classification for deployment models: public, private, hybrid and community clouds [2]. Public clouds are openly available for the general public and frequently they are deemed as the only possible cloud type. Anyway, when a private datacenter offers features such elasticity it can be classified as a private cloud. As not all private clouds can keep the level of idle resources needed for ondemand allocation, they can buy capacity from a public cloud and thus create a hybrid one. Community clouds may also be considered when multiple users share resources that are freely available to each other. Though many providers offering different cloud services exist and the adoption of private clouds by organizations has been growing in the last years, there is still little understanding and no available best practices about elasticity management. The key

challenge is how and when to add new resources to match the demand and to release them whenever it is not needed so that users will not be incurred in unnecessary costs. First of all, one needs to understand what elasticity really is, which mechanisms to deploy, which choices must be made and which tradeoffs are involved. Usually, controlling elasticity involves adopting a management strategy based on rules that contain metrics and thresholds that must be configured in order for actions to be taken. In this paper we analyze computing resources that are allocated in units of virtual machines (VM). Public providers, like Amazon, offer value-added services for managing elasticity. Yet, there is no in-depth available information on how to configure those services and it might be highly dependent on specific applications. For example, operating system metrics - such as CPU and memory that are directly managed by the provider may be used. Alternatively, application metrics – such as workload can be captured and fed into the elasticity mechanisms. On the other hand, there is little understanding about hybrid clouds, where elasticity actions from the private cloud to the public cloud are called cloudburst. This paper evaluates the performance of a Smart Grid scenario [3] where negotiation between customers and power distribution companies has highly variable intensities according to daily consumption trends. We implemented this architecture using a private cloud on OpenNebula in our university lab, the public Amazon EC2 service, a hybrid of both using cloudburst and PlanetLab, which we consider a community cloud. Our results show that metrics and thresholds must be carefully chosen in order to obtain the dual effect of providing adequate performance levels at the same time that costs are kept as low as possible. We advocate that elasticity management is a problem where details make all the difference and its benefits may be far less significant than may be obtained when more attention is put in the configuration of its mechanisms. Particularly, we found out that CPU and memory usage are not adequate elasticity metrics because they generate high instability. Thus, our experiments used an application metric, workload, so

that we could obtain stable results. Results point out a sensitivity of customer level quality and cost to particular values configured to thresholds. A performance-cost index makes it clear the existence of certain limits beyond which increasing or decreasing resources is not worthwhile. Also, hybrid clouds may be successfully used for equalizing demand to available resources, but the choice of the type of virtual machine in the provider plays a key role on the overall cost-effectiveness of the process. In the remainder of the paper, section 2 presents background and related work. Section 3 presents the methodology and section 4 presents our main. Lessons learned are discussed in Section 5 and finally Section 6 draws conclusions and topics for future work.

2 Background and Related Work The use of elasticity in cloud computing empowers users to define policies for automating and enforcing the fulfillment of some pre-specified performance guarantees. Elasticity allows changing automatically the amount of resources allocated to an application to adapt to workload variations in order to match demand at any point in time [4]. Computing is the most common resource to be considered, usually in the form of virtual machines (VM). A cloud service must provide different management options for controlling elasticity and procedures for analyzing the system effectiveness and efficiency with respect to avoiding over and underprovisioning as much as possible. Underprovisioning must be avoided to protect users from poor performance and overprovisioned must also be avoided to keep costs as low as possible. Existing elasticity solutions and proposals can be classified into four dimensions [5]: scope (IaaS, PaaS, SaaS); policy (manual, automatic); purpose (capacity, performance, cost, energy); and method (replication, redimensioning, migration). In this paper we focus on an IaaS scope and use automatic policy and replication for obtaining adequate performance levels. Of particular interest is the elasticity policy, which governs how and when resources are allocated and deallocated. Automatic elasticity policy can be implemented by a control-theoretical approach or a rule-based approach [6]. Since control-based policy is complex to design and understand, most services adopt a rule-based one, where policies are fired under certain conditions to take actions. Conditions match the value of metrics against predefined thresholds to take actions that we call positive and negative elasticity, when more resources are added and removed respectively.

Amazon is currently the major player among public cloud providers and its mainstream IaaS EC21 service can be managed by another elasticity service (AutoScaling2) that is fed by a monitoring service (Cloud Watch) and can be connected to a load balancing service (Elastic Load Balance). Although elasticity is most commonly associated to public clouds, it can also be used by private and hybrid clouds. Amazon AutoScaling allows users to choose different metrics and configure thresholds for elasticity actions. Unfortunately this feature is not directly offered by private cloud platforms, e.g. OpenNebula3 and OpenStack4, so that specific in-house customized solutions must be developed. Private clouds can offer limited elasticity since they maintain a much lower amount of idle resources than public ones. A compromise between budget constraints and performance guarantees can be reached by means of cloudbursting instant excess workload to a public cloud, i.e. creating a hybrid cloud [2]. Existing literature does not follow the same approach presented in this paper, which offer useful information for configuring elasticity metrics and thresholds. The work presented in [4] introduces some metrics for understanding the effects of elasticity, such as time and resources involved in under or overprovisioned states. It assumes the existence of a match function that precisely returns the resources needed in order to meet pre-specified system performance requirements under certain workloads. Although there might be many issues in defining such a function, no one is presented as an example and no further discussion is provided. This work was extended in [7] to add new metrics and present results of a performance analysis, though at a small scale. Again, it does not precisely define how over and underprovised states are detected and do not present a deep discussion on how to define metrics and to configure thresholds. A small-scale performance analysis in a public cloud is presented in [6] using rule-based elasticity policy, but no in-depth discussion on the effect of different metrics and parameters is presented. CloudFlex manages cloud resources to meet over capacity demands, but does not particularly target elasticity mechanisms [8]. DEPAS is a decentralized and probabilistic algorithm for auto scaling with a seamless integration to a peer-to-peer network [9]. This paper analyzes elasticity using simulation and experimentation, but results do not target the choice of metrics and thresholds. 1 2 3

aws.amazon.com/ec2 aws.amazon.com/autoscaling opennebula.org 4 openstack.org

3 Methodology 3.1 Scenario and Environment The scenario for evaluating elasticity is a management system for the Smart Grid [2], which is deployed into three cloud infrastructures: a privatee cloud in our university lab, the Amazon EC2 public cloud service and PlanetLab that we classify as a coommunity cloud. The system is comprised of three moddules (Figure 1): generation, distribution and consuumption. Power transmission is assumed to be implem mented together with generation in our model. Generration is a very simple module that simulates the am mount of electric power available to the customers. It is implemented in Java and hosted in our private cloud inffrastructure.

The distribution module can also work in a hybrid cloud model using Amazon Elasstic Compute Cloud (EC2) instances for elasticity when n the capacity of the private cloud is exceeded. Here we consider 3 instances types (small, medium and a large), aimed at evaluating the power needed by an instance for our application as far as elasticity is con ncerned. The consumption module generates demand requests to the distribution modu ule to use available power. This module continuously sends s requests to the distribution module so that custom mers can benefit from the best power prices. Customerrs are simulated as PlanetLab nodes, also called clien nts, where each node can represent multiple customerss depending on the request rate a customer is configurred to make in a real Smart Grid scenario. For example, if we consider that each Smart Grid customer sends one request every 2 minutes and we configure each node to send one request every 3 seconds, we have a 1:40 ratio, i.e., each node represents 40 customers.

3.2 Metrics

Figure 1 – Cloud-based Smart G Grid Scenario The distribution module is rresponsible for negotiating power requests coming from customers since the price varies seasonably inn different time scales as hours, days and months. The workload received by the distribution module varries significantly and therefore it runs on a cloud datacenter and manages the creation and teardown off VMs using an elasticity mechanism. Distribution is ddeployed in our private cloud managed by OpenNebuula, mainly used for creation and teardown of VMs mannaged by the Xen hypervisor. As elasticity mechanisms aare not provided directly by OpenNebula, they were im mplemented as a series of scripts that measure, moonitor and take decisions whether to create or teaardown a VM according to upper and lower tthresholds. The distribution service is replicated overr different VMs and runs a Tomcat server that iis continuously monitored for obtaining high availabbility. Those are lightweight VMs with 256 MB of RA AM, a carefully chosen configuration so that theey are easily overloaded, thus making it easier to ttrigger elasticity actions. In order for the distribution serrvice to be aware of the any change in the number and identification of VMs, elasticity is tightly integrated to an Apache load balancer, running apart from the other V VMs.

The basic metrics used in our experriments are: • Response time: time elapsed while the customer sends a request and waits for a response from the server (distribution). We co ompute the mean, median and 90th percentile for all clients’ response times every 10 seconds. • Computing power: number of VMs V allocated by the distribution module to answer customer c requests. • Workload: number of simultan neous requests being handled by the load balancer. • Number of nodes: number of o PlanetLab nodes actively generating requests at any a given time. • CPU usage: average percentagee of the virtual CPU used by all active VMs every 30 0 seconds. • Memory usage: average perceentage of the virtual memory used by all active VMss every 30 seconds.

3.3 Managing Elasticity The purpose of our analysis is to understand and tune elasticity actions according to custo omer response times. For that scenario, we consider that response times up to 10 seconds are acceptable and sh hould work for most cases, because negotiations are do one automatically by some kind of home-based agent off the Smart Grid, and not directly by the user. Howeverr, we achieve much lower response times, such as 2 to 3 seconds, even considering that our PlanetLab nodes simulating customers are spread all over the world. w Elasticity actions are positive, when w a new VM must be created because the average loaad of all active VMs is causing longer response times, and negative, when active VMs are idle. This approacch requires a metric

3.4 Experiments Experiments consist of PlanetLab nodes sending power negotiation requests to the distribution module running in private and public cloud infrastructures. We used 100 PlanetLab nodes but in order to do this, hundreds of nodes were assessed and many of them were discarded due to inadequacy to our purposes. Each PlanetLab node sends requests according to an Exponential distribution with a mean of 3 seconds (i.e. a Poisson arrival rate). Experiments consist of increasing and decreasing workload to force the elasticity management system to create and teardown VMs. One by one nodes start sending requests every 10 seconds and after the last node is started, one by one they are stopped every 10 seconds again. Each experiment lasts for 45 minutes with additional 15 minutes to download logs from PlanetLab nodes. Altogether we spent 230 hours running experiments. Each experiment was replicated 10 times and 99% asymptotic confidence intervals were computed, but most results in section 4 present time series using a single replication that well represents their average. Averages are also presented together with confidence intervals. We used 5 thresholds of average

simultaneous requests for positive elasticity (250, 300, 350, 400, 600) and 3 thresholds for negative elasticity (100, 125, 150). These particular levels for the thresholds were obtained in preliminary experiments, and there are aimed at showing different behaviors. Graphs showing time series only present results for positive and negative thresholds of 350 and 150 respectively, due to space constraints, which were identified as an adequate tradeoff between cost and performance in terms of VM creation and teardown.

4 Results We present results for elasticity management in private and hybrid clouds and a cost-performance index.

4.1 Elasticity in a Private Cloud Figure 2 depicts a time series of different metrics obtained from an experiment made in our private cloud where the request arrival rate increases and decreases over time. For this experiment, elasticity thresholds were set to 350 and 150 respectively. Out of the 10 replications of the experiment, a typical one is presented as a time series. Figure 2a presents the response time and one can observe that the mean and median are mostly kept under 3 seconds and the 90th percentile is under 9 seconds, so that our 10 second limit for the majority of requests is easily fulfilled. Better than this, the response time is maintained almost constant throughout the experiment, even though the workload varies considerably, from zero to 3500 simultaneous requests (Figure 2c). Particularly the median is kept close to 1 second, which is the minimum average response time considering the PlanetLab nodes are spread all over the world. As a result of the increasing workload, new VMs are created to keep up with the demand, maximum of 10, and they start to be released some time after the workload starts to decrease (Figure 2b). The number of PlanetLab nodes is consistent with the workload, as shown by Figure 2d. 9

Mean Median 90th Percentile

6 Seconds

and two thresholds to be defined. We use workload as the metric although ideally user-side metrics, e.g. response time, should be used because that is what is being optimized. However, it is not practical, thus server-side metrics are commonly used. Public cloud services, such as Amazon AutoScaling, provide machine-level metrics to be directly used in their elasticity services, as well as connecting metrics from user’s applications. Our experience with extensive preliminary experiments showed that CPU and memory are not adequate elasticity metrics, since there is no clear mapping between variations in CPU and memory usage and response times. CPU usage responds too fast to small variations in the workload and memory usage is not reliable since applications frequently fail to release memory giving the false impression of resource depletion. On the other hand, when we started using the number of requests we could find a clear correspondence to response times, which allowed us to correctly tune the needed number of VMs to keep up with customers’ expectations. Also, two thresholds must be defined: higher thresholds for triggering positive elasticity and lower, for triggering negative elasticity. Once we defined a metric for elasticity preliminary experiments were conducted to have a precise mapping between number of requests that a VM can handle and the response time. Further we defined ranges for thresholds and conducted experiments giving them different values.

3

0

0

10

20 Minutes

a)

30

Response time

40

Finally, Figure 2e and Figure 2f corroborate with the rationale presented in section 3.3. Memory usage is constant over the whole experiment making it inadequate for managing elasticity as well as CPU usage that showed an unstable behavior. From now on, only response time and number of VMs will be presented due to space constraints and because the other metrics do not bring significant new insights.

12

VMs

9 6 3 0 0

10

20 30 Minutes

40

4.2 Elasticity in a Hybrid Cloud

b) Computing power (VMs)

Requests

4000 3000 2000 1000 0

0

10

20 Minutes

c)

30

40

Workload

75

70 50

60

0

Mean Median 90th Percentile

50

25 0

10

20 Minutes

30

Seconds

PlanetLab nodes

100

Now we show results for experiments where we limit the number of VMs in the private cloud to 8 and after that new VMs are created in the public Amazon EC2 cloud, using cloudburst. Here we aim at understanding cloudburst and preliminary experiments with a higher number of VMs did not reveal significant differences, so that the 8 VM limit does not interfere in the results. Figure 3 shows results when instances (VMs) of the type small are used. Even though 3 new VMs were created, they were not enough for dealing with the requests and generated unstable and higher response times during the whole period when the public cloud is used.

40

40 30 20 10

d) Number of PlanetLab nodes

0

100

0

10

20 Minutes

RAM (%)

80

a)

60 12

40

0

10

20 Minutes

e)

30

40

Memory usage

6 4 2

100

0

80 CPU (%)

Response Time

8 VMs

0

40

Amazon Private

10

20

30

60

0

10

20 Minutes

30

40

b) Computing power (VMs) Figure 3 – Elasticity in a hybrid cloud with SMALL instance

40 20 0 0

10

20 Minutes

30

40

f) CPU usage Figure 2 – Elasticity in private cloud; thresholds: 350 (positive) and 150 (negative)

When the workload decreases and the VMs in Amazon are released, the response time returns to its original values. Therefore, we can conclude that VMs of the type small were not able to meet the 10-second limit requirement of our scenario. Furthermore, comparing Figure 2b to Figure 3b we observe that in

the private cloud experiment 10 VMs were created compared to 11 VMs in the hybrid cloud (8 in the private cloud and 3 in the public cloud). Experiments involving VMs of the type medium in the public cloud show a remarkable difference for the response time compared to small ones (Figure 4). It becomes evident that doubling the cost for new VMs was worthwhile for our scenario5. Now cloudburst did not generate instability of any kind, even though only 2 VMs were created in the public cloud. A comparison of results in Figure 4 and Figure 2 shows that response times are similar, even though all request arrive to our load balancer and are redirected to Amazon.

thresholds and private and hybrid cloud. Figure 6a shows response times for various thresholds for the private cloud, where one can observe no significant variations, except for an upper threshold of 600 average simultaneous requests. For that configuration, a significant increase in response times is observed because the creation of a new VM is postponed up to a point where the drop in performance is irreversible. The reduction in cost (8 VMs in average, Figure 6b) is not worth it since average response times are higher than our established 10-second limit. 10

4

Seconds

Seconds

Mean Median 90th Percentile

Mean Median 90th Percentile

8

6

6 4 2

2

0

0

10

0 0

10

20 Minutes

a) 12

a)

40

Response Time

8

VMs

6

2

4

0 0

2 20 Minutes

30

b) Computing power (VMs) Figure 4 – Elasticity in a hybrid cloud with MEDIUM instance As for the experiments with VMs of the type large in the public cloud, Figure 5 reveals that now using this level of computing power is not worth it, because doubling the cost from medium to large only brings small performance gains. Even the number of new VMs is the same for both types in all 10 replications of the experiment (even though only one is shown). Maybe different experiments might justify allocating large instances in Amazon EC2, but for our scenario they are not needed whatsoever as far as costeffectiveness is concerned. Figure 6 depicts the averages of the 10 replications for all experiments we conducted, including all 5

When we used EC2 one hour of a small and a medium instance was charged US$ 0.06 and US$ 0.12 respectively.

10

20 Minutes

30

40

b) Computing power (VMs) Figure 5 – Elasticity in a hybrid cloud with LARGE instance

40

For the experiments with cloudburst, Figure 6c shows similarity between that response times for the private cloud (with a 350:150 threshold pair) and for the hybrid cloud with medium and large instance types. The use of VMs in the Amazon public cloud only generated a slightly higher variability (see confidence intervals). 25 20

Seconds

10

Response Time

6 4

0

40

Amazon Private

10

8

0

30

12

Amazon Private

10 VMs

30

20 Minutes

15 10 5 0

Thresholds

c)

Response Time (private cloud)

d) Computing power (private cloud) 10 Seconds

8 6 4 2 0 private

e)

small

medium

large

Response time (private and hybrid)

12

VMs

11 10 9 8 private

f)

small

medium

simply states that the lower the CPI, the better the performance relative to the cost. Although in theory one might accept to have a very high cost in exchange for a very high performance, in practice response times have a lower limit that cannot be easily lowered even with high expenses in processing power. On the other hand, performance is frequently not acceptable beyond a lower limit (as in our scenario), even when one cannot afford its cost. Figure 7 presents the results for CPI for different experiments in private and hybrid clouds. In Figure 7a we can observe that most thresholds pairs generate small variations in CPI. Except for the experiment with the higher threshold set to 600, for the other ones CPI varied between 15 and 21, i.e., in general small changes up and down in the response times are compensated by similar movements in the opposite direction in the number of VMs. Please remember that we did not consider variability in CPI, which will make the difference even less significant. On the other hand, for 600 CPI was 160, revealing that configuring thresholds is something to be made very carefully, because expenditures savings up to some point result in an excessive low quality. Figure 7b reveals again that VMs of the type small are not adequate for our scenario and type large are not worth the additional cost charged by the provider.

large

Computing power (private and hybrid) Figure 6 – Average response times and computing power (VMs)

Figure 6d shows that the number of simultaneous active VMs is higher and present higher variability when using instances of the type small. Using instances of type medium and large decreases the average number of active VMs needed (with a small difference between medium and large) but it is not significant due to the confidence interval.

CPI

VMs

14 12 10 8 6 4 2 0

160 140 120 100 80 60 40 20 0

a)

As we tried to go deeper inside the behavior of both private and hybrid elastic cloud infrastructures, it became clear the existence of a trade-off between the cost of computing power to process the requests and the performance of the response times. Aiming at identifying the relative advantage of adding more VMs to the benefit of lower response times we computed a cost-performance index (CPI), by multiplying the average number of active VMs by the average response time of all replications of an experiment. For simplicity reasons, CPI does not make a difference between VMs in the private and public cloud, because an adequate pricing model for that would be more complicated. The value of CPI does not have a particular meaning, but

CPI

4.3 Comparing alternatives: cost vs. benefit

Private cloud (different thresholds)

70 60 50 40 30 20 10 private

small

medium

large

b) Hybrid and private cloud Figure 7 – Cost-Performance Index (CPI)

5 Discussion The ideal of utility computing requires the dynamic management of computing resources allocated to applications. In other words, it requires elasticity

management. Elasticity can be positive, when new resources are added, and negative when resources are released. Although cloud computing has already passed the buzzword phase and now is increasingly used, there is no widely available information about tricks, pitfalls and best practices for elasticity management. This paper presents performance evaluation results for managing elasticity both in private and hybrid clouds and four key points are highlighted: a) metric selection; b) threshold selection; c) cloudburst usage in hybrid clouds; d) selecting a type of VM in the public cloud. Even though public cloud providers offer elasticity services and metrics for controlling it, there is no available knowledge about choosing metrics and configuring thresholds. We found out that the choice of the metric for controlling elasticity is of fundamental importance. Amazon AutoScaling service provides machine-level metrics, such as CPU and memory usage, as well as it make it possible to capture application metrics. Following this approach, we started using CPU and memory but with unsatisfying results. Next we found out that an application metric, the average number of requests, can be used with very positive results and offers a direct mapping to a metric used by customers (response time). Even though literature and providers usually exemplify elasticity using CPU and memory, we did not find any results that invalidate our point of view. In other words, based on our findings we think that currently the choice of elasticity metrics might be taken for granted so that there may be room for improving efficiency in this area so that more effort must be put on this challenge. After a metric is selected, values must be provided for upper and lower thresholds, for increasing and decreasing resources allocated to applications. Our results show that client cost and performance are very sensitive to the thresholds. An approach of decreasing the cost by raising upper and lower thresholds has some intrinsic limits from which the penalty in performance makes it not worthwhile. Our costperformance index (CPI) clearly shows it. Our experience with a hybrid cloud reveals that cloudburst works well and resources of a public cloud can be effectively used for temporarily complementing the capacity of a private cloud, which usually does not have an average idleness level to fulfill demands of all its user base. Keeping a significant number of servers underutilized might be frequently much more expensive than temporarily buying extra capacity from a third-party provider. Also, when cloudburst is used the choice of the type of VM (or instance) in the public cloud has a decisive impact on keeping performance levels while keeping cost as low as possible. For example, our results show that using Amazon instances of the type small is deleterious of the performance of

our application. On the other hand, instances of the type large add no significant value to the performance of instances of type medium, but their cost is double. The bottom line is that small instances are simply too limited for applications with high workloads, but in our scenario the choice of medium or large instances has a significant different in cost.

6 Conclusions Elasticity management is a key issue for providing the feature of utility computing in a cloud environment. Since there is little information available on this subject, it becomes clear that cloud users do not have precise information for guiding them on how to make choices and configuring parameters. In this paper we show that the right choice of metrics and thresholds is fundamental for equalizing quality and costs. An application metric, workload, helped us in having stability in the process of creating and releasing virtual machines. Also, cloudburst can be used for in a hybrid cloud, but the choice of the virtual machine has a significant impact. As future work we intend to analyze metrics for detecting over and underprovisioned states and come up with guidelines and templates for helping administrators to configure elasticity parameters in private and hybrid clouds.

References [1] Armbrust, M. et al., “A View of Cloud Computing”, Communications of the ACM, 53(4), pp. 50-58, 2010. [2] Mell, P., Grance, T., “The NIST Definition of Cloud Computing”, NIST Special Publication 800-145, 2011. [3] Ipakchi, A., Albuyeh, F., “Grid of the Future: Are We Ready to Transition to a Smart Grid?”, IEEE Power & Energy Magazine, 7(2), pp. 52-62, March 2009. [4] Herbst, N. R., Kounev, S., Reussner, R., “Elasticity in Cloud Computing: What It Is, and What It Is Not”, ICAC 2013, June 2013. [5] Galante, G., de Bona, L.C.E., "A Survey on Cloud Computing Elasticity", IEEE Utility and Cloud Computing (UCC 2012), November 2012. [6] Ghanbari, H., Simmons, B., Litoiu. M., and Iszlai. G., "Exploring Alternative Approaches to Implement an Elasticity Policy", IEEE Cloud 2011, July 2011. [7] Coutinho, E. F., Gomes, D. G., Souza, J. N., "An Analysis of Elasticity in Cloud Computing Environments Based on Allocation Time and Resources", IEEE LatinCloud 2013, December 2013. [8] Seung, Y., Lam, T., Li, L. E., Woo, T., “Cloudflex: Seamless scaling of enterprise applications into the cloud”, INFOCOM 2011, pp. 211–215, April 2011. [9] Calcavecchia, N., Caprarescu, B., Di Nitto, E., Dubois, D., and Petcu, D., “DEPAS: a decentralized probabilistic algorithm for auto-scaling”, Computing, 94, pp. 701–730, 2012.