Energy-aware framework for virtual machine consolidation in Cloud ...

4 downloads 26145 Views 740KB Size Report
Energy-aware framework for virtual machine consolidation in Cloud computing. Zhibo Cao. School of Computer Science and Engineering,. Communication ...
2013 IEEE International Conference on High Performance Computing and Communications & 2013 IEEE International Conference on Embedded and Ubiquitous Computing

Energy-aware framework for virtual machine consolidation in Cloud computing Zhibo Cao

Shoubin Dong

School of Computer Science and Engineering, Communication & Computer Network Lab of Guangdong South China University of Technology Guangzhou, 510641, P.R.China [email protected]

School of Computer Science and Engineering, Communication & Computer Network Lab of Guangdong South China University of Technology Guangzhou, 510641, P.R.China [email protected]

Abstract—Virtual machine (VM) consolidation in Cloud computing environments provides a great opportunity for energy saving. However, the obligation of providing suitable quality of service to end users leads to the necessity in dealing with energy-performance tradeoff. In this paper, we propose a redesigned energy-aware heuristic framework for VM consolidation to achieve a better energy-performance tradeoff. The main contribution of us is a Service Level Agreement (SLA) violation decision algorithm, which is used to decide a host is overload with SLA violation or not. Finally, we have evaluated our framework through simulation on large-scale experiments driven by workload traces from more than a thousand VMs, and the results show that our framework outperforms previous work, which has 11.8%~27% decrease in energy consumption, 57.9%~78.4% decrease in SLA violation, 63.2%~84.1% decrease in energy-performance metric which is a product metric of the energy consumption and SLA violation.

become a hotspot in Cloud computing. To our knowledge, a VM includes three main parts: virtual CPU, virtual memory and virtual storage. Besides, there is a virtual network among VMs. Accordingly, VM consolidation should take into account these four virtual factors as much as possible in practice. However, In view of the virtual CPU is the main reason of energy consumption in data centers, so we only consider the virtual CPU in this work for VM consolidation. The remainder of this paper is organized as follows. In section  we discuss the related work. In section  we discuss the motivation for this paper and propose a problem for the implemented framework in CloudSim. In section  we propose our redesigned framework for VM consolidation. In section we deduced the qualifications for the SLA violation decision algorithm. In section  we design experiments for the redesigned framework. Finally, we discuss the future work and conclude this paper in section .

Keywords-SLA violation; energy-aware; VM consolidation; cloud computing; energy consumption

II. I.

INTRODUCTION

Modern data centers are requested to be able to deal with a diversity of applications including serial and parallel types, short and long periods of time types. For finishing these features, data centers will consume huge energy and make higher outlays in cloud computing. According to statistics, each data center in world consumes as much energy as 250,000 households on average [1]. And in another report by McKinsey, the overall estimated energy bill for data centers in 2010 is $11.5 billion and energy costs in a typical data center double every five years [2]. In 2014, based on the trends from the American Society of Heating, Refrigerating and Air-Conditioning Engineers (ASHRAE) [3], it has been predicted that infrastructure and energy costs would contribute about 75%, whereas IT would contribute only 25% to the total cost of operating a data center [4]. On the other hand, the Virtualization technology, which Cloud computing greatly relies on, provides the ability to consolidate VMs between physical nodes [5]. This enables the dynamic VM consolidation to the minimum physical nodes according to the current resource requirements. As a result, the idle nodes can be turned off or put into a sleep mode to save energy consumed by data center, but also it will inevitably lead to some SLA (Service Level Agreement) violation. Therefore, through VM consolidation to reduce energy consumption while keeping SLA at a low-level have 978-0-7695-5088-6/13 $31.00 $26.00 © 2013 IEEE DOI 10.1109/HPCC.and.EUC.2013.271

1890

RELATED WORK

Making high efficiency of energy consumed in virtual data centers for VM consolidation has already become an urgent issue to solve. Therefore, there are lots of works in the study of this issue. To our knowledge, the first work to manage power efficiently in virtual data centers has been done by Nathuji and Schwan in [8]. Nathuji et al. propose a VirtualPower Management approach (VPM) for VM consolidation. The VPM can support the isolated and independent operation assumed by guest VMs running on virtualized platforms, and make it possible to control and globally coordinate the effects of the diverse power management polices applied by these VMs to virtualized resources. Different from the VPM. In view of workload consolidation traditionally driven by performance objectives (SLA) in virtualized environments, Verma et al. [9] propose an architecture of a power-aware application consolidation framework (pMapper), which can incorporate various scenarios including power and performance management using Virtualization technology. It also provides the solution to the most practical possibility, i.e. power minimization subjected to a fixed performance requirement. Srikantaiah et al. [10] have studied the interrelationships between energy consumption, resource utilization, and performance of consolidated workloads. And the study shows the energy performance trade-offs for consolidation, and then they model the consolidation

Step 1. Choose Overload hosts in data center, then select VMs from those hosts to migrate (denote the VMs as VmsToMigrate) until the Overload hosts become Underload. Step 2. Choose suitable placements for VmsToMigrate on Underload or Idle hosts based on the Minimum Power policy [7]. As a result, some Underload hosts may turn off and become Idle, and some Idle hosts may turn on and become Underload. Step 3. Firstly, denote the overall Underload hosts as OUH, then exclude those Underload hosts (denote as EUH) who receive any migration from VmsToMigrate, and denote the remaining Underload hosts as RUH. Finally, turn off all the RUH which can migrate all VMs (residing on them) to the remaining RUH or the EUH without resulting in Overload. For these steps, the proposed framework [6] in CloudSim significantly reduces energy consumption, while ensuring a high level of adherence to the SLA. Although the framework has a great performance improvement in terms of energy consumption and SLA violation reduction, we still deem it has a lot room for improvement. For the framework, we have some problem: In step 1, the framework selects VMs from an Overload host until the host becomes Underload. Does it necessarily select a VM until the host becomes Underload? The Overload host does not mean the host will generate SLA violation, while it only means the host may generate SLA violation. If the Overload host does not generate SLA violation, then the migration will be in vain and result in high energy consumption. Therefore, we need a more suitable method to decide the VM selection in this step. In view of the above problem, we redesign the framework for VM consolidation in the next section.

problem as a modified multi-dimensional bin-packing problem. However, the above two works are workload dependent, whereas our framework is workload independent and can be deployed in generic Cloud environment. There are also some similar works can be found in[11][12]. There are also some works that treat VM consolidation as multi-objective optimization problem, or use prediction to minimize the number of running servers for energy-ware. In literature [13], Xu et al. treat VM consolidation as a multiobjective optimization problem while minimizing energy consumption and SLA violation simultaneously and propose an improved genetic algorithm with fuzzy multi-objective evaluation for VM consolidation. Duy et al. [14] integrate a neural network predictor in a Green scheduling algorithm to predict future resource requirements based on historical data. The algorithm turns off unused servers and restarts them to minimize the number of running servers, and then get less energy consumption. There are also some similar works can be found in[15][16]. Anton et al. [17] have defined an architectural framework and principles for energy-aware Cloud computing and develop algorithms for energy-aware mapping of VMs to suitable Cloud resources in addition to dynamic consolidation of VM resource partitions. However, the fixed threshold in [17] for vm consolidation is not suitable for virtual environment with variable workloads. Therefore, they [7] illustrate that VM consolidation should be optimized continuously in an online manner due to the variability of workloads experienced by modern applications. Then they propose novel adaptive heuristics for dynamic consolidation of VMs based on an analysis of historical data for the resource usage of the VMs. Experiments show that the allocation and selection algorithms can immensely save energy consumption. However, we deem the SLA violation and energy consumption produced by the framework has a lot room to improve. Therefore, we propose another redesigned energy-ware heuristic framework for VM consolidation in CloudSim, which performs better than the framework implemented in CloudSim for energyperformance tradeoff. III.

IV. THE REDESIGNED FRAMEWORK FOR VM CONSOLIDATION TABLE I.

NOTATIONS FOR THE HOSTS’ STATUS IN VIRTUAL DATA CENTER

Notation Over OverS OverNS Under Idle Critical

MOTIVATION

In virtual data center, energy consumption is enormous. Therefore, the energy consumption has become an urgent issue to be solved. In previous works, the solution to this issue is to migrate and consolidate VMs in data center. Although it can greatly reduce energy consumption, it will result in SLA violation. Obviously, we cannot eliminate the SLA violation by this way, but we can reduce SLA violation as much as possible while reducing energy consumption, which is called energy-consumption tradeoff and also the focus of this article. Generally speaking, in virtual data center, the host’s statuses include three types: Overload, Underload and Idle. Overload means the host may generate SLA violation; Underload indicates the host is in use but generates no SLA violation; Idle means the host is available (not in use). And then VM consolidation needs three main steps in CloudSim [7]:

Description The Overload host’s status The Over host with SLA violation The Over host with No SLA violation The Underload host’s status The Idle host’s status The host does not send or receive any migration, it’s a status between OverNS and Under.

Before illustrating the redesigned framework, we need to define some notations that will be used in it.Table illustrates the notations for the hosts’ status in virtual data center. In response to the two problems proposed in the last section, we redesign the original framework in CloudSim. The steps about the redesigned framework as follows: Step. Choose Over hosts in data center by Overload decision algorithms such as IQR, LR, LRR and MAD proposed in CloudSim. Then classify these Over hosts into OverS hosts and OverNS hosts. Afterwards, select VMs from the OverS hosts until they become OverNS or Critical, then put the VMs into the VmsToMigrate (the same as the notation in the Step 1 in the last section). Finally, select VMs from the OverNS hosts through the SLA violation decision

1891

algorithm proposed by us until they become Critical, and put the VMs into the VmsToMigrate. Step  and Step  are the same as the step 2 and 3 in the last section. Step ~ Step  and figure 1 describe the redesigned framework on the whole. In the redesigned framework, we use the Overload decision algorithms to classify the hosts into three host’s statuses: Over, Under and Idle; then divide the Over hosts into OverS and OverNS hosts through SLA violation, and then use the SLA violation decision algorithm to decide VM selection on the OverNS hosts; finally, apply the MinPower and MaxUtilizaiton policy to the Under and Idle hosts until there is no VM need migrating in the VmsToMigrate.

of

Mi

MUH i is set to xi (for each H i ). Then we can

easily get (2) from the formula (1). TABLE II.

In the next section, we will illustrate the implementation of the SLA violation decision algorithm in details, also we will briefly illustrate the Overload decision algorithm, VM selection algorithm implemented in CloudSim. METHODOLOGY AND METRICS FOR THE REDESIGNED FRAMEWORK

A. SLA violation decision algorithm For simplicity, the algorithm is referred as to SLAVDA. The SLAVDA is an algorithm used to decide if a host can greatly possibly generate SLA violation. Then, in table, some necessary parameters are defined to deduce the qualifications for the algorithm. First of all, we need to find out the necessary and sufficient condition for SLA violation. When the total request utilization of the VMs on H i is greater than the allocated utilization for them by the host, the host will generate SLA violation. If they are equal, it will be assumed to generate no SLA violation. Then we can easily deduce formula (1). Actually, the total allocated utilization by H i can never exceed the maximum utilization of the host. In other words, the total allocated utilization at most equals to the maximum utilization of the host. It also means if the request utilization of the VMs on H i exceed the maximum one, the host will definitely generate SLA violation. For simplicity, the ratio

THE DESCRIPTION OF THE PARAMETERS DEFINED IN ALGORITHM SLAVDA

Parameter Hi

Description the ith host in the data center

Vij

the jth virtual machine on the ith host

N

the number of hosts in the data center the number of virtual machines on the ith host (j belongs to{1,……., M i })

Mi

Figure 1. The process of the redesigned framework for VM consolidation

V.

∑ j =1 RUVij

MUH i

The Maximum Utilization (MIPs) of host H i

AUH i

The Total Allocated Utilization for VMs by H i

MUVij

The Maximum Utilization of the jth vm on H i

RUVij

The Request Utilization of the jth vm on H i

RUV j

The Request Utilization for the jth vm in VmsToMigrate

Mi ⎧ Mi ⎪ ∑ AUVij ∑ RUVij j =1 ⎪ j =1 < , SLA violation ⎪ MUH i MUH i ⎪M Mi ⎪ i ⎪ ∑ AUVij ∑ RUVij ⎪ j =1 j =1 = , No SLA violation ⎨ MUH i ⎪ MUH i Mi ⎪ Mi ⎪ ∑ AUVij ∑ RUVij ⎪ j =1 j =1 impossible ⎪ MUH > MUH , i i ⎪ ⎪⎩

(1)

SLA violation ⎧ 1.0 < xi , ⎪ ⎨0 ≤ xi ≤ 1.0, No SLA violation ⎪ x < 0, impossible i ⎩

(2)

As shown in the formula (2), the necessary and sufficient condition for SLA violation is xi > 1.0 , where xi is less than or equal to the ratio of

∑ j =1 MUVij Mi

MUH i . When

xi > 1.0 , the host H i is in OverS. When 0 < xi ≤ 1.0 , the host H i is in OverNS or Under. And when xi = 0 , the host H i is in Idle. The SLAVDA only handles the OverNS ( 0 < xi ≤ 1.0 ) status and decides if the OverNS hosts may greatly transform into OverS ( xi > 1.0 ). Afterwards, we need to find out the conditions which will lead into the above result (OverNS transformed into OverS). Firstly, xi is the ratio of the total request utilization of the M i VMs on H i to the maximum utilization of the host. So xi is a random variable, and the mean and standard deviation of xi can be easily calculated with the historical records, then denote the mean and standard deviation as μ i and δ i respectively. Additionally, denote the p ( xi ) as the probability density function for xi . Finally, we can calculate

1892

VI. PERFORMANCE EVALUATION

the probability of the condition that xi is greater than 1 in the next time in formula (3).

As the targeted system is an IaaS in cloud computing, it’s better to evaluate the redesigned framework on a large-scale virtualized data center infrastructure. However, it’s very difficult to conduct repeatable large-scale experiments on a real infrastructure. Therefore, to ensure the repeatability of the experiments, simulations have been chosen as a suitable way to evaluate the redesigned framework. For our experiments, the CloudSim toolkit [6] has been chosen as a simulation framework. The toolkit has been developed by the Cloud Computing and Distributed Systems (CLOUDS) Laboratory, University of Melbourne. It supports both system and behavior modeling of cloud system components such as data centers, virtual machines (VMs) and resource provisioning policies. Currently, it supports modeling and simulation of Cloud computing environments consisting of both single and inter-networked clouds (federation of clouds), and also supports energy-efficient management of data center resources.

P{xi ≥ 1} = P{xi − μi ≥ 1 − μi }(set 1 − μi =ε i ) = P{xi − μi ≥ ε i } 2

⎛ xi − μi ⎞ ⎜ ⎟ p ( xi ) dxi xi − μi ≥ε i ⎝ εi ⎠ 2 1 ∞ = 2∫ ( xi − μi ) p( xi )dxi ε i ε i + μi

≤∫



1 ε i2

=

δi 2 ε i2



∫0 ( xi − μi )

(3)

2

p( xi )dxi

In the formula (3), if the mean is greater than 1, the host H i will be assumed to be OverS in the next time. If the

mean is less than 1, the host’s status will be decided by the ratio δ i 2 ε i2 . Actually, we deem that the ratio should be greater than 1, then the host will generate SLA violation with great probability. Therefore, we can find out the qualifications for the result (OverNS transformed into OverS) by formula (4). ⎧ ∀δ i , μi ≥ 1.0 ⎨ ⎩δ i ≥ 1 − μi , μi < 1.0

(4)

B. The Overload Decision Algorithm For simplicity, the algorithm is referred as to ODA. The ODA is used to decide a host Over or not, while SLAVDA is used to decide a host OverS or not. Up to now, there have been already implemented four ODAs in CloudSim[6], and they are IQR, LR, LRR and MAD respectively. C. The VM Selection algorithm The algorithm is simply referred as to VMSA. The VMSA is used to select VMs from Over hosts to prevent them from being Over. There have been already implemented four VMSAs in CloudSim , and they are MMT, MU, RS and MC respectively. D. Metrics Meeting QoS requirements is very important for cloud computing. QoS requirements are commonly formalized as SLAs, which can be determined in terms of characteristics such as minimum throughput, maximum response time or minimum bandwidth and so on in practical system. These characteristics are workload or application dependent. However, the redesigned framework belongs to IaaS layer in cloud computing and should be workload independent. Therefore, we use those SLA-related metrics defined in the literature [7] to evaluate the redesigned framework in our experiments: SLATAH(SLA violation Time per Active Host), Migrations, Energy, PDM (Performance Degradation due to Migrations), SLAV(SLA Violation), ESV(Energy and SLA Violation).

A. Experiment Setup We simulated a data center with 800 nodes, half of which are HP ProLiant ML110 G4 (denote as G4) servers, and the other half consists of HP ProLiant ML110 G5 (denote as G5) servers. For better evaluating the effect of VM consolidation, the two selected servers both have two cores. The frequency [18] of the G4 server’s CPU is 1860 MIPS for each core, which has memory of 4096 MB. And the frequency of the G5 server’s CPU is 2660 MIPS for each core, which has memeory of 4096 MB. Both are modeled to have a bandwidth of 1 Gbit/s. We use PowerModels provided by the website [18] for the G4 and the G5, and the PowerModels can be simply described in the table . With reference to Amazon EC2 Instance Types [20], we modeled 4-type VMs in table . Additionally, taking into account workload data (see section) used in our experiments, each type has been set 1 core. Initially, the VMs are allocated according to the resource requirements (the maximum utilization of the VMs) defined by the VM types. Afterwards, during their lifetime, each VM is allocated due to its request utilization which is provided by the workload data. TABLE III. VM types Type 1 Type 2 Type 3 Type 4

THE FOUR VM TYPES USED IN EXPERIMENTS

Cores

Capacity (MIPs)

RAM (MB)

Storage (GB)

Bandwidth (Mbit/s)

1 1 1 1

2500 2000 1000 500

870 1740 1740 413

2.5 2.5 2.5 2.5

100 100 100 100

B. Workload Data and the Evaluated Frameworks In order to make the results more convincing, we need to use workload data obtained from real system. For our experiments, we use workload data provided by the CloudSim package. The only characteristic recorded in the data is the CPU utilization of VMs. The data is provided as a part of CoMon project, a monitoring infrastructure for PlanetLab [19], and it is about CPU utilization by more than a thousand VMs from servers located at more than 500

1893

values are 121.839 kWh and 129.886 kWh respectively. Compared with the Origin, the SLAVDA has 11.8%~27% decrease for energy consumption for the 16-type combinations respectively. 4) The PDM evaluation: In figure 2(d), for the Origin, the minimum and maximum evaluation values are 0.126% and 0.194% respectively. For the SLAVDA, the values are 0.072% and 0.109% respectively. Compared with the Origin, the SLAVDA has 42.9%~62.9% decrease for the 16-type combinations for the PDM respectively. 5) The SLAV evaluation: In figure 2(e), for the Origin, the minimum and maximum evaluation values are 0.707‰ and 1.379‰ respectively. For the SLAVDA, the values are 0.297‰ and 0.663‰ respectively. Compared with the Origin, the SLAVDA has 57.9%~78.4% decrease for the 16-type combinations for the SLAV respectively. 6) The ESV evaluation: In figure 2(f), for the Origin, the minimum and maximum evaluation values are 10.06‰ and 23.25‰ respectively. For the SLAVDA, the values are 3.7‰ and 8.34‰ respectively. Compared with the Origin, the SLAVDA has 63.2%~84.1% decrease for the 16-type combinations for the ESV respectively. In a nutshell, the SLAVDA outperform the Origin for the metrics.

places around the world. The interval of utilization measurements for the data is 5 minutes. There are 10-day CPU utilization records during March and April 2011 in the data. Additionally, in table  of the literature [7], there is a brief analysis about the workload. In the redesigned framework, the SLAVDA is an improvement for the step 1 (section 3) in the original framework. Therefore, for different improvements, the frameworks can be classified into two types: Origin and SLAVDA. C. Results and Analysis In the frameworks, we use four-type ODAs (IQR, LR, LRR and MAD) and four-type VMSAs (MC, MU, MMT, RS). Therefore, there are 16-type combinations policies for the ODA and the VMSA. According to the simulation results from reference [7], we set a suitable constant parameter for each ODA, and they are 1.5 for IQR, 1.2 for LR, 1.2 for LRR and 2.5 for MAD respectively. For simplicity, we give a sign to each combination alphabetically(A IQR_MC_1.5, B IQR_MU_1.5,C IQR_MMT_1.5, D IQR_RS_1.5, E LR_MC_1.2, F LR_MU_1.2, G LR_MMT_1.2,H LR_RS_1.2, I LRR_MC_1.2, J LRR_MU_1.2, K LRR_MMT_1.2, L LRR_RS_1.2, M MAD_MC_2.5, N MAD_MU_2.5, O MAD_MMT_2.5, P MAD_RS_2.5). 1) The SLATAH evaluation: In figure 2, we use the 10day workload to evaluate the four frameworks for the 16type combination policies. In each subfigure of the figure, each candlestick represents 10 results from the 10-day workload, of which the hollow stick represents 80% of the results (the same below). For better comparison, we use the average of the 10 results as an evaluation value (the same below). Therefore, the minimum and the maximum evaluation values for the Origin can be easily figured out and they are 5.5% and 7.87% respectively in figure 2(a). For the SLAVDA, the values are 4.1% and 6.74% respectively. Compared with the Origin, the SLAVDA has 25.4%~47.8% decrease for the 16-type combinations for the SLATAH respectively. 2) The Migrations evaluation: In figure 2(b), for the Origin, the minimum and maximum evaluation values are 31701 migrations and 65762 migrations respectively. For the SLAVDA, the values are 15615 migrations and 33417 migrations respectively. Compared with the Origin, the SLAVDA has 33.3%~67.9% decrease for migrations for the 16-type combinations respectively. 3) The Energy evaluation: In figure 2(c), for the Origin, the minimum and maximum evaluation values are 141.933 kWh and 171.444 kWh respectively. For the SLAVDA, the TABLE IV. Power Models G4(W) G5(W)

VII. CONCLUSION To obtain quick ROI (Return On Investment) in Cloud computing, Cloud providers need to reduce energy consumption for computing resources as much as possible while keeping SLA violation at a low-level, which is also called energy-performance tradeoff. Energy saving means data centers have to choose better cooling systems and construct better VM consolidation frameworks. SLA violation indicates data centers should take user requirements as qualifications for the VM consolidation framework. In this paper, we redesign an energy-aware heuristic framework for VM consolidation to make better energy-performance tradeoff. We classify the host status overload into two types: overload with SLA violation, overload with no SLA violation. And design the SLAVDA for hosts in data center to decide the host is overload with SLA violation or not. Finally, we have evaluated our framework and the Origin framework through simulation on large-scale experiments driven by workload traces collected from more than a thousand PlanetLab VMs, and the results show that our framework gets a better energy-performance tradeoff (the ESV metric).

POWER CONSUMPTION BY POWER MODELS AT DIFFERENT LOAD LEVELS IN WATTS 0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

86 93.7

89.4 97

92.6 101

96 105

99.5 110

102 116

106 121

108 125

112 129

114 133

117 135

1894

Figure 2. The four metrics (PDM,SLATAH, SLAV and ESV), Migrations and energy consumption evaluations for the two frameworks, the hollow bars in the six subfigures represent 80 percent results from the workload

REFERENCES [1] J. Kaplan, W. Forrest, N. Kindler, Revolutionizing Data Center Energy Efficiency, McKinsey, July, 2009. [2] http://searchstorage.techtarget.com.au/articles/28102-Predictions-29-Symantec-s-Craig-Scroggie [3] ASHRAE Technical COmmittee 99. Datacom equipment power trends and cooling applications 2005. [4] C. Belady, In the data center, power and cooling costs more than the equipment it supports, 2007. http://www.electronicscooling.com/articles/2007/feb/a3/ [5] R. Buyya, A. Beloglazov, J. Abawajy, Energy-efficient management of data center resources for cloud computing : a vision, architectural elements, and open challenges, in PDPTA 2010 : Proceedings of the 2010 International Conference on Parallel and Distributed Processing Techniques and Applications, CSREA Press, United States of America, pp. 6-17. [6] R. N. Calheiros, R. Ranjan, A. Beloglazov, C. A. F. D. Rose, R. Buyya, CloudSim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms, Software: Practice and Experience,2011, 41 (1):23-50. [7] A. Beloglazov, R. Buyya, Optimal Online Deterministic Algorithms and Adaptive Heuristics for Energy and Performance Efficient Dynamic Consolidation of Virtual Machines in Cloud Data Centers, Concurrency and Computation: Practice and Experience (CCPE), Wiley Press, New York, USA, 2011 [8] R. Nathuji, K. Schwan, Virtualpower: Coordinated power management in virtualized enterprise systems. ACMSIGOPS Operating Systems Review 2007, 41(6):265–278. [9] A. Verman, P. Ahuja, A. Neogi, pMapper: power and migration cost aware application placement in virtualized systems, in: Proceeding Middleware '08 Proceedings of the 9th ACM/IFIP/USENIX International Conference on Middleware, Springer-Verlag New York, Inc. New York, NY, USA,2008, pp. 243-264. [10] S. Srikantaiah, A. Kansal , F. Zhao, Energy aware consolidation for cloud computing, Proceedings of the 2008 conference on Power aware computing and systems, San Diego, California, 2008, pp.1010. [11] M. Cardosa, M.R. Korupolu, A. Singh, Shares and utilities based

power consolidation in virtualized server environments, IFIP/IEEE International Symposium on Integrated Network Management, Long Island, NY, USA, 2009, pp.327-334. [12] Z.H. Gong, X.H. Gu, PAC: Pattern-driven Application Consolidation for Efficient Cloud Computing, Proceeding MASCOTS’ 10 Proceedings of the 2010 IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, IEEE Computer Society Washington, DC, USA,2010, pp.24-33. [13] J. Xu, J.A.B. Fortes, Multi-Objective Virtual Machine Placement in Virtualized Data Center Environments, 2010 IEEE/ACM Int'l Conference on Green Computing and Communications & Int'l Conference on Cyber, Physical and Social Computing, Hangzhou, China, 2010, pp.179-188. [14] T.V.T. Duy, Y. Sato, Y. Inoguchi, Performance evaluation of a Green Scheduling Algorithm for energy savings in Cloud computing, 2010 IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum (IPDPSW), Atlanta, GA, 2010, pp.1-8. [15] E. Feller, L. Rilling, C. Morin, Energy-Aware Ant Colony Based Workload Placement in Clouds, Proceeding GRID’11 Proceedings of the 2011 IEEE/ACM 12th International Conference on Grid Computing, IEEE Computer Society Washington, DC, USA, 2011, pp.26-33. [16] K. Mills, J. Filliben, C. Dabrowski, Comparing VM-Placement Algorithms for On-Demand Clouds, 2011 IEEE Third International Conference on Cloud Computing Technology and Science (CloudCom), Athens, 2011, pp.91-98. [17] A. Beloglazov, J. Abawajy, R. Buyya, Energy-Aware Resource Allocation Heuristics for Efficient Management of Data Centers for Cloud Computing, Future Generation Computer Systems (FGCS), Elsevier Science, Amsterdam, May 2012, 28(5): 755-768. [18] SPECpower_ssj2008 Results res2011q1, http://www.spec.org/power_ssj2008/results/res2011q1/ [19] K.S. Park, V.S. Pai, CoMon: a mostly-scalable monitoring system for PlanetLab.ACM SIGOPS Operating SystemsReview2006, 40(1):74. [20] Amazon EC2 Instance Types, http://aws.amazon.com/ec2/instance-types/

1895