Data center energy efficient resource scheduling - CiteSeerX

Cluster Comput DOI 10.1007/s10586-014-0365-0

Data center energy efficient resource scheduling Junaid Shuja · Kashif Bilal · Sajjad Ahmad Madani · Samee U. Khan

Received: 24 September 2013 /Revised: 09 December 2013/ Accepted: 26 February 2014 © Springer Science+Business Media New York 2014

Abstract The information and communication technology (ICT) sector has grown exponentially in the recent years. An essential component of the ICT organizations is constituted by the data centers that are densely populated with redundant servers and communicational links to ensure the provision of 99.99 % availability of services; a fact responsible for the heavy energy consumption by data centers. For energy economy, the redundant elements can be powered off based on the current workload within the data center. We propose a Data Center-wide Energy-Efficient Resource Scheduling framework (DCEERS) that schedules data center resources according to the current workload of the data center. Minimum subset of resources to service the current workload are calculated by solving the Minimum Cost Multi Commodity Flow (MCMCF) using the Benders decomposition algorithm. The Benders decomposition algorithm is scalable: it solves the MCMCF problem in linear time for large data center environments. Our simulated results depict that DCEERS achieves better energy efficiency than the previous data center resource scheduling methodologies. J. Shuja Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia e-mail: [email protected] K. Bilal · S. U. Khan (B) Department of Electrical and Computer Engineering, North Dakota State University, Fargo, ND 58108-6050, USA e-mail: [email protected] K. Bilal e-mail: [email protected] S. A. Madani COMSATS Institute of Information Technology (CIIT), Abbottabad 22060 , Pakistan e-mail: [email protected]

Keywords Energy efficiency · Data centers · Resource scheduling · Cloud computing · Data intensive workloads

1 Introduction Data centers are integral to most of the Information and Communication Technology (ICT) sector organizations. Large data centers with thousands of servers have been deployed by renowned ICT organizations, like IBM, Microsoft, Amazon, and Google, to provide cloud computing services [1]. The phenomenal increase in the size and number of data centers and resultant energy utilization has been a driving force in carrying out research studies that dwell on the energy efficiency techniques, energy consumption, and future consumption estimates for data centers [2–6]. The estimates of aforecited studies agree on the future escalation in energy consumption by data centers. The study conducted by the Environmental Protection Agency (EPA) [2] reported that data centers consumed about 61 Tera Watt hour (TWh) of electricity in 2006, amounting to 1.5 % of the total electricity sales in the US the same year; an annual growth of 16 % in the preceding five years. The study estimated that the power consumption will double in every five years. The breakdown of the electricity consumption within a data center is: (a) ICT equipment (40 %), (b) cooling systems (45 %), and (c) power distribution systems (15 %). The study lists network devices to account for 5 % of the consumption of the ICT share. However, Kliazovich et al. [7] put the share of network elements as high as 33 % of the ICT equipment. The EPA study estimated that around 70 % energy savings are possible by applying the state-of-the-art efficiency measures at the cooling, airflow, power distribution, and resource management systems of the data center.

123

Cluster Comput Table 1 Annual savings in 2011 using state-of-the-art techniques ICT Component

2011 electricity usage (billion kWh)

2011 electricity usage with state-of-the-art techniques (billion kWh)

Infrastructure Network Devices Storage

42.1 4.1

18.1 1.7

4.2

1.8

Servers

33.7

14.5

Total

84.1

36.1

The data centers are also responsible for GreenHouse Gases (GHGs) emissions. Electricity production process emits large amount of GHGs, especially when fossil fuels, like coal, oil, and natural gas are used. Moreover, data center devices also emit GHGs during utilization [8]. The ICT sector is responsible for about 2 % of the worldwide GHG emissions: a quantity that the 2006 estimates [2] expect to increase by 6 %, annually. Moreover, the cooling units deployed to maintain the temperature and humidity of the data center at the operational level also emit GHGs. Therefore, the data centers are one of the major contributors to worldwide GHG emissions. Implementing energy efficient resource scheduling at the data centers can have three immediate positive consequences, viz: (a) significant decrease in the operational expenses (OPEX), (b) lesser cooling energy consumption, (c) lesser GHG emissions, and (d) lower device failure rates [2,9]. Table 1 shows the estimated electricity consumption of data center elements, for the year 2011, along with the energy savings that can be achieved using the stateof-the-art energy efficiency techniques [1]. Energy efficient resource scheduling techniques have often either only focused on server [10,11] or network [12, 13] components. Moreover, recent studies have not used formal flow models to solve complex network flow problems [7]. Heller et al. [12] have considered formal network flow model in ElasticTree, however. the heuristics used within ElasticTree have not been proved efficient for solving complex network flow problems [14]. Efficient and scalable heuristics are required to solve real world data center models with thousands of servers and communicational links. This study makes the following significant contributions. – We propose a Data Center Energy Efficient Resource Scheduling (DCEERS) framework that provides data center-wide energy savings. The data center resources are scheduled on the basis of current workload within the network flow model. – The minimum subset of required resources is calculated by solving the minimum cost multi commodity flow (MCMCF) problem. The MCMCF finds the optimal sub-

123

set of data center resources resulting in increased energy efficiency. – The DCEERS obtains optimal solution using the Benders decomposition algorithm; one of the fastest heuristic for solving complex MCMCF problems [14]. The Benders decomposition algorithm runs in linear time, thus the DCEERS responds quickly to workload fluctuations. – We compared DCEERS with GreenCloud [7], DENS [15], and ElasticTree [12]. DCEERS provides better energy efficiency than the aforecited resource scheduling methodologies. – We demonstrate that class base queuing can lead to lesser end-to-end delay for Data Intensive Workloads (DIW). The rest of the paper is organized as follows. Section 2 presents the basics of tiered data center network architectures, energy efficiency techniques, data center workloads, and related work. The details of DCEERS framework and the Benders decomposition algorithm for solving the MCMCF are presented in Sect. 3. Section 4 presents scalability comparison, energy efficiency, server workload distribution, and network parameters of various energy efficiency methodologies.

2 Background 2.1 Data center network architectures Typically, the data center network architectures are treebased and may be two-tiered, three-tiered, and three-tiered high speed [7]. These network architectures are deployed according to the size and Quality of Service (QoS) requirements of the applications hosted by the data center. An illustration of the three-tiered data center architecture is drawn in Fig. 1 (after [15]). In general, network switches are interlinked across other tiers and also connected with switches in the same tier. The servers are connected to the access layer via Top-of-Rack (ToR) switches. The access layer is connected to the core layer which provides connectivity to the backbone network. The three-tiered network architectures have aggregate layer between access and core networks. The aggregate layer provides content switching, load balancing, and various security measures such as firewall capability [16]. A three-tiered high-speed architecture provides 10 times more bandwidth at all layers through high speed (10 Gigabit Ethernet and 100 Gigabit Ethernet) transceivers. The redundant paths from one layer to another provide fault tolerance, load balancing, and availability. These multiple paths are limited by the EqualCost Multi-Path routing ECMP [17] implementation. ECMP routing is used to balance the load of traffic from one tier to another [16]. The ECMP routing performs load balancing by

Cluster Comput

Fig. 1 Three-tier (3T) network architecture.

calculating the hash of layer 2 (L2) and layer 3 (L3) headers on per-flow basis. The ECMP routing also limits to the maximum number of core routers in a tiered architecture to 16 by 16-way ECMP implementation [18]. High-end enterprise switches commonly deployed in data center environments support ECMP routing [16]. The difference in the uplink and downlink capacities of a switch leads to oversubscription that reduces the available cross-sectional bandwidth of a server. In tree based network architecture, the oversubscription ratio is usually higher than 1 which implies that the actual throughput is always less than the theoretical throughput [16]. Conventional data centers deploy modular switches in which modules in the form of linecards (ports) can be added on demand. The switching fabric of each linecard is also a source of oversubscription, which can range from 1:1 to 4:1 depending on the hardware specifications [16]. Researchers have proposed methodologies to achieve 1:1 oversubscription ratio, which does not limit the aggregate throughput of the data center by deploying commodity switches and unconventional data center network architectures [19,20]. These network architectures adopt mesh, CLOS, and hierarchical topologies, which accommodate more links and switches at the access and aggregate layers to achieve 1:1 oversubscription. Moreover, unconventional data center networks adapt routing protocols to support such architectures [21,18,22]. The only limitation of these network architectures is that they are restricted to research and not deployed in any real world scenarios [7]. Some network architectures are specifically proposed to provide energy efficiency in data center networks [12]. These proposals require changes in network architectures, hardware, and protocols, which are hard to accomplish in an operational data center environments [23].

(DVFS) [15,24]. Resource consolidation is further categorized into: (a) virtualization, and (b) workload consolidation. Virtualization is the most adopted energy efficiency technique in data center environments [25]. Virtualization aims to consolidate data center workload on a minimum number of physical servers using virtual machine live migration in order to provide energy efficiency. The server and memory resources are dynamically acquired according to the fluctuating QoS requirements of different applications hosted by the virtual machines (VM) [26]. Workload consolidation consolidates data center workload on minimum number of physical servers so the rest of servers can be powered off. Most of resource consolidation strategies only consider servers for energy optimization as powering off network elements is considered taboo due to performance constraints [7,27]. Resource consolidation strategies require an Autonomic Power Manager (APM) to dynamically power on/off data center elements according to workload requirements. The DVFS techniques are based on the fact that data center elements can be switched to low power state by scaling either input voltage or switching frequency. The DVFS technique requires hardware support and Advanced Configuration and Power Interface (ACPI) standard implementation. DVFS techniques have been implemented in both the server [28] and switch domains [29]. Switches operate a link at lower frequency such that it provides the required bandwidth while consuming lesser energy. DCEERS is a workload consolidation technique that optimizes set of active data center resources according to current workload for energy efficiency.

2.2 Energy efficiency techniques

Modern data centers provide different services like web applications hosting, Video on Demand (VoD), content sharing, and cloud computing facilities [15]. These services have different computational and communicational requirements. The workloads at a data center may be classified into three

The energy efficiency techniques, employed at data centers, can be broadly classified into two categories: (a) resource consolidation, and (b) Dynamic Voltage/Frequency Scaling

2.3 Workload classification

123

Cluster Comput

major categories [7,30]: (a) Balanced Workload, (b) Computation Intensive Workload (CIW), and (c) Data Intensive Workload (DIW). Balanced workloads are generated by applications that have both communicational and computational requirements such as, geographical information systems. CIW are generated by high performance computing (HPC) applications. Data centers hosting such applications have high demands of computational power (servers), while communicational requirements are minimal. Energy efficiency techniques in such data centers focus on switches as servers must be powered on to meet the computational requirements. An Example of a DIW is the one generated by content and VoD applications. YouTube is one of the largest such user-generated content (UGC) application of VoD data [31]. The DIW require high bandwidth and lesser end-to-end delay for efficient data transfer, while the computational requirements are minimal. Energy efficiency techniques in such data centers focus on servers as the switches have to be powered on to meet the high data transfer requirements [32–34].

tion of resources based on utilization levels of different components [15]. Flow scheduling requires switch reconfigurations that can be accomplished by specific switch design (OpenFlow [46]) while utilization based resource selection requires complex controller and workload predictor [7]. ElasticTree [12] selects minimum power subsets of data center network calculated by three heuristic strategies and requires OpenFlow [46] switches to update the routing paths and avoid network performance degradation due to powered off switches. Compared with all data center resource scheduling frameworks listed in this section, DCEERS: (a) encompasses both server and network components contrary to server centric [10,47] or network centric [12,13] approaches, (b) saves almost 70 % of energy in the data center environment, and (c) DCEERS adopts quikly to workload fluctuations as the Benders decomposition solves the optimization problem in linear time.

2.4 Related work

3.1 DCEERS framework

Resource scheduling techniques in the data centers can be generally classified into: (a) virtualized resource scheduling [35,36], (b) thermal-aware resource scheduling [37–39], and (c) network-aware resource scheduling [12,21,15]. The virtualization often leads to non-optimal placement of virtual machines (VMs) inside the data center network: a situation in which two VMs with large mutual bandwidth are placed at multi-hop distance from each other in the data center network [40]. Such phenomenon increases the amount of intra-data center traffic and effects the performance of the data center [41]. The optimal placement of VMs within a data center network is a NP-hard problem [40] and requires live migration of VMs for resource consolidation. The VM live migration further effects the data center network performance as large amount of VM data is migrated from one physical host to another through same network infrastructure that provides services to the clients [42]. Cooling infrastructure of the data center consumes almost same amount of energy as the computing infrastructure [37]. Most of the research in thermal-aware data centers emphasizes on workload assignment to resources [38,39] according to the thermal map of the data center. Thermal-aware resource scheduling techniques require thermal sensors [38], CARC and air flow management [43], data center physical design considerations [44], and geographical location selection for data centers [45]. All these proposals require changes in basic cyber-physical infrastructure of the operational data center environments. Network aware resource scheduling techniques are either based on flow scheduling [12,21] or selec-

In this subsection we present the implementation framework of DCEERS for data center environments. The data centerwide energy efficient resource scheduling (DCEERS) framework minimizes the energy consumed by scheduling only a subset of data center resources required for the current workload. The operational flow of DCEERS framework is depicted in Fig. 2. The DCEERS framework consists of two basic modules: (a) workload calculator, and (b) the resource/ priority scheduler. The workload calculator calculates the core bandwidth required based on: (a) the number of distinct user requests measured at the front-end servers, and (b) the required bandwidth per request. The core bandwidth required, bwcor e−r eq , is calculated from Eq. 1.

123

3 The proposed strategy

Fig. 2 DCEERS framework

Cluster Comput

bwcor e−r eq =

n 

(r equests × bw per −r equest )

(1)

i=1

Video streaming websites such as YouTube deliver DIWs to the users in form of Video on Demand (VoD) data. The throughput per second for VoD data ranges from 700 Kbps to 1200 Kbps [48]. Each user generated request is received at the data center by a front end servers. The front end server calculates the video content and the required bandwidth per second from the request URL and maps the requested video to one of the data center server hosting the content. The resource scheduler schedules minimum number of resources based on the core bandwidth required, bwcor e−r eq . In case bwcor e−r eq is greater than the data center core capacity, bwcor e , then the data center is already serving more users than its capacity and no energy savings are possible. Powering off data center resources in such a throughput demand scenario can affect the Service Level Agreements (SLAs) and end to end delays of the videos. In case when throughput demand, bwcor e−r eq , is less than the data center core capacity, bwcor e , the resource optimizer finds the minimum number of server and network resources required to fulfill the throughput demand. The rest of the resources can be powered off to achieve energy efficiency. Studies have reported that the average workload on a data center remains around 30 % of its peak potential, therefore, there is always a possibilty of resource optimization [25]. The resource optimizer optimizes the resources by solving the Mixed Integer Linear Programming (MILP) formulation of the MCMCF. The resource optimizer takes two parameters as input: (a) the throughput demand calculated by the workload calculator, and (b) the oversubscription ratio in the data center network. The throughput demand and the oversubscription ratio define the constraints on the MCMCF optimization problem described in Sect. 3.2. The MCMCF problem finds the minimum subset of vertices and corresponding links to satisfy a set of flows through a network. In our scenario, the minimum subset of vertices and links are the energy efficient solution in response to a particular workload within the data center. The MCMCF optimization generates minimum subset of data center resources required that results in resource optimization. The MCMCF problem concentrates a set of flow on minimum links. We assume that a flow can be satisfied by any server inside the data center. The heuristic solutions to MCMCF problem do not scale with large data center models with thousands of servers. The DCEERS uses the Benders decomposition algorithm [14] to solve MCMCF problem is linear time. Moreover, the DCEERS priority scheduler schedules packets based on assigned priority class. DIWs have the requirement of lesser end-to-end delay to maintain user satisfaction and SLAs.

DCEERS framework is simulated using the GreenCloud simulator [7]. Previous methodologies, GreenCloud [7] and DENS [15] did not consider a network flow model for data center resource optimization. ElasticTree [12] utilized network flow models for data center resource optimization, however, the affect of oversubscription ratio and ECMP routing paths was not considered. The oversubscription ratio considerably limits the available bandwidth within the data center network. In tiered data center architectures, oversubscription ratio varies between any two tiers. The available bandwidth from access to aggregate tier depends on: (a) capacity of outbound links from access tier, and (b) capacity of outbound links of aggregate tier [49]. Moreover, the ECMP routing protocol limits the multiplicity of outbound paths from a switch to two for access layer and sixteen for aggregate layer switch [18]. Based on these constraints, we modeled the data center as a multi-layered graph with the MCMCF optimizing the number of graph vertices and links. The oversubscription ratio and ECMP routing constraints increase the complexity of the MCMCF optimization, therefore, Benders decomposition heuristic is utilize to solve the optimization problem in linear time. The Benders decomposition provides a more scalable solution to the MCMCF problem than the Topology-ware and Greedy heuristics proposed in ElasticTree [12]. Although the graph modeling approach is simplistic, the constraints defined over the MCMCF problem capture the actual data center dynamics. Additional to these changes from previous methodologies, we have minimized the end-to-end delay of DIW with class based queuing. Moreover, recent studies have not discussed the effects of resource optimization on network parameters. We studied the affect of resource optimization on instant throughput and end-to-end packet delay. 3.2 Data center flow model The tiered data center architecture can be modeled is a multilayered graph with set of links and nodes [14]. The multi commodity flow inside the data center considering oversubscription and ECMP constraints describes the flow model. The data center is modeled as a flow network represented by a directed graph, G (N, A) with set of nodes N , set of links (i, j) ∈ A and link capacities ti j . Let Nser ver and Nswitch be subsets of the set N . At a particular instance, k ∈ K commodities (user demands) have to be transported from source servers rk to sink core switches sk for demand dk . We define ρ = (i, j) ∈ A as an alternative representation for a link. For each link ρ, the associated cost is bi j and for each node i ∈ N the associated cost is cn . A flow that satisfies all flow demands, dk , while transporting commodities (user demands) from source servers rk to core switches sk subject to the capacity constraint is called multi commodity flow (MCF). A minimum cost multi commodity flow (MCMCF) is

123

Cluster Comput

an optimization problem that minimizes the cost of links and nodes while providing the multi commodity flow. We define   two sets of links F Si (A ) and RSi (A ), such that A ⊆ A, for links exiting node i and set of links entering node i respectively. F Si (A ) = {ρ ∈ A : ρ = (i, j) for some j ∈ N }, ∀i ∈ N (2) 



RSi (A ) = {ρ ∈ A : ρ = ( j, i) for some j ∈ N }, ∀i ∈ N (3) We also define F Si max (A ) as the maximum number of links exiting from node i. We define two binary decision variables, xρ = 0 for a link inclusion or 1 otherwise

(4)

wn = 0 for a node inclusion or1 otherwise

(5)

We define yρ k as the amount of flow through link ρ for the kth origin-destination demand. The objective function is, min z = ρ∈A xρ × bi j + i∈N wn × cn

(6)

subject to, ∀ρ ∈ A, k∈K yρ k ≤ tρ xρ

(7)

∀k ∈ K , ρ∈F Sr k (A ) yρ k − ρ∈R Sr k (A ) yρ k = dk

(8)

∀ρ ∈ A, ∀k ∈ K , yρ k ≥ 0

(9)

∀ρ ∈ A, ∀k ∈ K , yρ k = yρ k ÷ O

(10)

∀Nswitches ∈ N , F Si max (A ) ≤ 16||2

(11)



Eq. 7– 11 define the set of constraints on the multi commodity flow of the data center network. Eq. 7 is the link capacity constraint, which states that the flow through a link cannot exceed link capacity, and powered off links have no flow. Eq. 8 is the demand satisfaction constraint, which states that the source and sink nodes send and receive flow equal to demand. Eq. 9 enforces non negativity on the amount of flow. Eq. 10 states that the flow through each link is divided by the corresponding oversubscription ratio. Eq. 11 defines maximum allowable exiting links F Si max (A ) from a switch. The maximum allowable exiting links from an aggregate switch is sixteen and constrained by the ECMP routing protocol [18], while maximum allowable exiting links from a access switch is two. 3.3 Benders decomposition algorithm The mixed integer linear programming (MILP) formulation of the MCMCF defined in previous section is solved by a Benders decomposition algorithm. Integer solution of MCMCF problem is NP-hard [50]. Researchers have proposed many heuristics and approximation algorithms to solve MCMCF problem in linear time while trading optimality of the solution [51,52]. Studies have proved computational efficiency

123

of the Benders decomposition algorithm in solving complex MCMCF problems [14,50]. The Benders decomposition algorithm decomposes the original optimization problem into a master problem (MP) and multiple sub problems (SP). The computational efficiency of the Benders decomposition algorithm applied to various optimization problems lies in the selection of the MP with initial constraints and variables while considering the problem domain. For MCMCF problems the addition of cover constraints to the original optimization problem has been proved to provide time efficiency [14]. We devised a cover constraint such that the total capacity of arcs coming out of a origin node must not be less than the origin-destination demand for that node. ∀ρ ∈ A, k∈K dk ≤ tρ xρ

(12)

The cover constraint is used as a preprocessor to the Benders decomposition algorithm. The Eq. 12 as a preprocessing step generates the values of xρ , i.e., we get the arcs that are part of the optimal solution. Further, we initialize the MP with constraint Eq. 7, i.e., the values of the variable xρ obtained from cover constraint (Eq. 12) are used to obtain values of yρ k  . A constraint is added to the MP at each iteration. The primal and dual formulation of the master and the sub-problems at any instance give the upper and the lower bound to the original problem respectively. We solve the sub-problem dual for the initial assignment of yρ k  during each iteration. If at the end of an iteration, the sub-problem does not have feasible solution, then the MP does not have the feasible solution and algorithm is terminated. Otherwise let β ≥ z  be the solution to the dual of sub-problem and β yρ k  (yρ k) be the bounding function. The value of yρ k l is updated and a constraint is added to the MP after each iteration. For detailed study of primal and dual formulations of the MP, please see [53]. The Benders Decomposition algorithm for finding the MCMCF in the Data Center Network is given below.

4 Simulation results 4.1 Simulation setup We utilized GreenCloud simulator [7] and IBM CPLEX 12.01 for performance evaluation of data center resource optimization heuristics. GreenCloud is a packet-level simulator based on Ns22 that captures packet level details of simulations in trace files. The GreenCloud simulator supports different tiered data center architectures described in Sect. 2.1. 1

http://www.ilog.com

2

http://www.isi.edu/nsnam/ns/

Cluster Comput

Algorithm 1 Benders Decomposition Algorithm for Data Centers 1: Preprocessor: Use cover constraint to obtain the value of xρ  2: Initialize yρ k in original problem  3: z ← −∞ 4: l ← 0  5: while sub problem dual has feasible solution β ≥ z do  6: Derive lower bound function β yρ k  (yρ k) with β yρ k  (yρ k ) ← β 7: l ← l + 1 {Add another constraint}  8: yρ k l ← yρ k 9: Add z ≥ β yρ k  (yρ k) to master problem 10: end while 11: if master problem is infeasible then 12: Stop the original problem is infeasible 13: else   14: Let (z , yρ k ) be the optimal value and solution to the master problem 15: end if   16: Return (z , yρ k )

Table 2 Energy consumption parameters Parameters

Power consumption (W) Servers

Server peak

301 Switches Access network

Core and aggregate network

Chassis

146

1.5K (10G)

15K (100G)

Linecard

-

1K (10G)

12K (100G)

Port transceiver

0.42

0.3K (10G)

1.6K (100G)

Moreover, GreenCloud incorporates detailed energy models of data center resources. GreenCloud is used to evaluate energy efficiency, throughput, end-to-end delay, and packet loss ratios of resource optimization heuristics. The parameters for energy consumption calculation adopted from standard server and switch devices are given in Table 2 [7]. IBM CPLEX is an optimization tool for solving constrained optimization problems. IBM CPLEX is used to evaluate scalability of resource optimization heuristics. All heuristics were simulated on a system of 2.4 G Hz processor and 2 GB memory. Data center workloads tend to depict sine-wave demand that fluctuates at specific hours during a day, specific days during a week and specific weeks during a month [54]. However, average data center workload remains at 30 % of data center capacity [7]. As data center traffic traces are not available publicly [21], most of studies have emulated traffic distributions used in published research [12,54–56]. We evaluated resource optimization heuristics on varying data center workloads. The optimization heuristics work on two data file that specify the data center topology and work load

in the from of arcs and nodes. The arc data file contains four columns (from, to, cost, capacity), while the node data file contains two columns (supply, demand). The workload in the GreenCloud simulations was varied with the help of dcw or kload parameter. The dcw or kload parameter changes the number of cloud users and cloud application to vary data center workload from 0–100 %. 4.2 Scalability results We evaluated four heuristics of data center resource optimization using two different approaches. The MCMCF problem was solved with the built-in linear programming (LP) optimization technique of CPLEX while rest of heuristics were implemented in C callable libraries. The time complexity of the LP technique becomes intractable, (O(n 3.5 )), for large data center scenarios with thousands of nodes and arcs, thus, making it infeasible for practical deployments. Therefore, the MCMCF problems in large-scale data centers need optimization heuristics to bound the solution in linear time. The DENS methodology [15] works by selecting best fit data center resources with sufficient communicational capability. Heller et al. [12] proposed a data center flow model and used MCMCF optimization to obtain minimum subset of required data center device resources. Greedy and topologyaware heuristics were used to solve MCMCF optimization in data centers [12]. Greedy heuristic solves the optimization problem in approximately O(n 2 ) while topology-aware heuristic takes O ≈ (n) time. We have used a more classical and proven approach to solving MCMCF problem, the Benders decomposition algorithm with additional cover constraints. The heuristics optimization of MCMCF problem results in a near optimal network topology subset which is equivalent to optimized data center resources. The simulations proved that the Benders decomposition runs in linear time and faster than the topology-aware heuristic for largescale data centers. The comparison of the time complexity of

Fig. 3 Time complexity of data center energy efficient heuristics

123

Cluster Comput Fig. 4 Energy efficiency comparison of tiered data center architectures at 30 % data center workload

DENS, greedy, topology-aware, and the Benders decomposition algorithm is presented in Fig. 3. Data center application demands change over time. Data center workload prediction can help schedule the required resources in advance to avoid any performance penalty.

123

DCEERS response time to change in workload depends on, (a) time required to compute the throughput demand, (b) time required to compute optimal data center resource subset using a heuristic, and (c) time required for servers and switches to boot according to the heuristic-based optimal data center

Cluster Comput Fig. 5 Energy consumption on varying workload

subset. The time required to calculate the throughput demand from Eq. 1 is negligible. Moreover, the time required to boot the data center servers and switches ranges from 30-185 seconds for low end systems to high end systems [12]. Furthermore, time complexity of the underlying MCMCF heuristic algorithm is the only varying factor in DCEERS response time. Therefore, for large data center environments, finding efficient and linear time heuristics for solving MCMCF problems greatly reduces the time response of the resource optimization frameworks. The Benders decomposition proves to be the fastest heuristic to solve MCMCF optimization in data center environments [14].

4.3 Energy efficiency results We evaluated energy efficiency of optimized network topology subsets obtained from: (a) Topology-aware heuristic, (b) Greedy heuristic [12], (c) GreenCloud Scheduler [7], and (d) Benders (DCEERS). The detailed energy consumption results of server and switch devices for the three tiered data center architectures on 30 % data center workload are shown in Fig. 4. The DCEERS achieved better energy efficiency than other methodologies by concentrating the flows on a minimum number of links and switches. If a link is underutilized, the flows through the link are transfered to adjoining links and the link is powered off. The server and switch devices are also powered off in case they are underutilized. The MCMCF optimal solution finds the minimum subset of data center resources. The mixed integer linear programming (MILP) formulation of MCMCF helps eliminate underutilized resources: the solution set will either contain a device with ≈ 100 % utilization, or the device will not be part of the optimal solution. Hence, the workload is transferred to a minimum subset and rest of devices are powered off. The

GreenCloud methodology operates devices in three states: power on, power off, and power scaled. The power scaled devices can be switched to power on state with minimum downtime as compared to power off devices. However, power scaled devices still consume energy. DCEERS does not operate devices in power scaled state, thus, achieves better energy efficiency than the GreenCloud methodology. Similarly, solution sets of greedy and topology-aware heuristics contain devices in power scaled state that results in higher energy consumptions. The workload on data center does not remain constant and changes with time. Energy efficiency comparison of different heurtistics on varying workloads in provided in Fig 5. The DENS methodology is not compared as it is less energy efficient than the GreenCloud [15]. Data center resources can be optimized efficiently for smaller workload as lesser number of flows can be concentrated on energy efficient subset of resources. However, most of the data center devices have to be powered on to satisfy the throughput demand as it increases.

4.4 Performance results We studied the effects of DCEERS methodology on network parameters through the trace files generated by GreenCloud simulations. Any energy efficiency scheme in data center scenario ought to affect the average throughput and end-to-end delay of the packets. The throughput of a normal data center is compared with the throughput of DCEERS and GreenCloud schedulers in Fig. 6. The instant throughput results were generated for 30 % data center worklaod. The throughput decreases gradually with the decrease in number of powered on devices. The end-to-end delay of packets ought to increase in any energy efficient scenario as some switches are switched off.

123

Cluster Comput

Fig. 6 Instant throughput

The increase in end-to-end delay of the packets is not good for data centers providing DIW. DIW have the requirement of greater throughput and lesser end-to-end delay. DCEERS implements a priority scheduling scheme for DIW to lessen its end-to-end delay. The data was divided into three classes: video, audio, and data class, for Class Based Queuing (CBQ) implementation. The video class was given the highest priority. For comparison of average end-to-end delay we evaluated three scenarios: (a) FIFO: Round-Robin resource scheduler with no packet priority scheduling, (b) Priority schedFig. 7 Average end-to-end delay

Fig. 8 Packet delivery ratio

123

uler: Round-Robin resource scheduler with packet priority scheduling, and (c) Priority scheduler + Resource scheduler: DCEERS with packet priority scheduling. The results of average end-to-end delay for the three scenarios are presented in Fig. 7 against different packet sizes. The results of priority scheduling with other resource scheduling methodologies were similar. The results show that the average endto-end delay is largest for FIFO scheduling and least for priority scheduling. The case of Priority scheduler + Resource scheduler advocates the use of priority scheduling in energy efficient scenarios: the average end-to-end delay with priority scheduling and lesser number of resources is still lower than that of FIFO. The reason of higher delay in case of Priority scheduler + Resource scheduler is that some switches and links have been powered off for energy efficiency purpose. The packet delivery ratio decreases about 3–5 % in CBQ scenarios. We have compared three packet scheduling scenarios: (a) FIFO scheduling (b) priority scheduling scenario, and (c) Priority scheduler + Resource scheduler (DCEERS). Fig. 8 shows that the packet delivery ratio is affected more in the later scenario as the number of switches and links are reduced for energy efficiency.

Cluster Comput

5 Conclusion Data center power consumptions have increased in recent years due to increase in size and number of data centers. We proposed a data center wide energy efficient resource scheduling framework (DCEERS) that schedules resources according to the throughput demand. We modeled the data center as a multi commodity flow network with multiple source and sink nodes and defined constraints on flow from one tier to another. The optimization of minimum cost multi commodity flow inside the data center gives the minimum number of data center resources required for the current workload demand. The Benders decomposition algorithm, one of the fastest heuristic for complex MCMCF problems, was used to find the optimal solution for scalable data center environments. Simulation results show that DCEERS framework saves more energy as compared to other heuristics. We demonstrated that the average end-to-end delay can be reduced by priority scheduling of DIWs, although it effects the packet delivery ratio. The throughput varies for resource scheduling heuristics depending on number of resources in optimal subset. These effects can be considered as tradeoff for energy efficiency. We simulated our algorithm on the GreenCloud simulator for all data center network architectures to prove the consistency of our results.

References 1. Masanet, E., Brown, R., Shehabi, A., Koomey, J., Nordman, B.: Estimating the energy use and efficiency potential of u.s. data centers. Proc. IEEE 99(8), 1440–1453 (2011) 2. Brown, R.: Report to congress on server and data center energy efficiency public law 109–431. Environ. Prot. 109, 431 (2007) 3. Bolla, R., Davoli, F., Bruschi, R., Christensen, K., Cucchietti, F., Singh, S.: The potential impact of green technologies in nextgeneration wireline networks: Is there room for energy saving optimization? Commun. Mag. IEEE 49(8), 80–86 (2011) 4. Webb, M.: SMART 2020: Enabling the low carbon economy in the information age. Tech. rep, Climate Group on behalf of the Global eSustainability Initiative (GeSI) (2008) 5. Koomey, J.G.: Worldwide electricity used in data centers. Environ. Res. Lett., IOPscience 3(3), 1–8 (2008). 6. Wang, L., Khan, S.U.: Review of performance metrics for green data centers: a taxonomy study. J. Supercomput. 63(3), 639–656 (2013) 7. Kliazovich, D., Bouvry, P., Audzevich, Y., Khan, S.: Greencloud: A packet-level simulator of energy-aware cloud computing data centers. In: GLOBECOM 2010, 2010 IEEE Global Telecommunications Conference, pp. 1–5 (2010). 8. Despins, C., Labeau, F., Le Ngoc, T., Labelle, R., Cheriet, M., Thibeault, C., Gagnon, F., Leon-Garcia, A., Cherkaoui, O., St Arnaud, B., et al.: Leveraging green communications for carbon emission reductions: Techniques, testbeds, and emerging carbon footprint standards. Communications Magazine, IEEE 49(8), 101– 109 (2011). 9. Khan, S.U., Wang, L., Yang, L.T., Xia, F.: Green computing and communications. J. Supercomput. 63(3), 637–638 (2013)

10. Khargharia, B., Hariri, S., Szidarovszky, F., Houri, M., El-Rewini, H., Khan, S., Ahmad, I., Yousif, M.: Autonomic power performance management for large-scale data centers. In: Parallel and Distributed Processing Symposium, 2007. IPDPS 2007. IEEE, International, pp. 1–8 (2007). 11. Wang, L., Khan, S.U., Chen, D., Kołodziej, J., Ranjan, R., Xu, Cz, Zomaya, A.: Energy-aware parallel task scheduling in a cluster. Future Gener. Comput. Syst. 29(7), 1661–1670 (2013) 12. Heller, B., Seetharaman, S., Mahadevan, P., Yiakoumis, Y., Sharma, P., Banerjee, S., Mckeown, N.: Elastictree: Saving energy in data center networks. In: IN NSDI (2010). 13. Da Costa, G., Gelas, J.P., Georgiou, Y., Lefevre, L., Orgerie, A.C., Pierson, J.M., Richard, O., Sharma, K.: The green-net framework: Energy efficiency in large scale distributed systems. In: Parallel Distributed Processing, 2009. IPDPS 2009. IEEE International Symposium on, pp. 1–8 (2009). 14. Garg, M., Smith, J.C.: Models and algorithms for the design of survivable multicommodity flow networks with general failure scenarios. Omega 36(6), 1057–1071 (2008) 15. Kliazovich, D., Bouvry, P., Khan, S.: Dens: Data center energyefficient network-aware scheduling. In: Green Computing and Communications (GreenCom), 2010 IEEE/ACM Int’l Conference on Int’l Conference on Cyber, Physical and Social Computing (CPSCom), pp. 69–75 (2010). 16. Cisco: Cisco data center infrastructure 2.5 design guide. Cisco press (2007). 17. Raiciu, C., Barre, S., Pluntke, C., Greenhalgh, A., Wischik, D., Handley, M.: Improving datacenter performance and robustness with multipath tcp. SIGCOMM Comput. Commun. Rev. 41, 266– 277 (2011) 18. Al-Fares, M., Loukissas, A., Vahdat, A.: A scalable, commodity data center network architecture. SIGCOMM Comput. Commun. Rev. 38, 63–74 (2008) 19. Greenberg, A., Hamilton, J.R., Jain, N., Kandula, S., Kim, C., Lahiri, P., Maltz, D.A., Patel, P., Sengupta, S.: Vl2: a scalable and flexible data center network. Proceedings of the ACM SIGCOMM 2009 conference on Data communication. SIGCOMM ’09, pp. 51– 62. ACM, New York, NY, USA (2009) 20. Bilal, K., Khan, S.U., Zhang, L., Li, H., Hayat, K., Madani, S.A., Min-Allah, N., Wang, L., Chen, D., Iqbal, M., et al.: Quantitative comparisons of the state-of-the-art data center architectures. Concurr. Comput.: Pract. Exp. 25(12), 1771–1783 (2013) 21. Al-fares, M., Radhakrishnan, S., Raghavan, B., Huang, N., Vahdat, A.: Hedera: Dynamic flow scheduling for data center networks. In: In Proceedings of Networked Systems Design and Implementation (NSDI) Symposium (2010). 22. Greenberg, A., Hamilton, J., Maltz, D.A., Patel, P.: The cost of a cloud: research problems in data center networks. SIGCOMM Comput. Commun. Rev. 39, 68–73 (2008) 23. Bilal, K., Malik, S.U.R., Khalid, O., Hameed, A., Alvarez, E., Wijaysekara, V., Irfan, R., Shrestha, S., Dwivedy, D., Ali, M., et al.: A taxonomy and survey on green data center networks. Future Gener. Comput. Syst. (2013). 24. Valentini, G.L., Lassonde, W., Khan, S.U., Min-Allah, N., Madani, S.A., Li, J., Zhang, L., Wang, L., Ghani, N., Kolodziej, J., et al.: An overview of energy efficiency techniques in cluster computing systems. Clust. Comput. 16(1), 3–15 (2013) 25. Liu, J., Zhao, F., Liu, X., He, W.: Challenges towards elastic power management in internet data centers. In: Distributed Computing Systems Workshops, 2009. ICDCS Workshops ’09. 29th IEEE International Conference on, pp. 65–72 (2009). 26. Wang, L., von Laszewski, G., Tao, J., Kunze, M.: Grid virtualization engine: design, implementation, and evaluation. Syst. J. IEEE 3(4), 477–488 (2009)

123

Cluster Comput 27. Kołodziej, J., Khan, S.U., Wang, L., Byrski, A., Min-Allah, N., Madani, S.A.: Hierarchical genetic-based grid scheduling with energy optimization. Clust. Comput. 16(3), 591–609 (2013) 28. Horvath, T., Abdelzaher, T., Skadron, K., Liu, X.: Dynamic voltage scaling in multitier web servers with end-to-end delay control. Comput. IEEE Trans. 56(4), 444–458 (2007) 29. Bilal, K., Khan, S., Madani, S., Hayat, K., Khan, M., Min-Allah, N., Kolodziej, J., Wang, L., Zeadally, S., Chen, D.: A survey on green communications using adaptive link rate. Clust. Comput. 16(3), 575–589 (2013) 30. Shuja, J., Madani, S.A., Bilal, K., Hayat, K., Khan, S.U., Sarwar, S.: Energy-efficient data centers. Computing 94(12), 973–994 (2012) 31. Cha, M., Kwak, H., Rodriguez, P., Ahn, Y.Y., Moon, S.: I tube, you tube, everybody tubes: analyzing the world’s largest user generated content video system. In: Proceedings of the 7th ACM SIGCOMM conference on Internet measurement. IMC ’07, pp. 1–14. ACM, New York, NY, USA (2007) 32. Meisner, D., Sadler, C.M., Barroso, L.A., Weber, W.D., Wenisch, T.F.: Power management of online data-intensive services. In: Computer Architecture (ISCA), 2011 38th Annual International Symposium on, pp. 319–330. IEEE (2011). 33. Wang, L., Tao, J., Ranjan, R., Marten, H., Streit, A., Chen, J., Chen, D.: G-hadoop: Mapreduce across distributed data centers for dataintensive computing. Future Gener. Comput. Syst. 29(3), 739–750 (2013) 34. Wang, L., Tao, J., Marten, H., Streit, A., Khan, S.U., Kolodziej, J., Chen, D.: Mapreduce across distributed clusters for data-intensive applications. In: Parallel and Distributed Processing Symposium Workshops & PhD Forum (IPDPSW), 2012 IEEE 26th International, pp. 2004–2011. IEEE (2012). 35. Urgaonkar, R., Kozat, U.C., Igarashi, K., Neely, M.J.: Dynamic resource allocation and power management in virtualized data centers. In: NOMS, pp. 479–486 (2010). 36. Beloglazov, A., Buyya, R.: Energy efficient resource management in virtualized cloud data centers. In: Cluster, Cloud and Grid Computing (CCGrid), 2010 10th IEEE/ACM International Conference on, pp. 826–831 (2010). 37. Wang, L., Khan, S., Dayal, J.: Thermal aware workload placement with task-temperature profiles in a data center. J. Supercomput. 61(3), 780–803 (2012) 38. Wang, L., von Laszewski, G., Dayal, J., He, X., Younge, A.J., Furlani, T.R.: Towards thermal aware workload scheduling in a data center. In: Proceedings of the 2009 10th International Symposium on Pervasive Systems. Algorithms, and Networks, ISPAN ’09, pp. 116–122. IEEE Computer Society, Washington, DC, USA (2009) 39. Wang, L., von Laszewski, G., Huang, F., Dayal, J., Frulani, T., Fox, G.: Task scheduling with ann-based temperature prediction in a data center: a simulation-based study. Eng. Comput. 27(4), 381–391 (2011) 40. Meng, X., Pappas, V., Zhang, L.: Improving the scalability of data center networks with traffic-aware virtual machine placement. In: INFOCOM, 2010 Proceedings IEEE, pp. 1–9 (2010). 41. Wang, G., Ng, T.S.E.: The impact of virtualization on network performance of amazon ec2 data center. In: Proceedings of the 29th conference on Information communications. INFOCOM’10, pp. 1163–1171. IEEE Press, Piscataway, NJ, USA (2010) 42. Voorsluys, W., Broberg, J., Venugopal, S., Buyya, R.: Cost of virtual machine live migration in clouds: A performance evaluation. In: Proceedings of the 1st International Conference on Cloud Computing. CloudCom ’09, pp. 254–265. Springer-Verlag, Berlin, Heidelberg (2009) 43. Banerjee, A., Mukherjee, T., Varsamopoulos, G., Gupta, S.: Cooling-aware and thermal-aware workload placement for green hpc data centers. In: Green Computing Conference, 2010 International, pp. 245–256 (2010).

123

44. Parolini, L., Sinopoli, B., Krogh, B.H., Wang, Z.: A cyber-physical systems approach to data center modeling and control for energy efficiency. Proc. IEEE PP(99), 1–15 (2011). 45. Le, K., Bianchini, R., Martonosi, M., Nguyen, T.: Cost-and energyaware load distribution across data centers. Proc. HotPower pp. 1–5 (2009). 46. McKeown, N., Anderson, T., Balakrishnan, H., Parulkar, G., Peterson, L., Rexford, J., Shenker, S., Turner, J.: Openflow: enabling innovation in campus networks. SIGCOMM Comput. Commun. Rev. 38(2), 69–74 (2008) 47. Wang, L., Tao, J., von Laszewski, G., Marten, H.: Multicores in cloud computing: research challenges for applications. J. Comput. 5(6), 958–964 (2010) 48. Zink, M., Suh, K., Gu, Y., Kurose, J.: Characteristics of youtube network traffic at a campus network-measurements, models, and implications. Comput. Netw. 53(4), 501–514 (2009) 49. Bilal, K., Manzano, M., Khan, S.U., Calle, E., Li, K., Zomaya, A.Y.: On the characterization of the structural robustness of data center networks. IEEE Transaction on Cloud Computing (2013). 50. Geoffrion, A.M., Graves, G.W.: Multicommodity distribution system design by benders decomposition. Manag. Sci. 20(5), 822–844 (1974) 51. Multicommodity network flow models and algorithms in telecommunications. Handbook of optimization in telecommunications, pp. 163–184. Springer, New York (2006) 52. Wang, L., Anta, A.F., Zhang, F., Hou, C., Liu, Z.: Routing for energy minimization with discrete cost functions. arXiv, preprint arXiv:1302.0234 (2013). 53. Costa, A.M.: A survey on benders decomposition applied to fixedcharge network design problems. Comput. Oper. Res. 32(6), 1429– 1450 (2005) 54. Chen, G., He, W., Liu, J., Nath, S., Rigas, L., Xiao, L., Zhao, F.: Energy-aware server provisioning and load dispatching for connection-intensive internet services. In: Proceedings of the 5th USENIX Symposium on Networked Systems Design and Implementation. NSDI’08, pp. 337–350. USENIX Association, Berkeley, CA, USA (2008) 55. Jiang, N., Parashar, M.: Enabling autonomic power-aware management of instrumented data centers. In: Parallel Distributed Processing, 2009. IPDPS 2009. IEEE International Symposium on, pp. 1–8 (2009). 56. Bianzino, A.P., Chaudet, C., Rossi, D., Rougier, J.L.: A survey of green networking research. CoRR (2010).

Junaid Shuja is a PhD student at the Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia. He completed his BS in Computer and Information Science from PIEAS, Islamabad, in 2009 and MS in Computer Science from CIIT Abbottabad, in 2012. His research interests include mobile virtualization, ARM based servers, and energy efficient data centers networks.

Cluster Comput Kashif Bilal received his BS and MS in Computer Science from COMSATS Institute of Information Technology, Pakistan. He received a campus medal in his BS. He served as a lecturer at COMSATS Institute of Information Technology from 2004 to 2011. Currently, he is pursuing his PhD in Electrical and Computer Engineering at North Dakota State University, Fargo ND, USA. Kashif’s research domain encompasses topics related to data center networks, distributed computing, wireless networks, and expert systems.

Sajjad Ahmad Madani received an MS degree from Lahore University of Management Sciences, Lahore, Pakistan, and PhD from Vienna University of Technology, Vienna, Austria. Currently, he is Associate Professor at the department of computer science at COMSATS Institute of Information Technology, Abbottabad, Pakistan. Dr. Madani’s research interests include wireless sensor networks, routing protocols, cloud computing, and social networks. He has over 60 publications in well reputed journals and conferences. He is member of the IEEE Computer Society, IEEE industrial electronics society, ACM, and Pakistan Engineering Council.

Samee U. Khan is an assistant professor of electrical and computer engineering at the North Dakota State University. Samee’s research interests include optimization, robustness, and security of: cloud, grid, cluster and big data computing, social networks, wired and wireless networks, power systems, smart grids, and optical networks. His work appears in over 225 publications. He maintains the GreenCloud simulator. Samee is Chair of the Steering Committee of the IEEE Technical Area in Green Computing, a member of the Executive Committee of the IEEE Technical Committee on Scalable Computing, and a member of the IEEE Technical Committee on Self-Organized Distributed and Pervasive Systems. He is an associate editor of the IEEE Communications Surveys & Tutorials, Scalable Computing, Cluster Computing, Security & Communication Networks, and International Journal of Communication Systems.

123