Increasing the Thermal Efficiency of an Operational Data Center Using ...

Increasing the Thermal Efficiency of an Operational Data Center Using Cold Aisle Containment Ali Habibi Khalaj

Thomas Scherer

Dept. of Mechanical Engineering School of Engineering The University of Melbourne Parkville, Melbourne, Australia Email: [email protected]

Cloud & Computing Infrastructure Dept. IBM Research – Zurich R¨uschlikon, Switzerland

Abstract—Cooling systems play a vital role in the design and operation of Data Centers (DCs), as they consume a considerable amount of energy. Hot air infiltration from the servers’ outlets into their inlets, which creates hot spots and flow short-circuiting, is one of the main sources of thermal inefficiency inside air-cooled DCs. This inefficiency increases the total energy consumption of DCs especially when the allocated workload and heat dissipation of the servers are increased. Therefore, efficient thermal management of DCs is considered as one of the main challenges of the DC industry. In order to highlight the importance of this inefficiency, an operational DC has been considered in this study. The thermal behaviour of the DC has been evaluated by conducting a numerical analysis of the flow and temperature fields based on the experimental measurements in a real DC. Numerical simulation highlighted a number of undesirable hot spots near the racks. Cold aisle containment, which is one of the efficient and industrially affordable methods of thermal inefficiency reduction, has been considered. The effectiveness of this method has been evaluated using three non-dimensional metrics known as Supply Heat Index (SHI), Rack Cooling Index (RCI) and the Coefficient of Performance (COP) of the cooling system. In this case study, by applying the proposed method, SHI, RCI and COP of the cooling system have been improved by more than 0.45, 17% and 19.5%, respectively. These results demonstrate the effectiveness of cold aisle containment for energy efficiency enhancement of DCs. Keywords—Data Center; Cooling System; Numerical Modelling; COP; CFD.

I. I NTRODUCTION A Data Center (DC) is a computing infrastructure facility housing large amounts of IT equipment required to process, transmit and store information. Uninterrupted and zerodowntime operation is the most crucial requirement for all DCs. Power and cooling are two crucial factors in this regard. Uninterrupted power is assured by installing several backup sources that would be automatically brought on line as soon as a power failure is detected. On the other hand, cooling is a more complex issue, since it must maintain the temperature and humidity of a DC in a proper condition according to the available standards [1–5]. According to a report of the United States Environmental Protection Agency (EPA) [6], DCs consume 1-2% of the world’s electricity production. Due to the their high c 978-1-4799-4598-6/14/$31.00 2014 IEEE

Jayantha Siriwardana and Saman Halgamuge Dept. of Mechanical Engineering School of Engineering The University of Melbourne Parkville, Melbourne, Australia

demand, this number is increasing at a rate of 12% per year. Furthermore, taking into account of the increase in unit cost of the electrical power, the total cost for powering DCs has become not only an operational problem but also a potential threat to the business profitability. The large portion of the electrical power utilized by a DC is converted into the heat energy which forces DCs to allocate a great fraction of their power consumption for cooling of the IT equipment. According to the report of the United States EPA [6, 7], a considerable fraction of the operational costs of a typical DC, which account for 40-50% of the total energy consumption of the DC, can be ascribed to its cooling system. Therefore, thermal management has recently become one of the main foci in design and operation of DCs to improve cooling efficiency and reduce energy cost. Thermal management techniques for DCs can be considered in multiple levels from chip to room. An extensive research has been accomplished in each level. A brief description of each level, the components and available thermal management solutions is given below [1–5, 8]: 1) Chip Level: Thermal management at this level comprises diverse techniques to augment heat dissipation of the chip. Available solutions can be categorized as applications of high conductivity thermal interface materials and effective thermal design of the heat sinks via micro heat exchangers for multiphase heat transfer [9–11]. 2) Server Level: Thermal management at this level depends on the manufacturing methodology of the chip carrier. One of the efficient solutions is liquid cooling of the servers and chips via cold plates and heat pipes, where buoyancy driven thermosiphons or pumped liquid loops are the common techniques utilized in this regard [12, 13]. 3) Chassis Level: In air cooled DCs, this level is utilized as a pathway to deliver air to the servers via installation of fans. This level plays a significant role in the implementation of different solutions to the rack level, particularly combined with the server levels [14]. 4) Rack Level: For both air and liquid cooling systems, the main purpose of the available solutions is to gain maximum

benefit of the prevailing cooled fluid in order to enhance the heat dissipation and decrease the temperature variability [15, 16]. 5) Room Level: At this level, computer room air conditioning (CRAC) units are utilized to distribute the cold air into the DC room through perforated tiles placed over a raised floor plenum. The hot air exhausting from the rear of the racks is accumulated and returned to the CRAC units. The major cooling inefficiency occurring at this level is due to the recirculation of hot air from the hot aisle into the cold supply air at the cold aisle [2–5]. This inefficiency is shown as hot spot in Fig. 1. The elimination of this inefficiency is the main purpose of thermal management at this level.

streams, creating hot spots around racks, as illustrated in Fig. 1. The CRAC units have to over-cool the DC room in order to supply sufficient cold air for these localized hot spots. This negatively affects the efficiency of the CRAC, not only through higher utilization, but also through reducing its Coefficient of Performance (COP). In order to save energy in a raised floor DC, the knowledge of heat recirculation, the CRAC efficiency and the magnitude of the over-cooling can be utilized to predict the energy savings for the available and proposed methodologies [2–5, 19]. In this study, an operational DC has been considered in order to highlight the importance of the thermal inefficiencies occurring in a real DC. A numerical simulation using experimental data has been conducted to investigate the thermal behaviour of the air distributed inside the DC. A number of undesirable hot spots near the racks have been observed in the simulation. In order to reduce the corresponding thermal inefficiencies, the effectiveness of cold aisle containment as one of the efficient and industrially affordable methods has been investigated in this study. II. E XPERIMENTAL M EASUREMENT

Fig. 1. General layout of a typical raised floor DC

6) Plenum Level: This level includes the air delivery plenum, typically below or above the DCs room space. Thermal management at this level plays a considerable role in cooling the current and future DCs, since chilled water pipes of the CRACs and any liquid cooled racks as well as electrical cabling often pass through this level. Therefore, effective use of this level along with the room level provides multiple opportunities in energy efficient design of DC’s cooling systems [17, 18].

In order to create an accurate numerical model of a DC, it is required to measure real experimental data inside the DC. This data can be used to define the boundary conditions of Computational Fluid Dynamic (CFD) model. In this study, a DC at the IBM Research – Zurich Laboratory has been considered for experimental data gathering and numerical modelling [20]. The layout of this DC, which has a raised floor area of 18 m × 15.9 m = 286.2 m2 with a height of 2.7 m, is depicted in Fig. 2. This DC houses 27 racks including high performance computers (HPCs), 9 racks with networking equipment and storage equipment, 6 CRAC and 4 power distribution units. Cooling air is provided through 92 perforated floor tiles and returned to the CRAC units through 25 ceiling vents above the hot aisles.

Thermal management solutions for each level can be considered as short- and long-term from implementation and payback perspectives. For instance, the majority of the solutions for chip, server, chassis and rack level are considered as long-term, since their implementation requires considerable modification in the DCs that may only be viable during significant facility renovations. Furthermore, corresponding modifications often require sizeable capital investments. On the other hand, solutions for room and plenum levels are considered as short-term due to requiring less modification, easy installation and low investment. Since short-term solutions are more affordable for the industry, they have been considered in this study. Fig. 2. Layout of the DC considered in this study [20].

Inefficiencies of the cooling system in the raised floor DC have a number of sources. The main source is heat recirculation caused by intermingling of the hot and cold

The temperature distribution of the operational DC was measured using Mobile Measurement Technology (MMT)

designed by IBM [21]. Using MMT, a three dimensional temperature map of the DC was created with a spatial resolution of 20 cm in x and y directions and 30 cm in z direction [20]. The volumetric airflow rates through the perforated tiles, racks, CRAC and ceiling vents were measured using a FlowHood. Furthermore, the power consumption of each rack was measured using the power meters at the power distribution unit. Further information regarding the experimental measurement is provided in our previous work [20]. III. N UMERICAL M ODELLING In order to simulate the complex behaviour of the airflow and temperature profile inside the DC, creating a high fidelity model of the DC is required. In a CFD model, the governing equations of the flow are the steady incompressible Navier-Stokes equations, and the temperature equation is given by the energy equation as follows [22–24]: Continuity equation:  u=0 ∇.

(1)

Momentum equation:  u = −∇P  + ∇.  τ + (ρ − ρ∞ ) g ρu.∇ Temperature equation:



(2)

Item

Value and/or Description

Material

Ideal gas

Reference state

25◦ C and 1 atm

Density

1.2 kg m−3

Molar mass

28.96 kg kmol−1

Specific heat at constant pressure

1.0044 kJ kg−1 K−1

Dynamic viscosity

1.831 × 10−5 kg m−1 s−1

Thermal conductivity

2.61 × 10−2 W m−1 K−1

TABLE II T HE E XPERIMENTAL DATA M EASURED AND U TILIZED AS THE B OUNDARY C ONDITIONS OF THE N UMERICAL M ODEL Parameter

Value

Total volumetric airflow into the DC through perforated tiles

4.725 m3 s−1

Total volumetric airflow into the DC through HPCs

7.670 m3 s−1

Total volumetric airflow out of the DC through ceiling vents

15.720 m3 s−1

Total volumetric airflow into the DC through leakage

3.325 m3 s−1

Minimum airflow rate at the server inlets

0 m s−1

Maximum airflow rate at the server inlets

0.73 m s−1

Flow rate of air leaking into the racks

0.139 m s−1

Supply air temperature



 = ∇.  λ∇T  ρCp u.∇T +S

TABLE I M ATERIAL P ROPERTIES OF THE F LUID USED IN OUR N UMERIACAL M ODELLING

(3)

 is the gradient vector, u is the velocity vector, ρ is the ∇ density of air, ρ∞ is the reference density, P is the pressure, τ is the stress tensor, g is the acceleration due to gravity, Cp is the specific heat at constant pressure, T is the fluid temperature, λ is the thermal conductivity, and S is the heat source. The properties of air utilized for our numerical modelling are given in Table I. Furthermore, an overview of the experimental data measured and utilized as boundary conditions for the numerical study is provided in Table II. IV. T HERMAL P ERFORMANCE M ETRIC OF DC We consider three non-dimensional metrics for thermal performance evaluation of the DCs, known as Supply Heat Index (SHI) [25], Rack Cooling Index (RCI) [26] and the Coefficient Of Performance (COP) of the DC’s cooling system [2, 5], respectively. The SHI was proposed based on rack inlet and exhaust temperature and is defined as Enthalpy rise in cold aisle due to hot air infiltration SHI = Total enthalpy rise at rack exhaust Tin.rack − Tref = , (4) Tout.rack − Tref

Total power consumption by IT equipment

16.5◦ C 219.73 kW

in which Tin.rack and Tout.rack are the average inlet and outlet temperature of the rack, respectively. Furthermore, Tref is the supply air temperature of the CRAC unit distributed via perforated floor tiles into the DC room. Tref is 16.5◦ C in the present study. The SHI is a value between 0 and 1. A value of 0 indicates that the inlet air temperature of the rack is equal to the CRAC supply air temperature and there is no air recirculation. The SHI can be calculated at different levels from individual server racks to the entire DC. The RCI is also an important non-dimensional metric introduced to investigate the cooling rate of the server rack. The RCI is defined as    RCI =

1−

(Tmax.in − Tmax.rec )Tmax.in >Tmax.rec (Tmax.all − Tmax.rec )n

×100%, (5)

in which Tmax.in , Tmax.rec and Tmax.all are the maximum inlet, recommended and allowable air temperature of the rack, respectively. Furthermore, n is the total number of the racks. The values of Tmax.rec and Tmax.all are based on ASHRAE guidelines [1], in which Tmax.all = 32 ◦ C and Tmax.rec = 27

◦

C. The interpretation of the RCI is as follows: a) RCI = 100%, if ∀ racks Tmax.in < Tmax.rec b) RCI < 100%, if for at least one rack Tmax.in < Tmax.rec

Further information about this metric can be found in Herrlin [26]. The last important non-dimensional metric is the COP of the DC’s cooling system that can be approximated based on the CRAC supply air temperature (Tref ). For this study we use an empirical formula that has been derived for a CRAC unit in a typical DC [27]. 2 COP = 0.0068Tref + 0.0008Tref + 0.458,

(6)

According to Eq. 6, providing sufficient cold air by reducing the supply temperature of the CRAC unit has the drawback of reducing its COP which leads to higher cooling energy cost. Using this metric, we can assess the effectiveness of any solution by evaluating the COP of the CRAC unit.

VI. N UMERICAL M ODELLING R ESULTS The numerical simulation in this study is conducted using the CFD software ANSYS CFX to determine the flow and temperature distribution, utilizing the Shear Stress Transport (SST) turbulence model. The fluid domain, as illustrated in Fig. 2, has been discretized into a number of control volumes in order to numerically solve the governing Eqs. (1-3), with the Boussinesq approximation for the buoyancy term in Eq. (2). The mesh with 627,076 equilateral hexahedral elements has shown an acceptable performance. Since increasing the mesh size has not significantly changed the simulation results, grid independence has been achieved. The convergence is reached in approximately 10 hours on a workstation with a 3.4 GHz Core i7 processor and 8 GB memory when setting the convergence criterion to 0.0001 for the maximum normalized values of the equation residuals. The result of this numerical study is visually depicted in Fig. 3 by temperature profiles across the DC from top and side views. It can be observed from the numerical results depicted in Fig. 3 that the inlet and outlet air temperature of some racks exceed the temperatures recommended by ASHRAE guidelines [1].

V. C OLD A ISLE C ONTAINMENT In an ideal air-cooled DC, the inlet air temperature of all racks should be equivalent to the air temperature supplied to the room by the CRAC units via perforated floor tiles. However, due to the differential pressure created across the DC because of its geometrical configuration, buoyancy force, rack layout, perforated floor tiles and ceiling vents, hot air from hot aisles is driven into the cold aisles, creating undesirable hot spots. This detrimentally affects the performance of the DC cooling system. The hot spots have to be cooled down by providing sufficient cold air by means of reducing the CRAC supply air temperature. According to Eq. (6), this will reduce the efficiency of the CRAC unit and increase the cooling cost. Containing the cold aisle is one of the efficient and industrially affordable methods to mitigate this problem. Cold aisle containment typically consists of physical barriers guiding the supplied cold air flow effectively into the racks by preventing hot air recirculation inside DC. This method avoids hot air short-circuits, reduces the volume of air to be cooled down, improves efficiency of the cooling system and consequently reduces its energy consumption. Using this method, the cold air is distributed into the contained cold aisle through perforated floor tiles. Then, this cold air, without being mixed with hot air, flows into the front of the servers, removes the heat from IT equipment and is exhausted as hot air through the rear of the racks. The hot air is finally removed by the CRAC units.

Fig. 3. Result of the numerical simulation depicted as temperature profile. a) Top view of the DC at 1.75 m, b) Side view of the DC

As seen in Fig. 3, hot spots mainly occur at the top and sides of the racks in which hot air recirculation is dominant. In order to reduce the effects of this infiltration, cold aisle containment has been considered in this study. We have considered the highlighted area in Fig. 2 in order to investigate the effectiveness of this method.

VII. E FFECTIVENESS E VALUATION OF THE M ETHOD In order to evaluate the effectiveness of cold aisle containment, the CFD model of the DC has been modified to include the aforementioned containment. After numerically resimulating the model, the results of the DC with containment have been compared to the original CFD model without any containment. The results of this comparison is depicted in Fig. 4, visualizing the outlet temperature profile of racks 13-17 and 20-24 with and without containment. As seen in Fig. 4, the outlet temperature of the servers in the upper part of racks 14, 16 and 17 in the original DC is undesirably high due to hot air recirculation. However, this thermal inefficiency is effectively reduced in the improved DC with cold aisle containment. According to Fig. 4, further workload can be allocated to the servers after containing the cold aisle, as the majority of them are now over-cooled and could effectively work under higher workload.

reduction of hot air infiltration. Using this method, the supply air temperature of the CRAC unit can be increased by 1.9◦ C without raising any issue for servers in the corresponding racks. When the supplied air temperature is increased from 16.5◦ C to 18.4◦ C, the COP of the CRAC unit, according to Eq. 6, increases from 2.32 to 2.78. This is an improvement of 19.5% as illustrated in Fig. 5.

Fig. 5. COP curve of the CRAC unit

Fig. 4. Outlet temperature profile comparison of the DC with and without containment: Outlet temperature profile of a) racks 13-17 in the original CFD model, b) racks 20-24 in the original CFD model, c) racks 13-17 in the DC with containment, d) racks 20-24 in the DC with containment.

The results of this comparison study between the DC with and without cold aisle containment have also been summarized in Table III, using the non-dimensional metrics introduced in Section IV. TABLE III E FFECTIVENESS E VALUATION OF C OLD A ISLE C ONTAINMENT IN I NCREASING THE COP OF THE CRAC UNIT T HROUGH H OT S POT R EDUCTION . R13-R17 & R20-R24

DC without Containment

DC with Containment

Total Gain

SHI

0.466

0.015

0.451

RCI

82.5%

100%

17.5%

Max Tout.rack

36.71◦ C

34.8◦ C

1.91◦ C

COP of the CRAC

2.32

2.78

19.58%

According to Table III, implementation of cold aisle containment can improve the SHI and RCI of the studied DC by more than 0.45 and 17%, simultaneously. Furthermore, this method can reduce the maximum outlet temperature of racks 13-17 and 20-24 by more than 1.9◦ C, which is due to the

VIII. C ONCLUSION Uninterrupted, reliable and economical operation of DCs significantly depends on the provision of an efficient cooling system. In this study, a numerical simulation of the air flow and temperature fields has been conducted based on experimental measurements from a real DC to evaluate its thermal behaviour. In order to decrease the thermal inefficiencies caused by undesirable air recirculation near the racks, effectiveness of cold aisle containment has been investigated in this study. The effectiveness has been evaluated through SHI, RCI and COP of the cooling system as they play an important role in thermal management of DCs. In this case study, by containing the cold aisle in an operational DC, SHI, RCI and COP of the DC’s cooling system has been improved by more than 0.45, 17% and 19.5%, respectively. The results of this method demonstrate the effectiveness of cold aisle containment in energy efficiency enhancement of DCs. Future work could involve further investigation on the effectiveness of other cooling solutions in room and rack levels for minimizing the hot air recirculation. ACKNOWLEDGEMENTS This work is partially funded by Australian Research Council grant LE120100117. Ali Habibi Khalaj is also fully funded by MIRS and MIFRS scholarships of The University of Melbourne. R EFERENCES [1] ASHRAE, “Thermal guidelines for data processing environments–expanded data center classes and usage guidance,” Whitepaper prepared by ASHRAE technical committee (TC) 9.9, 2011. [2] Y. Joshi and P. Kumar, Energy efficient thermal management of data centers. Springer, 2012.

[3] S. V. Patankar, “Airflow and cooling in a data center,” Journal of Heat Transfer, vol. 132, no. 7, p. 073001, 2010. [4] J. Rambo and Y. Joshi, “Modeling of data center airflow and heat transfer: State of the art and future trends,” Distributed and Parallel Databases, vol. 21, no. 2-3, pp. 193–225, 2007. [5] J. Dai, M. M. Ohadi, D. Das, and M. G. Pecht, Optimum Cooling of Data Centers. Springer-Verlag, 2014. [6] R. Brown et al., “Report to congress on server and data center energy efficiency: Public law 109-431,” Lawrence Berkeley National Laboratory, 2008. [7] J. Siriwardana, S. Jayasekara, and S. K. Halgamuge, “Potential of air-side economizers for data center cooling: A case study for key australian cities,” Applied Energy, vol. 104, no. 0, pp. 207 – 219, 2013. [8] S. V. Garimella, L.-T. Yeh, and T. Persoons, “Thermal management challenges in telecommunication systems and data centers,” Components, Packaging and Manufacturing Technology, IEEE Transactions on, vol. 2, no. 8, pp. 1307–1316, 2012. [9] S. V. Garimella, T. Persoons, J. Weibel, and L.-T. Yeh, “Technological drivers in data centers and telecom systems: Multiscale thermal, electrical, and energy management,” Applied Energy, vol. 107, pp. 66–80, 2013. [10] M. Ohadi, S. Dessiatoun, K. Choo, M. Pecht, and J. Lawler, “A comparison analysis of air, liquid, and twophase cooling of data centers,” in Semiconductor Thermal Measurement and Management Symposium (SEMITHERM), 2012 28th Annual IEEE. IEEE, 2012, pp. 58–63. [11] J. B. Marcinichen, J. A. Olivier, V. d. Oliveira, and J. R. Thome, “A review of on-chip micro-evaporation: Experimental evaluation of liquid pumping and vapor compression driven cooling systems and control,” Applied Energy, vol. 92, pp. 147–161, 2012. [12] E. Frachtenberg, A. Heydari, H. Li, A. Michael, J. Na, A. Nisbet, and P. Sarti, “High-efficiency server design,” in Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis. ACM, 2011, p. 27. [13] M. Ibrahim, S. Shrivastava, B. Sammakia, and K. Ghose, “Thermal mass characterization of a server at different fan speeds,” in Thermal and Thermomechanical Phenomena in Electronic Systems (ITherm), 2012 13th IEEE Intersociety Conference on. IEEE, 2012, pp. 457–465. [14] T. Dang, C. Serreli, H. E. Khalifa, R. R. Schmidt, and M. Iyengar, “Reduced rack exhaust temperature gradient via variable chassis flow rate,” in ASME 2011 Pacific Rim Technical Conference and Exhibition on Packaging and Integration of Electronic and Photonic Systems. American Society of Mechanical Engineers, 2011, pp. 451–459. [15] R. Hannemann and H. Chu, “Analysis of alternative data center cooling approaches,” in ASME 2007 InterPACK Conference collocated with the ASME/JSME 2007

[16]

[17]

[18]

[19]

[20]

[21]

[22] [23]

[24]

[25]

[26]

[27]

Thermal Engineering Heat Transfer Summer Conference. American Society of Mechanical Engineers, 2007, pp. 743–750. A. Radmehr, K. C. Karki, and S. V. Patankar, “Analysis of airflow distribution across a front-to-rear server rack,” in ASME 2007 InterPACK Conference collocated with the ASME/JSME 2007 Thermal Engineering Heat Transfer Summer Conference. American Society of Mechanical Engineers, 2007, pp. 837–843. J. W. VanGilder and R. R. Schmidt, “Airflow uniformity through perforated tiles in a raised-floor data center,” in ASME 2005 Pacific Rim Technical Conference and Exhibition on Integration and Packaging of MEMS, NEMS, and Electronic Systems collocated with the ASME 2005 Heat Transfer Summer Conference. American Society of Mechanical Engineers, 2005, pp. 493–501. S. Bhopte, B. Sammakia, M. Iyengar, and R. Schmidt, “Numerical and experimental study of the effect of underfloor blockages on data center performance,” Journal of Electronic Packaging, vol. 133, no. 1, p. 011007, 2011. A. H. Khalaj, T. Scherer, J. Siriwardana, and S. K. Halgamuge, “Multi-objective efficiency enhancement using workload spreading in an operational data center,” Applied Energy, vol. 138, no. 0, pp. 432 – 444, 2015. J. Siriwardana, S. K. Halgamuge, T. Scherer, and W. Schott, “Minimizing the thermal impact of computing equipment upgrades in data centers,” Energy and Buildings, vol. 50, pp. 81–92, Jul 2012. H. F. Hamann, J. A. Lacey, M. O’Boyle, R. R. Schmidt, and M. Iyengar, “Rapid three-dimensional thermal characterization of large-scale computing facilities,” IEEE Transactions on Components and Packaging Technologies, vol. 31, no. 2, pp. 444–448, June 2008. G. K. Batchelor, An introduction to fluid dynamics. Cambridge university press, 2000. F. White, Fluid Mechanics (Mcgraw-Hill Series in Mechanical Engineering). McGraw-Hill Science/Engineering/Math, 2010. B. R. Munson, D. F. Young, T. H. Okiishi, and W. W. Huebsch, Fundamentals of Fluid Mechanics. Wiley, 2009. R. K. Sharma, C. E. Bash, and C. D. Patel, “Dimensionless parameters for evaluation of thermal design and performance of large-scale data centers,” in 8th ASME/AIAA Joint Thermophysics and Heat Transfer Conference, 2002, pp. 1–1. M. K. Herrlin et al., “Rack cooling effectiveness in data centers and telecom central offices: The rack cooling index (rci),” Transactions-American Society of Heating Refrigerating and Air conditioning Engineers, vol. 111, no. 2, p. 725, 2005. J. D. Moore, J. S. Chase, P. Ranganathan, and R. K. Sharma, “Making scheduling” cool”: Temperature-aware workload placement in data centers.” in USENIX annual technical conference, General Track, 2005, pp. 61–75.