Experimental Characterization and Modeling of a Water-Cooled Server Cabinet 1
Kourosh Nemati1, Bruce T. Murray1, Bahgat Sammakia1 Department of Mechanical Engineering, Binghamton University-SUNY Binghamton, New York, USA, 13902 E-mail:
[email protected]
ABSTRACT Water-cooled server racks are typically closed cabinets, so no heat load is removed by the room cooling system in a raised floor data center. Instead, the cooling is provided by a closed system employing an air-water (fin-tube) heat exchanger. This kind of heat exchanger uses chilled water from a central plant, and there is the potential of using the waste heat that is returned. The goal of this study is characterize a specific sealed-door, water-cooled server cabinet under steady state and transient operation. The experimental part of the study is being performed on a Knurr CoolThermTM® rack. In this cabinet, the water/air heat exchanger is located at the bottom. Cooling air is circulated by three rear door mounted fans, the cooled air flows upward in front of the servers. The specific cabinet being tested has the capability to handle 25 kW and has a 30% cooling system energy efficiency. In the experiments,16 1U servers and a 9U load bank are employed for generating a heat load. Over 100 thermocouples are used to monitor air temperature within different parts of the cabinet. The operating conditions monitored include: water flow rate, water inlet temperature, air flow rate through the servers, air flow rate through the heat exchanger, air temperatures and server power. The internal temperature of the servers is also monitored. From the measured data, the effectiveness is calculated for four different water flow rates and different power dissipation levels. Computational modeling of the air flow and heat transfer in a simplified model of the cabinet is also performed.
INTRODUCTION In order to better represent the thermal performance of server cabinets in overall data center cooling studies, it is important to characterize different types of cabinets using a combination of experimental measurements and modeling. Computational modeling is an important tool in data center thermal analysis. However, experimental characterization is also very important and enables validation of computational models. A number of experimental studies have been performed on air-cooled server cabinets (see for example [1] and references within), but fewer studies have been performed on water-cooled cabinets [2]. There are five types of water cooling systems that are commonly used: rear door heat exchangers, in-row coolers, over-cabinet cooling units, fully-enclosed heat exchanger systems and in-server cold plate heat exchanger systems. The use of hybrid cooling systems offers advantages in energy efficiency, operation cost and energy recovery capability. For the closed cabinet studied here, the combination of a CDU and chilled water supply provide the cooling. The warm water from the cabinet heat exchanger is returned to the chiller. For the comprises a chilled water system. A schematic of a typical closed-loop system is shown in Fig. 1. The components of the refrigeration system can be located remotely from the data center. The chilled water is typically supplied at about 8-15°C (46-59°F). Chilled water systems are engineered to be extremely reliable since they introduce a large amount of liquid into the IT environment. [3]
KEY WORDS: water-cooled server cabinet, heat exchanger effectiveness, data center thermal management NOMENCLATURE Cp h 𝑚̇ T
Specific heat capacity, J/kg.K Heat transfer coefficient, W/m2·K Air flow rate, kg/s Temperature, Celsius
Greek symbols 𝜀 Effectiveness Subscripts a w i o
air water inlet outlet
Fig. 1: Liquid Cooling Systems-Loops within Data Center A schematic of the Knurr CoolThermTM® server cabinet [4] characterized in this study is shown in Fig. 2. The hot air exiting the servers is directed downwards through the heat exchanger where the heat is removed. After leaving the heat exchanger, the cooled air moves upwards in the front of the cabinet and is drawn in by the server fans. The air circulation is driven by three main fans that are located on the rear door of
the cabinet. The cabinet is designed such that the heated air travels through a duct back to the heat exchanger.
Fig. 4 rear view of Heat Exchanger
Fig. 2 Top and side view of server cabinet unit EXPERIMENTAL STUDY For the experiments the cabinet was populated with 16 DELL PowerEdge 1950 1U servers. Each server has two Quad-Cores processors operating up to 2.66 GHz and can dissipate a maximum power of about 450 W [5]. A internal view of the server components in shown in Fig. 3. In addition, a 9U high load bank is used to provide heat dissipation of up to an additional 10 kW. Empty parts of the cabinet rack are covered by a fiber glass blanking panel to prevent recirculation of cold air inside the cabinet.
Also, evident is the ducting that directs the air into the heat exchanger which is located on the bottom. In the right frame of Fig. 5, viewed from the front, the plenum that directs the cooled air upwards can be seen at the bottom. The air flow rate through the cabinet is controlled by a thermostat which is located at the upper front part of cabinet: below 16º C the air flow rate is 75% of the maximum. After 19º C, 100% of the maximum air flow rate is provided.
ii
2
1
thermocouples are attached to a thin wire frame as can be seen in the figure. The wireframe has a negligible effect on the air flow. There are three fans located on the rear door of the cabinet as shown by the right frame in Fig. 5.
i
4 3
iii
5
Fig. 3 Top view of DELL PowerEdge 1950 server For temperature monitoring, 5 thermocouples are installed inside each server, to monitor air temperature distribution inside all servers. The locations of the thermocouples are shown in Fig. 3 and are numbered as follows: 1) air inlet, 2) right CPU heat sink, 3) left CPU heat sink, 4) right outlet and 5) left outlet. The servers have a bank of 8 fans (labeled as i in Fig. 3). Also, highlighted in the figure is the location of the power supply (ii) and the array of memory cards (iii). The servers are running Debian Linux and the software Prime95 is run to exercise the processors and increase the power dissipation. The typical power dissipation of each server has been measured (idle-6 W, normal computation-250 W, complete loading-450 W). The maximum power generation that can be obtained inside the cabinet for the experiments is 14.7 kW. A total of 7 thermocouples are used to measure the air temperature at the inlet of the air/water heat exchanger and 9 thermocouples are used at the outlet. A rear view of the inlet side of the V-shaped heat exchanger is shown in Fig. 4. The
Fig. 5 Front and rear view of fully-enclosed cabinet In order to measure the water inlet flow rate and temperature a vortex flow meter (with temperature sensor) is installed on the supply pipe of heat exchanger. The water flow rate is controlled via a valve on the return side of the chilled water system. The water inlet temperature is kept constant in all the experiments at 16.7 ºC with +/- 0.5 ºC tolerance. The under floor location of the flow meter in shown in Fig. 6
Effectiveness
1 0.9 0.8 0.7 0.6 0.5 0.4 9500
11500
13500
Power[W]
15500
Fig. 6 Location of vortex flow meter on the supply pipe of heat exchanger
Fig. 8 Heat exchanger effectiveness values for Case 1 (maximum water flow rate)
RESULTS AND DISCUSSION One goal of the experiments was to determine the effectiveness of the heat exchanger. This was done for three different steady state operation scenarios (cases) by varying the water flow rate. At each flow rate, measurements are made for four different power levels. The effectiveness of the heat exchanger is calculated in terms of the ratio of the air and water inlet/outlet temperatures based on an energy balance. For the ɛ-NTU method of heat exchanger analysis, the effectiveness is defined by the following relationship:
All of the heat generated inside the closed cabinet should be remove by heat exchanger. It is possible to monitor the power required by the servers and load bank, so it is possible to perform an energy balance between the power in and the heat removed through the heat exchanger. There are four power distribution units inside of the cabinet. Fig. 9 shows the output from the power distribution units used to calculate the input power.
𝜀 = (𝑇𝑎𝑜 − 𝑇𝑎𝑖 )⁄(𝑇𝑤𝑖 − 𝑇𝑎𝑖 ) where 𝑇𝑎𝑜 is the air outlet temperature, 𝑇𝑎𝑖 is the air inlet temperature and 𝑇𝑤𝑖 is the water inlet temperature. Case 1 For the first set of experiments, the water inlet flow rate is the maximum at 50.1 LPM or 0.8383 kg/s. At this flow rate, the heat exchanger is working at the maximum capacity and should have the highest effectiveness. The rear door fans are operating such that 75% of the maximum air flow rate is achieved (1.425 kg/s). Temperature measurements were obtained for the following four power levels: 9.7 kW, 11.4 kW, 13.3 kW and 14.7 kW. The inlet and outlet temperatures of the air at the four power levels are shown in Fig. 7. Because of the capacity of the chilled water cooling, the power level has only a small effect on the outlet temperature. Naturally, the air inlet temperature increases with the power dissipation level. The calculated effectiveness for this case is shown in Fig.8. Note that the effectiveness values are essentially constant. The average value is 𝜀 = 0.83.
Temperature degC
40
Inlet
Outlet
30 25 20 11500
As it shown in figure above, the heat generated is calculated by monitoring the power distribution units. An energy balance equation is employed to calculate the quantity of heat removed by the heat exchanger, 𝑄 = 𝑚̇ cp ∆T
35
15 9500
Fig. 9 Power Distribution Unit remote monitoring
13500
Power[W]
15500
Fig. 7 Air temperature values for maximum water flow rate
Generated Heat Removed Heat
13700
1 0.9
Effectiveness
Power[W]
15700
0.8
11700
0.7
The input power values and the amount of power removed based on the measured heat exchanger temperatures and flow rates are shown in Fig. 10. Given the measurement uncertainties in the temperature and power values and the small amount of heat loss from the exterior of the cabinet, reasonable agreement is obtained for the power values. The largest difference is approximately 5 %.
Case 2 To investigate the effect of the water flow rate on the effectiveness, for the second case the water flow rate is decreased to 57% of the maximum (0.485 kg/s). Because the cooling capacity of the heat exchanger is decreased, higher outlet temperatures are obtained as shown in Fig. 11. Results are shown for the three higher power levels used in the first case. Temperature DegC
40
Inlet
Outlet
35
30 25 20 15 9500
11500 13500 15500 Power[W] Fig. 11 Air temperatures for 57% of maximum water flow rate The average effectiveness (shown in Fig. 12) decreases at the lower water flow rate. Note that for all of the experiments the air flow rate is unchanged. Only a small variation is obtained in the effectiveness values for the different power levels and the average effectiveness value is 0.74 for a water flow rate equal to 0.49 kg/s.
0.6 9500
11500
13500
15500
Power
Fig. 12 Effectiveness of heat exchanger for 57% water flow rate 15700
Power[W]
1 2 3 4 Fig. 10 Comparison of the power generated to the amount removed via the heat exchanger for the four power levels.
Generated Heat Removed Heat
13700 11700 9700
1 2 3 Fig. 13 Energy balance for the 57% maximum water flow rate Case 3 The lowest water flow rate studied is 42% of the maximum or about 0.35 kg/s. The capacity heat exchanger is about 10.5 kW in this case. The average inlet/outlet temperatures are measured for five different power levels in this case. Because the cooling capacity of the heat exchanger is further decreased in this case, higher outlet temperatures are obtained as shown in Fig. 14. The average inlet temperature of the heat exchanger for the 14.7 kW power level is about 37.5 ºC, while for Case 1 it was about 34 ºC. 40 Temperature DegC
9700
Inlet
Outlet
35 30 25 20 15 7500
9500
11500 Power
13500
15500
Fig. 14 Air temperature values for 42% of the maximum water flow rate The effectiveness of the heat exchanger for this case is shown in Fig. 15. Again the average value is decreased for the lower water flow rate (lower cooling capacity). The variation in the effectiveness at the different power levels results from the measurement accuracy. The average effectiveness value is 0.66 for the water flow rate equal to 0.35 kg/s. The effectiveness values calculated from the experimental
measurements is summarized in Table 1. Results from an additional water flow rate are included.
Effectiveness
1
Flow Rate Effectiveness
57% 0.74
82% 0.815
100% 0.83
In the computational model, the average air temperature of inlet and outlet of heat exchanger is computed from the local temperature values in the region near the heat exchanger. Fig. 17 shows a comparison of the experimental measurements and computational results for three different cabinet power levels for the maximum water flow rate. Very good agreement between the model and experimental data is obtained.
0.9 0.8 0.7 0.6 7500
9500
11500
13500
15500
40
Fig. 15 Effectiveness of heat exchanger for 42% water flow rate Energy balance for 42% water flow rate is shown in Fig. 16 that it shows good agreement. 15700 Generated Heat Removed Heat 13700
Temperature[C]
Power
Power[W]
42% 0.66
Experimental Computational
35 30 25 20 15 9500
10500
11500
12500
13500
Power[W]
11700
Fig. 17 Experimental and computational inlet (lower values) and outlet air temperatures of the cabinet heat exchanger
9700 7700 1
2
3
4
5
In order to compare the different experimental cases studied, Fig. 18 shows the average inlet and outlet air temperatures as a function of power level for three water flow rates. 40
COMPUTATIONAL MODELING A 3D flow and thermal model of the cabinet was built in FloTHERM 9.3 [6]. Due to high air velocity and size of the rack a fully turbulent flow model was utilized. A detailed representation of the servers is not included in the computational model. The central region of the rack is treated as a heat source with air flow corresponding to the overall air circulation in the cabinet. The overall size and geometry of the cabinet are represented accurately. External surfaces of the cabinet are treated as adiabatic. One of the heat exchanger models available in FloTHERM is employed. The recirculation model used by Schmidt et al. [2] is chosen to model the heat exchanger here. For the simplified cabinet model, solution convergence for both flow velocities and temperature were readily obtained.
35
In order to model the heat exchanger, a value for the heat transfer coefficient (h) is required. This value is calculated using the ɛ-NTU method and the air and water inlet temperatures. For a given amount of heat flow per unit area, the heat transfer coefficient is calculated from [7], 𝑄 = ℎ(𝑇𝑎𝑖 − 𝑇𝑤𝑖 ) Table 1 Summary of effectiveness values for different water flow rates
Temperature[C]
Fig. 16 Energy balance for 42% water flow rate
100% 57% 42%
30 25 20 15 7500
9500
11500
Power[W]
13500
15500
Fig. 18 Average inlet (lower values) and outlet temperatures of the heat exchanger for different power levels and flow rates CONCLUSION A combined experimental and computational study of a fullyenclosed server cabinet has been conducted. The cabinet is cooled using external chilled water and a water/air heat exchanger internally. Air circulation in the cabinet is provided by a series of rear door mounted fans. The commercially available cabinet has a 37U rack height capacity. A combination of actual 1U servers and one 9U load bank was used to provide the heat dissipation in the experiments. The most important part of a fully-enclosed server cabinet is heat exchanger. The heat exchanger effectiveness is one of the
best ways to characterize the performance for use in energy efficiency studies. By monitoring the air and water temperatures, the heat exchanger effectiveness was calculated for three cooling water flow rates and a number of different power levels. Good agreement between the measured input and output power levels was achieved providing some validation of the measured quantities. Results for the airside inlet and outlet temperatures from a simplified computational model of the closed cabinet are in good agreement with the experimentally measured values. Both the experimental and computational results presented here are for steady-state operation. In the next stage of the research, the transient behavior of the cabinet will be studied.
REFERENCES [1] J. Rambo and Y. Joshi, “Thermal Performance Metrics for Arranging Forced Air Cooled Servers in a Data Processing Cabinet”, J. Elec. Packaging, vol. 127, pp. 452-459, 2005. [2] R. Schmidt, et al., “Maintaining Datacomm Rack Inlet Temperatures with Water Cooled Heat Exchangers”, Proc. of InterPACK, Paper number IPACK2005-73468, July 17th22nd, San Francisco, California, 2005. [3] ASHRAE Pub. 2011 “Thermal Guidelines for Data Processing Environments – Expanded Data Center Classes and Usage Guidance” [4] CoolTherm Manual, Server Cabinet with integrated liquid cooling, 12 – 25 / 35 KW, www.knurrusa.com [5] Dell PowerEdge 1950 User Manual, http://www.livehosting.ro/Dell-PwoerEdge-specification/ [6] Flotherm 9.3 Reference Manual, www.flotherm.com. [7] T. Gao, et al., “Comparative Thermal and Energy Analysis of a Hybrid Cooling Data Center With Rear Door Heat Exchangers”, Proc. of InterPACK, Paper number IPACK2013-73101, July 16th-28th, San Francisco, California, 2013.
