Apr 11, 2017 - â¢Practices to build energy-efficient servers. â¢A case study of the energy consumption of virtual machine I/O. â¢Data center power consumption.
An Overview of Server and Data Center-level Power Optimizations Dr. Tao Lu ECE NJIT April 11, 2017
Examples of power consumption 1, 10, or 100 watts ? Iphone7 in standby mode
< 1 watt
MacBook Pro 15.4”
20 - 70 watts
Human being in normal activities
~ 100 watts *
Low-end server
120 - 200 watts
• Source: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2816633/ • Source https://en.wikipedia.org/wiki/Human-powered_transport
8 watts * 3 hour / 24 hour = 1 watt
Terms • Server • A node undertakes computation, communication, and data storage tasks.
• Data center • A large group of networked servers.
Outline • Why power became a salient issue • Server power breakdown by components • Practices to build energy-efficient servers • A case study of the energy consumption of virtual machine I/O • Data center power consumption • Practices to build energy-efficient data centers
Why power optimizations became important • Tens of millions of servers (75 million as of 2014) • Higher performance and energy consumption • Exploding data storage requirements • Increasing server density in DC • Heating • Cooling
• Total cost of ownership (TCO) • Power availability
Power issue: an HPC example # 1 CHALLENGE: Energy efficiency
“The goal is to achieve exascale using 20 MW of power, yet existing circuits consume an order of magnitude too much power to meet this goal. Without much more energyefficient circuits, architecture, power conversion, power delivery and cooling technologies, the total cost of ownership for exascale systems could be 10 times higher than today.”
Outline • Why power became a salient issue • Server power breakdown by components • Practices to build energy-efficient servers • A case study of the energy consumption of virtual machine I/O • Data center power consumption • Practices to build energy-efficient data centers
Power consumption of servers Server class
2000
2001
2002
2003
2004
2005
2006
Volum e
186
193
200
207
213
219
225
Midrange
424
457
491
524
574
625
675
Highend
5534
5832
6130
6428
6428
7651
8163
Estimated average power use (Watt) per server, by server class, 2000 to 2006 Average Power Draw Assumptions for Mid-Range and High-End Servers
Source1: Jonathan G. Koomey, ESTIMATING TOTAL POWER CONSUMPTION BY SERVERS IN THE U.S. AND THE WORLD, 2007 Source2: Arman Shehabi, et.al., United States Data Center Energy Usage Report, 2016
System organization
CPU
Disk controller
USB controller
Graphic adapter
Memory …... Network
Network
A simplified modern computer system.
…...
Sources of server power consumption 45-200 w CPU
5w USB controller
12 w Disk controller
8w
10 w Graphic adapter
Memory …... Network
Network
4 w
A simplified modern computer system.
…...
Source: Intel Labs, 2008
Outline • Why power became a salient issue • Server power breakdown by components • Practices to build energy-efficient servers • A case study of the energy consumption of virtual machine I/O • Data center power consumption • Practices to build energy-efficient data centers
Practices for energy-efficient servers • Improving power delivery efficiency • Reducing processor and chipset current leakage • Avoiding unnecessary wastes • • • •
Right provisioning Operating at lowest required power Sleep at idle Proper infrastructure management • Consolidating workloads and shutdown idle servers • Throttling the speed of fan • Shutdown surplus air conditioning units
Reducing current leakage: power reduction with high-k gate dielectric materials • 83% reduction in floor space • 87% reduction in energy cost • Full payback on the new servers in less than 2 years
Projected Data Center Power Consumption (for CPU nodes) Source: https://people.eecs.berkeley.edu/~istoica/classes/cs294/09/CERN_Whitepaper_r04.pdf
Right-provisioning: Lamp with multiple brightness levels
https://www.amazon.com/TaoTronics-Eye-caring-Dimmable-Charging-Control/
Operating at lowest required power: Power reduction with CPU C–state support
Source: ACPI specification 6.1
System power states States S0 S1 S2 S3 S4 S5
Description Working Sleeping with processor context maintained Sleeping without processor context being maintained; DRAM context maintained Similar to S2, but deeper sleeping DRAM context is NOT maintained Soft off, without saving any context, requires a complete boot when awakened
Processor power states States C0 C1 C2 C3
Description Executing instructions Maintain the context of the system caches Similar to C1, mainly for multi-core processors Prevent system bus masters from writing into memory
Power reduction with CPU C-state support (cont’d) WebBench Power Measurements Approximate CPU utilization (DBS off)
Typical CPU Utilization 15%
30%
45%
System Power with DBS 258 disabled
291
316
System Power with DBS 201 enabled
220
240
DBS power saving per system
24%
24%
22%
DRAM power savings • Dynamic CKE: 100+ nanosecond • Self refresh: 100+ microseconds
Server DRAM power per DIMM Source: Intel Labs, 2008
Summary of DRAM power savings Item
Lowest power option
Power savings per DIMM
Number of DIMM/ Component density
Use less DIMM. Use one 16 GB instead of 2 x 8GB or 4 x 4GB
5-21 W
Memory configuration/ Memory rank
Choose the highest number of ranks ~8W for a given capacity. Choose x8 instead of x4
DIMM speed
Use DIMM with lower frequency. Use DDR3-1067, instead of DDR31333. Note this reduces performance.
1.5-2.5 W
Latency timing
Use DIMM with higher latency DDR timings. Note this reduces performance as well.
0.3-0.5 W
Memory power management
Enable CKE or self refresh if it’s applicable
Idle power reduction varies
Saving storage (e.g. HDD) power: DPM
Power-state machine.
Power at each state • Read/Write 6.00 Watts • Idle 5.50 Watts • Standby 0.80 Watts
Switch to Standby mode root@debdev:~# hdparm -y /dev/sdb
DPM-enabling techniques • Power-aware caching and buffering • Workload consolidation by replicating data across disks • Popular data concentration by migrating data across disks
Application VFS Page Cache Flash Cache Disk FS / Blk. Dev. File Generic Blk. Layer I/O Sched. Blk. Dev. Drv. HDD/SSD
Write Off-Loading: a practice of power-aware buffering
Source: Narayanan, et. al., Write Off-Loading: Practical Power Management for Enterprise Storage, FAST’08
Power management and performance: A real story
• Cassandra key-value database: request timeout • 10+ engineers debugged for 2+ weeks, no clear idea • A weekend, an engineer was reading something related to CPU power saving and performance, which gave him some hints to suspect BIOS power setting was the culprit
Outline • Why power became a salient issue • Server power breakdown by components • Practices to build energy-efficient servers • A case study of the energy consumption of virtual machine I/O • Data center power consumption • Practices to build energy-efficient data centers
Virtual machine • Show VirtualBox
Traditional
Storage stacks: traditional vs. virtualization Application VFS Page Cache Flash Cache Disk FS / Blk. Dev. File Generic Blk. Layer I/O Sched. Blk. Dev. Drv. HDD/SSD
~2us
~10us
40+ us
DRAM caches: VM-side vs. Hypervisor-side vs. Native Benchmark
Benchmark Benchmark
VM
VM
Host &. Hypervisor
Host &. Hypervisor
Host &. Hypervisor
Storage
Storage
Storage 29
Power consumption of DRAM caches Benchmark
Benchmark
VM Hypervisor Storage Benchmark
Hypervisor Storage
30
Power consumption (watts)
VM Hypervisor Storage
Native
VM-side
Hypervisor-side
25
170%
20 15
10 5 0 1
2
3
4
5
6
7
8
9
10
IOPS (K) 30
Outline • Why power became a salient issue • Server power breakdown by components • Practices to build energy-efficient servers • A case study of the energy consumption of virtual machine I/O • Data center power consumption • Practices to build energy-efficient data centers
Data center organization
1U server
A small cluster
A 7-feet rack A row of servers in a Google data center Source: L. Barroso et. Al., The Datacenter as a Computer An Introduction to the Design of Warehouse-Scale Machines, Second Edition, 2013
Data center services: IaaS • Google compute engine
Outline • Why power became a salient issue • Server power breakdown by components • Practices to build energy-efficient servers • A case study of the energy consumption of virtual machine I/O • Data center power consumption • Practices to build energy-efficient data centers
Virtual Machine Migration
35
Data center-wide power management: VM Migration
Comparison of Energy Consumption GreenCloud Architecture Source: Liang Liu et. al., GreenCloud: A New Architecture for Green Data Center, ICAC’09
Data center-wide power management: Power Capping
A real-world case study of how Dynamo prevented a potential power outage. Source: Qiang Wu et. al., Dynamo: Facebook’s Data Center-Wide Power Management System, ISCA’16
Thanks
