UDP will consume most of the bandwidth. 70% BW ... Assume core is over-
provisioned and manage ... o Leverage Linux traffic control mechanism (HTB) for.
Gatekeeper: Supporting Bandwidth Guarantees for Multi-tenant Datacenter Networks Henrique Rodrigues , Yoshio Turner , Jose Renato Santos , Paolo Victor , Dorgival Guedes
HP Labs
WIOV 2011, Portland, OR
The Problem: Network Performance Isolation
Suppose that you have a datacenter…
…
Suppose that you have a datacenter…
And you are an IaaS provider … 70% BW
And you are an IaaS provider … 70% BW 30% BW
... and your network faces this traffic pattern: 70% BW 30% BW
TCP
TCP
... and your network faces this traffic pattern: 70% BW 30% BW
TCP
TCP
... and your network faces this traffic pattern: 70% BW 30% BW
TCP
TCP
TCP is flow-based, not tenant-aware...
It becomes worse with these transport protocols:
70% BW 30% BW
TCP
UDP
It becomes worse with these transport protocols:
70% BW 30% BW
TCP
UDP
UDP will consume most of the bandwidth
It becomes worse with these transport protocols:
70% BW 30% BW
TCP
UDP
Using rate limiters at each server doesn’t solve the problem…
It becomes worse with these transport protocols:
70% BW 30% BW
If you limit the rate of each at 30% TCP
UDP
Using rate limiters at each server doesn’t solve the problem…
It becomes worse with these transport protocols:
70% BW 30% BW
The aggregate RX is 90% TCP
UDP
Using rate limiters at each server doesn’t solve the problem…
The Problem: Network Performance Isolation • How can we enforce that all tenants will have at least the minimum amount of network resources they need to keep their services up?
o
In other words, how to provide network performance isolation to multi-tenant datacenters?
Practical requirements for a traffic isolation mechanism/system
Requirements for a practical solution • Scalability Datacenter supports thousands of physical servers hosting 10s of thousands of tenants and 10s to 100s of thousands of VMs
• Intuitive Service Model Straightforward for tenants to understand and specify their network performance needs
• Robust against untrusted tenants IaaS model allows users to run arbitrary code as tenants, giving users total control over the network stack. Malicious users could jeopardize the performance of other tenants
• Flexibility / Predictability What should we do with the idle bandwidth? Work conserving vs non-work conserving?
Existing solutions don t meet all these requirements
Solution
Scalable
Flexibility / Predictability
Intuitive Model
Robustness
TCP
✔
✗
✗
✗
BW Capping (policing)
✔
✗
✔
✗
Secondnet
✔
✔
✗
✔
Seawall
✔
✗
✗
✔
AF-QCN
✔
✗
✗
✔
Our approach
Assumption
Bisection bandwidth should not be a problem: • Emerging multi-path technologies will enable high bandwidth networks with full-bisection bandwidth • Smart tenant placement: tenant VMs placed close to each other in the network topology • Results on DC traffic analysis show that most of the congestion happens within racks, not at the core
Our approach • Assume core is over-provisioned and manage bandwidth at edge o
Addresses scalability challenge: Limited number of tenants in each edge link
Tenant Performance Model Abstraction VM
BW1
VM
VM
BW2
VM
BW3 BW4
BW10
VM
BW5
BW9
VM
VM
BW8 VM
BW7
BW6
VM
VM
• Simple abstraction to tenant o
Model similar to physical servers connected to a switch
• Guaranteed bandwidth for each VM (TX and RX) o
Minimum and Maximum rate per vNIC
Gatekeeper • Provides network isolation for multi-tenant datacenters using a distributed mechanism
• Agents implemented at the virtualization layer coordinate bandwidth allocation dynamically, based on tenants guarantees
Gatekeeper • Agents in the VMM control the transmission (TX) and coordinate the reception (RX)
Gatekeeper - Overview 70% BW 30% BW
TCP
UDP
Gatekeeper - Overview 70% BW 30% BW
Congestion! TCP
UDP
Gatekeeper - Overview 70% BW 30% BW
TCP
UDP
OK! Reducing TX
OK! Reducing TX
OK! Reducing TX
Gatekeeper Architecture
Gatekeeper Prototype o
Xen/Linux
o
Gatekeeper integrated into Linux Open vSwitch
o
Leverage Linux traffic control mechanism (HTB) for rate control
Example - RX 2 Tenants share a gigabit link: • Tenant A
o 70% of the link, o 1 TCP • Tenant B
Flow
o 30% of the link, o
3 Flows (TCP or UDP)
Example - TX 2 Tenants share a gigabit link: • Tenant A
o 70% of the link, o 1 TCP • Tenant B
Flow
o 30% of the link, o
3 Flows (TCP or UDP)
Example – Results without Gatekeeper
Transmit (TX) Scenario 1000 900 800 700 600 500 400 300 200 100 0
no control TX rate cap
none TCP UDP
none TCP UDP
Type of traffic for tenant B
Tenant A (TCP)
Tenant B
Example – Results without Gatekeeper
Transmit (TX) Scenario 1000 900 800 700 600 500 400 300 200 100 0
Receive (RX) Scenario
no control TX rate cap
none TCP UDP
1000 900 800 700 600 500 400 300 200 100 0
none TCP UDP
none TCP UDP
Type of traffic for tenant B
no control RX rate cap
none TCP UDP
Type of traffic for tenant B Tenant A (TCP)
Tenant B
Example – Results without Gatekeeper
•
Bandwidth Capping doesn t reallocate unused bandwidth (non work-conserving)
•
UDP consumes most of the switch resources
Transmit (TX) Scenario 1000 900 800 700 600 500 400 300 200 100 0
Receive (RX) Scenario
no control TX rate cap
none TCP UDP
1000 900 800 700 600 500 400 300 200 100 0
none TCP UDP
none TCP UDP
Type of traffic for tenant B
no control RX rate cap
none TCP UDP
Type of traffic for tenant B Tenant A (TCP)
Tenant B
Example – Results with Gatekeeper
Receive (RX) Scenario
Transmit (TX) Scenario Gatekeeper Gatekeeper 1000 900 800 700 600 500 400 300 200 100 0
no control TX rate cap
predictable
flexible
1000 900 800 700 600 500 400 300 200 100 0 none TCP UDP
none TCP UDP
none TCP UDP
Type of traffic for tenant B
none TCP UDP
Tenant A (TCP)
Gatekeeper no control RX rate cap predictable
none TCP UDP
none TCP UDP
none TCP UDP
Type of traffic for tenant B Tenant B
Gatekeeper flexible
none TCP UDP
Summary • Gatekeeper provides network bandwidth guarantee at the server virtualization layer § Extends hypervisor to control RX bandwidth
• Prototype implemented and used to demonstrate Gatekeeper in simple scenario • Future work § Evaluate Gatekeeper at larger scales § HP Labs Open Cirrus testbed (100+ nodes)
§ Further explore the design space § Functions to decrease/increase rate, etc
§ Evaluate Gatekeeper with more realistic benchmarks and applications
Gatekeeper: Supporting Bandwidth Guarantees for Multi-tenant Datacenter Networks Contacts:
{hsr,dorgival}@dcc.ufmg.br {yoshio_turner,joserenato.santos}@hp.com
Acknowledgements:
Brasil WIOV 2011, Portland, OR
