Service Profile / Fabric Selection ... Bandwidth Scheduler. Bandwidth Scheduler .... Assign each Class of Service a mini
VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus 10GE Networking Brad Hedlund, CCIE #5530 Data Center Architect Cisco Systems
[email protected] BRAD HEDLUND .com
v 2.5
- About the Author Brad Hedlund Technical Solutions Architect, Data Center Cisco Systems, Inc. CCIE #5530 http://bradhedlund.com/about/
Blog: http://bradhedlund.com Twitter: http://twitter.com/bradhedlund E-mail:
[email protected]
Comments welcome.
BRAD HEDLUND .com
Cisco UCS Networking Overview SAN A
ETH 1
ETH 2
SAN B
MGMT
MGMT
6100 Fabric A
6100 Fabric B
F E X
Chassis 1
F E X
F E X
Chassis 2
F E X
A
CNA B200
B
A
CNA CNA B250
B
BRAD HEDLUND .com
Emulex/Qlogic CNA FEX
FEX
Fabric B
Fabric A
10GE
10GE
CNA
vHBA
vNIC
vNIC
vHBA
vhba0
vmnic0
vmnic1
vhba1
vSwitch / Nexus 1000V BRAD HEDLUND .com
Emulex/Qlogic CNA – Logical View Fabric Interconnect B
Fabric Interconnect A vfc 1
vEth 1
vEth 2
FEX
vfc 2
FEX
CNA
vhba0
vmnic0
vmnic1
vSwitch / Nexus 1000V BRAD HEDLUND .com
vhba1
Service Profile / Fabric Selection
BRAD HEDLUND .com
Cisco VIC “Palo” FEX
FEX
Fabric B
Fabric A
10GE
10GE
Cisco VIC
vHBA
vNIC
vNIC
vNIC
vNIC
vNIC
vNIC
vNIC
vNIC
vHBA
vhba 0
vmnic 0
vmnic 1
vmnic 2
vmnic 3
vmnic 4
vmnic 5
vmnic 6
vmnic 7
vhba 1
vSwitch / vDS / Nexus 1000V BRAD HEDLUND .com
58
Cisco VIC QoS: Example FEX
FEX
Fabric B
Fabric A
Cisco VIC 10GE
Strict Priority
7
10GE
TX
TX
Bandwidth Scheduler
VIC Mgmt
10%
0
40%
2
1
3
10%
6
0
1
2
3
10%
4
VIC Mgmt
40%
5
6
(8) COS Queues
No Drop
RR
Limit TX 1GE
vNIC 1
40%
10%
No Drop
RR
7
Bandwidth Scheduler
40%
5
4
Strict Priority
Limit TX 1GE
vNIC 2
COS 1
vHBA 1
vNIC 3
vNIC 4
COS 3
COS 4
COS 5
vNIC 5
vNIC 6
COS 1
BRAD HEDLUND .com
vHBA 2
vNIC 7
vNIC 8
COS 3
COS 4
COS 5
58
Cisco VIC – Logical View Fabric Interconnect B
Fabric Interconnect A vfc 1
vEth 1
vEth 2
vEth 3
vEth 4
vEth 5
FEX A
vEth 6
vEth 7
vEth 8
FEX B Cisco VIC
vhba 0
vmnic 0
vmnic 1
vmnic 2
vmnic 3
vmnic 4
vmnic 5
vSwitch / Nexus 1000V BRAD HEDLUND .com
vmnic 6
vmnic 7
vhba 1
vfc 2
I/O Consolidation with 10GE VM Data
vmnic 0
VMotion
vmnic 1
CONCERN: Bandwidth Availability 10GE
MGMT
IP Storage
vmnic 2
vmnic 3
BRAD HEDLUND .com
SOLUTIONS: 1. MAX bw per NIC Rate Limits 2. MIN bw per NIC QoS 3. Do Nothing Let it rip
Network Switch
I/O Consolidation with 10GE
CONCERN: Bandwidth Availability
VM Data VMotion MGMT IP Storage
vmnic 1
SOLUTIONS: 1. MAX bw per Class Rate Limits 2. MIN bw per Class QoS 10GE 3. Do Nothing Let it rip
BRAD HEDLUND .com
Network Switch
MAX Bandwidth – Rate Limits 10GE VM Data
vmnic 0
2.5G
VMotion
vmnic 1
2.5G
vmnic 2
2.5G
vmnic 3
2.5G
MGMT
IP Storage
Network Switch
Sum of Limits must not exceed physical link speed Inefficient use of unused BW This slide animated @ http://bradhedlund.com/2010/08/16/cisco-ucs-qos-vs-hp-virtual-connect-rate-limiting/ BRAD HEDLUND .com
MIN Bandwidth – QoS 10GE VM Data
vmnic 0
VMotion
vmnic 1
Network Switch
vmnic 2
MGMT
IP Storage
vmnic 3
Minimum BW guarantee with no Maximum Efficient: unused BW available to other NICs This slide animated @ http://bradhedlund.com/2010/08/16/cisco-ucs-qos-vs-hp-virtual-connect-rate-limiting/ BRAD HEDLUND .com
MIN Bandwidth – QoS Nexus 1000V Classifies Traffic
Cisco VIC Enforces Policy 10GE
UCS 6100 Nexus 5000 Enforces Policy
VM Data
VMotion Network Switch
vmnic 1
MGMT
IP Storage
BRAD HEDLUND .com
Traffic Considerations Required • Guest VM Data • Management – VMKernel – Service Console
• vMotion • Central Storage Access
Optional • Fault Tolerance (FT) • NFS • iSCSI • FCoE • Realtime – Video/Voice, Market Data
BRAD HEDLUND .com
Cisco UCS + VIC Design (1): Required Traffic Fabric Interconnect B
Fabric Interconnect A vfc 1
vEth 2
vEth 1
vhba 0
vmnic 0
vEth 6
vEth 3
vmnic 2
vmnic 4
vmnic 1
vmnic 3
vEth 7
vEth 8
vmnic 5
vfc 2
vhba 1
vSwitch or vDS
MGMT
VM DATA
vMotion
FC
Primary
vmnic1
vmnic2
vmnic5
MP
Backup
vmnic0
vmnic3
vmnic4
MP
Min BW
10% 1GE Max
40%
10%
40%
Class
Bronze COS 1
Best Effort COS 0 BRAD HEDLUND .com
Silver COS 2
COS 3
UCS Manager QoS System Class Design #1
• Assign each Class of Service a minimum bandwidth weighting • System wide policy
BRAD HEDLUND .com
Apply QoS Policy to vNIC • Create QoS Policy • Assign Policy to vNIC
BRAD HEDLUND .com
Cisco UCS + VIC Design (2): Req + 1 Opt Fabric Interconnect B
Fabric Interconnect A vfc 1
vEth 1
vhba 0
vEth 2
vmnic 0
vEth 3
vmnic 2
vEth 4
vmnic 4
vEth 5
vmnic 6
vmnic 1
vEth 6
vmnic 3
vEth 7
vmnic 5
vmnic 7
vEth 8
vfc 2
vhba 1
vSwitch or vDS
MGMT
VM DATA
vMotion
NFS
FC
Primary
vmnic1
vmnic2
vmnic5
vmnic7
MP
Backup
vmnic0
vmnic3
vmnic4
vmnic6
MP
Min BW
10% 1GE Max 30%
10%
10%
40%
Class
Bronze COS 1
Silver COS 2
Gold COS 4
COS 3
Best Effort COS 0 BRAD HEDLUND .com
UCS Manager QoS System Class Design #2
• Assign each Class of Service a minimum bandwidth weighting • System wide policy BRAD HEDLUND .com
Cisco UCS + VIC Design (3): Req + 2 Opt + Realtime Fabric Interconnect B
Fabric Interconnect A vfc 1
vhba 0
vEth 1
vmnic 0
vEth 2
vEth 3
vmnic 2
vmnic 4
vEth 4
vEth 5
vmnic 6
vmnic 8
vEth 6
vEth 7
vEth 8
vmnic 1
vmnic 3
vmnic 5
vEth 9
vmnic 7
vEth 10
vmnic 9
vfc 2
vhba 1
vSwitch or vDS
MGMT
vMotion
Realtime
FT
VM DATA
NFS
FC
Primary
vmnic0
vmnic1
vmnic2
vmnic5
vmnic6
vmnic9
MP
Backup
vmnic1
vmnic0
vmnic3
vmnic4
vmnic7
vmnic8
MP
Min BW
1GE Max
10%
10%
20%
10%
10%
40%
Class
Bronze COS 1
Bronze COS 1
Plat COS 5 Gold COS 4 BRAD HEDLUND .com
Best Effort COS 0
Silver COS 2
COS 3
UCS Manager QoS System Class Design #3
• “Platinum” Class for Realtime traffic • Lossless Ethernet and Multicast optimized • “Silver” and “Bronze” for vMotion & NFS with Jumbo Frames BRAD HEDLUND .com
Design #4: (2) 10GE NICs with Nexus 1000V Cisco UCS or Nexus 5000 Fabric Interconnect A / Nexus 5000 #1 vfc 1
Fabric Interconnect B / Nexus 5000 #2 vfc 2
vEth 7
vEth 4
QoS Enforcement
Cisco VIC or Other adapter Cisco UCS server or non-Cisco vhba 0
vmnic 0
QoS Enforcement (if Cisco VIC)
vmnic 1
QoS Enforcement (if N1K v1.4)
NEXUS 1000V
MGMT
vMotion
QoS Classification
Realtime
FT
VM DATA
NFS
vhba 1
FC
Primary
vmnic0
vmnic1
vmnic0
vmnic1
vmnic0
vmnic1
MP
Backup
vmnic1
vmnic0
vmnic1
vmnic0
vmnic1
vmnic0
MP
Min BW
1GE Max
10%
10%
20%
10%
10%
40%
Class
Bronze COS 1
Bronze COS 1
Plat COS 5 Gold COS 4 BRAD HEDLUND .com
Best Effort COS 0
Silver COS 2
COS 3
vNIC QoS Policy for Design #4
• Preserve and use COS markings from Nexus 1000V • Same QoS System Class settings as Design #3
BRAD HEDLUND .com
Design #5: (2) 10GE NICs, no Cisco VIC, no N1K Nexus 5000 Nexus 5000 #2
Nexus 5000 #1 vfc 1
Eth 1/1
Eth 1/1
QoS Enforcement
vfc 2
QoS Classification
Cisco VIC or Other adapter Cisco C-Series or non-Cisco vhba 0
vmnic 0
vmnic 1
VMware NetIOC ? FCoE?
vSwitch or vDS
MGMT
vMotion
Realtime
FT
VM DATA
vhba 1
NFS
FC
Primary
vmnic0
vmnic1
vmnic0
vmnic1
vmnic0
vmnic1
MP
Backup
vmnic1
vmnic0
vmnic1
vmnic0
vmnic1
vmnic0
MP
Min BW
1GE Max
10%
10%
20%
10%
10%
40%
Class
Bronze COS 1
Bronze COS 1
Plat COS 5 Gold COS 4 BRAD HEDLUND .com
Best Effort COS 0
Silver COS 2
COS 3
Narrative posted at: http://bradhedlund.com/?p=1650
BRAD HEDLUND .com