Application-Specific Customization and Scalability of ...

Scalable Network Virtualization Using FPGAs

Deepak Unnikrishnan, Ramakrishna Vadlamani, Yong Liao, Abhishek Dwaraki, Jérémie Crenne, Lixin Gao and Russell Tessier Funded by National Science Foundation Grant CNS-0831940

Electrical and Computer Engineering University of Massachusetts, Amherst USA

European University of Brittany UBS – Lab - STICC Lorient, France

Outline  Network virtualization  Conventional approaches to network virtualization  Architecture of FPGA based virtualization system  Scalable implementation  Results  Conclusions

2

Introduction  Internet growing to include new applications and services  Cloud computing, Data center networking  Challenges  Lack of innovation at network core – Fixed internet routers  Coupled infrastructure-service provider model

 Solution  Network virtualization  Router hardware shared across multiple networks

3

Network Virtualization  Many logical networks over a physical infrastructure

B A

F E

C

 Virtual nodes  Shared network resources among multiple virtual networks

D

D

B A

F C

E

 Reduces costs  Independent routing policies

B A

C

D F E 4

Virtual Router  Independent routing policies for each virtual router  Key challenges • Isolation • Performance • Flexibility • Scalability

Physical router Virtual router A

Virtual router B

Routing Control

Routing Control

Forwarding Table

Forwarding Table

DEMUX

MUX

5

Traditional Network Virtualization Techniques  Software  Full virtualization  Container virtualization  Limitations  Limited performance (~100Mbps)  Limited isolation  ASIC  Supercharging PlanetLab Platform1  Juniper E series  Possible Limitations  Flexibility  Scalability 1

VM

VM

HW Full virtualization

OS instance

OS instance

HW Container virtualization

“Supercharging PlanetLab – A High Performance, Multi-Application, Overlay Network Platform”, J. Turner et al., 6 SIGCOMM 2007

Virtualization using FPGAs  A novel network virtualization substrate which • Uses FPGA to implement high performance virtual routers • Introduces scalability through virtual routers in host software • Exploits reconfiguration to customize hardware virtual routers Virtual Router 1 Virtual Router 2

A FP G

Virtual Router 3

Virtual Router 4

7

System Overview Software Virtual Router

NIC

Software Virtual Router

Software Virtual Router

NIC

Software bridge Kernel driver PCI

Linux

SDRAM

SRAM HW VRouter

SDRAM

HW VRouter

PHY

1G Ethernet I/F

SRAM

NetFPGA 8

Scalable Network Virtualization  Approaches for implementing scalable virtual routers • Single receiver approach • All packets routed through NetFPGA hardware • Described in paper • Multi-receiver approach • Use a switch to separate packets to NetFPGA and software

9

Architecture PHY MAC

MAC RX Q CPU RX Q

PHY MAC

MAC TX Q Dynamic Design Select

Input Arbiter

MAC RX Q

Output Queue Hardware Virtual Router 2

CPU RX Q PHY MAC

MAC RX Q

VIP 0 1 2 3

CPU RX Q

NetFPGA

TYPE HW HW SW SW

NetFPGA driver Bridge

To Switch

PC NIC

Host Workstation

CPU TX Q MAC TX Q

MAC TX Q CPU Transceiver

Software Virtual Router 2

PHY MAC

PHY MAC

CPU TX Q PHY MAC

CPU TX Q

PCI BUS CONTROL Software Virtual Router 1

Kernel space NetFPGA PCI I/F

ID 0 1 0 1

PHY MAC

CPU TX Q

MAC RX Q CPU RX Q

PHY MAC

MAC TX Q

Hardware Virtual Router 1

Kernel space NetFPGA driver

NetFPGA PCI I/F

Bridge PC NIC

From Switch

User space

10

Single receiver approach PHY MAC

MAC RX Q CPU RX Q

PHY MAC

MAC TX Q Dynamic Design Select

Input Arbiter

MAC RX Q

Output Queue Hardware Virtual Router 2

CPU RX Q PHY MAC

MAC RX Q

VIP 0 1 2 3

CPU RX Q

NetFPGA

TYPE HW HW SW SW

NetFPGA driver Bridge

To Switch

PC NIC

Host Workstation

CPU TX Q MAC TX Q

MAC TX Q CPU Transceiver

Software Virtual Router 2

PHY MAC

PHY MAC

CPU TX Q PHY MAC

CPU TX Q

PCI BUS CONTROL Software Virtual Router 1

Kernel space NetFPGA PCI I/F

ID 0 1 0 1

PHY MAC

CPU TX Q

MAC RX Q CPU RX Q

PHY MAC

MAC TX Q

Hardware Virtual Router 1

Kernel space NetFPGA driver

NetFPGA PCI I/F

Bridge PC NIC

From Switch

User space

11

Multi Receiver Approach MAC RX Q CPU RX Q PHY MAC

MAC RX Q

MAC TX Q Dynamic Design Select

Input Arbiter

Output Queue Hardware Virtual Router 2

CPU RX Q PHY MAC

MAC RX Q

VIP 0 1 2 3

CPU RX Q

Source

NetFPGA

TYPE HW HW SW SW

ID 0 1 0 1

PHY MAC

CPU TX Q

MAC RX Q CPU RX Q

PHY MAC

MAC TX Q

Hardware Virtual Router 1

PHY MAC

CPU TX Q MAC TX Q

PHY MAC

CPU TX Q MAC TX Q

CPU Transceiver

PHY MAC

CPU TX Q

Destination

PHY MAC

PCI BUS CONTROL NetFPGA PCI I/F

Software Virtual Router 1

NetFPGA driver Bridge

Switch

PC NIC

Host OS

Kernel space

NetFPGA driver

NetFPGA PCI I/F

Bridge Software Virtual Router 2 User space

PC NIC

Switch

Kernel space

12

Virtual Router Customization  Multiple virtual routers share the substrate  Individual virtual routers may need customization  Challenge • Minimize the impact on traffic in shared hardware virtual routers during modification

13

Dynamic Reconfiguration Reconfigure FPGA MAC RX Q

New HW Hardware Virtual Virtual Router 1 Router

CPU RX Q PHY MAC

MAC RX Q CPU RX Q

PHY MAC

MAC RX Q

VIP 0 1 2 3

CPU RX Q

Source

NetFPGA

Output Queue

TYPE HW HW SW SW

ID 0 1 0 1

PHY MAC

CPU TX Q MAC TX Q

PHY MAC

CPU TX Q MAC TX Q

CPU Transceiver

PHY MAC

CPU TX Q

PCI BUS CONTROL NetFPGA PCI I/F

SwSoftware Virtual Virtual Router Router 1

NetFPGA driver Bridge

Switch

Address Remap

MAC TX Q

Hardware Virtual Router 2

MAC RX Q

PHY MAC

CPU TX Q

Dynamic Design Select

Input Arbiter

CPU RX Q PHY MAC

MAC TX Q

Destination

PHY MAC

PC NIC

Host OS

Kernel space

NetFPGA driver

NetFPGA PCI I/F

Bridge Software Virtual Router 2 User space

PC NIC

Switch

Kernel space

14

Experimental Approach  Metrics • Throughput • Latency

Source NetFPGA Pktgen/ iPerf

1Gbps

Virtual Router

1Gbps

Sink NetFPGA Pktcap/ iPerf

 Packet generation • NetFPGA packet generator • iPerf  Ping utility used for latency measurements  Software virtual routers run on 3Ghz AMD X2 6000+ processor, 2GB RAM, Intel E1000 GbE in PCIe slot 15

Throughput of Single Hardware Virtual Router  Hardware virtual router 1 to 2 order better than the software virtual router

 Consistent forwarding rates vs packet size Throughput (Mbps)

1000

Hardware virtual router

100

Software virtual router

10 1 64

128 512 1024 Packet size (bytes)

1470

16

Scalability - Average Throughput of Virtual Routers  Aggregate throughput of 1Gbps upto 4 h/w virtual routers  Adding software virtual routers causes drop in aggregate throughput

Throughput (Mbps)

1000 Avg hardware and software

100

Software only

10

1 1

2

3

4

5

6

7

8

9 10 11 12 13 14 15

Virtual routers

17

Scalability – Average Latency of Virtual Routers  Latency of h/w virtual router an order better than software virtual router 1.2 Software only

Latency (ms)

1 0.8

Avg hardware and software

0.6 0.4 0.2 0 1

2

3

4

5

6

7

8

9 10 11 12 13 14 15

Virtual routers

18

Dynamic Reconfiguration Overhead  Experiment • • • •

Source pings destination through a single h/w virtual router Migrate traffic to software Reconfigure FPGA Migrate traffic to hardware

 12 seconds reconfiguration overhead • Instantiation of software virtual router (4 seconds) • Bitstream download (5 seconds) • Address remapping back to hardware (3 seconds) 19

Resource Usage and Power Consumption  Virtex 2 can support • 5 h/w data planes (without support for software planes) • 4 h/w data planes (with support for software planes)

 Virtex 5 can support up to 32 virtual planes  A single h/w virtual router consumes • 156mW static power • 688mW dynamic power

 Clock gating saves 10% power 20

Conclusion  A novel heterogeneous network virtualization substrate using FPGAs • High performance • 2 orders of magnitude faster than software virtual routers • Scalable • Number of virtual routers not limited by FPGA logic capacity

• Flexible • Reconfiguration and dynamic migration • Isolation • Dedicated resources for each virtual router

21