A Flexible Switch-Router with Reconfigurable Forwarding and Linux ...

A Flexible Switch-Router with Reconfigurable Forwarding and Linux-based Control Element István Moldován

Pál Varga

Budapest University of Technology and Economics Inter-University Cooperative Research Centre, Magyar Tudósok krt. 2, H-1117 Budapest, Hungary [email protected]

Budapest University of Technology and Economics Dept. of Telecommunications and Media Informatics Magyar Tudósok krt. 2, H-1117 Budapest, Hungary [email protected]

Abstract— The practical validation of investigating new router architectures and control algorithms is troublesome. It is partially due to the lack of programmable architectures that are fast enough for current networking, widely programmable, and equipped with the great variety of protocols that routers and switches has to handle. The idea of open, software-based routers running on commodity hardware have spread in recent years, resulting several practical implementations of so called softrouters. Beside the economic advantages of having the Forwarding Element functions tackled by commodity Network Interface Cards (NICs), this approach suffers from issues of suboptimal performance. This is partly due to the fact that all forwarding decisions are made by the main CPU instead of involving local processors at the NICs. Separating the functions into Forwarding Element and Control Element by the ForCES framework enabled the development of new control algorithms and protocol implementations for routers, however there are no widely available implementations. This paper describes a solution that is utilizing programmable hardware accelerators at the Forwarding Element, while keeping the advantages of open, Linux-based software-based router implementations. The hardware accelerators are implemented in the C-board, a reconfigurable, FPGA-based networking platform. The interoperability with existing Control Elements are described together with an analysis of features and capabilities of this combined solution.

implementation provides both functions, implemented all in software, providing a great flexibility. However, interworking with switching or routing hardware is not considered as of yet. In this paper we aim to achieve a seamless integration between the C-board [1] Generic Packet Forwarding architecture and the control protocols running on the Linux-based PC board. Figure 1 depicts the functional separation and the place of the defined - general - interface. The solution should fulfill the following requirements: • • • • •

Decoupling of packet forwarding at Layer 2 and 3. Redirect control plane traffic to Linux, and allow sending control plane traffic on the network interfaces Provide a consistent view on network interfaces (both real and virtual) towards Linux Reflect control plane changes to the data plane in a fast and reliable manner achieve these through allowing most control protocol implementations at Linux to run without changes

Keywords - switching; routing; Linux; hardware;

I.

INTRODUCTION

The implementations of such openly programmable software-based routers, which are able to run in commodity hardware have spread in recent years. These solutions enable researchers to experiment with new and open software of networking control algorithms. The aim of this paper is to suggest an interface between hardware (HW) and software (SW) components: which functions are more efficient to be done in HW, and which should be left for SW. This separation allows accelerating the development process (e.g. the complete set of protocols for soft-routers can be used), while provides better runtime performance (e.g. HW-based forwarding). The two types of intelligent, fundamental elements of current packet-switched networks are switches (or in other terminology, multi-port bridges), which work at OSI layer 2, and routers, working at OSI layer 3. The Linux networking

Figure 1. - Finding the optimal interface between reconfigurable hardware and Linux-based switching and routing software

II.

BACKGROUND AND MOTIVATION

Delegation of specific high speed function to hardware, and separating control and data planes has been in interest for long time. Let us summarize the main milestones of this effort. Open, standardized interface was sought and one of the most complete solutions was P.1520, which is an IEEE standards development project created in early 1998 [2]. Its aim is to establish an open architecture for network control and define the interface between network control and management functions. Nevertheless, the Linux routing implementation is based on a newer IETF standard. The possibility of creating open router architectures, and better structures through

experimenting with various algorithms was enabled by the ForCES framework [3][4]. This defined the separation of the Forwarding Element (FE) for executing the packet forwarding; and the Control Element (CE) for implementing the decision making functions. Such an interface is already in use in Linux systems – it is called Netlink [5]. The Linux Netlink protocol fulfils the requirements of the mandatory building blocks within the ForCES charter (i.e., IPv4 and QoS). Netlink is used to transfer information between kernel modules and user space processes. It consists of a standard sockets based interface for user processes and an internal kernel API for kernel modules. Rtnetlink allows the kernel's routing tables to be read and altered [6]. It is used within the kernel to communicate between various subsystems and for communication with user-space programs. Network routes, IP addresses, link parameters, neighbor setups, queuing disciplines, traffic classes and packet classifiers may all be controlled through NETLINK_ROUTE sockets. Lately, the Linux kernel makes possible to alter also bridge interface related information as well. There are many existing and upcoming protocol standards that can be implemented in user space. These implementations will require manipulating the routing and knowing what is being modified by other processes. Examples of such protocols are the dynamic routing protocols including the Routing Information Protocol (RIP), Open Shortest Path First (OSPF) and Exterior Gateway Protocol (EGP) that - beside other features - actively manage the routing environment of a host. In the literature there are few open source hardware routing protocol implementations. The NetFPGA reference router uses a modified OSPF routing protocol thus only supports OSPF. As for switching hardware, even if there are existing switch hardware in open-source devices like in the OpenWRT supported routers with internal switch chip, the cooperation with the control plane (namely the spanning tree protocol) is not implemented. Nevertheless, the VLAN functions are used in this implementation as well. ForTER is a ForCES Router implementation [7], however it is hardware dependent and not publicly available. Another possibility nowadays is the use of the OpenFlow protocol on the supporting hardware. Although its approach is successful nowadays, OpenFlow is aiming to achieve a different goal: total separation of control plane, and keeping the forwarding element as a black/box. In our approach both the FE and the CE is programmable and

reconfigurable. This gives complete freedom both for researchers and for developers who wish to investigate new routing architectures. The interface we propose keeps the development of the CE in shape by providing the supportive Linux environment with all its open and pre-written protocol handlers. On the other hand, the FE can also be enhanced and optimized, since the FPGA-based hardware provides a flexible design space. Another advantage is that its processing power and memory capacity is independent from the resources of CE. III.

GENERIC PACKET FORWARDING ARCHITECTURE

The C-board is a prototype platform for the communication infrastructure-related applications [1]. The main purpose of the C-board is to provide a basis for high-speed data processing and manipulation. It could either host or serve monitoring, switching, routing, filtering and other applications that require real-time processing of 10 Gigabit Ethernet traffic. The main components of the device are the following: • XFPs for 10 Gbps Ethernet traffic reception, • SFPs (Small Form/factor Pluggables) to handle various Gigabit Ethernet ports, for output to devices, • four FPGAs connected in a bidirectional ring, used for packet capture and manipulation, including interface handling, traffic flow handling firmware blocks, etc., • memory for packet buffering and flow tables, • extra processor for on-board processing and management software. A. Architecture The Generic Packet Forwarding Architecture is a packet processing framework for the reconfigurable hardware capable of forwarding packets in a scalable way. To achieve this goal, the packets are stored in a dynamic RAM (DDR2) and all queuing are done using packet headers only. Its main strength is the modular structure, which ensures scalability and also the possibility and extensions. The modular structure of the architecture is depicted by Figure 2. Packet forwarding is determined by the lookup module. The lookup module can perform either Layer 2 forwarding (switching) or Layer 3 forwarding (routing) or both. Both forwarding methods however require control algorithms in order to maintain the forwarding tables up-to-date. Interworking with higher level protocols running on the integrated PC are required.

Figure 2. – Generic Packet Forwarding

B. Lookup process The lookup module implements a multi layer packet forwarding: switching and routing. The switch module has interfaces towards the Linux PC and IP lookup. Figure 3 shows the main processes inside the lookup module.

Figure 3. – Lookup module of the Generic Packet Forwarding architecture

IV.

INTEGRATION

The optimal integration of a reconfigurable hardware with the Linux running on PC would be achieved if all higher level protocols (available by/through Linux) were able to run without modifications, while packet forwarding is running on the Cboard hardware/firmware. This of course requires that all interfaces must be visible for the Linux-based implementation.

Cboard driver

EthN

Eth2

Eth1

The integration is achieved by using an extended Linux driver, which provides a transparent view of the hardware interfaces to the upper layers, while mirrors control plane decisions to the board. Figure 4 depicts the internal functions and connections of the C-board driver.

Net_device handler

Netlink socket Netlink listener

PCIe interface

PCIe Figure 4. – Internal functions of the extended Linux device driver

A. IP forwarding For IP forwarding the C-board device driver must provide the following functions: create the IP interfaces, make possible the sending/receiving of control messages through these interfaces and reflect the routing table changes to the hardware lookup. First, the driver creates interfaces according to the configuration. Then it uses the internal packet header information of the Generic Packet Forwarding Architecture to differentiate between packets of different IP interfaces. When receiving a packet, the incoming port information is used to

send on the right Linux interface. Also, packets from a given Linux interface get a header with proper outgoing port identifier. The IP forwarding in the Linux kernel uses a common routing table, which is shared between the routing protocols. B. Ethernet Switching As previously mentioned, for switching there is no elegant solution. Switching is done in software – for which different bridging implementations exist; just to name a few: Linux kernel Ethernet Bridge, the Open vSwitch [8] and the LISA [9]. The biggest problem lies in the handling of the STP port state changes. The Spanning Tree protocol determines the state of a port, which can be Disabled/Discarding, Forwarding, Blocking/Listening, or Learning state. These port states are determined by the bridging code running on Linux, and it must be also propagated to the C-board as well. There is no universal solution in Linux, however the Rtnetlink has been extended to provide also bridge interface related information as well. The other solution would be to provide a method for setting port states in the driver and modify the bridging implementation to update the port states on the C-board accordingly. This is a good solution in case of Open vSwitch, which claims to support hardware switches as well. C. Virtual interfaces – VLANs, bridges Besides the physical interfaces, several virtual interfaces are also needed in some cases. Such interfaces are the VLAN interfaces and the bridge interfaces. Similarly to bridges, the VLANs may also have virtual interfaces assigned, according to the VLAN implementation selected. In the Linux kernel VLAN implementation all VLANs are handled as virtual interfaces. However, this may not be scalable, therefore other implementations like LISA uses a method similar to the commercially used industrial leader devices, avoiding to create virtual interfaces for all VLANs. In our HW implementation both methods are supported, however while LISA software implementation requires modifications in the Linux VLAN handling – ours handles it natively. D. QoS and Linux Traffic Control Linux provides extensive QoS support through the Linux Traffic Control framework, using queuing disciplines, filters, and classes. These elements are implemented in the kernel, and they provide great flexibility. Although the Rtnetlink provides an interface to pass configuration information to the kernel, in our current work we did not consider the HW implementation of a configurable traffic control module. V.

INTEROPERABILITY WITH EXISTING SOLUTIONS

There are several projects exploiting the networking possibilities offered by the Linux from high-speed routers (Vyatta [10], Open vSwitch [8]), commodity hardware projects (one-disk Linux routers) to low speed embedded devices (OpenWRT [11]). However most of them rely on the generic packet handling functions, with a few exceptions mostly related to the Layer 2 protocol handling. The generic Linux packet handling supports bridging through a kernel module with user-space control functions

(bridge-utils), and IP forwarding based on a common routing table maintained by user-space routing protocols [16]. For VLAN handling, the MAC-VLAN [12] implementation has been already integrated into the kernel. In our implementation we sought compatibility with these functionalities. A. Switching Since kernel version 2.6.25, the bridging implementation has Rtnetlink support, providing notifications on changes in interface states. Therefore the driver subscribing to these notifications will be able to mirror the port state information towards the hardware. Open vSwitch is multilayer virtual switch designed to enable massive network automation through programmatic extension, while still supporting standard management interfaces and protocols. The Open vSwitch also provides compatibility with the kernel bridging. Therefore the Rtnetlink communication based notifications will work with our code. LISA stands for LInux Switching Appliance. It's a project that aims at building a network multilayer switching device that runs on Linux. Both the RSTP (Rapid Spanning Tree Protocol) protocol implementation and the command line are user-space, and use IOCTL (Input/Output ConTroL) calls to communicate with the kernel module. Since our solution is Rtnetlink based, interworking without modifications in our code is not possible. B. Routing protocol implementations and platforms GNU Zebra and Quagga [13] are free software that manages TCP/IP based routing protocols. They support the most important routing protocols. Quagga is a fork of Zebra project which has been inactive since 2005. Both Zebra and Quagga implement Rtnetlink as a dynamic routing update channel between them and the kernel, therefore interoperability with our solution is ensured. XORP is the eXtensible Open Router Platform [14]. The XORP web page says their goal is “to develop an open source software router platform that is stable and fully featured enough for production use, and flexible and extensible enough to enable network research”. The Vyatta OFR (Open, Flexible Router) implementation is also based on XORP. The modularity and independency of the lower-level from the userlevel subsystem allows for its easy replacement with other solutions including high-end hardware-based forwarding engines. When using traditional kernel forwarding, Rtnetlink is used to update the kernel routing tables, which is compatible with our solution. The BIRD project [15] aims to develop a fully functional dynamic IP routing daemon primarily targeted on (but not limited to) UNIX-like systems. The kernel synchronization module is system dependent and implements the netlink protocol which performs synchronization of BIRD's routing tables with the OS kernel, therefore compatibility is ensured again. VI.

CONCLUSIONS

We have shown an architecture for HW-assisted packet forwarding with SW-implemented control plane supported by

open source Linux. In our approach of splitting the router functionality, both the Forwarding Element and the Control Element are programmable and reconfigurable. This provides a unique freedom for those who wish to investigate new routing architectures. Our proposed interface keeps the supportive Linux environment with all its pre-written protocol handlers and allows its integration with flexible, programmable, HWbased Forwarding Element. The processing power and memory capacity of the Forwarding and Control Elements are independent from each other, which is also advantageous. The reconfigurable C-board HW is responsible for layer 2 and 3 forwarding, decoupling control traffic and forwarding it to the Linux-based software. Moreover, this HW presents a control interface for link state and forwarding table management. We have used the already existing standard, control plane – data plane separation solution available in Linux: the Rtnetlink interface. The software integration only requires an extra device driver, which integrates into the kernel as an interface driver and also integrates into the Rtnetlink notification chain, automatically mirroring modifications related to interface states, bridge port status and routing table. As we have shown, all current open source routing implementations are compatible with our method, but some switch-implementations may require modifications in order to support Rtnetlink-based updating. Furthermore, our solution supports those current and future routing approaches that use Rtnetlink. Essentially we provided an enabler for researching and then implementing better, faster and more optimized algorithms and architectures for routers and switches. REFERENCES [1]

[2] [3] [4] [5] [6] [7]

[8] [9] [10] [11] [12] [13] [14] [15] [16]

P. Varga, I. Moldován, D. Horváth and S. Plósz, “A Low Power, Programmable Networking Platform and Development Environment”, in Advances in Network-Embedded Management and Applications, Springer, Proceedings of NEMA 2010 J. Biswas, et al., “Application Programming Interfaces for Networks”, IEEE P.1520 Whitepaper, January 1999 L. Yang, R. Dantu, T. Anderson, R. Gopal, “Forwarding and Control Element Separation (ForCES) Framework”, IETF RFC 3746, April 2004 H.Khosravi, T. Anderson, - “Requirements for Separation of IP Control and Forwarding”, IETF RFC 3654, November 2003 J. Salim, H. Khosravi, A. Kleen, A. Kuznetsov, “Linux Netlink as an IP Services Protocol”, IETF RFC 3549, July 2003. A. Udugama, “Manipulating the Networking Environment Using RTNETLINK”, Linux Journal, Mar 30, 2006 Wei-Ming Wang, Li-Gang DongBin Zhuge, „Analysis and Implementation of an Open Programmable Router Based on Forwarding and Control Element Separation”, July 8, 2008. Open vSwitch, online: http://openvswitch.org/ LISA Project, online: http://lisa.mindbit.ro/ Vyatta.org open networking community, online: http://www.vyatta.org/ OpenWRT, online: http://openwrt.org 802.1Q VLAN implementation for Linux, online: http://www.candelatech.com/~greear/vlan.html Quagga Project, online: http://www.quagga.net/about.php XORP: Xtensible Open Router Platform, online: http://www.xorp.org/ The BIRD project, http://bird.network.cz Hege Trosvik, Open Source Routing Suites, http://www.interop.com/lasvegas/exhibition/interoplabs/2006/oss/openso urcerouting.pdf