such as hubs, switches, or RJ45 jacks connecting to the outside world. Each wired link in CORE is implemented as an underlying Netgraph pipe node. The pipe ...
CORE: A REAL-TIME NETWORK EMULATOR Jeff Ahrenholz, Claudiu Danilov, Thomas R. Henderson, Jae H. Kim Boeing Phantom Works P.O. Box 3707, MC 7L-49, Seattle, WA 98124-2207 {jeffrey.m.ahrenholz; claudiu.b.danilov; thomas.r.henderson; jae.h.kim}@boeing.com
ABSTRACT We present CORE (Common Open Research Emulator), a real-time network emulator that allows rapid instantiation of hybrid topologies composed of both real hardware and virtual network nodes. CORE uses FreeBSD network stack virtualization to extend physical networks for planning, testing and development, without the need for expensive hardware deployments. We evaluate CORE in wired and wireless settings, and compare performance results with those obtained on physical network deployments. We show that CORE scales to network topologies consisting of over a hundred virtual nodes emulated on a typical server computer, sending and receiving traffic totaling over 300,000 packets per second. We demonstrate the practical usability of CORE in a hybrid wired-wireless scenario composed of both physical and emulated nodes, carrying live audio and video streams. Keywords: Network emulation, virtualization, routing, wireless, MANET 1. INTRODUCTION The Common Open Research Emulator, or CORE, is a framework for emulating networks on one or more PCs. CORE emulates routers, PCs, and other hosts and simulates the network links between them. Because it is a live-running emulation, these networks can be connected in real-time to physical networks and routers. The acronym stems from the initial use of this emulator to study open source routing protocols, but as we describe below, we’ve extended the capabilities of CORE to support wireless networks. CORE is based on the open source Integrated Multiprotocol Network Emulator/Simulator (IMUNES) from the University of Zagreb [1]. IMUNES provides a patch to the FreeBSD 4.11 or 7.0 operating system kernel to allow multiple, lightweight virtual network stack instances [2][3][4]. These virtual stacks are interconnected using FreeBSD’s Netgraph kernel subsystem. The emulation is controlled by an easy-to-use Tcl/Tk GUI. CORE forked from IMUNES in 2004. Certain pieces were contributed back in 2006, and the entire system will soon be released 978-1-4244-2677-5/08/$25.00 ©2008 IEEE
under an open source license. In addition to the IMUNES basic network emulation features, CORE adds support for wireless networks, mobility scripting, IPsec, distributed emulation over multiple machines, control of external Linux routers, a remote API, graphical widgets, and several other improvements. In this paper we present and evaluate some of these features that make CORE a practical tool for realistic network emulation and experimentation. The remainder of this paper is organized as follows: we present related work in Section 2. Then we provide an overview of CORE’s features in Section 3, and highlight the implementation of wireless networking in Section 4 and distributed emulation in Section 5. We then examine the performance of the CORE emulator for both wired and wireless networks in Section 6 and present a typical hybrid emulation scenario in Section 7, and end the paper with our conclusions. 2. RELATED WORK In surveying the available software that allows users to run real applications over emulated networks, we believe that CORE stands out in the following areas: scalability, ease of use, application support, and network emulation features. Simulation tools, such as ns-2 [6], ns-3 [7], OPNET [8], and QualNet [9] typically run on a single computer and abstract the operating system and protocols into a simulation model for producing statistical analysis of a network system. In contrast, network emulation tools, such as PlanetLab [10], NetBed [11], and MNE [12] often involve a dedicated testbed or connecting real systems under test to specialized hardware devices. CORE is a hybrid of the two types of tools, emulating the network stack of routers or hosts through virtualization, and simulating the links that connect them together. This way it can provide the realism of running live applications on an emulated network while requiring relatively inexpensive hardware. Machine virtualization tools, such as VMware [13], Virtual PC [14], or Parallels [15], have become increasingly popular, mainly due to the availability of hardware that can drive multiple operating systems at the
1 of 7
Figure 1. CORE Graphical User Interface
same time with reasonable performance. Operating system virtualization tools, such as Xen [16], UML [17], KVM [18], and OpenVZ [19], are mainly used for isolating multiple Linux server environments driven by the same hardware machine. CORE belongs to the class of paravirtualization techniques, where only part of the operating system is made virtual. In this case, only the isolation of processes and network stacks is employed, resulting in virtual machine instances that are as lightweight as possible. Machine hardware such as disks, video cards, timers, and other devices are not emulated, but shared between these nodes. This lightweight virtualization allows CORE to scale to over a hundred virtual machines running on a single emulation server. From a network layering perspective, CORE provides high-fidelity emulation for the network layer and above, but uses a simplified simulation of the link and physical layers. The actual operating system code implements the TCP/IP network stack, and user or system applications that run in real environments can run inside the emulated machine. This is in contrast to simulation techniques, where abstract models represent the network stack, and protocols and applications need to be ported to the simulation environment.
Because CORE emulation runs in real time, real machines and network equipment can connect and interact with the virtual networks. Unlike some network emulations, CORE runs on commodity PCs. 3. CORE OVERVIEW A complete CORE system consists of a Tcl/Tk GUI, FreeBSD 4.11 or 7.0 with a patched kernel, custom kernel modules, and a pair of user-space daemons. See Figure 2 for an overview of the different components. 3.1. CORE GUI The graphical user interface is scripted in the Tcl/Tk language which allows for rapid development of X11 user interfaces. The user is presented with an empty drawing canvas where nodes of various types can easily be placed and linked together. An example of a running CORE GUI is shown in Figure 1. Routers, PCs, hubs, switches, INEs (inline network encryptors) and other nodes are available directly from the GUI. Effects such as bandwidth limits, delay, loss, and packet duplication can be dynamically assigned to links. Addressing and routing protocols can be configured and the entire setup can be saved to text-based configuration file. A start button allows the user to enter an
2 of 7
“Execute” mode which instantiates the topology in the kernel. Once running, the user may double-click on any node icon to get a standard Unix shell on that virtual node for invoking commands in real-time. In addition, several other tools and widgets can be used to interact with and inspect the live-running emulation. virtual images (vimages)
Tcl/Tk GUI
core_span core_wlan
tunnels CORE API
userspace FreeBSD kernel netgraph system
ng_wlan NIC
Figure 2. Overview of CORE Components
3.2. Network stack virtualization CORE uses the FreeBSD network stack virtualization provided by the VirtNet project [4], which allows for multiple virtual instances of the OS network stack to be run concurrently. The existing networking algorithms and code paths in FreeBSD are intact, but operate on this virtualized state. All global network variables such as counters, protocol state, socket information, etc. have their own private instance [5]. Each virtual network stack is assigned its own process space, using the FreeBSD jail mechanism, to form a lightweight virtual machine. These are named virtual images (or vimages) by the VirtNet project and are created using a new vimage command. Unlike traditional virtual machines, vimages do not feature an entire operating system running on emulated hardware. All vimages run the same kernel and share the same file system, processor, memory, clock, and other resources. Network packets can be passed between virtual images simply by reference through the in-kernel Netgraph system, without the need for a memory copy of the payload. Because of this lightweight emulation support, a single host system can accommodate numerous (over 100) vimage instances, and the maxmimum throughput supported by the emulation system does not depend on size of the packet payload, as we will demonstrate in Section 6. 3.3. Network link simulation Netgraph is a modular networking system provided by the FreeBSD kernel, and a Netgraph instantiation consists of a number of nodes arranged into graphs. Nodes can implement protocols or devices, or may process data. CORE utilizes this system at the kernel level to connect
multiple vimages to each other, or to other Netgraph nodes such as hubs, switches, or RJ45 jacks connecting to the outside world. Each wired link in CORE is implemented as an underlying Netgraph pipe node. The pipe was originally introduced by IMUNES as a means to apply link effects such as bandwidth traffic shaping, delay, loss, and duplicates. One could create a link between two routers, for example, having 512 kbps bandwidth, 37 ms propagation delay, and a bit error of 1/1000000. These parameters can be adjusted on the fly, as the emulation runs. CORE modifies this pipe node slightly for implementing wireless networks and also adds a random jitter delay option. 3.4 External connectivity CORE provides users with a RJ45 node that directly maps to an Ethernet interface on the host machine, allowing direct connectivity between the virtual images inside a running emulation and external physical networks. Each RJ45 node is assigned to one of the Ethernet interfaces on the FreeBSD host, and CORE takes over the settings of that interface, such as its IP address, etc., and also transfers all traffic passing through that physical port to the emulation environment. This way, the user may physically attach any network device to that port and packets will travel between the real and emulated worlds in real time. 4. WIRELESS NETWORKS CORE provides two modes of wireless network emulation: a simple, on-off mode where links are instantiated and break abruptly based on the distance between nodes, and a more advanced model that allows for custom wireless link effects. Nodes, each corresponding to separate vimages, may be manually moved around on the GUI canvas while the emulation is running, or mobility patterns may be scripted. In the current version, CORE wireless emulation does not perform detailed layer 1 and 2 modeling of a wireless medium, such as 802.11, and does not model channel contention and interference. Instead, it focuses on realistic emulation of layers 3 and above, while relying on the adoption of external RF models by providing a standard link model API. The implementation of the on-off wireless mode is based on the Netgraph hub node native to FreeBSD, which simply forwards all incoming packets to every node that is connected to it (Figure 3, left). We added a hash table to the Netgraph hub and created a new wlan node, where a hash of the source and destination node IDs determines connectivity between any two nodes connected to the wlan. The hash table is controlled by the position of the nodes on the CORE GUI. We represent this wlan node as a small wireless cloud on the CORE canvas. Vimage nodes can be joined to a wireless network by drawing a link
3 of 7
between the vimage and this cloud. Nodes that are moved a certain distance away from each other fall out of range and can no longer communicate through the wlan node (Figure 3, center). forward to all
hash lookup on/off
hub
1
tag packet 55ms
wlan
1
wlan
1 55 ms
2
2 3
4
2 3
4
37 ms
3
4 29 ms
Figure 3. WLAN Kernel Module
The advanced wireless model allows CORE to apply different link effects between each pair of wireless nodes (Figure 3, right). Each wireless vimage is connected to the wlan node with a pipe that is capable of applying different per packet effects depending on the source and destination of each packet. The wlan kernel module hash table stores, in addition to node connectivity, the parameters that should be applied between each pair of nodes. A tag is added to the packet as it passes through, being read by the pipe. The pipe then applies the link effects contained in the tag instead of its globally-configured effects. To determine more complex link effects between nodes, we use a modular C daemon to compute the distance and link effects calculations, instead of using the Tcl/Tk GUI. This allows for swapping out different wireless link models, depending on the configuration. Wireless link effects models can set the statistical link parameters of bandwidth, delay, loss, duplicates, and jitter. The CORE GUI and the link effects daemon communicate through an API. When the topology is executed, the GUI sends node information to the daemon, which then calculates link effects depending on the configured model. For example, a simple link effects model available in the CORE default setup is an increasing delay and loss model as the distance between two nodes increases. Different link models can use the same API to interact with the CORE GUI for emulating various layer 1 and 2 wireless settings. Once the link effects wireless daemon computes the appropriate statistical parameters, it configures the wlan kernel module directly through the libnetgraph C library available in FreeBSD, and informs the GUI of links between nodes and their parameters for display purposes. 5. DISTRIBUTED EMULATION The in-kernel emulation mechanism ensures that each CORE virtual image is very lightweight and efficient; however, the applications that are potentially running on the virtual nodes, such as routing daemons, traffic
generators, or traffic loggers, even though running independently, need to share the memory and CPU of the host computer. For example, in an emulation environment with all nodes running the OSPF routing daemon available in the Quagga open source package, we were able to instantiate 120 emulated routers on a regular computer. To increase the scalability of the system, we developed the capability to distribute an emulation scenario across multiple FreeBSD host systems, each of them independently emulating part of the larger topology. When using a distributed emulation, each emulated node needs to be configured with the physical machine that will be used to emulate that node. The controller GUI uses this information to compute partial topologies composed of nodes running at individual emulation hosts. The control GUI then distributes these partial topologies to the emulation hosts, which in turn emulate the partial topologies independently. When a link connects two nodes that are emulated on different FreeBSD hosts, a tunnel is created between the two physical machines to allow data packets to flow between the two emulated nodes. We use a separate C daemon, named Span, to instantiate these tunnels. 5.1 Connecting emulation hosts The CORE Span tool uses the Netgraph socket facility to bridge emulations running on different machines using a physical network. One way to connect two CORE emulations would be using the RJ45 jack described earlier in this paper. However, this limits the number of connections to the number of Ethernet devices available on the FreeBSD machine, and requires the emulation hosts to be physically collocated in order to be directly connected. Span allows any number of Netgraph sockets to be created and tunnels data using normal TCP/IP sockets between machines. Each Netgraph socket appears as a node in the Netgraph system, which can be connected to any emulated virtual image, and a user-space socket on the other end. Span operates by managing the mapping between these various sockets. Span also runs on Linux and Windows systems, and sets up a TAP virtual interface as the tunnel endpoint. This allows Linux or Windows machines to participate in the emulated network, as any data sent out the virtual interface goes across a tunnel and into the CORE emulated network. A different way to connect CORE machines together is by using the Netgraph kernel socket or ksocket. This allows opening a socket in the kernel that connects directly to another machine’s kernel. The sockets appear as Netgraph nodes that can be connected to any emulated node. CORE uses the ksocket to connect together WLAN nodes that belong to the same wireless network, but are emulated on
4 of 7
different machines. The WLAN node forwards all data to a connected ksocket without performing the hash table lookup. It also prepends the packet with the source ID of the originating Netgraph node. When receiving data from a ksocket, the remote wlan node uses the source ID tag from the packet for the hash table lookup. This allows emulation of a large wireless network with some wireless nodes distributed over multiple emulation machines.
number of router hops in the emulated network was increased from 1 to 120. The resulting iperf measurements are shown in Figure 4. In Figure 5, we plot the total number of packets per second handled by the entire system. This is the measured end-to-end throughput multiplied by the number of hops and divided by the packet size. This value represents the number of times the CORE system as a whole needed to deal with sending or receiving packets.
6. PERFORMANCE The performance of CORE is largely hardware and scenario dependent. Most questions concern the number of nodes that it can handle. This depends on what processes each of the nodes is running and how many packets are sent around the virtual networks. The processor speed appears to be the principal bottleneck.
100 90
Throughput (Mbps)
80
Here we consider a typical single-CPU Intel Xeon 3.0GHz server with 2.5GB RAM running CORE 3.1 for FreeBSD 4.11. We have found it reasonable to run 30-40 nodes each running Quagga with OSPFv2 and OSPFv3 routing. On this hardware CORE can instantiate 100 or more nodes, but at that point it becomes critical as to what each of the nodes is doing.
70 60 50 40 30 20 10 0 0
20
40
60
80
100
120
Number of Hops mss=1500
mss=1000
mss=500
mss=50
Figure 4. iperf Measured Throughput
Because this software is primarily a network emulator, the more appropriate question is how much network traffic it can handle. In order to test the scalability of the system, we created CORE scenarios consisting of an increasing number of routers linked together, one after the other, to form a chain. This represents a worst-case routing scenario where each packet traverses every hop. At each end of the chain of routers we connected CORE to a Linux machine using an RJ45 node. One of the Linux machines ran the iperf benchmarking utility in server mode, and the other ran the iperf client that connects through the chain of emulated routers. TCP packets are sent as fast as possible to measure the maximum throughput available for a TCP application.
Th roughput (total packets/sec)
350000 300000 250000 200000 150000 100000 50000 0 0
20
40
60
80
100
120
Number of Hops mss=1500
mss=1000
mss=500
mss=50
Figure 5. Total Packets per Second
For this test, the links between routers were configured with no bandwidth, delay, or other link restrictions, so these tests did not exercise the packet queuing of the system. The two Ethernet interfaces connected the Linux machines at 100M full-duplex. Only emulated wired links were used inside of CORE, and by default each emulated router was running the Quagga 0.99.9 routing suite configured with OSPFv2 and OSPFv3 routing. The iperf utility transmitted data for 10 seconds and printed the throughput measured for each test run. We changed the TCP maximum segment size (MSS) value, which governs the size of the packets transmitted, for four different MSS values: 1500, 1000, 500, and 50. The
The measured throughput in Figure 4 shows that the CORE system can sustain maximum transfer rates (for the 100M link) up to about 30 nodes. At this point the CPU usage reaches its maximum of 100% usage. Even when emulating 120 nodes, the network was able to forward about 30 Mbps of data. Figure 5 shows a linear increase of the number of packetsper-second with the number of hops as the link is saturated. Then the packets-per-second rate levels off at about 300,000 pps. This is where the CPU usage hits 100%. This suggests that the performance of the CORE
5 of 7
CORE 2
CORE 1
Video Server
Video Client
Figure 6. Hybrid Scenario
system is bounded by the number of packet operations per second; other factors such as the size of the packets and the number of emulated hops are not the limiting performance factor, as send or receive operations are implemented in the kernel simply as reference transfers. These tests consider only the performance of a single system. The FreeBSD 7.0 version of CORE supports symmetric multiprocessing (SMP) systems, and with CPU usage being the main bottleneck, a multiprocessor system should perform even better. The current version has somewhat limited SMP support but development of the kernel virtualization continues with the focus on adopting virtual network stacks in the –CURRENT FreeBSD development branch, so a separate patch will not be required. As described in Section 5, we have also added support to distribute the emulation across multiple physical machines, allowing for greater performance; but this introduces a new performance bottleneck – the available resources of the physical networks that tunnel data between emulations hosts. 7. HYBRID SCENARIO CORE has been used for demonstrations, research and experimentation. One frequent use of CORE is to extend a network of physical nodes when a limited amount of hardware is available. In this section we show a typical use case of CORE. The scenario includes eight Linux routers communicating with 802.11a wireless radios, shown as rectangular black systems in Figure 6. Each Linux router also features an Ethernet port used by CORE as a control
channel. CORE has been extended to remotely control these Linux routers, and can govern the actual connectivity of the wireless interfaces by inserting and removing iptables firewall rules in Linux as links are created and broken from the GUI. The identical Quagga OSPFv3MANET [20] routing protocol code is run on the Linux routers and on the FreeBSD emulated routers. The network is expanded by including six emulated wired routers and ten additional emulated wireless routers, for a total of 24 routing nodes. The wired routers are shown near the top left of Figure 6, and the wireless routers appear in the bottom left of Figure 6. Span is used in this scenario to link together the two physical CORE servers (CORE 1 and CORE 2), each responsible for emulating portions of the network. A laptop, labeled “Video Client” in Figure 6, is used to display a video stream transmitted by one of the Linux routers labeled “Video Server”. The video stream first traverses the physical 802.11 network and then into a Span tunnel that sends the data into one of the CORE machines. The packets are forwarded through the emulated network, first in an OSPFv2 wired network and then into an OSPFv3-MANET wireless network. Finally, the video stream enters another Span tunnel that connects to the virtual interface of the Windows laptop where the video client displays the video. This path is depicted with a green line in Figure 6. Performance of the video stream can be viewed on the laptop screen as the wireless nodes are moved around, in
6 of 7
either the real wireless network or the emulated one. In this scenario we observed that the OSPFv3-MANET routing protocol behaves similarly between the real Linux systems and the emulated FreeBSD nodes, as we would expect from the same code running in both platforms.
[8] “OPNET Modeler: Scalable Network Simulation”, http://www.opnet.com/solutions/network_rd/modeler.html
CONCLUSION AND FUTURE DIRECTIONS The CORE network emulator was introduced and briefly compared with other emulation, simulation, and virtualization tools. The CORE GUI and FreeBSD kernel components were described, along with two modes of wireless networks and distributing the emulation across multiple FreeBSD systems. The performance of the system was characterized with a series of throughput tests. Finally, the practical usability of the system was demonstrated by presenting a hybrid wired-wireless scenario that combined physical and emulated nodes.
[11] “Emulab - Network Emulation Testbed Home”, http://boss.netbed.icics.ubc.ca/
The key features of CORE include scalability, ease of use, the potential for running real applications on a real TCP/IP network stack, and the ability to connect the live running emulation with physical systems.
[18] “Kernel Based Virtual Machine”, http://kvm.qumranet.com/kvmwiki
[9] “Scalable Network Technologies: QualNet Developer”, http://www.scalable-networks.com/products/developer.php [10] “PlanetLab: an open platform for deploying…”, http://www.planet-lab.org/
[12] “Mobile Network Emulator (MNE)”, http://cs.itd.nrl.navy.mil/work/proteantools/mne.php [13] “VMware Server”, http://www.vmware.com/products/server/ [14] “Microsoft Virtual PC”, http://www.microsoft.com/windows/products/winfamily/virtualpc/default.mspx [15] “Parallels Workstation”, http://www.parallels.com/en/workstation/ [16] “Xen Hypervisor”, http://www.xen.org/xen/ [17] “The User-Mode Linux Kernel”, http://user-mode-linux.sourceforge.net/
[19] “OpenVZ Wiki”, http://wiki.openvz.org/Main_Page [20] P. Spagnolo and T. Henderson, “Comparison of Proposed OSPF MANET Extensions,” in Proceedings – IEEE Military Communications Conference MILCOM, vol. 2. IEEE, Oct. 2006.
Future work continues on the CORE tool to make it more modular. The wireless daemon is being improved with better support for pluggable wireless models. Experiments are being performed to merge CORE emulation with existing, validated simulation models for layers 1 and 2. Management of instantiating and running the emulation is being moved to a daemon, away from the monolithic Tcl/Tk GUI. Components of this daemon are being developed to take advantage of Linux virtualization techniques in addition to the existing FreeBSD vimages. The CORE system will be released as open source in the near future.
REFERENCES [1] “Integrated Multi-Protocol Emulator/Simulator”, http://www.tel.fer.hr/imunes/ [2] Zec, M. “Implementing a Clonable Network Stack in the FreeBSD Kernel”, USENIX 2003 Proceedings, November 2003. [3] Zec, M., and Mikuc, M. “Operating System Support for Integrated Network Emulation in IMUNES”, ACM ASPLOS XI, October 2004. [4] “The FreeBSD Network Stack Virtualization Project”, http://imunes.net/virtnet/ [5] Zec, M. “Network Stack Virtualization”, EuroBSDCon 2007, September 2007. [6] “The Network Simulator - ns-2”, http://www.isi.edu/nsnam/ns/ [7] “ns-3 Project”, http://www.nsnam.org/
7 of 7
