Immersive Real-Time Large-Scale Network Simulation: A Research Summary∗ Jason Liu School of Computing and Information Sciences Florida International University Miami, Florida 33199
[email protected] Abstract Immersive real-time large-scale network simulation is a technique that supports simulation of large-scale networks to interact with real implementations of network protocols, network services, and distributed applications. Traffic generated by real network applications is carried by the virtual network simulated in real time where delays and losses are calculated according to the simulated network conditions. We emphasize network immersion so that the virtual network is indistinguishable from a physical testbed in terms of network behavior, yet allows the flexibility of simulation. In this paper we provide a summary of current research in immersive real-time large-scale network simulation, particularly in areas of hybrid network traffic modeling and scalable emulation infrastructure design.
1. Introduction Today’s Internet has become a de facto lifeline that connects people and businesses across the globe. New network applications nowadays emerge at a stunning pace. For example, the use of VoIP is rising sharply to replace the traditional telephony, video streaming is working its way to redefine home entertainment, and peer-to-peer file sharing has long been the dominant source of Internet traffic. The increasing importance of Internet underscores the necessity of supporting experimental networking research so that it can continue to bring innovations and discoveries for both today’s and future network design. Several problems exist today for large-scale network experimentation. The root cause of these problems is the fact that Internet is an operational environment, not a platform for direct experimentation. It is difficult to conduct globalscale network experiments. Since no one owns the Internet ∗ This work is supported in part by the National Science Foundation grants CNS-0546712 and HRD-0317692.
entirely, it is difficult to find the necessary resources to deploy the applications and carry out the experiments. Also, Internet cannot afford to support applications that could cause service disruptions (such as studying Internet worm propagation behaviors). There are four types of tools we use today for experimental networking research: analytical models, physical testbeds, simulation, and emulation. Analytical models provide mathematical formulation of networking problems, which bring key insight to the system design. However, analytical models often do not include implementation details that may be of crucial importance to performance evaluation and design validation. Physical testbeds can accurately capture detailed network transactions. Normally, these testbeds are costly to build and thus difficult to be replicated as a general evaluation platform accessible to the research community at large. Overlay network testbeds (such as PlanetLab [15]) allow different experiments to share resources and, more importantly, include live network traffic. Major problems associated with this approach are scalability and flexibility. The networking condition of a physical testbed is by and large constrained by the physical setup of the system. To solve some of these problems, NSF recently created GENI, an initiative that aims to establish a large-scale network experimentation platform [14]. Simulation and emulation can be seen as the middle ground between analytical models and physical testbeds. The major distinction is that simulation is purely virtual, where time advancement in simulation bears no direct relationship to the wall-clock time. Emulation focuses on interaction with real applications; real network traffic that goes through the emulation system is added with artificial packet delays and losses. Comparatively, simulation offers more flexibility and efficiency at modeling the traffic behavior, while emulation provides more realism as it interacts with physical entities and real implementations. While network emulation has made steady progress recently (for example, EmuLab [17] currently supports hundreds of projects with thousands of users running over
18,000 experiments per year), network simulation has failed to meet the challenge. Especially when operational realism and deployment issues are of principal concerns, network researchers would favor emulation or physical testbeds, and bypass network simulation as an impractical solution. One important reason is that special efforts must be made to create a separate model to fit into the simulation environment. This can significantly prolong the development time, due to the complexity of the protocol development process that is both labor intensive and error prone. We introduce real-time network simulation, which combines the advantages of both simulation and emulation by alleviating the burden of model development and increasing fidelity as real systems are included in simulation. With this approach, real-world distributed applications and network services can run together with the simulator that operates in real time. Real packets are injected into the simulation system and subject to the simulated network conditions computed as a result of both real and virtual traffic traversing the network and competing for network resources. In the following we summarize the important characteristics of real-time network simulation: Accuracy: Simulation can accurately capture detailed packet-level transactions on the Internet, e.g., the layer-3 packet forwarding behaviors. Realism: Real implementation of existing network applications can be incorporated directly in simulation rather than using a separate implementation specifically design for simulation purposes. In addition, real traffic can be included in the study and interact with simulated network entities. Efficiency: Compared with emulation, where one typically implements packet forwarding by directing the packet across a physical link (an I/O operation), simulation involves only a few changes to the state variables in the model, resulting much less overhead. Scalability: Using parallel and distributed simulation techniques, network simulation can scale up far beyond what can be supported by emulation testbeds. Repeatability: The virtual network is represented by simulation events, and thus can be used to produce repeatable networking conditions for testing and evaluating real network applications. Controllability: The simulated network condition is completely under the control of the modeler, who can change model parameters and inspect state variables with relative ease. Flexibility: Simulation can be used for both analyzing the performance of existing systems and for evaluating new design alternatives. Once a simulation model has been developed and validated, it takes little effort to
conduct simulation experiments under different operating settings. It is comparatively easy to address the what-if questions (e.g., by injecting network failures); thus, real-time simulation can be used to explore a wide spectrum of design space. The goal of our research project is to investigate fundamental technologies that enable immersive real-time largescale network simulations and to develop a real-time network simulation environment. Since real applications operate in real time, real-time network simulation must meet real-time requirements. In essence, the performance of a large-scale network simulation must be able to keep up with the wall-clock time and allow real-time interactions with many real applications. To achieve our goal, we focus on two main research areas: hybrid network traffic modeling and scalable emulation infrastructure. We discuss recent advances in these two areas in the next two sections.
2. Hybrid Network Traffic Modeling Fluid network simulation models network traffic and network transactions as fluid flows rather than individual packets. There are two common fluid models. A discreteevent fluid model is based on scheduling events to represent changes in the flow rate, modeled as a piece-wise constant function (e.g., [12]). An advantage of this approach is that it is natural to integrate events of the fluid model with those from the packet-driven simulation in a hybrid simulation (see, for example, [4] and [16]). Conceptually, this gives rise to an important distinction between packetoriented foreground traffic and fluid-based background traffic. The foreground traffic is the traffic stream we are interested in and needs to be modeled with high fidelity. In contrast, the background traffic, which represents the bulk of the network traffic, is of secondary interest and does not require significant accuracy. It is nonetheless important as the background traffic interferes with the foreground traffic as they both compete for network resources. The drawback of this approach is that smoothing mechanism must be in place to prevent the so-called “ripple effect”, where the number of events indicating flow rate changes may increase sharply in a congested network. Results show moderate speedup of approximately an order of magnitude can be achieved over the traditional packet-oriented simulations [13]. The other type of fluid model is based on analytical models, such as the one that uses a set of ordinary differential equations (ODE) to describe the behavior of persistent TCP flows [11]. ODEs can be solved numerically with great computational efficiency; therefore, the model can scale well to a large number of TCP flows. One potential problem of this approach is the difficulty of integrating with the discrete-event packet-oriented simulation. Previous work [3, 19] assumes that simulation maintains two
Figure 1. Router queue lengths.
Figure 2. Execution speed.
separate networks: a fluid network representing the network whose state evolution is dictated by the solution of differential equations, and a packet network where network transactions are modeled as discrete events. The interaction happens at the boundary. Packets entering the fluid network are smoothed and transformed into a fluid flow (as a piece-wise constant function) and then compete with the pure fluid flows for network resources according to the set of differential equations. The packet delay and drop probability are computed cumulatively within the fluid network and used when packets departing from the fluid network. This approach physically separates fluid and packet traffic representations, and therefore can only be used to model end-to-end behavior of packet flows. The detailed state of the fluid network (such as the dynamics of the queue size of a router inside the fluid network) cannot be accessed directly to interact with the packet flows. Our integration approach [7] allow the interaction between fluid and packet flows to happen within the network queues. Thus, the emulated network traffic is able to reach any part of the virtual network and mingle seamlessly with the background fluid flows. The pure fluid flows arriving at each network queue are augmented with packet flows arriving at the queue. The queuing dynamics are still governed by the set of differential equations only modified to include both types of flows. These differential equations are solved using a time-stepped Runge-Kutta method. Packets entering a network queue are simulated as discrete events and observe the delay and drop probability according to the state of the queue calculated by the fluid model. We incorporate three practical techniques [9] in the implementation of the hybrid traffic model to improve its performance: i) we use a simple caching technique to reduce the search time of the state variables used by the differential equations from O(n) to nearly O(1); ii) we reduce the number of fluid updates along the flow paths by scheduling events if only flow rates are changed beyond a given threshold; and iii) we use a Runge-Kutta method with adjustable step size, which depends on the function of state evolution over time.
We developed a prototype of the hybrid traffic model. Figure 1 shows the result of an experiment with a simple dumbbell network model. It shows that the queue lengths at a bottleneck router computed by the fluid model match well with those from the pure packet-oriented simulation. The spike at around time 50 results from a sudden increase in the offered load. Figure 2 shows the effective number of packet events processed per second wall-clock time. For fluid-based simulation, it represents the equivalent number of packets being forwarded on the virtual network during a second if one chose to use packet-oriented simulation. We observe that the packet-oriented discrete-event simulation simulates packet forwarding at about 500,000 packets per second, where the rate increases steadily in the fluid model with increasing number of aggregated TCP flows, finally reaching over 566 million packet events per second. The result indicates that the fluid model can achieve significantly better performance than packet-oriented simulation (over three orders of magnitude). It should be noted that the result is approximately 5.34 times faster than a previous attempt of running a large network on 1,536 processors [2], albeit with a much simpler network scenario. The hybrid traffic model can be parallelized to achieve even greater performance [8]. We observe that the time it takes to propagate fluid characteristics (such as flow rate, accumulative delay, and accumulative loss) along the path taken by traffic flows has a lower bound equal to the minimum link delay as manifested by the governing ordinary differential equations. This is an important observation, which means that a better lookahead (when compared with the Runge-Kutta step size) can be used to allow parallel simulation of the hybrid model to run without more synchronization overhead than the corresponding discreteevent packet-oriented model. In [8], we developed an analytical model to show that the flow update events propagated across the processor boundaries can be sent along the events that represent packets on the virtual network. As such, the parallel hybrid model can achieve the same scalability as the traditional parallel packet-oriented simulation. We are
currently investigating the performance of the hybrid model with extremely large network models.
3. Scalable Emulation Infrastructure Real-time network simulation requires that simulation of large-scale networks run no slower than real time. To allow a large number of real applications to be connected to the virtual network during an experiment, we must create an emulation infrastructure that supports high throughput and low latency for emulation traffic to flow between the real applications and the real-time simulator. In addition, the emulation infrastructure must also be flexible at embedding various applications and yet remain transparent to the applications to allow network immersion. There are several ways to incorporate real applications into the emulation environment, the decision of which to use largely depends on where the interactions take place. Several techniques exist that allow running unmodified software, which include using packet capturing techniques (such as libpcap, IP table, and IP tunnel), preloading dynamic libraries, and modifying the binary executables. In certain cases, moderate software modifications are necessary to allow more efficient direct-execution. We use an open systems approach based on Virtual Private Network (VPN), which is customized to function as a gateway to bridge traffic between the physical client machines and the simulated network [10]. Figure 3 illustrate the VPN emulation infrastructure, which consists of three components: the VPN clients, the simulation gateways, and the real-time simulator. Real machines running real applications establish their connections to the simulation gateways as VPN clients (by each running an automatically generated VPN configuration script). VPN creates logical network interfaces (namely, TUN/TAP devices), assigns IP addresses, and changes the routing table accordingly, so that applications running on the client machine can forward packets to the virtual network through the logical network interfaces. The VPN client itself, which runs as an application on the client machine, upon receiving packets from the logical network device, sends the packets via UDP to the VPN server running at the simulation gateway. Each simulation gateway runs a modified VPN server and a daemon process called ssfgwd, which is responsible for delivering the packets passed on from the VPN server to the real-time simulator via a dedicated TCP connection. The real-time simulator, presumably running on parallel computers. The real-time simulator spawns two I/O threads for communicating with the ssfgwd process at the simulation gateway. Packets arriving at the simulator are injected into the event list as simulation events; the simulator simulates the packets being forwarded on the virtual network as if they were created by the corresponding
Figure 3. VPN emulation infrastructure. virtual node that has the same IP address assigned to the VPN client. Traffic targeting the emulated client machines works its way in the reverse direction, i.e., from the realtime simulator, to the ssfgwd process, to the VPN server, and eventually arrives at the designated client machine. One distinct advantage of this approach is that the emulation infrastructure does not require special hardware to set up. It is also secure and scalable, which are merits inherited directly from the underlying VPN implementation. Multiple simulation gateways can run simultaneously. In order to produce accurate results, however, the emulation infrastructure needs a tight coupling between the emulated entities (i.e., the client machines) and the real-time simulator. In particular, the segment between the client machines and the real-time network simulator must consist of only low-latency links. To maintain throughput, the segment must also provide sufficient bandwidth to carry the emulation traffic. With these constraints, the physical latency between the clients and the simulator can actually be made transparent in the network model [6]. The idea is to allow an emulation packet in simulation to preempt other simulated packets in the network queues so that the packet can be delivered ahead of its schedule in order to compensate for the physical delays. This emulation infrastructure has been used to analyze the performance of both end applications (such as TCP variants) and in-network services (such as CDN). Here we describe a recent work [5] that extends the emulation infrastructure to support scalable routing experiments with an open-source routing software called XORP [18]. We use a forwarding plane offloading approach to separate packet forwarding (performed by the real-time network simulator) from routing (performed by XORP), thereby eliminating the I/O overhead otherwise caused by the bulk of network traffic being forwarded through the emulation infrastructure. As a case study, we conducted an intra-domain routing experiment (with OSPF), which was originally used by VINI [1] to demonstrate its capability to conduct elabo-
100
Ping round-trip time (ms)
90 80 70 60 50 40 30 0
10
20
30 40 Time (seconds)
50
60
70
Figure 4. OSPF route convergence.
progress in the research areas of hybrid network traffic modeling and scalable emulation infrastructure design. Currently we are investigating methods that can dynamically change the modeling resolution based on real-time response of the simulator by switching between the fluid representation and the packet-oriented model. We are also investigating more efficient integration schemes to embed real applications. We are particularly interested in applying virtual machine technologies to real-time network simulation. The goal is to multiplex a large number of network applications to run on separate virtual machines and have them interact with the simulator with little overhead.
References
Figure 5. TCP transfer. rate routing tests on PlanetLab. The experiment consists of eleven routers representing the backbone of the Abilene network. We injected a link failure followed by a recovery, and measured the effect on the round-trip time and data throughput. Figure 4 shows the round-trip time measured by ping. When the link went down at 14 seconds, OSPF instances recomputed a longer route. When the link was repaired at 43 seconds, OSPF re-established the shortest path after the routes converged. Figure 5 shows the number of bytes received at the receiver end in a separate run. The TCP transfer was interrupted when the link failure occurred at 18 seconds and then resumed when an alternative route was found at 32 seconds. At time 50, the broken link was repaired, the shortest path was re-established, and a better throughput was achieved (demonstrated by a steeper slope in the figure). This experiment shows that the emulation infrastructure can correctly capture the routing dynamics in large network settings.
4. Conclusions and Future Work In this paper, we introduce the technique of immersive real-time large-scale network simulation that allows network simulation to interact with real network applications, in order to support experimental networking research with better accuracy, realism, efficiency, scalability, repeatability, controllability, and flexibility. We present our recent
[1] A. Bavier et al. In VINI Veritas: Realistic and controlled network experimentation. SIGCOMM, 2006. [2] R. Fujimoto et al. Large-scale network simulation – How big? How fast? MASCOTS, 2003. [3] Y. Gu, Y. Liu, and D. Towsley. On integrating fluid models with packet simulation. INFOCOM, 2004. [4] C. Kiddle et al. Hybrid packet/fluid flow network simulation. PADS, 2003. [5] Y. Li, J. Liu, and R. Rangaswami. Real-time network simulation support for scalable routing experiments. Submitted for publication. [6] M. Liljenstam et al. RINSE: the real-time interactive network simulation environment for network security exercises. PADS, 2005. [7] J. Liu. Packet-level integration of fluid TCP models in realtime network simulation. WSC, 2006. [8] J. Liu. Parallel simulation of hybrid network traffic models. PADS, 2007. [9] J. Liu and Y. Li. On the performance of a hybrid network traffic model. Submitted for publication. [10] J. Liu et al. An open and scalable emulation infrastructure for large-scale real-time network simulations. INFOCOM, 2007. [11] Y. Liu et al. Fluid models and solutions for large-scale IP networks. SIGMETRICS, 2003. [12] D. Nicol, M. Goldsby, and M. Johnson. Fluid-based simulation of communication networks using SSF. European Simulation Symposium, 1999. [13] D. Nicol and G. Yan. Discrete event fluid modeling of background TCP traffic. TOMACS, 2004. [14] NSF Global Environment for Network Innovations (GENI). http://www.geni.net/. [15] L. Peterson et al. A blueprint for introducing disruptive technology into the Internet. HotNets-I, 2002. [16] G. Riley, T. Jaafar, and R. Fujimoto. Integrated fluid and packet network simulations. MASCOTS, 2002. [17] B. White et al. An integrated experimental environment for distributed systems and networks. OSDI, 2002. [18] XORP. http://www.xorp.org/. [19] J. Zhou et al. MAYA: integrating hybrid network modeling to the physical world. TOMACS, 2004.
