PrimoGENI for Hybrid Network Simulation and Emulation Experiments ...

1

PrimoGENI for Hybrid Network Simulation and Emulation Experiments in GENI Nathanael Van Vorst, Miguel Erazo, and Jason Liu School of Computing and Information Sciences Florida International University Emails: {nvanv001, meraz001, liux}@cis.fiu.edu Abstract—The Global Environment for Network Innovations (GENI) is a community-driven research and development effort to build a collaborative and exploratory network experimentation platform—a “virtual laboratory” for the design, implementation and evaluation of future networks. The PrimoGENI project enables real-time network simulation by extending an existing network simulator to become part of the GENI federation to support large-scale experiments involving physical, simulated and emulated network entities. In this paper, we describe a novel design of PrimoGENI, which aims at supporting realistic, scalable, and flexible network experiments with real-time simulation and emulation capabilities. We present a flexible emulation infrastructure that allows both remote client machines, local cluster nodes running virtual machines, and external networks to seamlessly interoperate with the simulated network running within a designated “slice” of resources. We present the results of our preliminary validation and performance studies to demonstrate the capabilities as well as limitations of our approach. Index Terms—network simulation, emulation, real-time simulation, virtualization

F

1

INTRODUCTION

The success of Internet can be attributed to many factors. However, many argue that the very success has brought the ossification of its own design that makes it difficult to sometimes realize even the simplest changes. Political and economic factors notwithstanding, the success of transforming academic research to empirical implementation depends on the availability and trustworthiness of network testbeds to correctly expose important design and operational issues. Considerable progress has been made in recent years in network testbed design to foster network innovations. The GENI initiative (The GENI Project Office, 2011a) capitalizes on the success of many previous efforts and strives to create a single collaborative and exploratory platform for future Internet research, bringing together the latest advances of network testbed research. The GENI design is built on four premises (The GENI Project Office, 2011b): 1) Programmability: the ability to execute custom software on GENI-enabled facilities deep inside the network hierarchy; 2) Resource sharing: sharable resources among multiple users and experiments, possibly through virtualization;

3) Federation: interoperability among various resources provided by different organizations; and 4) Slice-based experimentation: supporting network experiments running independently within “slices” (a subset of GENI resources). While GENI offers a cross-platform virtualization solution for physical and emulation testbeds, simulation is ostensibly missing in its original design. Although physical and emulation testbeds provide the necessary realism in prototyping and testing new network designs, simulation is more effective for studying complex behaviors of large-scale systems, which are both difficult to handle using realistic settings and intractable to closeform mathematical and analytical solutions. PrimoGENI is a project that aims to incorporate real-time network simulation capabilities into the GENI “ecosystem”. Real-time network simulation refers to simulation of potentially large-scale networks in real time so that the virtual network can interact with real implementations of network protocols, network services, and distributed applications. PrimoGENI uses our existing real-time network simulator, called PRIME (Modeling and Networking Systems Research Group, 2011), which is a real-time extension to the SSFNet simulator, capable of running on parallel and distributed machines for large-scale network simulations (Cowie et al., 1999).

2

We augment PRIME with the GENI aggregate API, through which the experimenters can allocate resources and run experiments. PRIME can interoperate with the physical, simulated and emulated network components. In particular, network applications or protocols can run unmodified on emulated hosts, which are implemented as virtual machines, and interact with the simulated network. Network traffic generated by these applications or protocols is “conducted” on the simulated network operating in real time, with appropriate packet delays and losses calculated according to the simulated network conditions. In this respect, the simulated network is considered indistinguishable from a physical network. In this paper, we present the design of PrimoGENI, which uses a current GENI control framework and extends the existing emulation infrastructure to support simulation and emulation experiments on a compute cluster. We highlight a set of major design decisions that direct our development and prototyping activities with the objective of substantially broadening GENI’s capability in supporting realistic, scalable, and flexible experimental networking studies. The contributions of this paper include the following: 1) PrimoGENI is the first of its kind to provide an end-to-end solution for building a real-time network simulation testbed that allows users to construct, deploy, and run network experiments involving physical, simulated and emulated network components in GENI. 2) PrimoGENI features an infrastructure that allows multiple network experiments to run independently within their own slices, each possibly consisting of remote client machines and local cluster nodes running virtual machines that seamlessly interoperate with the simulated network. 3) We conducted preliminary validation and performance studies to demonstrate the capabilities and limitations of our approach. The rest of the paper is organized as follows. In Section 2 we present the background. We then describe in detail the PrimoGENI design in Section 3. More specifically, we focus on the emulation infrastructure of PrimoGENI in Section 4. We present the results of our validation and performance studies from our preliminary implementation in Section 5. Finally, we conclude this paper with a discussion of our ongoing and future work in Section 6.

2

BACKGROUND

In this section we describe related work and review the current GENI architecture.

2.1

Options for Network Experiments

Network testbeds are commonly used for prototyping, evaluating, and analyzing new network designs and services. Physical testbeds, such as WAIL (Barford and Landweber, 2003) and PlanetLab (Peterson et al., 2002), provide an iconic version of the network for experimental studies, sometimes even with live traffic. They provide a realistic testing environment for network applications, but with limited user controllability. Further, being shared facilities, they are constantly overloaded due to heavy use, which can severely affect their availability and accuracy (Spring et al., 2006). An emulation testbed can be built on a variety of computing platforms, including dedicated compute clusters, such as ModelNet (Vahdat et al., 2002) and EmuLab (White et al., 2002), distributed platforms (such as VINI (Bavier et al., 2006)), and special programmable devices, such as ONL (DeHart et al., 2006) and ORBIT (Raychaudhuri et al., 2005). While most emulation testbeds provide basic traffic “shaping” capabilities, simulation testbeds can generally achieve better flexibility and controllability. For example, ns-2 (Breslau et al., 2000) features a rich collection of network algorithms and protocols that can be selected to model a myriad of network environments, wired or wireless. Simulation can also be scaled up to handle large-scale networks through parallelization, e.g., SSFNet (Cowie et al., 1999), GTNeTS (Riley, 2003) and ROSSNet (Yaun et al., 2003). It would be otherwise difficult and costly to build a large-scale physical or emulation testbed. Real-time simulation is the technique of running network simulation in real time and thus can interact with real implementations of network applications (Liu, 2008). Most existing real-time simulators, e.g., Fall (1999); Liu et al. (2003); Simmonds and Unger (2003); Zhou et al. (2004), are based on existing network simulators extended with emulation capabilities. PRIME is a discrete-event simulator designed to run on parallel and distributed platforms and handle large-scale network models. PrimoGENI uses PRIME for real-time simulation. In addition, PRIME also provides several important functions: •

•

PRIME has an emulation infrastructure based on VPN that allows distributed client machines to remotely connect to the real-time simulator and exchange traffic (Liu et al., 2009). PRIME provides models for 14 TCP variants (mostly ported from the Linux TCP implementation). They have been validated carefully through extensive simulation and emulation studies (Erazo

3

•

2.2

et al., 2009). PRIME uses multi-scale modeling for large-scale simulation. PRIME implements a fluid traffic model, which has been integrated with simulated and emulated network traffic. The hybrid traffic model can achieve a speedup of more than three orders of magnitude over the traditional packetoriented simulation (Liu and Li, 2008). GENI Architecture

GENI is a network research infrastructure that combines several prominent network testbeds, such as EMULAB (White et al., 2002), PlanetLab (Peterson et al., 2002), and ORBIT (Raychaudhuri et al., 2005), as a single collaborative and exploratory platform for implementing and testing new network designs and technologies. The GENI design consists of three main types of entities: clearinghouses, aggregates, and principals. A clearinghouse provides the central management of GENI resources for experimenters and administrators; specifically, it provides registry services for principals, slices and aggregates, and authentication services for accessing the resources. An aggregate represents a group of components encapsulating the GENI sharable resources (including computation, communication, measurement, and storage resources). When an experimenter from a research organization (i.e., a principal) decides to conduct a GENI experiment, she negotiates with the clearinghouse and the associated aggregate managers through an elaborate resource discovery and allocation process. In response to the experimenter’s request, each participating aggregate allocates the requested resources for the experimenter, which are called slivers. Jointly, these slivers from all participating aggregates form a slice, which represents the set of allocated resources in which the experimenter will run network experiments. A experimenter acquires resources from the GENI aggregates using a control framework run by the clearinghouse. GENI supports several control frameworks, including ProtoGENI (Flux Research Group, 2011), PlanetLab (Peterson et al., 2002), ORBIT (Raychaudhuri et al., 2005), and ORCA (ORCA-BEN Project, 2011). These control frameworks implement the GENI aggregate manager API that allows the experimenters with proper authentication to query for available sources, allocate resources upon the resource specification of an experiment, set up the resources for the experiment, and reclaim the resources after the experiment has finished. PrimoGENI extends the ProtoGENI control framework to manage, control and access the underlying

resources. ProtoGENI is developed based on EmuLab (White et al., 2002), which is an emulation testbed implemented on a compute cluster. ProtoGENI extends EmuLab with key GENI functions, such as the registry services for principals, slices, and aggregate/component managers. To interoperate with other GENI facilities, PrimoGENI functions as a GENI aggregate; as such, the experimenters can use the well-defined GENI aggregate API to remotely control and realize network experiments consisting of physical, simulated and emulated network entities exchanging real network traffic.

3

THE PRIMOGENI APPROACH

PrimoGENI runs a network model in real time, and thus can interact with real applications. Real-time simulation can provide accurate results since it is able to capture detailed packet-level transactions. It is also flexible since one can easily explore the simulation model to answer what-if questions. Further, it supports large-scale network experiments potentially with millions of simulated entities (hosts, routers, and links) and thousands of emulated hosts running unmodified network protocols and applications. 3.1

Slingshot

PrimoGENI has an integrated development environment, called slingshot, as the front-end for the experimenters to manage the “life cycle” of a network experiment, which includes model construction, configuration, deployment, execution, monitoring and control, data collection, visualization and analysis. Slingshot provides a Python console for constructing and configuring network models interactively. One can also specify the network models using Java or XML. A network model describing the target system may consist of the topology of the network and a specification of simulated traffic on the network. One can designate some network entities as emulated hosts, which are either virtual machines or physical machines with specified operating systems to run unmodified network protocols and applications. Additionally, one can add one or more traffic portals which enable traffic from real networks to interact with traffic within the virtual network. Slingshot provides a persistent storage for the network models using a database so that they can be inspected, analyzed and possibly reused after the experiment. Slingshot also provides a graphical representation of the network model for the experimenters to conveniently inspect and change the detailed configurations. Once a network model is determined, the experimenter needs to specify the number of compute nodes to run the

4

3.2

PrimoGENI Aggregate

A PrimoGENI cluster consists of a set of compute nodes connected by one or more high-speed switches. PrimoGENI uses the ProtoGENI (Flux Research Group, 2011) control framework to manage, control and access the underlying resources. We designate one compute node to run the ProtoGENI aggregate manager and another one to run as the file server. The aggregate manager is responsible for allocating the resources, instantiating and setting up the virtual network, and connecting with the emulated hosts on the virtual machines and physical machines on the PrimoGENI cluster on behalf of the experimenter. Our design makes a distinction between what we call meta resources and virtual resources. Meta resources include compute nodes and network connectivities between the compute nodes (such as a VLAN setup). We call them “meta resources” to separate them from the physical resources (which are known as the substrate). Meta resources also include virtual machines and/or virtual network tunnels in addition to ProtoGENI resources which are not PrimoGENI compute nodes. Meta resources are managed by ProtoGENI. PrimoGENI translates the experiment specification into meta resource requirements and instructs ProtoGENI to allocate the resources for the experiment. Virtual resources are elements of the target virtual network, which include both simulated entities (such as hosts, routers and links), emulated hosts, and traffic portals. Emulated hosts can be implemented as virtual or physical machines, which run unmodified applications, such as web servers or software routers. These emulated hosts communicate with other hosts and routers on the

Experiment Layer

EH

EH

SIM

SIM

EH

EH Simulation and Emulation Execution Layer

Aggregate Manager

SIM

Emulation Gateway

EMU

EMU SIM

EMU

EMU

OpenVZ Kernel OpenVZ Kernel

Resource Allocation & Multiplexing

network experiment. Slingshot uses the METIS graph partitioner (George Karypis Lab, 2011) to partition the network model by dividing the network model into subnetworks of approximately equal size and at the same time trying to maximize the communication latencies between the subnetworks (in order to achieve better lookahead for parallel simulation). Emulated hosts are given more weight in the network graph since it is important to spread the virtual machines evenly among the compute nodes as they require more resources. After that, the user can deploy and launch the experiment on a PrimoGENI cluster, which we describe in detail momentarily. During an experiment, the user can use slingshot to monitor the progression of the experiment and, if needed, change the state of the network entities when the experiment is running. The experiment results can be collected and displayed during or after the experiment.

Meta Resource Layer

Physical Resource Layer (Substrate)

Fig. 1. PrimoGENI aggregate

virtual network (either simulated or emulated) according to the network model. Traffic portals allow real networks to exchange traffic with the virtual network as if the virtual network were also real. Network traffic originated from or destined to an emulated host or external network is called emulated traffic. All emulated traffic is “conducted” on the virtual network in simulation with proper packet delay and loss calculations as it competes with the simulated traffic. The PrimoGENI aggregate can be viewed as a layered system, as illustrated in Fig. 1. At the lowest layer is the physical resources (substrate) layer, which is composed of compute nodes, switches, and other physical resources that constitute the EmuLab suite. These resources are made known to the clearinghouse(s) and can be queried by researchers during the resource discovery process. A compute node is set up to run the aggregate manager to export the aggregate API to experimenters and clearinghouses. PrimoGENI uses the ProtoGENI control framework, which provides common operations, such as creating, deleting and shutting down slivers. ProtoGENI uses secure socket layer (SSL) for authentication and encryption, and XML-RPC for remote procedure invocation. The meta-resource layer is composed of a subset of resources allocated from the physical resource layer as needed by the experiment. After the requested resources have been successfully allocated by the ProtoGENI control framework, PrimoGENI will bootstrap each compute node by running a customized OS image that supports virtual machines. The simulation and emulation execution layer is created according to the network model of the experiment. Each compute node serves as a basic scaling unit and

5

implements the operation of a subnetwork or a host that is mapped to it. A scaling unit for a subnetwork consists of a simulator instance, and zero or more emulated hosts each running as a virtual machine. The simulator instances on different scaling units synchronize and communicate with each other using parallel discrete-event simulation techniques. An emulated host can also run directly on a physical compute node. This is designed for cases where the emulated hosts have stronger resource requirements and thus cannot be run as virtual machines. PrimoGENI provides an emulation infrastructure to connect the emulated hosts running on virtual or physical machines to the corresponding simulator instances (see Section 4). After the experiment starts, the experimenter can interact with the emulated hosts and the simulated network entities on the experiment layer. To access the emulated hosts, the experimenter can directly log onto the corresponding computing nodes or virtual machines and execute commands (e.g., pinging any virtual hosts). To access the dynamic state of the simulated network, PrimoGENI provides an online monitoring and control framework. This allows the experimenter to query and visualize the state of the network. 3.3 Compute Node Configuration PrimoGENI uses the ProtoGENI control framework to manage the compute nodes in the cluster; ProtoGENI allows the compute nodes to be loaded with custom operating systems. To scale up emulation experiments, we use virtual machines (VMs) as emulated hosts. For PrimoGENI, we chose OpenVZ (OpenVZ, 2011) to run the virtual machines for the emulated hosts where unmodified network applications can run. OpenVZ is an OS-level VM solution that meets our minimum requirement in terms of providing necessary separation of CPU, memory, and network resources among the VMs. CPU and memory separation is needed to limit the interactions of applications running on different VMs. Network separation is necessary to manage network traffic to and from the VMs each with an independent network stack. OpenVZ compartmentalizes the system resources inside the so-called containers; applications can run within these containers as if they were running on dedicated systems. Each container has its own set of processes, file system, and manages its own set of users (including root), and network stack (including network interfaces and routing/forwarding tables). With OpenVZ, PrimoGENI is able to perform network experiments with a large number of emulated hosts. We preload the compute nodes with OpenVZ in the PrimoGENI cluster and provide an OS template from

which the containers will be created. The OS template consists of a root file system and common network applications that may be run on a container (such as iperf and tcpdump). Additional applications can be added to individual containers using the meta-controller framework, which we describe in Section 3.4. Since each container maintains its own file system, and the amount of storage space needed by the file system at each container may range from tens to hundreds of megabytes, which depends on the applications the experimenter wishes run; it could mean a lot of space for running network experiments with a large number of emulated hosts. To solve this problem, we choose to use a union file system (Stephane Apiou, 2011). A union file system consists of two parts: a read-only file system (RF), which is common to all containers, and a writable file system (WF), which is specific to a container. A copy-on-write policy is used: when a file in RF needs to be modified, the file will be first copied to WF. To prepare the file system for each emulated host, PrimoGENI unions the default OS-template with an initially empty writable base file system. In this way we need only to store changes that later occur for each container as opposed to storing the entire file system. 3.4

Meta-Controllers

As mentioned earlier, PrimoGENI uses the ProtoGENI control framework to allocate the necessary compute nodes to run experiments. After the compute nodes are booted from the customized OpenVZ image, we need a mechanism to configure the compute nodes, create the containers, set up the emulation infrastructure, start the simulation, and launch the applications inside the containers. One can use XML-RPC for this purpose. However, since the commands are originated from slingshot, which is the user interface experimenters use to configure, launch and control the experiments, we would like to have a mechanism that allows the compute nodes to be coordinated locally on the PrimoGENI cluster to achieve better efficiency. In this section, we describe a tiered command framework, which we use to set up a network experiment and orchestrate experiment execution. The command framework, as illustrated in Fig. 2, uses MINA, a Java framework for distributed applications (Apache MINA Project, 2011). MINA provides both TCP and UDP-based data transport services with SSL/TLS support. It is designed with an eventdriven asynchronous API for high-performance and highscalability network applications. Our implementation requires each compute node to run a meta-controller

6

SIM SIM SIM

EMU

Master Meta-Controller

EMU

EMU

EMU

EMU OpenVZ KernelSIM

EMU OpenVZ Kernel

OpenVZ Kernel

Slingshot

EMU

EMU

EMU SIM

EMU

EMU

OpenVZ Kernel OpenVZ Kernel

Slave Meta-Controllers

Fig. 2. Meta-controllers

daemon process when first booted. When the metacontroller starts, it waits for an incoming connection. At this point the meta-controller takes no role, but after the connection is made, the meta-controller will become either a master or a slave. The difference between a master meta-controller and a slave meta-controller is that the master meta-controller is the one connected with slingshot; commands sent from slingshot, if they are not destined for the master, will be relayed to the corresponding slave meta-controllers. After the compute nodes have been allocated for an experiment, slingshot will choose one compute node to be the master and establish a secure connection to it using the experimenter’s key obtained from the aggregate manager. After slingshot successfully connects to master meta-controller, slingshot instructs it to take control over the remaining compute nodes—the master establishes an secure connection to each of those compute nodes and instructs them to act as slaves. After that, slingshot sends commands to the master meta-controller, which properly distributes the commands to the slave meta-controllers, if needed. Each command specifies the target compute node or one of its containers on the compute node, where the command is expected to run. A command can be either a blocking command or a nonblocking command. If it is a blocking command, the meta-controller will wait until the the command finishes execution and sends back the result of the command (i.e., the exit status) to slingshot. If the command is a nonblocking command, the meta-controller forks a separate process to handle the command and immediately responds to slingshot. Our current implementation uses blocking commands to set up the containers and the emulation infrastructure, and uses nonblocking commands to start the simulator and the user applications within the containers. To set up the experiments correctly, the meta-controllers on the compute nodes need to run a series of commands: 1) Set up MPI. The master meta-controller creates the machine file and instructs the slave metacontrollers to generate the necessary keys for MPI to enable SSH logins without using passwords.

2) Create containers. The meta-controller creates a container for each collocated emulated host on the compute node. This step also includes creating the union file systems for the containers. 3) Set up the emulation infrastructure. This step includes installing and configuring necessary network devices and other software components (such as software bridges and TAP devices) in containers and on the physical host. 4) Run the experiment. The partitioned network model is distributed among the compute nodes. The master meta-controller initiates the MPI run, which starts the simulator instances on each compute node with the partitioned model. 5) Start applications within containers. Individual commands are sent to the meta-controllers to install and run applications at the emulated hosts. 6) Shut down the experiment. At any time, one can shut down the experiment by terminating the simulator, stopping the containers, and removing the emulation infrastructure. All these commands are issued from slingshot automatically using the meta-controller command framework. In addition to the commands that set up the compute nodes and run applications, the meta-controller is also responsible for transporting state set, fetch, and update commands, which allow users to inspect and modify state during experiment execution. Each simulator instance maintains an event channel with its associated slave controller. One can set or fetch the value of a variable during experiment execution. Both set and fetch commands are issued from slingshot and delivered to the simulator instance through this command channel. Update commands contain the variables’ runtime value; they are issued by the simulator instance and delivered to slingshot, which stores them as time-series data to be inspected during or after experiment execution.

4

E MULATION I NFRASTRUCTURE

The emulation infrastructure in PrimoGENI needs to support high-throughput low-latency data communication between the emulated hosts and the simulation instances. 4.1

Design Architecture

There are two kinds of emulated hosts: collocated and remote. Collocated emulated hosts run as virtual machines on the same compute node as the simulator instance that simulates the subnetwork containing the corresponding virtual hosts. In PrimoGENI, the compute nodes run

7

OpenVZ, which can support a large number of collocated emulated hosts and therefore allow PrimoGENI to conduct network emulations at scale. Remote emulated hosts are either physical compute nodes in the PrimoGENI cluster (not VMs), or machines that are not part of the PrimoGENI cluster and potentially at different geographic locations. Remote emulated hosts can run applications which otherwise cannot be run on a virtual machine due to either stringent resource requirements or system compatibility concerns. For example, an emulated host may require a special operating system (or version), or need specialized hardware that is not available on the compute node. In addition to emulated hosts, PrimoGENI provides traffic portals to exchange traffic with real networks. Traffic portals allow hosts and routers outside the virtual network to interact with simulated and emulated traffic within the virtual network. PrimoGENI’s emulation infrastructure consists of three major components: the virtual network interfaces, the emulation device drivers, and the interconnection mechanism. The virtual network interfaces (VNICs) are installed at the emulated hosts, and treated as regular network devices, through which applications can send or receive network packets. Packets sent to a VNIC are transported to the simulator instance that handles the corresponding virtual host. The simulator subsequently simulates the packets traversing the virtual network as if they were originated from the corresponding network interface of the virtual host. Upon packets arriving at a network interface of an emulated virtual host in simulation, the packets are sent to the corresponding VNIC at the emulated host, so that its applications can receive the packets. The emulation device drivers (EDDs) are collocated with the simulator instances running on the compute nodes. They are software components used by the realtime simulator to import real network packets sent from VNICs at the emulated hosts or traffic portals. Conversely, EDDs also export simulated packets and send them to the emulated hosts or traffic portals. Our emulation infrastructure supports different types of EDDs for remote emulated hosts, collocated emulated hosts, and traffic portals. PRIME provides functions designed specifically for interactive simulations, which include handling real-time events and performing conversions between fully formed network packets and simulation events. The interconnection mechanism connects VNICs on emulated hosts and EDDs with the simulator instances. Its design depends on whether it is dealing with traffic portals, remote or collocated emulated hosts. For remote

Container 0

Container 1

Container 2

Simulator Instance EDD

H1

Emulated Applications on H1

0 0

R1

tap1

tap2

1

veth2.1

R2

veth2.0

veth1.0

eth0

Emulated Applications on R1

eth0

eth1

bridge2 bridge1

OpenVZ Kernel

Fig. 3. Collocated emulated hosts

emulated hosts, PrimoGENI uses the VPN-based emulation infrastructure, which we developed previously in PRIME (Liu et al., 2009). In this scheme, OpenVPN clients are run at the the remote emulated hosts, each corresponding to one VNIC. PrimoGENI designates one or more compute nodes to run a modified OpenVPN server, which forwards IP packets to and from the EDDs collocated with the simulator instances. Detailed information about this interconnection mechanism can be found in Liu et al. (2009). The interconnection mechanisms for collocated emulated hosts and external networks are described in Sec. 4.2 and Sec. 4.3, respectively. 4.2

Collocated Emulated Hosts

Here we describe an interconnection mechanism designed for collocated emulated hosts, which uses Linux bridges and TAP devices. Fig. 3 depicts our design. For each collocated emulated host, we create a software bridge and a TAP device connected to the bridge. Multiple bridges are used to segregate the traffic of each emulated host and ensure the emulated traffic is routed properly. In particular, we do not allow collocated emulated hosts to communicate with each other directly without going through the simulator. There are two possible alternatives. One could use VLANs to separate the traffic, although processing the VLAN headers may introduce additional overhead. One could also use ebtables, which is a mechanism for filtering traffic passing through a Linux bridge (Bart De Schuymer, 2011). In our implementation, we use the TAP device for the EDD collocated at the simulator instance (in container 0) to send and receive packets to and from the VNICs1 . The EDD also acts as an ARP proxy responding to the ARP requests from the emulated hosts, so as to direct packets originated from the VNICs to the associated TAP device. The collocated emulated hosts are running as OpenVZ containers. We use a virtual Ethernet device (OpenVZ 1. The latest TAP device implementation prohibits writing IP packets to the kernel, possibly for security reasons. We choose to use a raw socket instead for the EDD to send IP packets.

8

VETH, 2011) for each VNIC on the emulated hosts. The virtual Ethernet device is an Ethernet-like device, which actually consists of two Ethernet interfaces—one in container 0 (the privileged domain) and the other in the container where the emulated host is. The two interfaces are connected to each other, so that a packet sending to one interface will appear at the other interface. We connect the interface at container 0 to the bridge which corresponds to the emulated host. We use an example to show how this mechanism works, by following the path of a ping packet from H1 via R1 and R2 shown in Fig. 3. Suppose H1 and R1 are emulated in container 1 and 2, respectively. Before container 1 can send a packet to R1, assuming its ARP cache is currently empty, it sends an ARP request out from eth0 asking for the MAC address of R1’s network interface eth0. The EDD responds with the MAC address of tap1, which is the TAP device connected to bridge1, the software bridge designated for container 1. Subsequently, container 1 is able to forward the ICMP packet to the EDD, which injects the packet into the simulator (by inserting an event that represents the packet sent out from network interface 0 of the simulated host H1). Once the simulation packet gets to R1 on the simulated network, the packet is exported from the simulator and sent by the EDD to eth0 in container 2 through tap2 and bridge2. Similarly, before container 2 forwards the packet onward to R2, it sends an ARP request out from eth1. The EDD responds with the MAC address of tap2, which is the TAP device connected to bridge2, the software bridge designated for container 2. In this way, the packet can find its way to the simulator, which forwards it on the virtual network. 4.3

Traffic Portals

A simulated network interface can be attached to an external network using the traffic portal. Traffic portal is an interconnection mechanism designed to allow physical networks to exchange traffic with the simulated network. Fig. 4 shows an example where we attach two physical networks to the simulated network: the 13.0.0.0/8 network is attached to interface 1 of R1 and the 14.0.0.0/8 network is attached to interface 2 of R3. In this example, each traffic portal manages the connection to a single physical network; in general, a traffic portal is capable of managing multiple external networks with the only restriction being that each external network can only be attached to a single traffic portal. The traffic portals are mapped to network interfaces on the compute node running the virtual network. In the

Simulator Instance 1 R1

EDD

1 2

R2

14.0.0.0/8 2 1

13.0.0.0/8

eth1

R4

R3 2

eth2 Linux Kernel

eth1

eth2

13.1.0.30

14.1.0.20

Fig. 4. Traffic portal example

previous example, interface 1 of R1 is mapped to to eth1 and interface 2 of R3 is mapped to eth2 on the compute node. The EDD uses raw sockets to read/write raw ethernet packets from/to the interfaces. Additionally, the EDD runs the ARP protocol and responds to ARP requests for any IP addresses that are within the simulated network or other attached external networks. For example, the ARP on eth1 will respond to ARP requests for any IP in the simulated network and in the 14.0.0.0/8 network. We use this example to show how this mechanism works, by following the path of a ping packet from 13.1.0.30 via R1, R2 and R3 to 14.1.0.20. Before 13.1.0.30 can send a packet to 14.1.0.20, assuming its ARP cache is currently empty, it sends an ARP request for the MAC address of 14.1.0.20. The EDD responds with the MAC address of eth1, which is the network interface mapped to the traffic portal for the 13.0.0.0/8 network. Subsequently, the EDD will receive the ICMP packet, which is injected into the simulator (by inserting an event that represents a packet received on network interface 1 of the simulated router R1). The packet is routed through the virtual network until it arrives at R3 at which point the packet is exported from the simulator and sent by the EDD via eth2 and routed by the real switch to 14.1.0.20. Similarly, before 14.1.0.20 can respond to the ICMP ECHO request, it must send an ARP for 13.1.0.30. The EDD responds with the MAC address of eth2. In this way, real packets can be routed back through the virtual network. In the previous example we have directly attached eth1 and eth2 to separate switches of the corresponding physical networks. One could also attach eth1 and eth2 to a single switch. However, the interfaces must segregate the traffic of the two physical networks; otherwise the underlying hardware could bypass the simulator and

9

route the traffic around the simulated network. We can simply use VLAN tags to segregate traffic from the two networks in this case.

5

EXPERIMENTS

In this section we describe a set of experiments aimed at validating the accuracy of our testbed. In order to show the utility of our approach, we also assess its performance limitations. The experiments described in this section are conducted on a prototype PrimoGENI cluster with eight Dell PowerEdge R210 rack-mount servers, each with dual quad-core Xeon 2.8 GHz processors and 8 GB memory. The servers are connected using a gigabit switch.

Scenario 1: Simulator

H1 R1 H2

H3 R2 H4

Container 0

Scenario 2: Simulator

H1 Emulated Host H2 Container 1

R1 H2

H3 Emulated Host H4

R2 H4

Container 0

Scenario 3: sim0

sim1

R1

R2

Ctnr 0

Ctnr 0

H1

H3

Emulated Host H2 Container 1

H2

Compute Node 0

5.1

Validation Studies

We validate the accuracy of the testbed by comparing the TCP performance between emulation and simulation. TCP is used in these experiments because it is highly sensitive to delay, jitter, and losses and therefore can magnify the errors introduced by the emulation infrastructure. Previously, we performed validation studies for the VPN-based emulation infrastructure; which is designed for connecting applications running on remote machines (Liu et al., 2009). In this study, therefore, we only focus on the emulation infrastructure for collocated emulated hosts and traffic portals. 5.1.1 Collocated Emulated Hosts The TCP congestion window trajectories achieved by the real Linux TCP implementations on the OpenVZ containers are compared against those from our simulation. We arbitrarily choose three congestion control algorithms: BIC, HIGHSPEED, and RENO; out of the 14 TCP variants we have implemented in the PRIME simulator (Erazo et al., 2009). We use a dumbbell model for the experiments. The dumbbell model has two routers connected by a bottleneck link with 10 Mb/s bandwidth and 64 ms delay. We attach two hosts to the routers on either side using a link with 1 Gb/s bandwidth and negligible delay. The buffers in all network interfaces are set to be 64 KB. In the experiment, we direct a TCP flow from one host to the other that traverses the two routers and the bottleneck link. For each TCP algorithm we test three scenarios, as illustrated in Fig. 5. In the first scenario, we perform pure simulation and use a simulated traffic generator. In the second and third scenario, we designate the two hosts as emulated hosts and use iperf to generate the TCP flow (and measure its performance). We run the emulation on one compute node for the second scenario,

Container 2

Emulated Host H4

H4

Container 1

Compute Node 1

Scenario 4: Simulator

H1 Physical Host H2

R1

CN 0

H3

Compute Node 1

Scenario 5:

sim0

CN 0

CN 2

sim1

H1 Physical Host H2

Physical Host H4

R2

H3 R1 CN 1

R2 CN 2

Physical Host H4 CN 3

Fig. 5. Dumbbell experiment scenarios

in which case the compute node runs the simulator in container 0 and uses the other two containers for the emulated hosts. In the third scenario, we use two compute nodes, each running one simulator instance and one emulated host. The emulated traffic in this case has to travel across the memory boundary between the two parallel simulator instances running on two compute nodes. (Scenarios 4 and 5 will be used by testing traffic portals). For simulation, we use a script to analyze the trace output; for emulation, we sample the (/proc/net/tcp) file at regular intervals to extract the TCP congestion window size. Fig. 6 shows very similar TCP congestion window trajectories between simulation and emulation. Next, we use a TCP fairness test to show whether our approach can correctly intermingle emulated and simulated packets. We generate two TCP flows in the same direction—one for each of the two hosts on the left side to one of the two hosts on the right side. We select the TCP HIGHSPEED algorithm for both flows. We start one flow 20 seconds after the other flow. We compare the first and the second scenario. The results are shown in Fig. 7. In both cases, we see that the congestion window

10

HIGHSPEED

180

180

160 140 120 100 80 60 40

pure simulation emulation, sequential run emulation, parallel run

20 0 0

20

40 60 Time (seconds)

80

RENO 250 Congestion Window Size (MSS)

200 Congestion Window Size (MSS)

Congestion Window Size (MSS)

BIC 200

160 140 120 100 80 60 40

pure simulation emulation, sequential run emulation, parallel run

20 0

100

0

20

40 60 Time (seconds)

pure simulation emulation, sequential run emulation, parallel run

200

150

100

50

0

80

100

0

20

40 60 Time (seconds)

80

100

Fig. 6. TCP congestion window trajectories

RENO

Congestion Window Size (MSS)

200

pure simulation, first flow pure simulation, second flow mixed experiment, first flow mixed experiment, second flow

150

C3 Flow 1 C1

S3 Flow 3

S1

Flow 2

100

C2

50

S2

Fig. 8. Cross-traffic example 0 0

10

20

30 40 50 Time (seconds)

60

70

80 7e+08

simulation physical mixed

6e+08 TCP Sequence Number

Fig. 7. TCP fairness size of the first TCP flow reduces when the second TCP flow starts; both flows eventually converge with a fair share of the bandwidth (at about 30 seconds after the second flow starts transmitting). Again, we see similar results between simulation and emulation.

5e+08 4e+08 3e+08 2e+08 1e+08 0 -1e+08 0

5.1.2 Traffic Portals We first investigate whether the traffic portal can correctly intermingle real and simulated flows. Using the setup shown in Fig. 8 we evaluated three scenarios. In the first scenario (called “simulated”), we simulate the entire model. In the second scenario (called “mixed”), we simulate all of the routers, as well as hosts S2, C2, S3 and C3. We use two real hosts, S1 and C1, and connect them to the simulator via the traffic portals. In the third scenario (called “physical”), we instantiated the whole network topology on Utah EmuLab (White et al., 2002). During the experiment, we first start Flow1 at time zero, which consists of a single TCP flow from S1 to C1. We then start Flow2 10 seconds later, which consists of two separate TCP sessions from S2 to C2. 10 seconds after Flow2, we start Flow3, which also consists of two TCP sessions from S3 to C3. Each TCP session in Flow2 and Flow3 transfers 100 MB. The TCP session in Flow1 transfers an object large enough so it does not complete during the experiment.

10

20

30 40 50 Time (seconds)

60

70

80

Fig. 9. TCP sequence numbers for Flow 1

In Fig. 9 we plot the TCP sequence numbers over time for Flow1. We conducted 25 runs per data point and plot the average with the 95% confidence intervals. The results match fairly well in the first 10 seconds and after 50 seconds. Between 10 and 50 seconds when Flow1 interacts with the other two flows, we observe some difference in throughput achieved for the mixed scenario. We attribute this to the additional delay imposed by the I/O system that we discuss in the next section. 5.2

Performance Studies

5.2.1 Collocated Emulated Hosts The emulation infrastructure inevitably puts a limit on the throughput of the emulated traffic. In the first experiment, we looked into this limiting effect on the

11

450

Aggregate TCP Throughput (Mbit/sec)

TCP Throughput (Mbit/sec)

900

pure simulation emulation, sequential run emulation, parallel run

400 350 300 250 200 150 100 50 0

pure simulation, 1 flow pure simulation, 10 flows sequential run, 10 flows parallel run, 10 flows sequential run, 1 flow parallel run, 1 flow

800 700 600 500 400 300 200 100 0

0

50

100 150 200 250 300 350 Bottleneck Link Bandwidth (Mbit/sec)

400

450

0

100

200 300 400 500 600 700 Bottleneck Link Bandwidth (Mbit/sec)

800

Fig. 10. TCP throughput for collocated hosts

Fig. 11. Cross-traffic TCP throughput

TCP behavior for collocated emulated hosts. For this experiment, we again use the dumbbell model. However, to increase the TCP throughput, we reduce the delay of the bottleneck link of the dumbbell model to 1 millisecond. We vary the bandwidth of the bottleneck link from 10 Mb/s to 450 Mb/s with increments of 40 Mb/s. As in the previous experiment, we direct a TCP flow from one host to the other through the bottleneck link and we compare the same three scenarios. Fig. 10 shows the results. While the throughput increases almost linearly with the increased bandwidth for the simulated flow, the error becomes apparent for emulated traffic at high traffic intensity. The throughput for the emulated traffic is kept below roughly 250 Mb/s for sequential runs and 130 Mb/s for parallel runs. The reduced throughput for parallel runs is due to the communication overhead as the TCP traffic gets exposed to the additional delay between the parallel simulator instances (over MPI). We were surprised to find the throughput of the sequential run was limited to roughly 250 Mb/s. We conducted preliminary evaluations using UDP and saw a throughput closer to 500 Mb/s. After an extensive investigation we discovered that the I/O system imposed significant delays now and then on packets when the packet rate was low. This would have a significant impact on TCP during its starting phase and cause TCP to throttle the throughput.

compare the results with those from the pure simulation case (Scenario 1). Fig. 11 shows the results. While the throughput increases linearly with the increased bandwidth for the simulated flow, it becomes apparent that the traffic with 10 TCP flows crossing through the traffic portal is capped at roughly 500 Mb/s for sequential runs and 180Mb/s for parallel runs. Again, the reduced throughput for parallel runs is due to the communication overhead imposed by the simulator as the TCP traffic goes across the boundary of the parallel simulator instances. This indicates there is room for future improvement. In the case of 1 TCP flow, the throughput is significantly lower. We discovered that this is due to the same timing issue at the I/O system that we mentioned earlier. The sporadic delay imposed on the packets caused TCP to throttle its congestion window size. As a result we also found significant variance in the throughput measurement for the sequential runs. This behavior is not obvious for the parallel runs because the throughput is limited by MPI communication overhead. In another experiment, we evaluated the ability for an external physical host to interact with a simulated host using TCP. The physical host acted as a client to simultaneously download 10 files of significant size from a simulated host. PrimoGENI provides full TCP interoperability. The results are shown in Fig. 12. We measured the aggregate throughput of all TCP flows and found the throughput is limited to about 250 Mb/s for sequential runs and about 120 Mb/s for parallel runs. This is mainly due to the simulator falling behind when generating large amounts of traffic. This indicates further room for improvement.

5.2.2 Traffic Portals We also take a closer look at the capacity of the traffic portal. We use the same dumbbell model as in the previous section (Fig. 5), however, this time with test scenarios 4 and 5. That is, both H2 and H4 are physical hosts, and we run the simulator either sequentially on one compute node or in parallel on two compute nodes. We direct either one or ten TCP flows between the two physical hosts through the traffic portal. We again

5.2.3 Emulation Capacity In the previous experiments we determined that the emulation infrastructure is placing an upper limit on the emulation traffic the system can support. Here, we

12

Aggregate TCP Throughput (Mbit/sec)

900

pure simulation emulation, sequential run emulation, parallel run

800 700 600 500 400 300 200 100 0 0

100

200 300 400 500 600 700 Bottleneck Link Bandwidth (Mbit/sec)

800

Fig. 12. Interactive TCP throughput

Aggregate Throughput (Mbit/sec)

300 250 200 150 100 50

one compute node two compute nodes four compute nodes

0 1

2

4 8 16 32 Number of Emulated TCP Flows

64

128

Fig. 13. Emulation capacity

use a set of experiments to measure the capacity of the emulation infrastructure as we increase the number of TCP flows while increasing the number of emulated hosts we run. Here we use the same dumbbell model (with a bottleneck link of 1 Gb/s bandwidth and 1 millisecond delay), and attach the same number of emulated hosts on each side of the dumbbell routers. We start a TCP flow (using iperf ) for each pair of emulated hosts, one from each side of the dumbbell. So the total number of TCP flows is half the number of emulated hosts. We make same number of the TCP flows to go from left to right as those from right to left. The measured throughput (from iperf ) is shown for all emulated TCP flows in Fig. 13. The curves shown in Fig. 13 correspond to three experiments, described as follows. In the first experiment, we assign all emulated hosts to run on the same compute node. For one flow, the throughput reaches about 250 Mb/s, as we have observed in the previous experiment for sequential runs. The aggregate throughput increases slightly for two flows, but drops continuously as we increase the number of flows all the way to 64 flows (that’s 128 VMs). The slight increase is probably due to TCP’s opportunistic behavior

that allows it achieve better channel utilization with more flows. We suspect that the drop in throughput is because the emulated hosts (OpenVZ containers) are competing for shared buffer space in the kernel network stack. With a smaller share when the number of containers gets bigger, the TCP performance degrades. In the second experiment, we divide the network between two compute nodes (splitting the model along the bottleneck link). The throughput for one emulated flow in this case is around 130 Mb/s, like we have observed before. As we increase the number of flows, the aggregate throughput increases slightly until 32 flows, after which we start to see a significant drop. Running the simulation on two compute nodes results in more the event processing power, the simulator is thus capable of handling more emulated flows than the sequential case. In the third experiment, we extend the dumbbell model by placing four core routers in a ring and connecting them using the bottleneck links. We attach the same number of emulated hosts to each core router. Each TCP flow was sent from an emulated host attached to one router to another emulated host attached to the next router in the ring. In this case, we spread the number of emulated flows evenly among the emulated hosts. Comparing with the scenarios with two compute nodes, the aggregate throughput for the four-node case is higher. Again we think this is due to the higher processing power of the parallel simulator. The throughput starts to drop for 128 flows (that’s 32 VMs per compute node) with increase contention for the shared buffer space.

6

CONCLUSION

PrimoGENI supports real-time network simulation and emulation on GENI enabled resources. It provides the necessary tools for the experimenters to construct and configure the network models, allocate the resources, deploy and run the experiments, and collect the experiment results. Each experiment is run within its own slice of resources, allocated using the GENI control framework. The resources may consist of the compute nodes in the cluster and possibly remote machines. Each compute node is treated as a scaling unit in PrimoGENI; it can be configured to run a parallel simulator instance together with a set of virtual machines for emulated hosts. The latter provides a realistic operating environment for testing real implementations of network applications and services. PrimoGENI applies parallel and distributed simulation techniques to synchronize the simulator instances running within the scaling units. The virtual machines are connected with the simulator instances using a flexible emulation infrastructure.

13

Our validation experiments show that PrimoGENI can produce accurate emulation results when the emulated traffic does not exceed the capacity of the emulation infrastructure. Our performance studies show that the throughput of the emulated traffic is dependent upon the number of virtual machines running on each scaling unit. We are currently investigating more efficient mechanisms to increase the emulation throughput. For example, we want to apply zero-copy techniques for moving data packets more efficiently between the virtual machines. We also want to have a fine-grained control of the I/O functions inside the real-time simulator to reduce the overhead. Another technique is to slow down the emulation speed using time dilation (Gupta et al., 2006), which controls a systems notion of how time progresses on the virtual machines. This is achieved by enlarging the interval between timer interrupts by what is called a time dilation factor (TDF), thus slowing down the system clock. If we proportionally slow down the realtime simulator, the applications can experience a higher networking speed than what can be offered by the emulation system.

ACKNOWLEDGMENTS We want to thank our undergraduate students, Eduardo Tibau and Eduardo Pe˜na, for the development of slingshot. The PrimoGENI project is part of GENI’s Spiral 2 prototyping and development effort, funded by the National Science Foundation (NSF) through GENI Project Office (GPO). The PrimoGENI project is built on PRIME, which is also funded by NSF through the grant CNS-0836408.

R EFERENCES Apache MINA Project. http://mina.apache.org/, 2011. Paul Barford and Larry Landweber. Bench-style network research in an Internet instance laboratory. ACM SIGCOMM Computer Communication Review, 33(3): 21–26, 2003. Bart De Schuymer. http://ebtables.sourceforge.net/, 2011. Andy Bavier, Nick Feamster, Mark Huang, Larry Peterson, and Jennifer Rexford. In VINI veritas: realistic and controlled network experimentation. In Proceedings of the 2006 ACM SIGCOMM Conference, pages 3–14, 2006. Lee Breslau, Deborah Estrin, Kevin Fall, Sally Floyd, John Heidemann, Ahmed Helmy, Polly Huang, Steven McCanne, Kannan Varadhan, Ya Xu, and Haobo Yu. Advances in network simulation. IEEE Computer, 33 (5):59–67, 2000.

James Cowie, David Nicol, and Andy Ogielski. Modeling the global Internet. Computing in Science and Engineering, 1(1):42–50, 1999. John DeHart, Fred Kuhns, Jyoti Parwatikar, Jonathan Turner, Charlie Wiseman, and Ken Wong. The open network laboratory. ACM SIGCSE Bulletin, 38(1): 107–111, 2006. Miguel Erazo, Yue Li, and Jason Liu. SVEET! A scalable virtualized evaluation environment for TCP. In Proceedings of the 5th International Conference on Testbeds and Research Infrastructures for the Development of Networks and Communities (TridentCom’09), April 2009. Kevin Fall. Network emulation in the Vint/NS simulator. In Proceedings of the 4th IEEE Symposium on Computers and Communications (ISCC’99), pages 244– 250, 1999. Flux Research Group. http://www.protogeni.net/, 2011. George Karypis Lab. http://www.cs.umn.edu/∼metis/, 2011. Diwaker Gupta, Kenneth Yocum, Marvin McNett, Alex Snoeren, Amin Vahdat, and Geoffrey Voelker. To infinity and beyond: time-warped network emulation. In Proceedings of the 3rd USENIX Symposium on Networked Systems Design and Implementation (NSDI’06), 2006. Jason Liu. A primer for real-time simulation of largescale networks. In Proceedings of the 41st Annual Simulation Symposium (ANSS’08), pages 85–94, 2008. Jason Liu and Yue Li. On the performance of a hybrid network traffic model. Simulation Modelling Practice and Theory, 16(6):656–669, 2008. Jason Liu, Yue Li, Nathanael Van Vorst, Scott Mann, and Keith Hellman. J. Systems & Software, 82(3): 473–485, 2009. Xin Liu, Huaxia Xia, and Andrew A. Chien. Network emulation tools for modeling grid behavior. In Proceedings of 3rd IEEE/ACM International Symposium on Cluster Computing and the Grid (CCGrid’03), 2003. Modeling and Networking Systems Research Group. http://www.primessf.net/, 2011. OpenVZ. http://wiki.openvz.org, 2011. OpenVZ VETH. http://wiki.openvz.org/Veth, 2011. ORCA-BEN Project. https://geni-orca.renci.org/trac/, 2011. Larry Peterson, Tom Anderson, David Culler, and Timothy Roscoe. A blueprint for introducing disruptive technology into the Internet. In Proceedings of the 1st Workshop on Hot Topics in Networking (HotNetsI), 2002. D. Raychaudhuri, I. Seskar, M. Ott, S. Ganu, K. Ra-

14

machandran, H. Kremo, R. Siracusa, H. Liu, and M. Singh. Overview of the ORBIT radio grid testbed for evaluation of next-generation wireless network protocols. In Proceedings of the IEEE Wireless Communications and Networking Conference (WCNC 2005), 2005. George F. Riley. The Georgia Tech network simulator. In Proceedings of the ACM SIGCOMM Workshop on Models, Methods and Tools for Reproducible Network Research (MoMeTools’03), pages 5–12, August 2003. Rob Simmonds and Brian W. Unger. Towards scalable network emulation. Computer Communications, 26 (3):264–277, 2003. Neil Spring, Larry Peterson, Andy Bavier, and Vivek Pai. Using PlanetLab for network research: myths, realities, and best practices. ACM SIGOPS Operating Systems Review, 40(1):17–24, 2006. Stephane Apiou. http://funionfs.apiou.org/, 2011. The GENI Project Office. http://groups.geni.net/, 2011a. The GENI Project Office. http://groups.geni.net/geni/ wiki/GeniSysOvrvw, 2011b. Amin Vahdat, Ken Yocum, Kevin Walsh, Priya Mahadevan, Dejan Kostic, Jeff Chase, and David Becker. Scalability and accuracy in a large scale network emulator. In Proceedings of the 5th Symposium on Operating Systems Design and Implementation (OSDI’02), pages 271–284, 2002. Brian White, Jay Lepreau, Leigh Stoller, Robert Ricci, Shashi Guruprasad, Mac Newbold, Mike Hibler, Chad Barb, and Abhijeet Joglekar. An integrated experimental environment for distributed systems and networks. In Proceedings of the 5th Symposium on Operating Systems Design and Implementation (OSDI’02), pages 255–270, 2002. Garrett Yaun, David Bauer, Harshad Bhutada, Christopher Carothers, Murat Yuksel, and Shivkumar Kalyanaraman. Large-scale network simulation techniques: examples of TCP and OSPF models. ACM SIGCOMM Computer Communication Review, 33(3):27–41, 2003. Junlan Zhou, Zhengrong Ji, Mineo Takai, and Rajive Bagrodia. MAYA: integrating hybrid network modeling to the physical world. ACM Transactions on Modeling and Computer Simulation (TOMACS), 14 (2):149–169, 2004.