Distributed Virtualization Scenarios Using VNUML - dit/UPM

Distributed Virtualization Scenarios Using VNUML Fermín Galán Márquez1,2, David Fernández Cambronero2 1

Telefónica I+D Departamento de Ingeniería de Sistemas Telemáticos (DIT), Universidad Politécnica de Madrid (UPM) {galan,david}@dit.upm.es

2

Abstract. This paper describes a distributed approach to deploy and manage a virtual scenario (described in a high-level specification language) over a set of deployment hosts. The proposed system is as transparent as possible from the point of view of the user. It is based on Virtual Network User Mode Linux (VNUML), a virtualization tool with a solid background on research and educational projects. After a brief introduction to VNUML, the architecture and its operation is thoughtfully described. Additionally, an example is provided, showing the practical application and feasibility of the proposed solution. Keywords: distributed management, virtualization, VNUML, UML.

1 Introduction Virtualization techniques allow running over a physical host a set of virtual machines, which behave very close (in terms of transparency and performance) to conventional real ones. Nowadays, virtualization is widely used in datacenters to concentrate the organization service infrastructure (both classical services, like web or email, and advanced ones, like Service Oriented Architecture, SOA) into a reduced set of physical servers. This approach is called server consolidation and its purpose is to reduce infrastructural and operational cost, along to provide a high flexibility degree in service management (quick provisioning, high availability, etc.). Another (maybe less known) field of application of virtualization is networking testbeds. A testbed is a development and testing oriented platform usually employed in research, pre-production and educational contexts. The goals are basically the same than in server consolidation but there is a significant difference with virtual networking testbeds: not only virtual machines have to be provided, but also virtual networks interconnecting them. Note that in the case of server consolidation, network is usually seen as a commodity and complex topologies are not addressed. Although there are some systems oriented to virtualize networking testbeds (VNUML [1], NetKit [2] or MLN [3]), able to specify and deploy virtualization scenarios involving complex virtual network topologies, they assume a centralized approach. In other words, the virtual scenario is deployed and managed inside just one physical host. The main drawback of this centralization is scalability, being difficult to build large scenarios (like the one described in [4], with more than 50 nodes).

In order to solve the shortcomings of these tools, this paper goes one step further defining a distributed system that allows the management of a virtual scenario (to be deployed in a set of physical hosts) transparently as a whole from a user high-level perspective. The solution is based on Virtual Network User Mode Linux (VNUML [1]), an open source tool with a solid background in research and education. VNUML was created in the context of Euro6IX project to emulate complex IPv6 network scenarios, but has later been used in several research projects, for example, to experiment different constraints based routing algorithms over the ADRENALINE testbed [5], a GMPLS based Automatic Switched Optical Network (ASON). Moreover, VNUML has been extensively used in university computer network laboratories to allow students to work complex network scenarios [4]. See [1] for a detailed list of initiatives using VNUML. This paper describes an improvement of the VNUML tool, adding distributed management capabilities to its virtualization scenario specification language. The rest of the paper is structured as follows. First, a brief introduction to VNUML is provided in Sect. 2. The proposed solution is described in the next two sections, firstly providing a detailed description of the physical deployment architecture in Sect. 3, then describing the operation of the key architectural piece (the deployment controller) in Sect. 4. A practical example is introduced in Sect. 5. Next, Sect. 6 discusses related work and its relation with our approach. Finally, Sect. 7 closes the paper with the conclusions and future work lines.

2 VNUML VNUML [1] is a general purpose open source tool designed to help building virtual network scenarios or network testbeds automatically. Basically, VNUML is a frontend to UML (User Mode Linux) [6] –a virtualization technique to run virtual Linux machines over a Linux host– that allows the user to define, start and interact with virtual scenarios made of Linux virtual machines interconnected through virtual networks. Virtual scenarios created with VNUML can be interconnected with external equipment and networks, opening the possibility to create mixed virtual/real testbeds. Virtual network scenarios are mainly suited for functional tests. Performance evaluation is difficult to achieve under virtualization due to the fact that physical resources are shared among the virtual machines. However, some rough evaluations can be obtained when carefully planning and monitoring tests. An interesting comparison of resource consumption and performance of different virtualization techniques implementing a virtual network scenario can be found in [7]. VNUML relieves the users of all the complex UML details that the creation of virtual network scenarios requires, allowing them to concentrate on the purpose of the testbed rather than on its creation and management. The tool is made of two main components: the XML (eXtended Markup Language) based VNUML language that allows describing the virtual network scenario; and the interpreter of the language that parsers the description and builds and manages the scenario. Figure 1 shows the general use case for VNUML tool. First, the user writes the specification of the testbed using the VNUML language by means of a standard XML

based editor or, alternatively, using the VNUMLGUI tool [8]. Later, it invokes VNUML parser to process the specification and start the virtual network scenario (scenario building mode). Finally, the user interacts with the scenario to achieve his goals. Interaction with virtual machines can be achieved accessing them through their consoles or by executing programmed command sequences on them with the help of the VNUML tool (execution mode). Eventually, user dismantles the scenario, releasing the host resources, running VNUML in scenario releasing mode. Interaction with virtual scenario

Testbed design Virtual scenario build, exec, release

Network Scenario Spec

Net0

Net1

Net2

VNUML Parser Net3

Internet

Figure 1. VNUML use case A VNUML specification is made of two main sections (see the simplified example on Sect. 5): virtual networks and virtual machines. There are two other sections (global and host) but they are not described here for the sake of briefness. A complete reference of the VNUML language can be found in [1]. The virtual networks that interconnect virtual machines among them or with the host network interfaces are defined using the tag. Virtual networks are implemented by means of virtual software bridges running on the host machine. Later, the tag is used to describe each virtual machine. The language allows describing their main characteristics in terms of: the kernel and the filesystem they run ( and tags), the network interfaces they have ( tag) and the addresses associated with them, either IPv4 or IPv6 ( and tags), as well as the IP routes ( tag). Besides, virtual machine specifications can include and tags to define files to be copied from the host to the virtual machines and user defined commands to be executed on them. At present, the whole virtual scenario built using VNUML tool runs over a single host machine. The next two sections present a work-in-progress proposal to allow the distribution of a virtual scenario over several host machines.

3 Deployment Architecture The deployment architecture is composed of a set of N hosts, physically connected to an interconnection backplane. This backplane is composed of one or several programmable 802.1q VLAN-based interconnected network switches. The host-tobackplane physical connections (and the switch-to-switch ones, in the case of using

several switches) are trunks, able to multiplex several simultaneous VLANs. This architecture is illustrated in Fig. 2. The VNUML tool is installed in each one of the deployment hosts, along with the additional components it requires for proper operation (root filesystems, kernels, etc.).

Deployment hosts

command interface

Deployment controller

… trunk

Interconnection backplane management VLAN

Figure 2. Deployment architecture

In addition to deployment hosts, there is also one deployment controller (also physically connected to the interconnection backplane through a trunk interface) which purpose is to manage the distributed virtualization scenario, providing the user interface and implementing the procedures and operations described in Sect. 4. In order to do so, IP connectivity to each one of the deployment hosts is required, typically implemented through a dedicated management VLAN. The deployment controller is a logical entity, it can be set up in a separate physical machine (as shown in Fig.2) or in one of the deployment hosts.

4 Deployment Controller Operation This section describes the proposed mechanism to provide distributed virtualization management, based on implementing the virtual scenario described in VNUML language specification in a set of distributed physical hosts. The main goal is transparency, in the sense that the user defines the scenario from a high-level perspective, without concerning about how it would be finally deployed (i.e., which particular host will allocate which particular virtual machines). Therefore, the interface provided to users is as closer as possible to VNUML, involving the same three operation modes described in Sect. 2: scenario building, command sequence execution and scenario releasing. The workflow for these three modes is shown in Fig. 3. Note that all cases implement a common starting step, named segmentation, in which the VNUML specification is split in several per-host sub-specifications. Then, these subspecifications are used as input by the VNUML parser installed in each one of the corresponding deployment hosts. In addition, the building (releasing) operation needs to perform additional configuration (unconfiguration) on the deployment architecture. The different steps are described in the following subsections.

Segmentation

Segmentation

Segmentation

Environment configuration

Per-host VNUML (execution)

Per-host VNUML (releasing) Environment unconfiguration

Per-host VNUML (building) Scenario building

Command sequence execution

Scenario releasing

Figure 3. Operation modes workflow

4.1. Segmentation Segmentation is the first step for all the cases and its purpose is to divide the original VNUML specification into several per-host sub-specification pieces, each one including a subset of the total virtual machines, thus partitioning the virtualization scenario among the deployment hosts. The segmentation process involves the following sub-steps: 1. Virtual machine mapping to deployment hosts. This step establishes a correspondence in which each one of the in the original VNUML specification is assigned to exactly one deployment host. There are several alternatives to implement this mapping, from the simplest ones (e.g., simple load balancing used in the example in Sect. 5) to complex approaches (like the one described in [9], trying to find the optimal assignment that minimizes inter-switch bandwidth and other metrics). 2. Per-host VNUML sub-specifications generation. A VNUML sub-specification is generated for each deployment host, including the tags that have been assigned to it. Regarding , there are two possibilities: • Intra-host virtual networks (i.e., the network connects virtual machines deployed in the same deployment host). The tag in the subspecifications is the same that the one in the original VNUML file. • Inter-host virtual networks (i.e., the network connects at least a virtual machine that is deployed in other host). In this case, an external attribute need to be included in the , specifying an opaque name for the virtual bridge that will implement the distributed virtual network (e.g., “brA”), as detailed in Sect. 4.2. Once the N sub-specifications files have been generated, they are installed in the corresponding deployment hosts, using the management VLAN (e.g., scp, rsync, etc.). 4.2. Environment Configuration and Unconfiguration In the case of building scenario operation, additional steps need to be accomplished after segmentation by the deployment controller in order to provide the support for inter-host virtual networks. In particular, the following actions have to be performed:

•

Choose a free VLAN identifier for each inter-host virtual network (the distributed controller maintains a list of currently used identifiers) and enable it in the interconnection backbone switches if needed (e.g., ID 120, 121, etc.) • For each VLAN ID, create the corresponding sub-interfaces in the trunk connection of each deployment host using the associated virtual network (e.g., eth0.120, eth0.121, etc.). This involves the remote use of the vconfig command through the management VLAN. • For each sub-interface, create a corresponding virtual bridge using the opaque name that was used during segmentation for the external attribute (e.g., “brA”, etc.). This involves the use of the brctl command through the management VLAN. In the case of scenario releasing operation, the corresponding “dual” unconfiguring steps are performed in inverse order (i.e., bridge releasing, sub-interface releasing and VLAN ID freeing). Finally, as show in Fig. 3, note that the environment configuration step precedes the per-host VNUML invocations in the case of scenario building (because the bridges need to be created before VNUML runs), while the order is the opposite in the case of scenario releasing (because of virtual machines need to be shutdown before releasing the bridges). 4.3. Per-host VNUML This step involves the invocation of the VNUML parser (installed in each deployment host, as detailed in Sect. 3) using the per-host sub-specification in the corresponding mode (i.e., building scenario, command sequence execution or scenario releasing), through the management VLAN. The parser behaves as described in Sect. 2, performing actions locally for the assigned sub-specification.

5 Example This section provides and utilization example, in order to illustrate the distributed virtualization mechanism previously described. In particular, let’s consider a 5 virtual machines and 3 virtual networks scenario to be built in an environment composed of two physical hosts (as shown in Fig. 4). The VNUML specification (simplified) corresponding to the scenario is shown in Table 1. Table 1. VNUML specification 10.0.0.1 default 10.0.0.2 default

10.0.0.3 10.0.1.1 10.0.2.0/24 10.0.1.2 10.0.2.1 default 10.0.2.2 default

vm2 .2 .2 vm5 Net1 Net2 Net0 10.0.2.0/24 10.0.0.0/2410.0.1.0/24 .1 .1 .2 .1 .3

vm1

host2

host1 vm2 vm1 Net0

vm4

Net2

vm5

vm3

vm4

vm3

brA

brA

Deployment controller

Interconnection backplane Net1 VLAN (e.g, ID=120)

Figure 4. Deployment example In order to deploy this scenario, the user issues the scenario building operation in the deployment controller, using the aforementioned specification (Table 1) as input. Then, the controller performs the following operations in sequence: • Segmentation. Considering a simple load balancing algorithm that provides a fair distribution of virtual machines among deployment hosts, one possible mapping is to assign vm1, vm2 and vm3 to host1 and vm4 and vm5 to host2. In that case, Net0 and Net2 are intra-host virtual networks (not requiring any modification), while Net1 is and inter-host network, requiring the addition of the external attribute in . The resulting per-host sub-specifications (shown in Table 2 and Table 3) are copied to each host using scp or rsync. • Environment configuration. There is one inter-host network (Net1), so the deployment controller reserves a free VLAN ID for it (let’s assume this ID is 120). Then the VLAN is enabled in the interconnection backplane switches, a sub-interface is created in the trunk interface in each host (considering eth0 is the trunk interface, ‘vconfig add eth0 120’) and a virtual bridge created with the value used in external attribute as name (‘brctl addbr brA’), bridging the sub-interface just created (‘brctl addif brA eth0.120’). • Per-host VNUML. Finally, the VNUML parser installed in each host is invoked (through SSH) in building scenario mode, using the subspecification derived during segmentation as input.

Table 2. VNUML sub-specification for host1 10.0.0.1 default 10.0.0.2 default 10.0.0.3 10.0.1.1 10.0.2.0/24

Table 3. VNUML sub-specification for host2 10.0.1.2 10.0.2.1 default 10.0.2.2 default

6 Related Work Nowadays, there are several existing modern general purpose virtualization systems, like VMware [10], UML [6], Xen [11] or KVM [12]. However, they only provide “bare” virtual machines but not virtual scenario management frameworks (neither centralized nor distributed). Exceptions are VirtualCenter [13], XenServer [14] or SmartDomains [15], which provide a management and monitoring interface for virtual machines deployed in a set of physical hosts, including some advanced features (like virtual machine migration between deployment hosts). However, they are both oriented to server consolidation and not to networking testbeds. Therefore, they do not allow to manage networking scenarios involving complex topologies. Netkit [2] and MNL [3] provide languages to describe virtual networking scenarios along with processing tools able to parse such specifications for deployment and management. They are front-ends of the actual virtualization system (in particular, UML and Xen) and, in this sense, they work similar to VNUML. However, they assume all the virtual machines are deployed in the same physical host and thus do not provide a distributed solution like the one described in this paper.

In a different context, the descriptions of virtual machines and networks in VNUML specifications can be associated with CIM (Common Information Model [16]) objects which are semantically close. As a matter of fact, a model-driven architecture based on testbed-independent deployment-agnostic scenario models (described using a specific CIM profile) is being work out [17]. The testbedindependent models are then transformed to testbed-specific deployment-oriented models. In this sense, segmentation can be seen as the transformation step and the set of per-host VNUML specifications as the testbed-specific model. Finally, regarding virtual machine mapping, the problem of efficiently allocating virtual topologies to deployment environments is treated in [9]. Optimal mapping (minimizing inter-switch bandwidth and other metrics) is a NP-hard problem and that paper proposes an algorithm that finds a sub-optimal solution in a reasonable time. Although authors start from a context different from virtual scenario distribution (in particular, the cluster-based Emulab testbed [18], in which each node is mapped to a physical PC), the solution is general enough to be applied to our case, complementing the segmentation step (described in Sect. 4.1) with a more powerful mapping algorithm than the simple load balancing used in the example in Sect. 5.

7 Conclusions and Future Work This paper has described an architecture to distribute virtual network scenarios over a set of physical hosts, thus providing a solution for the distributed virtualization management problem. Its main characteristic is transparency: user interacts with the scenario through the deployment controller using a well-defined high-level interface (i.e., build, exec and release operations), without knowing how the deployment will be made. The main component of the architecture is the deployment controller in charge of the segmentation of the scenario. The solution is based on VNUML tool, which has a long and solid background as a virtual testbeds deployment tool (see [1] for a list of initiatives using it). VNUML high-level language is used to describe the scenario to be distributed, and the processing engine is modified to implement virtual machines and network partitions in each host. The paper provides a complete example, showing the feasibility of the described solution which is at present being implemented. Regarding future work, there are several working lines, apart of the model-driven architecture introduced in Sect. 6. The simple mapping of virtual machines to deployment hosts (described in the example of Sect. 5) can be improved in several ways. One efficient algorithm is the one described in [9] but implementing it could be unnecessary complex (note that this algorithm was developed for Emulab, addressing the problem of fair utilization of a highly shared physical infrastructure, which is different from our distributed virtualization context). Instead, we are considering more affordable alternatives, like capability-based allocation (virtual machines describe the resources –CPU, memory, etc.– they need and the mapping process finds the most suitable physical host for them) or proximity-based allocation (if two virtual machines are “topologically near” the mapping process tries to assign them to the same physical host). Both imply extending the VNUML language.

Another working line is related with the evolution of the deployment architecture from local-area (i.e., the deployment environment is a cluster of hosts interconnected through LAN switches) to wide-area (i.e., deployment environment is geographically distributed, being the hosts interconnected through existing networks, like the Internet). Such environment implies new challenges: tunnel configuration among deployment hosts, security credential management or improved controller-to-host interface (XML-RPC or SOAP seem better alternatives than scp/SSH). However, from the user point of view, the system will be equally transparent in both local and wide area cases. Other well-known distributed deployment platforms (like the PlanetLab [19]) should be carefully studied to accomplish this goal. Finally, it could be interesting to study how to provide virtual machine migration between deployment hosts. Virtual machine migration is quite extended in server consolidation (mainly for load balancing and high availability) and it worth the effort to consider it also in the context of virtualized networking testbeds.

References 1. 2. 3. 4.

Virtual Network User Mode Linux (VNUML): http://www.dit.upm.es/vnuml NetKit, online document: http://www.netkit.org The MNL Project, online document: http://mln.sourceforge.net/ F. Galán, D. Fernández, J. Rúiz, O. Walid, T. de Miguel. “A Virtualization Tool in Computer Network Laboratories”, ITHET'04, Instambul, May 2004. 5. R. Muñoz, C. Pinart, R. Martínez, J. Sorribes, G. Junyent, A. Amrai et al.,“The ADRENALINE Testbed: Integrating GMPLS, XML and SNMP in transparent DWDM networks”, IEEE Communications Magazine, vol. 43(6), pp. s40–s48, August 2005. 6. Jeff Dike. “User Mode Linux (UML)”. Prentice Hall, 2006. 7. W. M. Fuertes, J.E. López de Vergara “A quantitative comparison of virtual network environments based on performance measurements”, Proceedings of the 14th HP Software University Association Workshop, Garching, Munich (Germany), July 2007. 8. VNUML Graphical User Interface: http://pagesperso.erasme.org/michel/vnumlgui 9. R. Ricci, C. Alfeld, J. Lepreau, “A solver for the Network Testbed Mapping Problem”, ACM SIGCOMM Computer Communication Review, vol. 33(2), pp. 65-81, April 2003. 10.J. Nieh, O. C. Leonard, “Examining VMware”, Dr. Dobb’s, August 2002. 11.P. Barham et al., “Xen and the art of virtualization”, SOSP’03, pp. 164–177, Bolton Landing, NY (USA), 2003. 12.M. T. Jones, “Virtual Linux: An overview of virtualization methods, architectures, and implementations”, IBM Whitepaper, December 2006. 13.VMware VirtualCenter, online document: http://www.vmware.com/products/vi/vc/ 14.XenServer, online document: http://www.xensource.com/products/xen_server/ 15.SmartDomains tutorial, online document http://smartdomains.sourceforge.net/ 16.Common Information Model (CIM), online document: http://www.dmtf.org/standards/cim/ 17.F. Galán, J. López de Vergara, D. Fernández, R. Muñoz, “A Model-driven Configuration Management Methodology for Networking Testbeds Infrastructures”, NOMS 2008, Salvador (Brazil), April 2008 (submitted). 18.B. White, et al., “An Integrated Experimental Environment for Distributed Systems and Networks”, OSDI’02, pp. 255-270, 2002. 19.L. Peterson, T. Anderson, D. Culler, T. Roscoe, “A Blueprint for Introducing Disruptive Technology into Internet”, HotNets-I, Princenton, NJ (USA), October 2002.