Seamless integration of cloud and fog networks - Wiley Online Library

INTERNATIONAL JOURNAL OF NETWORK MANAGEMENT Int. J. Network Mgmt (2016) Published online in Wiley Online Library (wileyonlinelibrary.com). DOI: 10.1002/nem.1944

Seamless integration of cloud and fog networks Igor Duarte Cardoso,1,, João P. Barraca,1,2 Carlos Gonçalves3 and Rui L. Aguiar1,2 1

Instituto de Telecomunicacoes, 3800-193 Aveiro, Portugal 2 Universidade of Aveiro, 3800-193 Aveiro, Portugal 3 NEC Laboratories Europe, 69115 Heidelberg, Germany

SUMMARY A way to merge cloud computing infrastructures with traditional or legacy network deployments, leveraging the best in both worlds and enabling a logically centralized control. A solution is proposed to extend existing cloud computing software stacks so they are able to manage networks outside the cloud computing infrastructure, the fog, by extending the internal, virtualized network segments. This is useful in a variety of use cases such as incremental legacy to cloud network migration, hybrid virtual/traditional networking, centralized control of existing networks, bare metal provisioning, and even offloading of advanced services from typical home gateways into the operator. Any organization can develop different ‘drivers’ to support new, specific networking equipment, not necessarily tied to a protocol or vendor, and leverage the fog network. Our conceptual solution is prototyped on top of OpenStack, including changes to the API, command-line interface (CLI), and other mechanisms. Test results indicate that there are low penalties on latency and throughput, and provisioning times are reduced in comparison with similar maintenance operations on traditional computer networks. Copyright © 2016 John Wiley & Sons, Ltd. Received 30 May 2015; Revised 4 June 2016; Accepted 6 June 2016

1. INTRODUCTION Trends like cloud computing, which exploit virtualization concepts, have dominated the service provision market, making the deployment of services easier and more flexible. Telecommunications operators have also started making use of cloud computing, alone or applied to network virtualization and software-defined networking (SDN) that together have created the network function virtualization (NFV) movement, with the purpose of simplifying tasks and operations, eventually decreasing time to market of new services. With network virtualization, providers have ultimate control and flexibility, and cloud tenants are now able to design and implement their own network topologies, with considerable flexibility, and attach them to their own virtual machines (VMs). Furthermore, it enables hybrid environments where networks are composed of remotely located virtual instances and locally located servers. Nowadays, for existing cloud computing stacks, to achieve the desired level of elasticity and ease of maintenance, the underlying network resources to be virtualized need to be as homogeneous as possible, while supporting specialized control interfaces (e.g., OpenFlow). Homogeneity of resources, by itself, can indeed alleviate a set of problems, improving efficiency in multiple aspects. However, it also limits what kind of use cases can be fulfilled by the cloud computing provider in comparison with a deployment consisting of heterogeneous network resources, in the sense that this potentially enables access to other protocols, dedicated networking features, or advanced administrative capabilities. Given these premises, this work seeks the midpoint between typical cloud computing infrastructures with network virtualization, based on homogeneous networking resources, and traditional network

Correspondence to: Igor Duarte Cardoso, Instituto de Telecomunicacoes, 3800-193 Aveiro, Portugal. E-mail: [email protected]

Copyright © 2016 John Wiley & Sons, Ltd.

I. D. CARDOSO ET AL.

deployments that rely on heterogeneous networking equipment (different types, vendors, and models), exploiting concepts that are parallel to typical SDN solutions while allowing the inclusion of legacy, for example, non-OpenFlow-enabled devices. We believe that this close integration is beneficial and actually required. The work hereafter presented aims to propose a solution that integrates existing cloud computing software stacks with already existing networks composed by devices having minimal management functions. Thus, it aims at extending on existing solutions to provide a better, more complete solution as a whole, with the best of both worlds, regarding integration of legacy networks and virtualization. We wish to make it possible to extend a network segment (usually virtual and deployed inside a cloud computing/network virtualization software stack) with another network segment that can live anywhere outside the first deployment, keeping control logically centralized just as with existing cloud resources. Moreover, heterogeneity support is one of the main pillars of our solution. Without this property, there are no substantial justifications to deploy the work in practice, precluding the accomplishment of any use case that justifies its own development. The document is organized in five sections. Section 2 explains the core motivation for this work, as well as the main use cases that can be made possible. Section 3 proposes a solution to materialize, or empower, the use cases mentioned. Section 4 evaluates and analyses the solution as implemented on top of OpenStack. Section 5 describes the state of the art related to the integration of networks into virtual computation environments, with focus in the integration of legacy devices and in what was presented as our work. Finally, Section 6 concludes, providing some closing thoughts on the solution and ideas for future work. 2. MOTIVATION AND USE CASES Our motivation for this work comes from the fact that different kinds of users have different requirements, although many of them can be improved in the same way. Our work aims at fulfilling a set of use cases that can serve them. We distribute use cases amongst three different types of users: telecommunications operators, enterprise/IT, and the final consumer or end-user. 2.1. For telecommunications operators Cloud computing paired with SDN, where network devices are represented as virtual instances, provides very interesting and useful features to customers. Cloud tenants or administrators are now able to design and implement their own network topologies, with considerable flexibility, and attach them to their own VMs. However, in order to achieve the desired level of elasticity and ease of maintenance, the resources to be integrated need to be very homogeneous. Although this is not really a disadvantage for new deployments, from the perspective of a telecommunications operator, there are undesirable consequences when providing a service like this to clients, namely, overhead and performance concerns [1] due to virtualization, and lack of backward compatibility with existing equipment. Also, upgrading all customer premises equipments (CPEs) is an expensive solution as a single provider will have an immense number of customers and, consequently, devices to maintain – some of them outdated. There are several use cases we envision that could benefit a telecommunications operator, by having a scalable solution integrating virtual and legacy networks, that is, where routers, CPE, or servers are able to be orchestrated together with cloud resources. Legacy to cloud migration is a use case that telecommunications operators can benefit from, namely, regarding the migration of the ‘old’ home gateways (HGWs) and even enterprise CPE. Another possibility for this migration is for bringing legacy, dedicated equipment to the cloud, like physical appliances meant for carrying out network functions. Figure 1 illustrates machines of a distant legacy network being brought to a virtualized network inside the cloud infrastructure. Centralized network control and flexibility can benefit telecommunications operators. Some use cases that come out as more specific instances of this are the ability to change network attributes in a global manner, for example, L3 addressing for end-users’ networks on a virtual private network (VPN) deployment or a virtual HGW. That way, administrators have maximum control over addressing, easily avoiding any conflicts, and issuing changes by executing non-ambiguous configuration commands. Copyright © 2016 John Wiley & Sons, Ltd.

Int. J. Network Mgmt (2016) DOI: 10.1002/nem

SEAMLESS INTEGRATION OF CLOUD AND FOG NETWORKS

Figure 1. Legacy resources being attached to a virtualized network Applying advanced services provided by the cloud computing framework to the physically deployed networks, for example, policy control and firewall, also contributes to the centralization of control and services because nothing else needs to be provisioned for the networks that typically are not under the umbrella of a cloud computing framework. Figure 4 further illustrates this use case. Home gateways, residential gateways, or some kinds of CPEs are used in households to provide users with broadband Internet access via an Internet Service Provider (ISP) as well as modern services like television’s channel streams’ rewinding. There have been progresses and changes to the typical HGW during the last years to accommodate for different access network technologies and new services, although not substantially. A possible use case that could be leveraged is the offloading of network services, usually presented in the HGW or other CPE, to the ISP’s infrastructure. Some have defended the idea of making an HGW more complex, like Rückert et al. [2], others in making it simpler, such as [3]. Cruz et al. [4] present a detailed framework for a fully virtualized HGW/residential gateway. Virtual HGWs can become a simple reality to deploy, offloading some services to the ISP side, for example, the local Dynamic Host Configuration Protocol (DHCP) server. Thus, ISPs can directly manage HGWs, which may prove helpful to reduce operating expenses (OpEx), time to market for new services, fight widespread security threats (as stated in [3]), and so on. Other advantages of this scenario are the lower cost of consumer equipment acquisition, increased energy savings because of the simplistic hardware, and reduced capital expenses, to name a few. Internet connectivity can be provided to the external hosts of the external network like any other host, such as a VM instance. Consequently, this use case can be especially useful for telecommunications operators betting on NFV. Besides HGWs, enterprise CPEs can likewise be integrated in the same scenario, with the same advantages and features as NFV. Figure 2 shows a deployment with two customers having their networks linked to the same cloud computing infrastructure. Because this work allows extending cloud-provided networks into the outside world (and vice versa), many different kinds of services may, as a result, be provided externally. Take the case of service chaining, an important NFV use case that can be leveraged. Existing equipment and services can be reused (which goes towards the incremental legacy to cloud migration) for service chaining purposes (like deploying a firewall or anti-virus and steering specific traffic through these services, located at existing equipment). Initially, service chains were deployed in a hard way, that is, specialized appliances that were then interconnected in predefined ways to achieve a desired purpose [5]. Recently, with the advent of SDN and NFV, service chains have become more flexible, with less associated OpEx, while easier to deploy and manage. However, legacy service chains are still the dominant portion of telecommunications operators’ core network infrastructures because it is impracticable to easily, and quickly, replace them with modern technology [5]. Consequently, a compromise between modern and legacy service chaining becomes an interesting possibility. Existing hardware, providing specific services, can thus be reused in a cloud context alongside ‘software-defined’ service chaining features implemented by recurring to, for example, L2 traffic steering [6], with minimum effort. Depicted in Figure 3 is a scenario where services both inside the cloud computing and the external infrastructures are being chained together. The arrows, along their numbered labels, show the direction and order of the chain amongst hosts. Copyright © 2016 John Wiley & Sons, Ltd.



Figure 2. Deployment where two kinds of customers are linked to a cloud computing infrastructure

Figure 3. Service chaining with external entities

Finally, telecommunications operators provide services to enterprise customers, and that may include the provision of VPNs. With this work, it is possible to deploy L2 VPN seamlessly with all the major advantages offered by this work: flexibility, control, and cloud integration and compatibility with heterogeneous equipment and technologies, all within the cloud computing platform. Bringing the local area network (LAN) to the wide area network is a possible use case for L2 VPNs that allows, for instance, to play a LAN-only multiplayer video game over the Internet. 2.2. For enterprise IT Consider a company that has its own datacenter, being responsible for its maintenance and administration. The equipment inside said datacenter may be very heterogeneous, consisting of different standards, vendors, technologies, requirements, and interoperability aspects. Besides that, some nodes have special functionality (like deep packet inspection) but are very hard (or costly) to replace by analogous, yet different, technology. The company has its reasons for moving into the cloud but faces multiple worries and blockers: Copyright © 2016 John Wiley & Sons, Ltd.



strict requirement to keep special functionality where it currently is, because of physical control, latency and performance requirements, or other aspects; desire to keep some existing equipment to maximize investments and avoid major expenses; difficulty in replacing the current network deployment as a whole; some services currently provided by the infrastructure may not be tolerant to downtimes, meaning that they must be migrated only when strictly necessary and as quickly as possible, with the cloud replacement already in place and properly tested; and there may be resources hampering or making the transition to cloud more difficult, at the time of the intention . . . . The points just discussed can also be applied to a telecommunications operator environment, assuming that type of cloud chosen would be a private one, handled by the operator itself. By changing to a cloud computing provider that supports the work we present, companies can move all resources to become virtual instances except the ones that must currently be kept as local hosts. Then, cloudprovided networks would be extended to merge legacy resources that were not moved to the cloud. This network extensibility ideally would only require calling special operations on the cloud computing provider and making sure there is a reachable point in the legacy network that is externally configurable/manageable in order to provide the linking point between the two. From the point of view of the company, their datacenter would be physically split, but topologically united. Figure 1 shows what can be accomplished in this use case: legacy resources, already part of a legacy network, become part of a cloud-managed virtualized network that is an extension of the legacy network, and vice versa. Centralized network control and flexibility is also very useful for enterprise IT clients. By having parts of the networking cloud spread across different physical/legacy segments (not in the cloud, as to say), while also offering the ability to reconfigure these parts via a logically centralized API with the benefits of cloud computing, network administration becomes more agile in scenarios where infrastructure keeps changing. One of the most useful realizations of centralized networking control and flexibility is the creation of a virtual campus. Besides the advantages of having the ability to change network attributes in a global manner (deployed across the whole location, company, and campus), for example, L3 addressing, central management of wireless access points (APs), either to attach them to virtualized networks or networks already accessible by Ethernet ports or even to create temporary service set identifiers (SSIDs) for specific, particular occasions is definitely a use case possible to achieve. The ability to merge different physical networks in distant physical locations into the same broadcast domain is an undertaking that benefits from being carried out in a centralized manner, ideally without manually connecting any equipment. Education is another use case benefiting from centralized network control, where the ability to set up networks for use in the classroom, especially useful for computer networking lab classes, leveraging most, if not all, of the use cases presented in [7], is a considerable benefit. Finally, easily setting up networks for guests or for research test beds, deployed on top of any connectivity interface available, just by accessing a control interface, can be made possible. This is a key principle of SDN, where networking devices are controlled by a so-called SDN controller [8], which we enhance by the fact that legacy devices can be considered too. Figure 4 illustrates this use case. Because of the overhead in operating VMs in an Infrastructure as a Service (IaaS) infrastructure (there have been improvements such as [9]), the ability to achieve heavy workloads, with high performance, becomes limited or compromised. Therefore, on scenarios where an IaaS infrastructure is in place, the need to provide bare metal machines, that is, operating systems installed on top of physical machines, has increased. Moreover, bare metal machines require network connectivity as well. Instead of implementing and deploying a specific way to reach and boot these machines via the network, this work allows to reuse the same functionality from VM networking for that end. For businesses that rely on network functions or advanced services for their networks, the ability to carry out service chaining may be useful. With this work, it is possible for existing equipment and services to be reused (which goes towards the incremental legacy to cloud migration) for service chaining purposes. Finally, and just like with telecommunications operators, a possible end scenario is the deployment of an L2 VPN. Copyright © 2016 John Wiley & Sons, Ltd.



Figure 4. Centralized network control and flexibility via unified API 2.3. For the end-user The end-user will benefit from legacy to cloud migration in the sense that some of their legacy equipments can have their life extended, eventually decreasing costs or OpEx for the user. The provisioning of L2 VPN services by the telecommunications operator or the enterprise IT business may benefit the end-user if the interfaces for this provisioning are provided to the end-users themselves. They can then leverage the flexibility and on-demand ease of these interfaces, which should have similar semantics as the remaining interfaces provided by the other cloud services. In the cases that the end-users are ISP customers, they can benefit from having additional services being provided to them, without additional hardware or visits to their homes, as the device they have at home could be remotely attached to the cloud run by the telecommunications operator, leveraging the virtual HGW use case. Certainly, other use cases can be extracted from this work, although some may not differ much in relation to the previous ones. In summary, the main points that this work leverages are as follows: (a) it allows typically virtualized networks to be extended to the outside (and vice versa); (b) it fits into a cloud computing infrastructure that provides other desirable advantages like multitenancy or different sets of advanced services; (c) it allows bringing external services to the cloud; and (d) it opens up the possibility to develop advanced features on top of it, like legacy network state monitoring.

3. INTEGRATING CLOUD AND FOG NETWORKS This solution is defined by a set of concepts that, together, are able to materialize or empower the use cases presented earlier. We name them as network segment extension, network connection awareness, and auto configuration. These concepts drive the general architecture we devised, and contribute to the flexibility and capability of our solution. Concept 1: Network segment extension At the root of our solution is the concept of an L2 network segment extension. These are L2 network segments ‘living’ in a virtualized network infrastructure, possibly managed by an IaaS provider, that can be extended beyond that basic infrastructure. The interpretation for Figure 5 is that the virtualized network segment is extended beyond the cloud by bridging it with the remote network segment. Consequently, all hosts, connected to either the virtualized or remote part of the network, will be in the same broadcast domain. Copyright © 2016 John Wiley & Sons, Ltd.



Figure 5. A virtualized network segment extends into a remote area

Figure 6. Configurations are "pushed" to the NAD so that its network segment becomes part of Neutron. Network state reports are obtained from the NAD for further processing

Concept 2: Network connection awareness It is beneficial to keep the state of the whole network up-to-date. Given that, this work is designed to be aware of all main entities logically connected to the cloud infrastructure. Specifically, it has knowledge about end devices that are externally connected, that is, L3 hosts in the remote network segment, and the devices that give connectivity to the former. Earlier, Figure 4 illustrates this transparency by what an administrator should generally be able to see and what interactions occur at a very high level. This awareness can be also augmented through the network state reporting interface that allows the report of specific events towards the IaaS provider, as can be seen in Figure 6. These reports are useful for reactive changes in the network and security policies or for other administrative purposes. Concept 3: Auto configuration This solution takes care of necessary configuration actions that need to be taken on the devices, which are responsible for fulfilling network segment extension at the remote infrastructure’s side. Figure 6 illustrates this process. Additional configurations can be made via the network state reporting interface that allows the report of specific events towards the IaaS provider. The solution is implicitly flexible because it makes available a generic interface for customizing each new network segment, depending on the technologies and configuration options available in each device. An example of this flexibility occurs when extending networks via a device that is actually a wireless AP. This solution is flexible to the point of setting up different SSIDs with different security configurations. Flexibility also means that segment extensions through the device can be part of different networks or the same one. Finally, the solution is heterogeneous because it is devised to be compatible with virtually any kind of device. The main idea is that the support for devices is driver based and pluggable, as depicted in Figure 7. Given that, there is a clear separation between the overall features and their implementation. Copyright © 2016 John Wiley & Sons, Ltd.



Figure 7. Heterogeneity attained by developing drivers for devices

Figure 8. Flexibility attained by configuring the device with any option supported by its driver The legacy devices that support foreign networks, which can be any switch, router, or other equipment with network bridging capabilities and a remote management interface accessible by IP, are called network attachment devices (NADs) and play a very important role in this work. First of all, they must be able to operate at L2, like all typical switches and also some routers that allow L2-specific configurations (e.g., managing internal switches). They also need to be remotely reachable, from where the cloud computing stack is deployed. Remote administration must be possible, by making use of communication and configuration protocols. The ability to instantiate tunnels, or otherwise another technology that allows traffic to be kept in the same broadcast domain while traversing the Internet, is necessary inside the NAD. The device must also allow the reconfiguration of internal switches, or some form of virtual switches, to manipulate what subnets are to be provided for each physical interface (Ethernet, wireless SSID, etc.), usually by the use of virtual LANs (VLANs) although not necessarily. Finally, a device must be able to associate the tunnels created to the subnets managed by the same, so internal traffic meant for the other part of the network (at the cloud infrastructure side) can be sent there (and vice versa) (Figure 8). 3.1. Architecture A cloud computing framework, able to orchestrate cloud instances and its networks, can be improved in a way that enables integration of the so-called fog networks, enhancing its instantiation by supporting the use cases discussed in the previous section. To this end, and by taking in consideration the previous concepts, a generic solution is proposed. The main objective is to provide a clear and simple method to integrate external devices into the virtualization framework, enabling direct communication between VMs and bare hosts residing in foreign network segments, supported by networking devices including legacy ones (with or without support for protocols such as OpenFlow). The solution aims to give cloud consumers the ability to extend their virtual networks out of the datacenter, and vice versa, Copyright © 2016 John Wiley & Sons, Ltd.



Figure 9. An overview of the proposed architecture

in a self-service manner. Furthermore, dedicated architectural components have the responsibility of dealing with different networking devices in an abstract and extensible manner. With this design, a cloud computing software stack can be used to control distant network segments, merging them with virtualized counterparts in an automated manner. Several root concepts drive the design pattern of this solution. Figure 9 provides an overview of the architecture, showing how the entities are organized and giving examples of the kinds of devices than can be seen at each level. The networking management side of a cloud computing framework is called a network virtualization stack (NVS). An example of an NVS is OpenStack’s Neutron, the networking API from the free and open-source software cloud computing software stack, OpenStack. The solution we propose is named external port extension (EPE) and translates as an extension to an NVS, which enables external ports (or hosts) to be brought into the cloud. NVS and EPE together form a deployable solution. All the new entities developed can be considered part of the EPE. From NVS’ sole perspective though, EPE is only an interface for extended functionality. They must be assigned to a specific driver in order to translate the operations ordered by the EPE. The drivers are managed and instantiated by a component we name as network point controller (NPC). These drivers are called external drivers (EDs) and have all necessary instructions to allow interaction with specific kinds of NADs, leveraging the communication against them to exchange state and configurations, further translating abstract operations into devices’ configurations. The NPC expects all drivers to follow the same interface so configurations and operations can be translated to each NAD. Furthermore, they must be reachable via IP to leverage network segment extension. Each NAD can provide multiple logical entities called network attachment points (NAPs). They can be interpreted as simple, logical L2 bridges, configured inside the parent NAD to bridge a set of its interfaces with a cloud network remotely reachable by the NAD. A Layer 2 Gateway [10] is a possible materialization of an NAP. This entity can be of a special type when it is also capable of reporting network state information, which can be utilized to trigger actions inside NVS or EPE. The figure also shows an NVS as an NAP provided by an NAD, which means that this work is theoretically able to bring other cloud computing platforms’ networks to a ‘master’ platform, through the intermediate NAP that can be, for instance, an agent installed on the second NVS’s server machine or a built-in feature/extension of that second cloud. Finally, network external ports (NEPs) refer to any external IP host: a server, smartphone, sensor, camera, and so on, directly connected/known to/by the NAP, which can be brought into the domain of a cloud network. NAPs are made up by a set of NEP resources. Copyright © 2016 John Wiley & Sons, Ltd.



Figure 10. An example of a Neutron EPE deployment

3.2. OpenStack as a network virtualization stack In order to validate and test our solution, a reference implementation was developed on top of OpenStack/Neutron, acting as an NVS. OpenStack is a set of services, with Neutron being the networking service, that together make up an IaaS cloud stack. OpenStack is written in Python and so was the EPE. From a deployment perspective, Figure 10 shows OpenStack, extended with an implementation of the EPE solution, and its corresponding API. A total of five attachment points are spread amongst two NADs. The first NAD is an OpenWrt-flashed HGW providing three attachment points, two of them materialized in different wireless SSIDs and the third one materialized in a specific Ethernet port. A cloud shape shows how clients of the laptops connected to the wireless AP’s SSID are also connected to the rest of the network elements (VMs) living inside OpenStack, in the same broadcast domain. Figure 10 also demonstrates a tunnel established between the Open vSwitch (OVS)’s br-int and the OpenWrt-flashed HGW NAD, fulfilling a network segment extension. Analogous clouds and tunnels can be drawn for the remaining attachment points of Figure 10 but have been omitted to keep the figure legible. Concerning the data model, resources of networks and ports were left untouched because they already exist on OpenStack, while the new models were added. About the API, the operations specified at Section 3.3 were added and the complete data model exposed via a Create, Read, Update, Delete (CRUD) interface. Full support was developed for the CLI project for Neutron, python-neutronclient, to make use of the new operations offered by the EPE. Copyright © 2016 John Wiley & Sons, Ltd.



Figure 11. Class diagram of solution and directly related classes

Figure 12. Sequence diagram when attaching a network segment 3.3. Data model and interfaces The data model for this solution, implemented on top of OpenStack as the NVS, requires the support of NADs, NAPs, NEPs, networks, and network ports (typically for VMs or other cloud-provided services). The class diagram present in Figure 11 illustrates how these entities relate to each other. The word ‘Network’ is omitted in the diagram, for the three new resources introduced. The first new data entity developed to support this work is the NAD. The data model for this entity is presented in Table 1. The ip_address attribute is used to reach the device, while driver is used to instantiate the proper software component to interact and configure the device. The NAP data entity is presented in Table 2. The identifier attribute is used to pass configuration options to the driver responsible for the NAD, free to be used by the cloud consumer. The technology field guarantees that both the device and the cloud computing framework agree in the technology used for extending the network, for example, Generic Routing Encapsulation (GRE) or Virtual Extensible LAN (VXLAN). Finally, the NEP data entity is in Table 3 and maps external hosts to the cloud-managed network. The other entities of networks and ports are supposed to be part of the NVS, where ports are anything that can be connected to a cloud network: a VM or another network service. Copyright © 2016 John Wiley & Sons, Ltd.



Table 1. Data model for network attachment devices Name

Type

Access

ip_address driver

string string

RW, all RW, all

Default

Validation ip_address string

Table 2. Data model for network attachment points Name

Type

Access

device_id identifier technology network_id index report_point

string string string string int string

RW, all RW, all RW, all RW, all RO, all RW, all

Default

Validation

generated True

N/A string string uuid_or_none int convert_to _boolean

Table 3. Data model for network external ports Name

Type

Access

Default

Validation

mac_address attachment _point_id port_id

string string string

RW, all RW, all RW, all

uuid_or_none uuid_or_none

mac_address string string

This solution inserts new operations in the NVS, meant for cloud tenants or administrators. The first one is attaching an NAP to a network, where a request for extending a network segment is issued (a Create or Update operation on an NAP resource, by specifying the Neutron’s network_id to attach to). The NVS (OpenStack) propagates the new request down the architecture, reaching NPC, where it will instantiate the desired ED and (re)configure an NAD. Figure 12 presents a high-level sequence diagram highlighting the important steps when attaching a segment. Detaching an NAP is the opposite operation, but following the same path in the architecture. The rest are simple CRUD operations against NADs, NAPs, and NEPs. It must also be noted that some NAPs can act as reporting points, meaning that, when attached, NPC calls a special monitor operation on the ED to carry out network state reporting. To keep this solution open and able to be further integrated, without breaking any existing functionality provided by the NVS, existing interfaces were not changed, only new ones explicitly created. In other words, Neutron’s upstream API was left untouched, specifically Neutron ports and networks, because those are the resources that have any relationship with the new resources herein proposed. Consumers will make use of the new features only if they intend to. The programmatic interface acts as an API for tenants or administrators. The interface specified between NVS and EPE mirrors the operations that consumers can request via the programmatic EPE’s API. That is, every operation that can be requested by a consumer via the EPE has a matching implementation inside NVS. These API operations consist on the CRUD combinations of the resources presented in Tables 1–3. Base interface of any external driver: Although more details about the EDs are provided in the next section, it is worth mentioning how they are structured and how they interface with the NVS. Figure 13 presents the ED’s model and associated methods that interface with the NVS and initiate operations against both the NVS infrastructure and the distant device. All setting up is located inside Python’s default object initialization method __init()__. Methods in bold represent the ones that were implemented in each of the classes. Copyright © 2016 John Wiley & Sons, Ltd.



Figure 13. ED model and interfacing methods When the API consumer, after having created all necessary resources, updates the NAP to become associated with a Neutron network, a new instance of the respective driver associated is created, and its attach() method is called. Conversely, detach() is called when removing the network from the NAP, reusing the same driver instance. The reason why there are driver instances is to potentially monitor configurations, keeping the attachment active, or to keep running the explicit monitor() method to provide reports back to the NVS. The private method _connect is also shown because it is dedicated to fulfill the connection to the remote device, where credentials are exchanged. In terms of the arguments expected by each of the methods:

nvs_addr is the IP address of the NVS; nad_addr is the IP address of the NAD; identifier comes directly from the NAP’s identifier field; technology comes directly from the NAP’s technology field; index is a unique number to segregate between different NAPs provided by the same NAD (e.g. might be rendered as GRE tunnel key or a VXLAN Network Identifier); ftext text to log at the NVS and additional arguments via args params parameters for the monitor() method; usr username used to connect to the NAD; pwd password used to connect to the NAD;

3.4. External drivers Support for heterogeneous network equipment is achieved by the use of EDs. These drivers are used by the EPE to interface with the respective NADs and configure them according to what networks are attached to which attachment points. The external agent (shown in Figure 10) is the entity that instantiates and controls these drivers, serving mainly as a mediator between what is defined in Neutron’s database and the implementation of these definitions by the NADs. Two working drivers are presented in this work: OpenWrt and Cisco EtherSwitch. Only the GRE tunneling technology is used, for both drivers. The first driver interfaces with typical switches or routers running OpenWrt.1 Connections made by the driver can be through either ssh or telnet. Regarding Cisco EtherSwitch IOS driver, it interfaces with the Cisco EtherSwitch line of switching modules. Connections are made through telnet.

1

https:// openwrt.org.




Table 4. OpenWrt driver identifier syntax Key iface usr pwd ssid ssid_pass eth_ports

Description Access interface Login username Login password SSID to instantiate Password for SSID Ethernet ports to bridge

Example value ssh root secretpass Staff 65846531 1,3,4

SSID, service set identifier.

OpenWrt This driver (named openwrt) interfaces with typical switches or routers running the latest stable version of OpenWrt at the time, 14.07 Barrier Breaker. Compatibility with this driver must be checked for each device, because OpenWrt is an operating system that runs on a wide variety of switches and routers. A NETGEAR WNDR3700v1 HGW was used for all debugging and testing regarding this driver. Connections made by the driver to the managed NADs can be through either ssh or telnet, depending on the identifier’s iface attribute. The syntax of the identifier field for the OpenWrt driver is a simple key-value list of configurations: =;=;.... Table 4 presents what keys were developed and gives example values: Possible sources of concern when looking at the keys defined in Table 4 are the pwd and ssid_pass fields, meant to define the password to access the remote device’s shell interface and to define the wireless password for the SSID being defined, respectively. They are stated in clear text because security was out of scope when developing this work. Security can be enhanced by supporting public key authentication for the remote shell access (when using Secure Shell (SSH)), assuming that the remote device has been pre-provisioned accordingly, while for the SSID password would require other methods (dependent on different drivers). The objective is to have maximum control over an unaware legacy networking device, and consequently, limitations will vary per device. Internally, the OpenWrt ED makes use of the UCI system [11] to inject network configurations into the NAD. These configurations primarily create new VLANs, reconfiguring the integrated switch and wireless radios to assign ports and SSIDs to these VLANs. Besides UCI configurations, new Linux interfaces of type gretap (packets become automatically encapsulated in GRE) are always created and connected to virtual Linux bridges (not the integrated switch) meant for specific VLANs. Furthermore, traffic coming from a specific VLAN will always arrive at the correct Linux bridge, via integrated switch’s ports, radio interfaces, or the gretap interface, and, afterwards, exit through another Linux bridge port (per the forwarding table). Devices using the OpenWrt driver must have a reachable IP address from within Neutron. Remote administration is possible thanks to SSH or telnet for obtaining access to the devices’ CLIs. The Python module used for SSH was paramiko2 and for telnet was telnetlib3 . GRE tunnels are possible to be established thanks to the gre kernel module available in OpenWrt. These devices allow reconfiguring the integrated switch for assigning VLANs with any combination of physical interfaces (Ethernet, wireless SSIDs, etc.) and associate GRE tunnels to the internal VLANs created. Cisco EtherSwitch IOS The Cisco EtherSwitch IOS driver (named etherswitch) interfaces with the Cisco EtherSwitch line of switching modules compatible with some Cisco equipment running IOS. Connections established by this driver to reconfigure the switch are made through telnet only. A Cisco C3640 with an EtherSwitch module (NM-16ESW) was used for all debugging and testing regarding this driver. The syntax of the identifier field for the Cisco EtherSwitch IOS driver is a simple key-value list of

2 3

http:// www.paramiko.org. https:// docs.python.org/ 2/ library/ telnetlib.html.




Table 5. Cisco EtherSwitch IOS Driver identifier syntax Key usr pwd eth_ports

Description

Example value

Login username Login password Ethernet ports to bridge

root secretpass fe0/0,fe0/1,fe0/2

Figure 14. Representative diagram of the test scenario configurations: =;=;.... Table 5 presents what keys were developed and example values: Internally, VLANs are created and then associated to a bridge group [12]. Besides that, GRE tunnels are created and associated to the bridge group as well. Matching with the NAD requirements stated in Section 3, devices using the Cisco EtherSwitch IOS driver in fact have a reachable IP address from within Neutron. Remote administration is possible thanks to telnet for obtaining access to the devices’ IOS CLIs (again making use of the telnetlib Python module). GRE tunnels are possible to be established because built-in support in the Cisco EtherSwitch modules is available [13]. These devices allow reconfiguring the integrated switch for assigning VLANs and any combination of Ethernet ports by making use of bridge groups. Finally, GRE tunnels can be associated to the bridge groups. 4. EVALUATION AND ANALYSIS We proceeded to evaluate our solution by carrying out tests to measure different characteristics of the implementation, considering its deployment on a simplified real-world scenario. We consider several hosts, connected to the Neutron managed networks as NEPs, having access to the external network like any other host, such as a Nova VM instance. In our scenario, there are no routers between the server running OpenStack and the NAD that provides NAPs, to focus on measurements that are, as much as possible, part of the network segment extension and not intermediate equipment’s bottlenecks. Figure 14 shows this test scenario where the OpenStack server is directly connected to the NAD that will provide NAPs. It must be noted that, for each physical machine and VM that form the test scenario, a specific amount of main memory is reserved, and there is no overcommit, so that memory swapping does not occur. A Neutron network is already provisioned inside the OpenStack node presented in Figure 14, with two Nova VM instances running – this virtualized network will be extended through an NAP. At the right-hand side, an OpenWrt switch/router is depicted – this is the NAD that will fulfill the attachment. The NAP to be attached/detached through the NAD will be present in VLAN 2, at port 4. The Copyright © 2016 John Wiley & Sons, Ltd.



server running OpenStack, the NVS, has an Intel Core i5-2450M central processing unit (CPU). Main memory totals 8 GiB at 1333 MHz (dual channel). The operating system in use is Arch Linux running on top of the Linux kernel 3.17.1 (64 bit). Finally, it uses VirtualBox 4.3.18 to run an Ubuntu VM, on top of which DevStack4 is installed. This VM is set to acquire network connectivity by bridging against the host’s network interface controller (NIC). All code developed in this work has been applied on top of Neutron’s stable/icehouse git branch as of mid-October 2014 (commit ID 1d4a3e33c169ea3072f205cb81386a6624cf37fe), which is the basis for all tests. The VM running DevStack is an Ubuntu 14.04.1 LTS on top of Ubuntu’s linux-generic kernel 3.13.0-37 (64 b). One CPU core is assigned to this VM, and a total of 5 GiB of main memory is allocated to it. Both Intel VT-x and Intel EPT hardware virtualization technologies,5 compatible with the host’s CPU, are enabled under VirtualBox. The VM instances running in Nova are Ubuntu 14.04.1 LTS images taken from Ubuntu Cloud Images, daily build 20141016 – these are deployed inside the virtual network that will be extended via an NAP. Depending on the test, either one or two of these machines were provisioned under Nova. Each VM has a total of 512 MiB of main memory allocated to it, and the kernel is Ubuntu’s linux-generic 3.13.0-37 (32 b). The virtual NIC driver in use by the instance is the virtio_net for improved network performance in virtualized guests/instances. Because these instances are already provisioned inside a virtualized environment, the default hypervisor (KVM6 ) does not make use of hardware acceleration, instead defaulting to Tiny Code Generator, that is, no nested virtualization. The physical NAD is a NETGEAR WNDR3700v1 having 64 MiB of random access memory. Barrier Breaker 14.07 is the OpenWrt version, which uses the Linux kernel 3.10.49. The remote computer is a traditional PC with the latest Ubuntu 14.04.1 LTS, linux-generic 3.13.0-37 (32 b), installed, and all packages are updated to the latest version as of 20 October 2014 – this is the EPE that will be inserted to the Neutron network through the physical segment attached (the NAP). The CPU is an Intel Celeron D with a clock speed of 2.66 GHz. Main memory totals 960 MiB at 400 MHz. 4.1. Attach and detach latency The first aspect to test is how long administrators/tenants need to wait before their orders for attaching or detaching NAPs become effective. Two requests have been measured: attach and detach. In order to faithfully measure these times, with a guaranteed upper bound of precision, a timer is started in an additional computer directly connected to the device at the exact moment the API request to attach a network is sent to Neutron (an Update request to the NAP resource, associating it to a Neutron network by setting the network_id field). This timer has been programmed to keep sending network pings to Neutron’s virtual router IP address in the network to be attached. Pings are sent every 100 ms, and as soon as a reply is obtained, the timer is stopped, and the total latency is obtained. Thus, the upper bound in precision when measuring the setup latency is 100 ms. Similarly, for the detach latency, the timer stops as soon as pings stop being received. Both tests of attach and detach latency have been repeated a total of 10 times each. Table 6 presents average and standard deviance of the attach and detach latency tests. It must be remembered that the measuring precision has an upper bound of 100 ms. Another test was carried out, measuring the attach time solely at the cloud side, which has no such limit on the bound of precision. In order to evaluate the scalability of this work in what concerns simultaneous attachments, a final test was made, evaluating the amount of time to attach a set of 1 to 50 NAPs in parallel. This test was also iterated 10 times, for each cardinality of simultaneous NAPs. Attach times have been observed to be around the 9-s mark, spread between establishing an SSH session with the OpenWrt device, executing specific commands and then committing all changes to the device. In a production scenario where CPU usage may reach higher levels and network may get congested, the attach time will certainly increase. However, the operations carried out are small,

4

http:// devstack.org/ . http:// ark.intel.com/ Products/ VirtualizationTechnology. 6 http:// www.linux-kvm.org. 5




Table 6. All test results acquired Test Attach time (s) Detach time (s) Attach time (cloud side only) (s) Local latency (between VMs) (ms) Remote latency (between NEP and VM) (ms) Latency between host and VM (ms) Latency between host and namespace (ms) Latency between namespace and VM (ms) Latency between NAP and NEP or host (ms) Total estimated latency between NEP and VM (ms) Local TCP traffic throughput (Mbps) Local UDP traffic throughput (Mbps) Remote upstream TCP traffic throughput (Mbps) Remote upstream UDP traffic throughput (Mbps) Remote downstream TCP traffic throughput (Mbps) Remote downstream UDP traffic throughput (Mbps)

Average 8.80 3.97 0.369 1.10 1.42 0.59 0.12 0.47 0.41 1.41 290.1 26.7 90.0 58.9 77.5 63.1

Std. Dev. 0.25 0.23 0.112 0.34 0.24 0.14 0.01 0.14 0.05 0.16 20.0 1.6 1.0 4.6 2.3 3.3

VM, virtual machine; NEP, network external port; NAP, network attachment point; UDP, User Datagram Protocol.

Figure 15. Distribution of attach time in local cloud and remote device

and the network traffic exchanged with the device is low, so these measurements are not likely to increase greatly. Waiting 9 s, or even 10 times that, to insert a new, external network segment to a virtual network managed by OpenStack is a significant achievement when looking at the workarounds that usually need to be made. For instance, and considering that this work is best suited for existing network equipment, usually without support for SDN protocols such as OpenFlow, rearranging networks so that a special host becomes available in a specific network will require manual configuration in each of the equipment involved, which, because of human nature, can easily take from some minutes to hours if configuration cannot be made to match properly during the first attempts, or other operational constraints. Because the attach time is highly dependent on the driver and device used, the additional test of measuring how long the OpenStack side managed to have all its resources ready, including the virtual switch configured for the new NAP, was performed. Table 6 contains the results for this test, attach time (cloud only), and was iterated 100 times. Figure 15 shows the same results, but comparing with the total time it takes to configure both the cloud and the device. As can be observed, configuring the remote device takes significantly longer than configuring the local cloud, but the amount of time is specific to the device used. Waiting less than 1 s between the API request for attaching an NAP and the switch becoming configured is an excellent value and would only be more significant if dozens of NAPs were attached/detached every second. About the detach time, it is around the 4-s mark, less then half of attach’s time. What has been said for the attach time also applies in this case. Much like attach time, having an ‘instant’ detach time would be better, although it still is better than traditional solutions of real-world deployments (mainly Copyright © 2016 John Wiley & Sons, Ltd.



Figure 16. Scalability of attaching multiple NAPs in parallel because of human-related reasons), save for critical networks with critical components. A specific downside of having a non-instant detach time is that, if a network administrator or monitoring software finds that a tenant is exploiting the infrastructure for illegal purposes by making use of this extension, it cannot instantly block it just by using the public API methods available. Still, it is better than typical solutions, especially considering the heterogeneity support of the work (which influences, in a negative or positive way, attach and detach times). The scalability test, whose results are in Figure 16, makes the relationship between number of NAPs (xx axis) and amount of time until the last NAP got connected to OVS (yy axis). The metrics start at the time the first request is issued and end after the last request has been processed by the API, with all requests being sent in parallel (from the machine that runs OpenStack). The relationship, at least up to 50 NAPs, grows in an approximately linear way, meaning that attaching 50 NAPs is achievable with typical personal computer hardware. We do not expect that attaching such an amount of NAPs will be a typical scenario, although it makes sense during the cloud computing infrastructure initialization, where many NAPs may need to be brought to the cloud at the same time, for the first time. Another point about these results is, when attaching at least 41 NAPs simultaneously (or 35 taking into account the standard deviance), that the time it takes to prepare the cloud-side network is likely to be higher than the time it takes to prepare the remote device (assuming the same device hardware and driver we have tested). 4.2. Traffic latency Traffic latency can be of two types: local and remote. Local latency measures the latency between two VMs hosted by Nova in a standard out-of-the-box DevStack deployment with Neutron. The relevancy of this test lies in the overall comparison of latency between different VMs hosted in a single node and latency between one of these VMs and an NEP physically distant from the Nova node. Latency is measured by executing a series of pings (Internet Control Message Protocol (ICMP) Echo Request/Response) between one of the VMs and the other. Ping properties are set up as the default ones used by the iputils’ ping utility,7 version 20121221, available in the Ubuntu VMs used. These properties consist in a 1-s delay between each ping, with each ICMP packet having a total of 48 B of ICMP data. This translates to a total of 102 B, including ICMP, IP, and Ethernet headers, plus the frame check sequence (FCS). It must be noted, though, that for packets traversing the tunnel that leads to/from attachment points, an additional 42 B of overhead is inserted (totalizing 144 B). Remote latency measures the latency between a VM host by Nova and a physical machine provided as an NEP. Carrying out this test enables comparisons with other kinds of deployments that do not provision NEPs. Namely, it is compared with the local latency test where two VMs in the same Nova node have their

7

http:// www.skbuff.net/ iputils/ .




Figure 17. Distribution of latency amongst segments of the basic path latency measured. The procedure for testing the latency is similar in both tests, save for the fact that the machines are in physically disparate locations in the remote latency scenario. An extra test carried out in the context of local latency is measuring how long it takes for the host running OpenStack to reach one of its Nova VMs. It also consists in exchanging a series of pings, as for the previous case. Both tests of local, remote latency and from OpenStack itself to one of its Nova VMs, have been carried out by sending a total of 100 pings between hosts. Local latency tests are hereafter named ‘VM–VM’ based on the fact that communication takes place between Nova VMs, bidirectionally. Similarly, remote latency tests are named ‘VM–PM’. Table 6 presents average and standard deviance variables of the local and remote latencies’ tests. In addition, extra tests were carried out to measure how long it takes for the OpenStack host to reach a virtual router’s Linux namespace, one of its Nova VMs, and how long it takes between an NEP and the NAD as well as between the NAD and the OpenStack host. The reason for measuring the time to reach the router’s namespace is to have an estimate for the time it takes to reach something connected to OVS but not a VM or Tap device. All results are presented in Table 6, prefixed by (Extra). Figure 17 shows the latency distribution of these segments. Table 6 also states the average and standard deviances. An important observation to notice in these tests is how the average latency changes when moving from a VM scenario to a VM to NEP scenario. The average of the former is 1.10, while the latter is 1.42 ms, an increase of approximately 29%. The first result is not influenced by the work proposed as there are no dependences between them. Given that both VMs are run by the same Compute/Nova node and the NEP computer is physically located outside this node and its network (and behind an OpenWrt Ethernet switch), this delay increase is not very significant and is within expectations for such a network deployment scenario. 4.3. Traffic throughput Traffic throughput tests aim to measure the overall throughput between different combinations of the kinds of nodes involved: Nova instances and NEPs. Unless otherwise stated, the User Datagram Protocol (UDP) buffer size is 160 KB, UDP datagram size is 1470 KB, TCP client’s window size is 48.3 KB, and TCP server’s window size is 85.3 KB. Local traffic throughput measures the average throughput between two VMs hosted by Nova. This test is analogous to the local latency test, except that it is meant to measure throughput instead of latency. Besides, it is decoupled in two different sub-tests: one for TCP and another for UDP. Analogously, there is a remote traffic throughput test. For the testing procedure, iperf8 is used. One of the instances is set listening for incoming connections. At the other VM, iperf is set as a client to send traffic to the first instance. In the case of UDP, the following command is used: iperf -u -c 10.0.0.5 -b 1000M. The remote traffic throughput test is actually undertaken twice because the hosts are not twins anymore: one having

8

https:// iperf.fr/ .




iperf listening at the Nova VM and the other at the computer hosted as an NEP. This way, throughput can be analyzed both in an upstream manner and a downstream manner. The UDP iperf command when VMs act as clients is different: iperf -u -c 10.0.0.5 -b 1000M -l 1430, where -l 1430 specifies a custom datagram size. The reason for adding a non-default datagram size is that OVS would otherwise drop the datagrams instead of fragmenting them to fit the GRE tunnel. Both tests of local, remote downstream and remote upstream traffic throughput have been iterated 20 times. Local traffic throughput tests are hereafter named ‘VM ! VM’ based on the fact that traffic is sent from one Nova VM to another one, which one is not important given their twin nature and network symmetry. Table 6 presents average and standard deviance variables of TCP and UDP’s local, remote downstream and remote upstream throughput tests. Local and remote traffic throughput are split in six different tests, aiming to measure throughputs in the three possible scenarios when there are two VMs and one NEP deployed, twofold: one for UDP and one for TCP. The results in Table 6 present an interesting finding: throughput between VMs is the highest for TCP but, conversely, the lowest for UDP. The cause for this lies in the fact that, for traffic between VMs, there is a higher processing load. Because there are two VMs involved instead of just one as in the remote traffic throughput tests, this load only becomes apparent for the local traffic throughput tests. They both run inside the same compute node, which stresses CPU usage and traverses the virtual network stack twice. Consistent with this reasoning is the throughput obtained for the remote traffic throughput tests, where only one of the VMs is involved, and is roughly two times the throughput of the test between VMs. The throughput between VMs for UDP is the lowest because there is a higher processing load. The actual reason for UDP being this heavy when packets are being sent/received between VMs, to the point it becomes a bottleneck, can be attributed to iperf’s default UDP buffer size of 160 KB, which fills up very quickly and stresses the machine. Both TCP and UDP throughput measurements were not very distant between each remote direction tested. In the case of TCP, the upstream direction is, on average, 16.1% faster than the downstream direction. For UDP, the downstream directions wins by 7.1%. This discrepancy can be attributed to some security rules imposed by Neutron to the traffic, as well as other non-symmetric mechanisms of sending and delivering traffic from/to Nova instances. All tests have shown sustained network speed, with quite low standard deviances. Although standard deviance is not high for the TCP test between VMs and the UDP throughput test from the physical machine (NEP) to the VM, sustained speed for these specific cases demonstrated to be more volatile. What these results mean for the deployability of the work is that throughput will be inferior than having only VMs inside one same machine. For datacenters with multiple compute nodes, there is likewise an impact on the throughput between VMs there provisioned. In addition, these VMs are usually interconnected (if part of the same virtualized network) via a tunneling technology, for instance, VXLAN. In other words, the use of tunnels does not negatively impact network performance/throughput more than it does today. The NAD being used is the one that may instill limitations, because of processing overhead and available internal bandwidth, but that is dependent on the hardware used. 4.4. Traffic overhead Traffic overhead is an implicit test. Although it is important to (explicitly) measure traffic overhead in this implementation, the only driver tested relies on the GRE protocol to achieve network segment extension. As such, traffic overhead (for the total packet size) is always fixed. The absolute traffic overhead per packet is the sum of Ethernet (14 B), IP (20 B), and GRE (8 B) headers, which total 42 B. In a typical network deployment with NIC interfaces set to use a maximum transmission unit (MTU) of 1500 B, the actual amount of data that can be transferred in a single packet (including transport layer headers) is 1480 B (when using the IP protocol because its header occupies 20 B from the MTU). In the same conditions, the implementation proposed in this work allows for an actual amount of data of 1438 B to be transferred in a single packet (1480 42 D 1438 B). Therefore, traffic overhead is approximately 1480=1438 D 2:92% for full-size packets, which results in the same increase of packet count when transferring sequential data. However, for traffic that consists in smaller packets, the traffic overhead proportion is greater. Sinha et al. provide a technical report on the distribution of packet sizes on the Internet in October 2005 [14]. Copyright © 2016 John Wiley & Sons, Ltd.



Table 7. Examples of packets and associated GRE overhead Packet description

Size (B)

Overhead (%)

Sequential transfer 1500-B packet 1300-B packet DHCP discover ICMP echo request TCP Ack IPv6 DNS query TCP Ack IPv4 40-B packet

1500 1500 1300 328 84 72 60 52 40

2.92 4.95 3.19 12.14 41.18 46.67 53.85 60.00 72.41

DHCP, Dynamic Host Configuration Protocol; ICMP, Internet Control Message Protocol; DNS,.

Figure 18. Packet overhead per individual packet size, using GRE They observed a strong mode of 1300-B packets (L2 data length) in some cases. Besides, packet sizes seemed to follow a mostly bimodal distribution for 40-B packets and 1500-B packets (at 40% and 20% of packets, respectively). Even though the abundance of 1500-B packets as reported by [14] is very likely to be tied with sequential data transfers, it is still interesting to analyze the overhead of individual packets with this size. It may seem that the packet overhead is very similar to the one obtained for traffic overhead in sequential data transfer (because the whole MTU is used), but what actually happens is that a second packet must be sent to transfer the whole 1500 B. Ignoring the processing overhead implied by a second packet being necessary and focusing only on total packet size overhead, the first packet is able to transmit 1458 B of IP data, and the second transmits the rest (42 B). It then results in a total overhead of .1518C18C42/=1518 D 3:95%. The other two numbers are important to analyze because of their frequent activity on the Internet. The packet overhead proportion for packets with a size of 1300 B (1280 B of IP data, 1318 B with FCS, and all headers included) would translate in 1360=1318 D 3:19%. For 40-B-packet instance, the overhead is 100=58 D 72:41%. Table 7 summarizes traffic/packet overheads for the cases described as well as for other typical packet kinds, with Ethernet data sizes specified. Moreover, Figure 18 presents a graph with the packet overhead per individual packet size (defined for Ethernet data lengths between 1 and 1458 B) as given by the GRE overhead formula present in equation (1), where n is the Ethernet data length, in bytes. The reasoning behind this equation is the ratio of a packet with GRE overhead against a normal packet, minus the Copyright © 2016 John Wiley & Sons, Ltd.



packet itself (100%). A normal packet has 18 more bytes because of the Ethernet and FCS, while the same packet via GRE has an additional 42 B of overhead, totaling 60 B. f .n/ D

n C 60 1 ; n 2 Œ1; 1458: n C 18

(1)

Usually, packet overhead is fixed because of the GRE tunneling technology used, which imposes an additional 42 B of headers. However, the proportion of overhead varies for each packet size, which results in a variable traffic overhead when multiple packets are being sent through the network. It has been shown that the overhead beyond L3, that is, IP data length, is approximately 2.92% for a typical network deployment relying on an MTU of 1500. This means that, for a sequential file transfer over the network, approximately 2.92% more packets will be sent (and probably acknowledged, depending on the transport protocol). Besides the total increase in traffic, CPU, networking hardware applicationspecific integrated circuits, and main memory usage at the end points and intervening nodes, such as switches and routers, will increase, resulting in lower overall performance. However, this increase is (arguably) negligible, especially when taking into account the rest of the benefits and potential benefits of this solution. Another important point is that packets with a size within the higher 42-b boundary of the MTU require a second packet to send the rest of the data (assuming fragmentation like this is not an issue, which most likely is thus requiring NEPs to customize their MTU to 1458 B either manually or via the Neutron DHCP server). This increases both traffic overhead and processing overhead (highly dependent on hardware and operating system). Additional latency may be imposed on the additional packet because it may not be sent exactly after the first, depending on network configuration and congestion status. Furthermore, for the remaining packet sizes, it can be seen that as the size decreases, the proportional overhead increases. Even though packet overhead may be too high for some packets, proportionally, it does not invalidate the deployability of this work with the GRE technology, unless in very specific scenarios with tight requirements. For large packets, they are usually followed by more packets of the same kind, mainly for stream transfers, keeping overhead safely below 10%. For medium packets, overhead will be more significant but still usually below 10%. For small packets, where packet overhead (and traffic overhead for that matter, as multiple packets may be transferred in a short time span) is proportionally more significant, it may not be a concern anyway because the packets are small. In other words, small packets with the GRE overhead are still small packets (only slightly larger) so they will most likely not disrupt packet latencies and consume all of the link’s capacity. However, one scenario where the high overhead of small packets is noticeable and undesirable occurs when they are very frequent and the dominating kind of packets, contributing to the exhaustion of the link’s capacity, effectively contributing to a steep decrease in traffic throughput in comparison with a traditional link. For exactly the same conditions, traffic without the GRE overhead would be quicker and lighter. Nevertheless, this is a very specific scenario, and the GRE overhead is compatible with the majority of scenarios and use cases, because it is used in multiple enterprise and cloud deployments, with the same applying to VXLAN. Plus, the advances in networking equipment and computers have led to processing power and bandwidth higher than ever, cheaper than ever. 5. STATE OF THE ART This document is an extension of a previous paper, ‘Seamless integration of cloud and Fog networks’ [15], published and presented in the first IEEE Conference on Network Softwarization (IEEE NetSoft) in April 2015. Newly added materials include additional material for clearer interpretation in Section (2) as well as a new view on these for telecommunications operators, enterprise IT, and end-users. The solution’s introduction (3) was extended, a new Section 3.4 diving into greater detail regarding the OpenStack solution, EDs how the interaction between components work, and so on, was added, and lengthier explanations and discussion on the test cases/results (4) as well as more figures to support it. Regarding the test cases/results, more tests have been carried out, evaluating the scalability of the work and measuring more specific attributes. Finally, other corrections and improvements were made throughout the text. Copyright © 2016 John Wiley & Sons, Ltd.



The work proposed is integrated in current developments related to the integration of the so-called fog networks. That is, the networking environment that surrounds a datacenter but is not directly managed by the cloud orchestration tool. Several other authors also aimed at the same, or similar, objectives, from which we describe the most relevant. Bruce Davie presents [16,17] a way for extending virtual networks from the datacenter to the outside, spanning both virtual and physical resources, while leveraging logically centralized management and control. The simplest use case achievable with the proposed implementation consists of extending network segments (L2) that interconnect existing VM instances deployed, towards the physical network that encompasses physical machines. The authors have gone even further and implemented services in higher layers, namely, distributed logical routing [17] (L3). Farias et al. propose [18], a way to use existing legacy infrastructures simultaneously with new/experimental/future network architectures, software, and protocols, by making use of OpenFlow [19]. The solution presented intends to keep the legacy part of the network intact, while enabling new network functionality and concepts through modern technologies, making it possible for entities to incrementally migrate to new technologies and test new approaches without sacrificing current infrastructure stability. Chan and Martin propose [7] a way to leverage a virtualized networking environment on top of a physical, legacy network, with the objective of optimizing computer networking lab classes in multiple fronts. This solution is an example of how to improve networking lab classes in terms of flexibility, technologies offered, capital expenses, and OpEx. In his master’s dissertation, Filipe Manco envisioned an advanced manner of virtualizing an entire legacy network infrastructure and attaching it to a cloud computing provider [20]. He mapped physical network elements to logical entities in order to create a network overlay on top of the legacy network. This design effectively addresses virtualization of legacy networks composed by heterogeneous networking elements and technologies, enabling new L2 networks on top of existing datacenter resources. Cloud computing integration was also taken into account, having OpenStack been chosen as the proposed software stack. Implementation-related details can be found in OpenStack blueprints [21,22]. A blueprint was submitted to OpenStack to enable what the authors call ‘external attachment points’ [23]. The root motivation for this work lies in the fact that there is no well-defined way to connect devices not managed by OpenStack directly into a Neutron network. The main use cases presented that justify this undertaking are the ability to create L2 gateways that make it possible to extend existing broadcast domains of Neutron networks, via existing datacenter switches. Returning to our work and how it compares with each element of the state of the art described, there are a few characteristics that justify the development of this new endeavor. Despite the qualities of Bruce Davie’s solution, it has a few shortcomings. First, it has no integration with an existing cloud computing network and compute infrastructure stacks. Second, it requires specialized hardware at the physical side of the network. Specifically, it requires switches that support the modern OVS Database Management Protocol, a fact that precludes integration with most legacy devices. Our work seamlessly integrates with Neutron, keeping all remaining services untouched and usable. Furthermore, it virtually supports any kind of switching hardware, at the distance of a pluggable driver, a seducing fact for the support of legacy hardware and distant fog resources. Farias et al. work presents a fresh solution for legacy networks. However, it aims at infrastructures where modern networking equipment is already in place. If we take as an example a company having a legacy network infrastructure, the solution is unable convert or integrate that network. Also, cloud computing integration has not been addressed or discussed, making this work difficult to assess, as a service. Our work integrates with cloud computing and does not require anything special or modern outside of the infrastructure dedicated to the cloud because it individually brings external (legacy) points to that infrastructure. Chan and Martin work about improving lab classes could have had integration with cloud computing solutions in order to provide cloud features. Another disadvantage is that this solution has a limited view of the network, that is, from the virtual server towards the physical devices. Also, logically centralized control of physical devices is not really addressed or discussed. Filipe Manco’s dissertation presents an ambitious way of dealing with the heterogeneous and legacy world but to the point that the architecture may become administratively complex, which may increase the complexity of problem tracing and performance evaluation, inverting the lower OpEx expectations. In our work, there was an attempt at finding a midpoint, enough that the physical/legacy network could be brought under the umbrella




of a cloud computing infrastructure. Also, no final implementation was developed and tested to ascertain the feasibility of this design. Finally, the OpenStack blueprint contribution has limitations in the following aspects: it limits itself to attachment points created by administrators, which increases the complexity of use cases related to legacy to cloud, especially for public clouds; it does not predict how distantly can network attachments function properly; and even though it has a high degree of heterogeneity by allowing the use of switches’ VLANs or OVS gateways, it still is not heterogeneous enough for clients that make extensive use of legacy networking equipment that require specific methods of configuration, likely in an interactive fashion. Our work aims at embracing anything that can extend the virtual networks, be it Top-of-Rack switches, HGW, or any other device with switching/tunneling capabilities along with a driver to support it. 6. CONCLUSION Starting with the premise that cloud computing has become, and still becoming, more popular, an important disadvantage of the concept was identified: the necessity for homogeneous resources. Therefore, we have made progress on turning existing heterogeneous networks into cloud-managed ones, with the ability to be transparently controlled and administrated in a centralized and cloud-like manner. With this work, we presented a novel way of combining the advantages of modern cloud computing software stacks, with the benefits of having an heterogeneous, bare-metal, or eventually legacy networks deployed. Given the ability of cloud computing to provide network connectivity as a service to tenants, coupled with inherent properties like flexibility and on-demand consumption, we were able to make both parts integrate and inter-operate in a seamless way, with control and administration in a logically centralized manner. Results have shown that these advantages are not eclipsed by major problems, again empowering the feasibility of the work. Although the proposed implementation can already be deployed, multiple paths of future work can be traced that further enrich and strengthen this solution. It can be extended in the future with new features and capabilities for notable scenarios, with specific requirements that may present new challenges and define a new generation in state of the art. The first interesting addition to this work is to implement a robust manner of detecting new NEPs, via an attachment driver (whose architecture already expects this functionality), for instance, by recurring to Link Layer Discovery Protocol (described in [24]) or Simple Network Management Protocol (described in RFC 1067 [25]). Alternatively, detection could be made by developing an internal mechanism that analyses traffic and detects when new hosts have joined, for instance, by recurring to OVS Database Management Protocol (described in RFC 7047 [26]) notifications and IP Flow Information Export (described in RFC 5101 [27]) notifications or by reacting to client DHCP. Another interesting addition to this work is inherent support for high availability, which proves especially helpful for telecommunications operators, but not limited to them. Significant robustness and security improvements (not just by relying on ED’s capabilities) are amongst other desirable characteristics not explored in this work, which may be a strict requirement for critical deployments as well as for telecommunications operators. Another interesting future work is the ability to integrate it with an orchestration layer, such as OpenStack’s Heat, to accelerate and streamline multi-cloud deployments that make use of hybrid networks based on this work. Naturally, there are other projects that might be interesting to further integrate with this work, materializing the use cases presented. An example is service chaining with NEPs, which might be implemented by integrating the traffic steering work presented in [6]. Finally, the ability to create external ports in a cross-cloud manner, that is, merging network segments from disparate mutual cloud computing deployments, is an interesting use case currently not achievable by this work.

REFERENCES 1. Vouk MA. Cloud computing issues, research and implementations. In ITI 2008 - 30th International Conference on Information Technology Interfaces, IEEE, Dubrovnik, 2008; 31–40. DOI: 10.1109/ITI.2008.4588381. 2. Ruckert J, Bifulco R, Rizwan-Ul-Haq M, Kolbe HJ, Hausheer D. Flexible traffic management in broadband access networks using software defined networking. In 2014 IEEE Network Operations and Management Symposium (NOMS), Krakow, 2014; 1–8. DOI: 10.1109/NOMS.2014.6838322.




3. Feamster N. Outsourcing home network security. In Proceedings of the 2010 ACM SIGCOMM Workshop on Home Networks - HomeNets ’10, New Delhi, India, 2010; 37–42. DOI: 10.1145/1851307.1851317. 4. Cruz T, Simoes P, Reis N, Monteiro E, Bastos F. An architecture for virtualized home gateways. In Proceedings of the 2013 IFIP/IEEE International Symposium on Integrated Network Management, IM 2013, Ghent, 2013; 520–526. 5. John W, Pentikousis K, Agapiou G, Jacob E, Kind M, Manzalini A, Risso F, Staessens D, Steinert R, Meirosu C. Research directions in network service chaining. In SDN4FNS 2013 - 2013 Workshop on Software Defined Networks for Future Networks and Services, Trento, Italy, 2013; 1–7. DOI: 10.1109/SDN4FNS.2013.6702549. 6. Goncalves C, Soares J. Traffic steering blueprint, 2014. Available from: https://review.openstack.org/#/c/92477/7/specs/ juno/traffic-steering.rst. [Accessed 03 July 2016]. 7. Chan KC, Martin M. An integrated virtual and physical network infrastructure for a networking laboratory. In 2012 7th International Conference on Computer Science & Education (ICCSE 2012), IEEE, The University of Melbourne, Melbourne, Australia, 2012; 1433–1436. DOI: 10.1109/ICCSE.2012.6295333. 8. Congdon P. Software-defined networking: the new norm for networks. Technical Report, ONF, 2012. Available from: https://www.opennetworking.org/images/stories/downloads/sdn-resources/white-papers/wp-sdn-newnorm.pdf. [Accessed 03 July 2016]. 9. Gordon A, Amit N, Har’El N, Ben-Yehuda M, Landau A, Schuster A, Tsafrir D. ELI: bare-metal performance for I/O virtualization. In ASPLOS ’12 Proceedings of the Seventeenth International Conference on Architectural Support for Programming Languages and Operating Systems4, London, UK, 2012; 411–422. DOI: 10.1145/2150976.2151020. 10. Chang YH. Wide area information accesses and the information gateways. In Proceedings of lst IEEE International Workshop on Community Networking, San Francisco, CA, USA, 1994; 21–27. DOI: 10.1109/CN.1994.337369. 11. OpenWrt. The UCI system, 2014. Available from: http://wiki.openwrt.org/doc/uciAccessed 03 July 2016]. 12. Cisco. Configuring transparent bridging, 2005. Available from: http://www.cisco.com/c/en/us/support/docs/ ibm-technologies/source-route-transparent-srt-bridging/10676-37.html [Accessed 03 July 2016]. 13. Cisco. Switch virtual interface for Cisco integrated services routers, 2012. Available from: http://www.cisco.com/c/ en/us/products/collateral/routers/1800-series-integrated-services-routers-isr/prod_white_paper0900aecd8064c9f4.html. [Accessed 03 July 2016]. 14. Sinha R, Papadopoulos C, Heidemann J. Internet packet size distributions: some observations. Technical Report ISI-TR2007-643, USC/Information Sciences Institute, May 2007. Available from: http://www.isi.edu/~johnh/PAPERS/Sinha07a. html[Accessed 03 July 2016]. 15. Cardoso I, Barraca JP, Goncalves C, Aguiar RL. Seamless integration of cloud and fog networks. In 1st IEEE Conference on Network Softwarization (NetSoft 2015), 2015; 1–9. 16. Davie B. Network virtualization gets physical, 2013. Available from: http://blogs.vmware.com/cto/network-virtualizationgets-physical/[Accessed 03 July 2016]. 17. Davie B, Fortier R, Duda K. Physical networks in the virtualized networking world, 2014. Available from: http://blogs. vmware.com/networkvirtualization/2014/07/physical-virtual-networking.html. [Accessed 03 July 2016]. 18. Farias FNN, Salvatti JJ, Cerqueira EC, Abelem AJG. A proposal management of the legacy network environment using OpenFlow control plane. In Proceedings of the 2012 IEEE Network Operations and Management Symposium, NOMS 2012, 2012; 1143–1150. DOI: 10.1109/NOMS.2012.6212041. 19. Mckeown N, Anderson T, Balakrishnan H, Parulkar GM, Peterson LL, Rexford J, Shenker S, Turner JS. OpenFlow: enabling innovation in campus networks. Computer Communication Review 2008; 38: 69–74. DOI: 10.1145/1355734.1355746. 20. Manco F. Network infrastructure control for virtual campus. MSc Dissertation, Universidade de Aveiro, 2013. 21. Manco F. Appendix D - campus network extension, blueprint, 2013. 22. Manco F. Appendix E - ML2 external port extension, blueprint, 2013. 23. Benton K. Neutron external attachment points, 2014. Available from: https://review.openstack.org/#/c/87825/13/specs/juno/ neutron-external-attachment-points.rst. [Accessed 03 July 2016]. 24. Congdon P. Link layer discovery – protocol and MIB – v0.0. Technical Report, IEEE, 2002. Available from: http://www. ieee802.org/1/files/public/docs2002/lldp-protocol-00.pdf. [Accessed 03 July 2016]. 25. Case J, Fedor M, Schoffstall M, Davin J. Simple network management protocol. RFC 1067, August 1988. Available from: http://www.ietf.org/rfc/rfc1067.txt, obsoleted by RFC 1098. [Accessed 03 July 2016]. 26. Pfaff B, Davie B. The Open vSwitch database management protocol. RFC 7047 (Informational), December 2013. Available from: http://www.ietf.org/rfc/rfc7047.txt. 27. Claise B. Specification of the IP flow information export (IPFIX) protocol for the exchange of IP traffic flow information. RFC 5101 (proposed standard), January 2008. Available from: http://www.ietf.org/rfc/rfc5101.txt, obsoleted by RFC 7011.




AUTHORS’ BIOGRAPHIES Igor Duarte Cardoso received his Master of Science degree in Computer and Telematics Engineering in 2014 from the University of Aveiro. Currently a Software Engineer working around Telecommunications, Network Functions Virtualization (NFV) and Cloud Computing, he was originally doing research on NFV at the Instituto de Telecomunicações (IT), Aveiro. He has participated in open-source projects including OpenStack, and is helping to shape Service Function Chaining in that project. Prior to his involvement with NFV, he’s also made research on Human-Computer Interaction and authored a fitness Android application. João P. Barraca received a Ph.D. degree in Informatics Engineering in 2012 from the University of Aveiro, where he developed work focused on the integration of social structures into self-management network functions. He is currently an Invited Assistant Professor at the University of Aveiro, in areas related to programming, networking and security. Most of his research is conducted through Instituto de Telecomunicaçöes (IT), to which he associated since 2003, and acts as coordinator of the Aveiro Telecommunications and Networking Group (ATNoG). During this time at IT he has published works in the areas of networking and software for computer systems. He has also acted as reviewer for several conferences and journals, mostly of international nature. He has extensive participation in both national and international funded projects, having participated in projects such as FP7 IST Daidalos, IST WIP, IST S(o)OS, IST-Onelab, IST Prose, CAPES DEVNF, and FCT CRUISE. He is also a member of the Telescope Manager consortium for the Square Kilometre Array (SKA) telescope, leading the tasks related to the cloud computation infrastructure (LINFRA) for the software management components.

Carlos Gonçalves is a R&D Engineer on the 5G Networks team at NEC Laboratories Europe in Heidelberg, Germany. He works in the areas of Network Functions Virtualization and Carrier-Cloud Operation & Management, developing novel technologies and tools for the design, deployment, operation and management of cloud-native virtualized network functions. He has been participating in open-source projects including OpenStack and OPNFV where he plays a committer role in OPNFV Doctor project. Carlos received his Master of Science degree in Computers and Telematics Engineering from the University of Aveiro, and prior to joining NEC he was a Researcher in the Institute of Telecommunications, Portugal.

Rui L. Aguiar received his degree in telecommunication engineering in 1990 and his Ph.D. degree in electrical engineering in 2001 from the University of Aveiro. He is currently a Full Professor at the University of Aveiro, responsible for the networking area. He is coordinating a research line nationwide in Instituto de Telecomunicações, on the area of Networks and Multimedia. His current research interests are centred on the implementation of advanced wireless systems, with special emphasis on 5G networks and Future Internet. He has more than 400 published papers in those areas, including standardization contributions to IEEE and IETF. He has served as technical and general chair of several conferences, from IEEE, ACM and IFIP, and is regularly invited for keynotes on 5G and FI networks. He sits on the TPC of all major IEEE ComSoc conferences. He is currently associated with the 5G PPP Infrastructure Association and is on the steering board of the Networld2020 ETP. He is a chartered engineer, senior member of IEEE, Portugal ComSoc Chapter Chair, and a member of ACM. He is associated editor of Wiley ETT, Springers Wireless Networks and of the recently launched Elsevier ICT Express.



Seamless integration of cloud and fog networks - Wiley Online Library

Seamless integration of cloud and fog networks - Wiley Online Library

Suggest Documents

Seamless Integration of Predictive Analytics and ... - Wiley Online Library

Temporal and spatial characteristics of fog ... - Wiley Online Library

CLOUD computing - Wiley Online Library

Ecohydrological and socioeconomic integration ... - Wiley Online Library

Leveraging fog computing and software defined ... - Wiley Online Library

Experimental evaluation of fog computing ... - Wiley Online Library

Assessment of fog collection as a sustainable ... - Wiley Online Library

Patientcentered Integrated Networks of ... - Wiley Online Library

Unravelling intravertebral integration, modularity ... - Wiley Online Library

Floral integration, phenotypic covariance ... - Wiley Online Library

2. Cloud thermodynamic phase - Wiley Online Library

Secure Cloud Computing: Reference ... - Wiley Online Library

Cloud water, rainwater, an - Wiley Online Library

Static and Dynamic Views of European Integration - Wiley Online Library

Cotton market integration and the impact of ... - Wiley Online Library

European integration and the pay policies of ... - Wiley Online Library

The vertical integration of Lisbon and ... - Wiley Online Library

Relationship of personality with integration and ... - Wiley Online Library

Redistribution of ice nuclei between cloud and ... - Wiley Online Library

Change of cloud amount and the climate ... - Wiley Online Library

Impact of cloud and radiation scheme ... - Wiley Online Library

Traditional and Online Support Networks in the ... - Wiley Online Library

Brazilian bibliometric coauthorship networks - Wiley Online Library

Complex communication networks - Wiley Online Library