that up to 92% of workloads will be cloud-based, with the bulk of the ..... back-end services in the management plane be up and running, they should be ...
Deploying a Multi-Tier Heterogeneous Cloud: Experiences and Lessons from the SAVI Testbed Thomas Lin, Byungchul Park, Hadi Bannazadeh, and Alberto Leon-Garcia Department of Electrical and Computer Engineering University of Toronto, Toronto, ON, Canada Email: {t.lin, byungchul.park, hadi.bannazadeh, alberto.leongarcia}@utoronto.ca
Abstract—The SAVI testbed is a multi-tier SDN-enabled heterogeneous cloud created as an application enablement platform for academic research. It was developed to investigate the hypothesis that the entire infrastructure can be virtualized and abstracted as a service. Key to this investigation was the creation of a resource management system based on the concept of Software-Defined Infrastructure (SDI), which enables integrated control and management over heterogeneous resources and exposes a programmable interface for users. In this paper, we present our experiences in designing and deploying the Canadian national SAVI testbed, and discuss the various lessons learned along the way. In addition, we describe our ongoing work to enhance the testbed management system to meet the demands and challenges of the IoT era.
I. I NTRODUCTION The number of end-devices communicating with cloudbased applications and services over the network has grown exponentially during the last two decades. This growth has been driven by a number of factors, including the rapid deployment of wireless access points (APs), the growth of mobile devices, and the ever-growing Internet of Things (IoT). From an end-to-end perspective, the resources involved in modern cloud applications are increasingly diverse and heterogeneous, where networked devices can range from tiny coinsized embedded devices to massive datacentre-grade servers. The growing number of inter-connected devices and services in turn generates large amounts of data and network traffic flowing to and from cloud datacentres. By 2020, it is estimated that up to 92% of workloads will be cloud-based, with the bulk of the workloads being related to video and IoT data analytics [1]. The growth of IoT has also resulted in an increasing number of novel applications and services with low-latency or localized processing requirements. To meet these requirements, there has been a trend towards distributing small cloud datacentres to the edge, closer to end-users [2][3][4]. The apex of this trend is realized by having equipment directly colocated with the users, where virtualized customer premises equipment (vCPE) [5] can be controlled and virtualized by the cloud provider. This may be desirable for organizations with strict policies that data does not leave their premises, while having the advantages of on-site cloud services. Foreseeing these trends and changes, the Smart Applications on Virtualized Infrastructure [6] (SAVI) project was created in c 2018 IEEE 978-1-5386-3416-5/18/$31.00
Fig. 1. Multi-tier cloud applications platform for IoT.
2011 to address the key challenges in realizing future cloud application platforms. A fundamental part to this research was the design and implementation of the SAVI testbed, a multi-tier heterogeneous cloud. The multi-tier aspect is illustrated in Fig. 1, in which the testbed’s high-level deployment architecture consists of a few traditional Core datacentres interconnected through a dedicated backbone with “Smart Edge” datacentres strategically located close to the end-users, as well as an access tier which allows users to connect their devices to the testbed network. We call the testbed Edge datacentres “Smart” since they contain a variety of heterogeneous resources (e.g. FPGAs, SDRs, wireless APs, CPUs, GPUs, and etc.) for enabling the dynamic and adaptive creation of intelligent applications and services. These resources are inter-connected by a fully SDN-enabled network at Layer2 Ethernet that allows experimentation with non-IP network stacks directly on the physical network. In recent years, the SAVI team has been researching ways to extend the testbed to function as an end-to-end IoT enablement platform. We foresee global-scale deployments of virtualized sensors, actuators, and gateways that will result in rapid growth of network traffic volumes as well as the need for cloudbased services to control, manage, and support them. We are currently integrating IoT resources into the testbed to enable novel research on end-to-end IoT applications and services. In this paper, we share our experience in designing and deploying the SAVI testbed to assist others looking to deploy their own SDN-enabled heterogeneous clouds and avoid common pitfalls, reducing the time from design to production. We will provide insight into our operational experience, the issues we faced, and the lessons learned. In Section II, we describe the components used in the control and management of the testbed, and in Section III we present an overview of how these components are deployed. Section IV discusses the challenges we faced, our solutions, and lessons we learned. Section V presents our on-going work to enhance the testbed, and Section VI concludes the paper.
Fig. 2. Application Enablement Stack
II. I NTEGRATED M ANAGEMENT OF H ETEROGENEOUS R ESOURCES Since the SAVI testbed contains many different types of resources, the efficient management of these resources would be best realized by having an integrated solution in order to satisfy the needs of end-to-end services and applications. Figure 2 shows a 5-layer stack of components needed to support applications and services. This section presents a brief overview of these components and presents an architecture based on the 5-layer application enablement stack.
Fig. 3. Software-Defined Infrastructure
other operations such as modifying packet headers or dropping packets altogether. B. Software-Defined Infrastructure
A. Resource-Specific Management At the base infrastructure, L1 in Fig. 2, the testbed is composed of heterogeneous resources. Some of these resources may be virtualized to create the illusion of multiple instances, e.g. a standard server can be virtualized to support multiple VMs or containers. These different resource types (e.g. switches, routers, access points, CPUs, etc.) are typically managed separately using resource-specific control software (L2 in Fig. 2). To control and manage compute resources such as CPUs and and disk (storage) space, we leverage projects from the OpenStack [7] ecosystem, a popular set of software projects for deploying Infrastructure-as-a-Service (IaaS) clouds. We deployed OpenStack Nova for virtualizing CPU resources and enabling the creation of virtual machines (VMs). Nova has an extensible architecture where plugins are used to ensure compatibility with lower-level technologies (e.g. libvirt, Xen, VMware, Hyper-V, and etc.). We extended Nova to support the virtualization of GPUs and FPGAs, enabling applications with high-performance computational requirements. For users who wish to have extra disk space for their VMs, we use the OpenStack Cinder project to virtualize large clusters of hard drives. Cinder allows users to create network-attached storage and to dynamically attach/detach them from different VMs. To control the network that interconnects the various resources in the testbed, we rely on a combination of OpenStack Neutron and OpenFlow [8]. OpenStack Neutron is used primarily as a database of the network state, storing information regarding the endpoint addresses (MAC and IP) of VMs and the tenants they are associated with. This information can then be used to enforce network access control policies (e.g. isolating resources belonging to different tenants). To enforce these policies in the network, we leverage OpenFlow controllers to install flow entries into the switches throughout the network. These flow entries dictate the behaviour of traffic and can be used to set up paths through the network, as well as
To service the needs of end-to-end applications, we need an infrastructure management system (L3 in Fig. 2) that can perform integrated control and management over multiple resource types. An integrated approach yields more efficiency and performance in comparison to managing each resource type in isolation. To enable users and organizations to build heterogeneous applications on the infrastructure, a set of open interfaces (APIs) are exposed which offer unique functionalities and capabilities (L4 in Fig. 2). These functionalities can be specific to a single resource, or involve multiple resources. To this end, the SAVI team proposed the concept of Software-Defined Infrastructure (SDI) [2], where an integrated management system coordinates and controls all infrastructure resources while exposing programmatic interfaces for the construction of unique applications and services (L5 in Fig. 2). The SAVI SDI-based architecture is shown in Fig. 3. The architecture contains a set of centralized processes, collectively referred to as the SDI management system. We now briefly describe the purpose of each of the components within the SDI management system: • SDI Manager: The SDI manager is the primary component responsible for controlling and managing the virtual compute and networking resources. It also acts as the gateway responsible for receiving all the API calls. The manager has a modular design [9] and so its actual control and management logic is implemented in various plugins. This design makes it agnostic to the infrastructure, the control algorithms, and the resource controllers. • Topology Manager: The topology manager keeps track of the infrastructure topology, and all the changes that occur within it. It keeps a record of all the resources, both virtual and physical, and tracks their relationships with one another via a graph-based database [10]. The SDI manager uses the topology manager to gain awareness of the current topology, which is crucial for any state-based control and management algorithms.
•
Monitoring and Analytics: This system collects and stores data regarding the utilization of resources in the infrastructure. It collects data for both physical and virtual resources, and can also be used to collect data about applications, e.g. within VMs [11]. The system allows the SDI manager as well as users to submit queries, in the form of jobs, to gain knowledge and insight and to make more informed management decisions.
III. SAVI T ESTBED D EPLOYMENT Currently the SAVI testbed consists of 2 Core datacentres and 9 Edge datacentres within Canada, located in 7 universities. The SAVI testbed has also deployed an Edge at POSTECH university in South Korea, and it is federated with the GENI [12] testbed in the United States, which allows users to utilize resources in both testbeds. Another benefit to federating with GENI is that we can provision temporary SAVI Core and Edge regions using GENI’s baremetal resources, enabling SAVI experiments with extended geographical footprint across both Canada and the US. A. Datacentre Management Each SAVI Core and Edge datacentres uses identical SDI control and management systems shown in Fig. 3. By distributing the control and management on a per-datacentre basis, we ensure that each can function autonomously. In our deployments, minor differences exist in the configuration and versions of the resource controllers (OpenStack services and OpenFlow controllers), depending on the types of resources available in each datacentre. For example, some datacentres contain a mix of switches supporting different OpenFlow versions, some contain specialized resources such GPUs, FPGAs, SDRs, etc. The SDI resource management system, however, need not be changed or reconfigured, as the differences in the underlying infrastructure are abstracted away by the resource controllers. B. Testbed-Wide Services The SAVI testbed provides four testbed-wide essential shared services to enable users to interact with all the regions. These services are: • Orchestration: The orchestration service in the SAVI testbed manages the lifecycle of entire applications. We utilize OpenStack Heat to realize the orchestration service. Its operation is based on the manipulation of humanreadable templates that describe the various resources necessary for the composition of a given application. • Image Registry: The registry service stores the location and information of VM images across the SAVI testbed. While the actual back-end storage of images are distributed in the various datacentres, there exists a central registry. This enables the ability to locate and download images regardless of which region they have been uploaded to. In SAVI, we leverage the OpenStack Glance component to realize the registry. • Identity and Access Management: The identity and access management (IAM) component is crucial for any
•
cloud. It stores the credentials of all the users and the details regarding which resources and tenants they are allowed to access. In SAVI, we use OpenStack Keystone for the IAM, which serves as both the authentication and authorization manager. Web Portal: The web portal provides users a graphical user interface (GUI) to interact with the SAVI testbed. For this, we leverage the OpenStack Horizon component.
C. Dedicated L2 Ethernet WAN To enable large-scale multi-region applications, the SAVI testbed’s various regions within Canada are interconnected by dedicated optical networks that provide Layer-2 Ethernet connectivity. Within the province of Ontario, our regions are connected by ORION (Ontario Research and Innovation Optical Network). Other regions within Canada are connected via CANARIE (Canadian Research and Education Networks), which is federated with ORION. For our Edge datacentre in South Korea, we use an OpenVPN connection. To achieve L2 connectivity between the SAVI and GENI testbed networks, we developed HyperExchange [13], a protocol-agnostic Software-Defined Exchange (SDX) that can maintain tenantbased isolation policies across different cloud infrastructures. The combination of L2 inter-region connectivity and OpenFlow capabilities within each region of the testbed allows users to create multi-region applications, including novel nonIP protocol experimentation without using overlays or tunnels. IV. E XPERIENCES AND L ESSONS In this section, we discuss our experiences and the lessons we learned in the course of designing and deploying the SAVI testbed. We group these into three broad categories: designing for scalability of the infrastructure and services; the necessity for a multi-level monitoring solution; and miscellaneous lessons. A. Scalability of Infrastructure and Services As a cloud testbed, our system should be scalable and elastic to meet the dynamic demands of users. This applies not only to the software services of the testbed, but also the testbed capacity as well. Scaling IAM Service: During periods of heavy demand, our Keystone IAM service can quickly become a bottleneck. This is because every request from clients, as well as interservice communications, must be able to authenticate each other, making the IAM a critical point of failure. To increase the service capacity of the IAM system, our initial approach was to horizontally scale the number of instances behind a reverse-proxy load balancer to create a cluster. However, our benchmark showed that the response time for each request was still slow. Further examination of the requests revealed that most of them are read-only queries, thus we deployed memcached, an in-memory caching solution that allows the IAM instances to quickly fetch “hot” data. Despite these steps, our experience has shown that power failures and BGP issues outside our domain of control can just as easily take down the entire IAM cluster. To protect
against these issues and increase fault-tolerance, we duplicate the clusters and locate them in different regions. This not only protects against issues which could affect entire clusters, but also reduces the network latency for distant regions. To keep the state in multiple regions consistent, we utilize Galera to create a multi-master database cluster. To direct clients to the nearest cluster, we leverage DNS round-robin, which also provides quick fail-over if a cluster is down. Scaling the Network Management: The SAVI testbed’s network is fully SDN-enabled, where all network decisionmaking is done by the SDI management system [14], and traditional SDN controllers are used as proxies for OpenFlow messages. In periods when many resources actively establish new network flows, the SDI management system can become a bottleneck. To alleviate this, we employed 3 strategies: 1) our network management proactively installs flows from all resources to their respective gateways; 2) we installed rules to drop all broadcast packets, making the network more secure and reducing the amount of traffic sent to the controller; and 3) we distributed light-weight SDN controllers throughout the network and configured them to store and cache decisions made by the SDI manager. [15] discusses our network management scheme in further detail, and in particular, how we handle DHCP and ARP without broadcasting. Elasticity of Testbed Capacity: When the testbed is in high demand, we may run out of capacity to host new VMs. Since certain regions of our testbed contains baremetal servers, we can temporarily and quickly increase our capacity by pushing the baremetals into service as new physical agents to host more VMs. Additionally, our federation with the GENI testbed allows us to acquire baremetal servers there as well, allowing us to quickly deploy new regions of SAVI. Streamlining the deployment of new agents and new regions can be achieved by packaging all the related software services into archived formats (e.g. tar, deb, rpm, etc.) to ease installation, and using orchestration methods such as Ansible playbooks to dynamically configure them. B. Multi-Level Monitoring Monitoring is crucial in any large-scale infrastructure, and even more so in highly dynamic environments such as the SAVI testbed. The management plane of the testbed comprises dozens of different services that must all function correctly in tandem. In cloud environments, we not only have to monitor the system of services that comprise the management plane, but also the physical and virtual infrastructure as well. Public API Monitoring: Our testbed exposes many features and functionalities to users over publicly accessible APIs. Under-the-hood, these API calls often branch out and involve multiple inter-communicating back-end services. For example, launching a single VM in OpenStack may involve as many as 28 different calls between various services [16]. A failure at any of the back-end services results in the failure of the overall task. Thus, monitoring these public APIs implicitly checks an entire chain of services. We created and deployed application-level monitoring jobs that periodically attempt to
spawn a new VM (targetting specific physical hosts in a roundrobin manner), SSH into it, and ping Google DNS to ensure Internet connectivity. If the job fails, we are alerted and can begin a process of diagnostics to narrow down the cause. Stateful Service Monitoring: Since it is crucial that all the back-end services in the management plane be up and running, they should be individually monitored. Our initial attempt utilized the Linux supervisord tool, which monitors process IDs and respawns processes when they disappear. However, we found that this is insufficient since many services often remain alive, but in a corrupted state with limited functionality. Thus, we created a stateful monitoring system that periodically queries the individual services’ internal APIs and parses the returned output to ensure they align with our expectations. This also implicitly ensures a viable network path to those services exist. Physical Infrastructure Monitoring: For high-level health monitoring of our physical servers, we use Nagios to monitor and alert us of issues related to CPU load, disk I/O load, free disk space, and network reachability. However, our experience has shown us that physical failures often occur in a number of different ways with different root causes. This prompted us to develop a comprehensive monitoring system for the physical infrastructure that provides details related to HDD health, RAM health, temperature, fan speed, power consumption, network utilization, and CPU utilization. A web-based visualizer was created that graphs some of these metrics in real-time. We obtain these metrics from a mix of out-of-band monitoring tools (e.g. IPMI) and in-band tools from within the OS of switches, routers, and servers. In addition, we deployed a log parsing system within our public-facing servers and routers in order to catch malicious attempts to scan or breach our systems, and to then actively blacklist the source IPs. Certain metrics (e.g. network flows, traffic statistics, CPU utilization, disk I/O, etc.) are also streamed into a distributed storage system (in our case, we use HDFS), which is then used by the SDI’s monitoring and analytic system for long term trending and analysis. This data allows us to calculate key performance indicators (KPIs) for our system, that provide insight into the utilization of testbed resources, enabling us to better plan for future expansions. C. Miscellaneous Lessons In this subsection we briefly review some other experiences and lessons learned. Preventing Data Corruption and Loss: All hard drives in active use eventually suffer from either data corruption due to natural wear-and-tear, or total failure. Our early experience with these issues taught us the importance of maintaining frequent backups of vital system state data, which would allow us to minimize the downtime incurred from total drive failures. Our initial na¨ıve approach was to outfit critical servers with two hard drives and configure them for RAID 1 (active mirroring of main drive to create an exact copy). However, this method was soon found to be insufficient. In one case, the backup drive in the server failed before the main drive without
our knowledge. This lesson prompted us to start employing active HDD health checks. In another case, an entire datacentre was lost due to flooding. Thus, we now maintain multiple copies of critical data such as databases storing system state, and configuration files for servers and routers. A common practice we employ is to keep one copy on a secondary drive within the same server (if available), and one or more copies on remote servers. Optimizing CPU Performance: As an academic testbed, our users are researchers engaged in various experiments, some of which are CPU intensive, while some are bandwidthintensive. Our early users reported certain experiments had inconsistent or irreproducible results. Investigations revealed this to be primarily due to the CPU frequency governor within the Linux kernel, which by default will adjust the CPU frequency depending on the current system load. Since our testbed widely employs Open vSwitch, a software-based OpenFlow switch, this also had an undesirable side-effect on the network throughput. For consistent performance, albeit at the expense of power consumption, it is advised to set Linux servers’ CPU frequency governor to ‘performance’ mode. Optimizing Network Performance for NFV: In recent years, Network Function Virtualization (NFV) has seen a growth in interest as it promises to spur the development of novel implementations of existing and new network functions. Key to this is the deployment of Virtualized Network Functions (VNFs) using programmable resources. While our physical network infrastructure can support multi-gigabit rates, our early attempts at creating VNFs in software observed throughput that was less than anticipated. Our investigation revealed two main causes of reduced throughput: 1) the CPU frequency scaling previously mentioned; and 2) the Linux network stack’s stateful firewalls such as iptables and ebtables. The solution to the first issue was previously discussed. The second issue arose from the fact that servers and VMs were traditionally considered as network endpoints, and thus filtering all packets became an ingrained function in the OS’ network stack. For VNFs, network traffic should enter untouched by third parties that are not part of the NFV service chain. To bypass the Linux network stack, we leveraged the Data Plane Development Kit [17] (DPDK) to directly access the network interface cards. We observed significant improvements for our software-based VNFs, maintaining 10 Gigabit line-rate for simple VNFs with uncomplicated workloads (e.g. signature matching at static offsets). However, for computationally intensive VNFs (e.g. full packet scans) we found the CPU remained as a performance bottleneck, thus we have recently integrated FPGA-based VNFs which allows us to further maintain linerate with complex workloads [18]. Optimizing Disk I/O: Certain applications, such as those involved in big data analytics, are often quite disk-intensive. Some of our early users engaged in big-data applications reported poor disk I/O bandwidth, prompting us to investigate methods for hypervisor-tuning. In the SAVI testbed, we use the KVM hypervisor, which features multiple options for disk read/write caching. In benchmark tests with mechanical
HDDs, we were able to improve I/O performance up to 6x for reads and 10x for writes when we enable host page caching and disk write caching, though this comes with a risk of data loss in the event of power failures. We are able to offer these caching options to users for them to decide on the performance vs risk trade-off. Time-Limited Resources: As a free-to-use testbed, our experience showed us that users are often not mindful of the limited capacity of the cloud, leaving experiments active and running. To ensure the SAVI testbed’s capacity is not wastefully reserved for an undue amount of time, we place a time limit of 30 days on all VMs acquired by regular users. We run a system that automatically tracks the lifetime of VMs and alerts their owners via email as the 30-day limit is close to running out. The users may then request an extension for their resources by another 30 days, else the VMs are automatically deleted. For specialized resources in high-demand, such as GPUs, we apply the same logic but with a reduced time limit of one week. V. W ORK I N P ROGRESS The control and management plane of the SAVI testbed has undergone numerous changes since its inception six years ago, and will continue to evolve in the near future as the testbed transitions to serve as an IoT applications enablement platform. Since IoT is predicted to feature a large diversity of networked devices, producing massive volumes of network data and traffic, all in highly dynamic and mobile environments with high churn, the SAVI testbed’s management system must be ready for the scalability and high-availability challenges to come. In this section, we discuss some of our work-in-progress to further enhance and evolve the testbed. A. Containerization of Services One of the directions we have been investigating over the past years has been the use of container technology (i.e. Docker and Kubernetes) to package management plane services as stand-alone containers. This process, called containerization, allows container orchestration engines to monitor container clusters and dynamically scale them as needed. We have already containerized the SAVI topology manager and the main SDI manager, allowing us to quickly deploy them as needed when bringing up new regional datacentres, or when recovering from a system fault (e.g. power outage or server crash). In addition, we have containerized our OpenFlow controllers, and are currently working towards containerizing the rest of our resource-specific controllers. For the various services within the OpenStack ecosystem, we are leveraging Stackanetes [19], an initiative to containerize and orchestrate the deployment of OpenStack services, allowing us to quickly deploy clusters of OpenStack services as needed. B. Auto-Scaling Management Services In [20], Khazaei et al. created a prototype auto-scaling engine for data processing pipelines based on a containerized ELKB (Elasticsearch, Logstash, Kibana, Beats) stack. The prototype, named Elascale, is able to implement a basic MAPE
loop [21] for autonomically and elastically scaling both VMs and containers. Although Elascale was initially proposed for monitoring and auto-scaling IoT sensor applications, we plan to adapt it with the SAVI testbed’s infrastructure and service monitoring system, and use it to auto-scale the containerized OpenStack and SDI services. This would represent a major step towards realizing a fully autonomous management plane. C. Distributed Management Services While we can use existing container technologies to quickly deploy containerized services, often times these services may need to be individually configured, or may exist behind NATs and load balancers which become bottlenecks when traffic rates increase. To meet these challenges, we are investigating ways to decentralize services over a P2P substrate, identifying and discovering services based on hashes instead of endpoints. This method creates location-agnostic services that benefits mobility, and enables the system to be self-configuring and self-healing. As a proof-of-concept, we have implemented a mechanism to enable self-healing for the OpenFlow controller services within our testbed. As mentioned in subsection IV-A and explained in [15], our network control architecture uses the SDI manager to make all decisions regarding which flows to permit in the network. These decisions are passed to a set of OpenFlow controllers for enforcement. The OpenFlow controllers are distributed throughout the infrastructure, as shown in Fig. 4(a), with one local controller process per switch. While distributing the OpenFlow controllers allows the SDI manager to scale, it currently requires manual deployment and configuration of each switch to connect to its own localized controller. When an instance of the controller dies, the switch it controls can no longer update its flow table, and any VMs connected to that switch can no longer establish any new communications. With many other OpenFlow controllers distributed throughout the infrastructure, a switch that loses connectivity to its local controller should be able to automatically discover a neighboring controller identical to its local controller, and immediately connect to it. To achieve this, we have developed a simple system that adapts IPFS [22], a P2P distributed file system, for discovering candidate controllers for a switch to connect to. The overall logic or our system is as follows: 1) Each node participating in the IPFS swarm hashes their local controller code. As our network uses a mix of two different controller implementations, we can use its hash to discern them; 2) Monitor the switch status. When a switch becomes disconnected from its controller, we query the IPFS swarm for a list of peers using the hash of its local controller, which should find peers with identical controller implementations; 3) A list of candidate peer nodes is returned along with latencies to each node. We order the list by ascending latency and progressively scan through them; 4) For each peer in the returned list, we obtain its underlying IP and probe the TCP port corresponding to Open-
(a) Distributed OpenFlow controllers (different types), one per switch
(b) Auto-discovery and connection to neighboring controller Fig. 4. Self-Healing of OpenFlow Controllers in the SAVI Testbed
Flow controllers in order to test if an active instance exists within that peer. If the probe is successful, we connect the switch to this new OpenFlow controller, else we continue down the list; 5) If we exhaust all candidate peers, we sleep and try again later. 6) Repeat from step 2 As shown in Fig. 4(b), we tested the proof-of-concept by manually killing OpenFlow instances and observing that the switches automatically discover and connect to a neighboring OpenFlow controller, ensuring continuous operation of the network. This proof-of-concept demonstrates the feasibility of using P2P-based discovery of distributed services to realize a self-healing management system. We believe the same general logic can be applied to all services to achieve scalability, highavailability, and support for mobility. VI. C ONCLUSION This paper shares our experience and lessons learned from designing and deploying the SAVI testbed, a multi-tier SDNenabled heterogeneous cloud. We described the various components that comprise our testbed, and discussed how we were able to deploy them to create a nation-wide multi-region cloud. In operation since 2012, the SAVI testbed went through many iterations of incremental improvements to optimize scalability, performance, and attain a thorough monitoring of our infrastructure. As the SAVI testbed transitions to become an IoT applications platform, the control and management plane must evolve to meet the challenges of scalability, highavailability, and mobility foreseen in the IoT era. Thus, we briefly discussed some of our ongoing work to support these characteristics in our testbed. We hope that our operational experiences can provide insight and lessons for those small organizations wishing to deploy their own similar SDN-enabled heterogeneous clouds.
R EFERENCES [1] Cisco, “Cisco Global Cloud Index: Forecast and Methodology 20152020,” in Cisco White Paper, 2016. [2] J.-M. Kang, H. Bannazadeh, H. Rahimi, T. Lin, M. Faraji, and A. LeonGarcia, “Software-Defined Infrastructure and the Future Central Office,” in Communications Workshops (ICC), 2013 IEEE International Conference on, pp. 225–229, June 2013. [3] L. Velasco, L. Contreras, G. Ferraris, A. Stavdas, F. Cugini, M. Wiegand, and J. Fernandez-Palacios, “A Service-Oriented Hybrid Access Network and Clouds Architecture,” Communications Magazine, IEEE, vol. 53, pp. 159–165, April 2015. [4] ETSI, “Mobile-Edge Computing - Introductory Technical White Paper,” in ETSI White Paper, 2014. [5] IDC, “The Business Benefits of Implementing NFV: New Virtualized vCPE Enterprise Services,” Nov. 2014. [Online; accessed 8-June2015]. Available: http://www8.hp.com/h20195/V2/getpdf.aspx/4AA56949ENW.pdf. [6] “Smart Applications on Virtual Infrastructure (SAVI).” [Online] Available: https://www.savinetwork.ca/ [Accessed: April 2017]. [7] “OpenStack is open source software for creating private and public clouds.” [Online] Available: https://www.openstack.org/ [Accessed: August 2017]. [8] N. McKeown, T. Anderson, H. Balakrishnan, G. Parulkar, L. Peterson, J. Rexford, S. Shenker, and J. Turner, “OpenFlow: Enabling Innovation in Campus Networks,” SIGCOMM Comput. Commun. Rev., vol. 38, pp. 69–74, Mar. 2008. [9] T. Lin, “Implementation and Evaluation of an SDN Management System on the SAVI Testbed,” Master’s thesis, University of Toronto, 2014. [10] J. M. Kang, H. Bannazadeh, and A. Leon-Garcia, “SDIGraph: Graphbased management for converged heterogeneous resources in SDI,” in 2016 IEEE NetSoft Conference and Workshops (NetSoft), pp. 88–92, June 2016. [11] J. Lin, “MonArch: Scalable Monitoring and Analytics for Visibility and Insights in Virtualized Heterogeneous Cloud Infrastructure,” Master’s thesis, University of Toronto, 2015. [12] “Global Environment for Network Innovations.” [Online] Available: https://www.geni.net/ [Accessed: August 2017]. [13] S. Arezoumand, H. Bannazadeh, and A. Leon-Garcia, “Hyperexchange: A protocol-agnostic exchange fabric enabling peering of virtual networks,” in 2017 IFIP/IEEE Symposium on Integrated Network and Service Management (IM), pp. 204–212, May 2017. [14] T. Lin, J. M. Kang, H. Bannazadeh, and A. Leon-Garcia, “Enabling SDN applications on Software-Defined Infrastructure,” in 2014 IEEE Network Operations and Management Symposium (NOMS), pp. 1–7, May 2014. [15] B. Park, T. Lin, H. Bannazadeh, and A. Leon-Garcia, “JANUS: Design of a Software-Defined Infrastructure Manager and its Network Control Architecture,” in 2016 IEEE NetSoft Conference and Workshops (NetSoft), pp. 93–97, June 2016. [16] “Request Flow for Provisioning Instance in Openstack.” [Online] Available: https://ilearnstack.com/2013/04/26/request-flow-for-provisioninginstance-in-openstack/ [Accessed: September 2017]. [17] “Data Plane Development Kit.” [Online] Available: http://dpdk.org/ [Accessed: May 2017]. [18] T. Lin, N. Tarafdar, B. Park, P. Chow, and A. Leon-Garcia, “Enabling Network Function Virtualization over Heterogeneous Resources,” in 2017 19th Asia-Pacific Network Operations and Management Symposium (APNOMS), pp. 58–63, September 2017. [19] “OpenStack on Kubernetes.” [Online] Available: https://coreos.com/blog/announcing-stackanetes-technical-preview.html [Accessed: September 2017]. [20] H. Khazaei, R. Ravichandiran, B. Park, H. Bannazadeh, A. Tizghadam, and A. Leon-Garcia, “Elascale: Autoscaling and Monitoring As a Service,” in Proceedings of the 27th Annual International Conference on Computer Science and Software Engineering, CASCON ’17, (Riverton, NJ, USA), pp. 234–240, IBM Corp., November 2017. [21] “An Architectural Blueprint for Autonomic Computing,” tech. rep., IBM, June 2005. [22] J. Benet, “IPFS - Content Addressed, Versioned, P2P File System,” CoRR, vol. abs/1407.3561, 2014.
