Virtual Machine Migration in heterogeneous Clouds: From OpenStack ...

Virtual Machine Migration in heterogeneous Clouds: From OpenStack to VMWare Dimitrios Kargatzis1, Stelios Sotiriadis1,2, Euripides G.M. Petrakis1 1

School of Electrical and Computer Engineering, Technical University of Crete (TUC), Chania Greece 2 Computer Engineering Research Group, University of Toronto, Toronto, Canada [email protected], [email protected], [email protected]

Abstract —Adopting a Cloud solution means binding to a specific platform and vendor, with proprietary protocols, standards and tools so usually running into a vendor lock-in. The fear of vendor lock-in is often cited as a major impediment to Cloud service adoption. In this work, we focus on the Virtual Machine (VM) migration between homogeneous and heterogeneous Cloud platforms. We focused on the technical parameters that are essential to be tuned, depending on the various types of virtualization engines used by the Cloud environments. The key difference is the configuration requirements mainly related with image formats and used hypervisors. To demonstrate heterogeneous VM migration we develop a tool that works for OpenStack and VMWare platforms. The experimental analysis demonstrates the effectiveness of our solution with regards to the HTTP response times, especially between the heterogeneous Cloud platforms. The tool achieves approximately two milliseconds response time for HTTP requests to Cloud APIs, excluding the time required to download and upload image files. Keywords—Cloud Computing Virtual Machine migration, Heterogeneous Virtual Machine Migration, Container based migration, OpenStack, VMWare

I.

INTRODUCTION

Cloud computing is an Internet based infrastructure that provides shared computer processing resources and data on demand. It offers elasticity (as the ability of Cloud computing scale up or down computing resources), on-demand selfservice so consumers can adjust computing resources, measurable services provision and pay per use. Today, Cloud computing is becoming a game changer for IT companies by offering scalable infrastructure and services for data storage and analytics [3]. Enterprises, that need complex and expensive IT technology to support business processes, Cloud provides an attractive alternative by which the compute resources are made available at a fraction of cost. Adopting Cloud computing can offer advantages (e.g. reduce infrastructure costs), but it’s important to choose the ideal Cloud hosting service and the Cloud solution for different business needs [4]. Determining which provider is the best depends largely on the business needs in terms of Cloud services, time of use, degree of required control over hardware and software for application development or hosting. Additional factors are the security accreditations of the providers. However, selecting a Cloud provider usually means binding with the specific

protocols, standards and tools that the Cloud provider supports, an issue also known as vendor lock-in [5]. Vendor lock-in is often cited as a major obstacle to adoption of Cloud computing. The fear of vendor lock-in is often cited as a major impediment to cloud service adoption. If the provider decides to raise its prices or change its security policies, the customer may have to consider moving workloads to another provider. However transition to a competitor platform is a challenging task. The complexity of the migration depends on the way a Cloud provider configures its infrastructure, making migration difficult and expensive. As described in [7], a key concern of Cloud Computing is the limited portability, as it is difficult and costly to transfer to a new Cloud provider due to the complexity of the already deployed application. The basic migration constraints between heterogeneous Cloud providers are their system architectures, the supported hypervisors and the containers and disk formats. Lately, various solutions develop platforms and tools to migrate from a Cloud provider to another but usually they focus on the same infrastructure (homogeneous Cloud VM migration) as in [8]. In this work, we focus on the problem of migrating VMs between different Cloud infrastructures taking into consideration the technical requirements of each platform that are mostly related with image formats and hypervisors. To demonstrate the heterogeneous migration, we implement a toolkit that allows VM migration and deployment from Openstack1 to VMware2. The implementation consists downloads the source Cloud provider VM, then it solves the technical migration constraints. Finally, the VM is deployed on the target Cloud provider. The experimental analysis presents a mechanism for VM migration between Openstack (that uses KVM hypervisor [9]) and VMWARE (that uses ESXi hypervisor [10]). The standards approach is to freeze the running instance and restart it in the new environment following a series of configuration steps according to the initial VM parameters (e.g. security groups, public keys etc.). Having discussed that, in Section II we present the review of the background works, in Section III the VM migration requirements, in IV the design of the migration service, in Section V the implementation of the toolkit, in Section VI the performance analysis of the homogenous and heterogeneous scenarios and in Section VII we conclude with the summary of our work. 1 2

https://www.openstack.org/ https://www.vmware.com/

II.

BACKGROUND SOLUTIONS

VM migration between homogeneous and heterogeneous platforms expose a number of technical requirements and configurations, so this section is focusing on describing related solutions. In our work, hypervisors have one of the most important roles on the VM deployment, as it is the software responsible for creating the virtual environment where the VM operates and is responsible for dynamically allocating hardware resources. The underlying hypervisor capabilities depend on the choice of the Cloud provider [5]. Our work targets on the following hypervisors KVM (used by Rackspace, FIWARE and others and based on OpenStack), XEN (Amazon Web Services), Hyper-V (Microsoft Azure) and ESXI (VMWare). These are considered as the most popular solutions as described in [8]. The following discussion is based on technology definitions as described in [1] and [9]. KVM3 is an open source project and runs on most Linux distributions. It is considered as the default hypervisor for virtualization so making it one of the most widely used hypervisors in the world. OpenStack, IBM, Google and other use it. XEN4 is an open source hypervisor, used by Amazon Web Services. XEN is not part of the Linux operating system, whereas KVM is part of the Linux kernel. Hyper-V 5 is a Microsoft product that is used by Microsoft Azure. Finally, ESXi 6 , is a VMware product, that supports any operating system. Since there are a variety of solutions, in this work we firstly focus on VM migration between: (a) OpenStack systems that use KVM hypervisors (homogeneous case) and (b) OpenStack and VMware thus migrating from KVM to ESXi hypervisors (heterogeneous case). Both are the industry’s most popular infrastructure suites that deliver comprehensive virtualization, management, resource optimization, and application availability capabilities. Another important factor to be considered is the disk format and the containers that are used by the different Cloud providers. The disk format is the format of the underlying disk image. Well known disk formats are (a) the RAW format (that is simple and easily exportable to all other formats), (b) the ISO that is usually used to install a virtual appliance (e.g. disk), (c) the VHD (Virtual Hard Disk) that it could contain disk partitions and a file system, (d) the VMDK (Virtual Machine Disk) is an open source and is one of the disk formats used in the Open Virtualization Format for virtual appliances and (e) the QEMU Copy On Write (QCOW2) that is file format used by a hosted virtual machine monitor (QEMU) that is primary software behind any virtualization environments and is usually associated with a specific guest operating system. Container format refers to the VM image that also contains a file with information about the VM (image metadata). Well known container formats are (a) the Bare that indicates no use of container or metadata envelope for the image (that is an XML file containing all the required parameters for the container to be deployed), (b) the Open Virtualization 3

https://www.linux-kvm.org/ https://www.xenproject.org 5 https://www.microsoft.com/en-sa/Cloud-platform/servervirtualization 6 http://www.vmware.com/products/esxi-and-esx.html 4

Format (OVF) that includes one OVF descriptor (XML) which describes metadata about virtual machine image and (c) the Amazon Machine Image (AMI) that include a template for all the required information to launch an instance (including the root volume for the instance with an operating system, servers and applications). It should be mentioned that Docker 7 container is not a format yet it wraps a piece of software in a complete file system (with everything that is needed to run, code, runtime, system tools, system libraries). To summarize the above discussion we present Table 1 that shows the comparison between the standards used by the most popular Cloud Providers including Amazon EC2, Microsoft Azure, Google Cloud, Rackspace, FIWARE, VMWare and IBM. We focused on Openstack and VMware that are the industry’s most popular full infrastructure suite that delivers comprehensive virtualization, management, resource optimization, application availability and operational automation capabilities is an integrated offering. Table 1: Comparison between disk formats,containers, hypervsoris,Command Line Interfaces, REST and XML/JSON standards of popular Cloud Providers Provider Amazon EC2 Microsoft Azure Google Cloud Engine Rackspace FIWARE VMWare IBM BlueMix

Disk Format AMI

Container

Hypervisor

CLI

AMI

XEN

VHD

BARE

RAW, AMI, VDI VHD QCOW2, AMI VMDK QCOW2, VDI,AMI

XML/JSON

✓

REST API ✘

Hyper-V

✓

✓

XML

BARE

KVM

✓

✓

JSON

BARE OVF, AMI, BARE OVF OVF, AMI

XEN KVM, XEN

✓ ✓

✓ ✓

XML/JSON JSON

ESXi KVM

✓ ✓

✓ ✓

XML/JSON JSON

XML

To help us with various image conversions we use the QEMU and the OVF tool. QEMU 8 is a generic and opensource disk conversion tool. It also provides a set of tools to create and convert disk images and supports many image file formats that can be used with virtual machines as well as with any of the tools, including QCOW2 (KVM and Xen), Raw, VDI, VHD (Hyper-V) and VMDK (VMware). The OVF Tool9 is a command line utility that generates OVA packages and converts OVF and other source formats to any target format10. III.

MIGRATION IN CLOUD

We define as heterogeneous Cloud migration the process of transferring processes and workloads between Cloud providers that may differ in architectures, image and disk formats and hypervisors. Next we present three VM migration categories that differ in terms of what part of the software is migrated (e.g. the whole image or just the application process or a container). These are (a) migrating a complete image of the VM between two Clouds, (b) migrating a process from one 7

https://www.docker.com 8 http://www.qemu.org 9 https://www.vmware.com/support/developer/ovf/ 10 https://www.vmware.com/support/developer/ovf/ovf350/ovftool350-userguide.pdf

VM to another and (c) migrating a container between two Clouds. Per category we discuss the key technical requirements. 1) Migrating a complete image of the VM VM image migration is the process of moving a running instance from one Cloud providers to another. Live migration requires a common infrastructure such as a shared storage between different Cloud providers, thus we focus on the stopand-copy that freezes the VM state and re-launching in another provider. In stop-and-copy VM migration, transferred data contain information for the operating system and application data that are in the virtual disk at that time and virtual versions of the supported devices (e.g. disks that are used by the VM). Since VMs are depended on the Cloud provider specifications (such as hypervisors), heterogeneous VM migration could be challenging. Basic requirements to taken into consideration are as follows. • Cloud provider specifications may differ in hypervisor type (e.g. OpenStack supports KVM and VMWare supports ESXi), image formats, container format, tools and APIs. To migrate a VM from OpenStack to VMWare requires changing the image format from QCOW to OVF, a process that is time consuming and requires the use of additional tools and storage. • A VM is tied up with Cloud provider specifications and it’s difficult to migrate to a Cloud provider of a different infrastructure without making basic configurations such as security groups, static float IPs and virtual networks (any network interfaces that have been created by the user at that time). To achieve VM migration between heterogeneous Cloud environments we develop a middleware to bridge all these specification gaps among Cloud providers. 2) Migrating a process between two VMs The method of moving a running application to a different host referred as process migration and can be achieved using various tools such as CRIU 11 . CRIU is a software tool for Linux operating systems to freeze a running application (or part of it) and checkpoint it as a collection of files on the disk. The checkpoint process saves a snapshot of the application’s state in the disk. It is generally used to increase fault-tolerance so long running applications will not need to restart from the beginning. Specifically, the tool stores a state of a process and restores it as it was during the time of the freeze on the same or another host with the initial process ID. CRIU separates the application from the underlying OS and freezes all the processes that are associated with the process. Information related to the process is stored in one or more image files that eventually could be migrated to another VM. These image files contain information, such as memory maps, pipes, file descriptors, inter-process communication and other. Basic requirements for process migration include the following.

• Depending on the application functionality, kernel configurations are required. Checkpoint process is not a straightforward process and differs depending on the application to be migrated and requires configuration of operating system related parameters. • Beyond kernel configuration and the appropriate checkpoint parameters, CRIU limitation is that for a successful migration both the source and destination system must have the same versions of libraries. • The process restore phase fails if the process ID is in use on destination system. Furthermore, checkpoint procedure freezes the process and its process tree; making impossible to checkpoint and restore a process on its own. 3) Container-based Migration Container-based migration is a software level migration method. In contrast to VM applications where the host operating system and some middleware are shared and transferred, in container-based migration only the application and some system libraries are migrated. It supports decomposition of the VM parts, that is an essential VM migration process since it supports splitting large monolithic applications to smaller parts. Basic requirements include. • Applications designed for containers are forced to be compatible with most systems that deploy applications in containers. • Docker uses the CRIU tool for checkpoints and it restores a container similar to migrating a process. IV.

VM MIGRATION SERVICE

Heterogeneous VM migration is a complicated process since a VM is transferred from a Cloud environment to another with different architecture, protocols and standards. Here we propose the use of a service middleware to bridge the different configuration gaps between these environments. The migration restrictions could be on the type of hypervisor, image and virtual machine description (meta-data) that each platform supports. We develop a VM migration service appropriate for homogeneous and heterogeneous Cloud environments. To demonstrate heterogeneity, the service utilizes the Openstack and VMware’s REST API to allow users to perform migration. The user uses a web interface to provide required information that will be used by the service for a successful migration. Figure 1 demonstrates the VM migration processes that include the migration tool, the image conversion tool and the instance configuration tool.

• Process migration is a kernel level migration and a technical constraint is that requires a Linux kernel of version 3.11 or newer, with some specific options set. 11

https://criu.org

Figure 1: VM migration process using the proposed tool

The user interacts with the service through the front-end interface and the service that performs VM migration. The migration service makes use of three sub-modules as follows. The “Migration Tool” that is responsible for connecting to the source Cloud provider and makes the required HTTP calls for collecting essential information (e.g. VM size, security groups, public keys etc.), the “Image Conversion Tool” that is responsible for converting images to the required format of the target Cloud provider and the “Instance Configuration Tool” that connects to the target system, configures and starts the new VM. The following list details the specification of each component. • The UI allows user to interact with the service and provide essential information (e.g. user credentials, tenant ID etc.). • The migration tool performs two processes, firstly the download procedure from source Cloud and the upload procedure to target Cloud. The tool guides the user to perform an instance migration between two homogenous or heterogeneous Clouds following few simple steps. On each step, the user provides required information and the tool performs the corresponding action on the appropriate Cloud platform. • The image conversion tool is responsible to convert the downloaded image to a portable format that matches the target Cloud configuration to be uploaded. • The instance configuration tool performs an action on target Cloud to initiate the VM. Specifically, the tool uploads a XML file (called OVF descriptor) on the target Cloud with initial instance details. • The back-end service offers required functionality as it temporarily stores the image while the image conversion tool changes to the required format. The current tool version offers these functionalities for OpenStack platforms (e.g. Rackspace, FIWARE etc.) and VMWare environments. Next we demonstrate the different steps for each case. A. Configuring Homogeneous Migrations The homogeneous migration process involves the following steps. • User Authentication: The user provides credentials and tenant name to be authenticated on source Cloud. Authentication generates a unique token for each and used for the rest of the HTTP calls. • Get Instances: The service retrieves all existing instances registered by the user. • Get Images: The service retrieves a list of public images and images registered to this user including snapshots. • Create Snapshot: User selects an existing running instance for migration. The service creates a snapshot that makes a copy of this instance. • Download Image: User selects an existing public image or a snapshot for downloading. • Authentication: User provides credentials and tenant names of the target Cloud. • Select Region Requirements: If appropriate the user selects the region to transfer the selected VM.

• Upload: The user provides details of the new VM e.g. name, and then the service performs actions automatically, to upload the selected image or snapshot. The service creates a new blank image with the name provided by the user. Finally, it uploads the data of the downloaded image to the blank image and launches the VM. B. Configuring Heterogeneous Migrations The heterogeneous migration process is similar to the homogenous and involves the same steps from 1 to 6. The rest are as follows. • Creation of a VM in VMWare platform: The user creates a new VM that is necessary to deploy a new virtual machine. • Upload of the OVF descriptor: VMware VM operates on the Open Virtualization Format (OVF) and also are exported in OVF format. User uploads the OVF descriptor to initiate this VM template that contains virtual machine’s meta-data such as hardware configuration and virtual machine image size. • Upload Image: Τhe user uploads the reference file (VMDK image) and user’s VM is ready to deploy on VMWare source Cloud. • The service performs automatically the conversion of downloaded image to VMDK format. V. IMPLEMENTING THE MIGRATION SERVICE In this section we present the implementation of our tool and the performance analysis. The OpenStack API usage is based on [1]. Also, the implementation aspects of the homogeneous VM migration can be found in [2]. The heterogeneous migration involves HTTP calls between OpenStack and VMWare APIs. The tool follows the next interactions between the two platforms. Step 1: The service performs authentication in the OpenStack Cloud, in which its the identity service generates and returns a token that represents the authenticated identity of a user and grants authorization on a specific project or domain. The next demonstrates the HTTP POST call request. URL: http://SOURCE_CLOUD_IP:PORT/v2.0/tokens Headers :{“Content-Type”: “application/json”} Body: {"auth": {"tenantName": "user Cloud","passwordCredentials": {"username": "[email protected]","password": "********" }}}

Step 2: The service retrieves the list of VM owned by the user on a specific region as shown in the next GET call. URL:http://SOURCE_CLOUD_IP:PORT/v2/$tenant-id/servers Headers: {“Content-Type”: “application/json”, “X-Auth-Token”: “$token”}

Step 3: The user selects one VM (running or idle), which she/he wants to migrate, and creates a snapshot of it as shown in the next POST request. URL:http://SOURCE_CLOUD_IP:PORT/v2/$tenant_id/servers/$ser ver_id /action Headers:{“Content-Type”:“application/json”,“X-Auth-Token”: “$token”} Body:{"createImage": {"name": "Snapshot", "metadata": {}}}

Step 4: The service retrieves and displays the list of images. This list contains public images that published either from other users, are private or are default by the Cloud platform as shown in the next GET request. URL: http://SOURCE_CLOUD_IP:PORT/v2/$tenant_id/ images Headers: {“Content-Type”: “application/json”, “X-Auth-Token”: “$token”}

Step 5: The user selects an image that she/he wants to download, in this case it is the snapshot that created earlier based on the next GET request. URL: http://SOURCE_CLOUD_IP:PORT/v2/images/$ima ge_ id/ file Headers: {“Content-Type”: “application/json”, “X-Auth-Token”: “$token”}

Step 6: The service connects to the VMWare Cloud and gets an authentication token to be used as a header in all subsequent API calls using the following POST call. URL: http://TARGET_CLOUD_IP/api/sessions Headers: {“Accept”: “application/*+xml;version=5.1”} Auth Headers: {username@organization_name:password}

Step 7: The service retrieves the organization id from the authentication response body and makes a get request to the target Cloud with this organization id as URL parameter as shown in the following GET request. URL http:// TARGET_CLOUD_IP/api/org/$organization _id Headers: {“Accept”:“application/*+xml;version=5.1”,“x-vCloudauthorization”: “$token” }

Headers: {“Accept”:“application/*+xml;version=5.1”,“x-vCloudauthorization”: “$token” }

Step 10: The service converts the downloaded image to a portable format (vmdk) for the target Cloud using the QEMU tool. The response body of “vApp” request also contains the link for the reference file. If the upload of the OVF descriptor succeeds, the user needs to upload the reference file and the snapshot to the target Cloud as shown in the next PUT request. URL: http:// TARGET_CLOUD_IP/api/$file_link Headers: {“Accept”:“application/*+xml;version=5.1”,“x-vCloudauthorization”: “$token” }

VI. PERFORMANCE ANALYSIS The proposed service allows homogenous and heterogeneous VM migrations. In this section we present the evaluation of the performance analysis with regards to the time needed to perform specific tasks in different Cloud platforms. Specifically, homogeneous migration is implemented to perform migration between two OpenStack environments where the IntelliCloud12 of the Technical University of Crete, Greece is the source Cloud and the FIWARE13 platform is the target Cloud. The heterogeneous migration is implemented to perform migration between OpenStack and a VMWare public Cloud provider. The performance evaluation of the service involves two experimental use cases that demonstrate the time required for each instance migration procedures. The homogeneous process, the user performs a migration of an instance from IntelliCloud to FIWARE system. Figure 2 demonstrates the homogeneous VM migration and the times required to execute the different calls of Section III B.

Step 8: The service sends the following POST request to create a new empty image that will host the source Cloud VM. URL: http:// TARGET_CLOUD_IP/api/$new_vdc_link Headers: {“Accept”:“application/*+xml;version=5.1”,”ContentType”: “application/vnd.vmware.vCloud.uploadVAppTemplate Params+xml”,“x-vCloud-authorization”: “$token” }

Next, the service extracts the link from the previous response body, and makes a POST request to initiate the new VM calles “vApp” with the following properties. BODY

Ubuntu vApp Template

Step 9: The response body of the new “vApp” request contains the link for the OVF descriptor. This is an XML file that contains initial instance properties such as virtual disk name and size, virtual disk information (capacity), virtual machine information (name, OS type) and virtual hardware requirements (resources, controllers, network). The next demonstrates the PUT request to the VMWare Cloud. URL:

http:// TARGET_CLOUD_IP/api/$ovf_link

Figure 2: VM migration times for different service HTTP calls in homogeneous OpenStack systems

The x-axis includes the different actions from Section III C and the y-axis is the HTTP response time in milliseconds from each Cloud provider. We perform two VM migrations, for an image of Centos 7 (958.4 MB) and for an image of Ubuntu 12.04LTS-64 (243.6 MB) and we deploy two small flavors VMs (1 VCPU, 20 GB disk and 2GB RAM). We can observe that the most time is spending during the downloading and 12

http://Cloud.intelliCloud.tuc.gr 13 https://www.fiware.org

uploading phase of the image between the two Clouds. The snapshot creation phase is also fast (executed in less that 3 seconds) if we consider that at that stage the VM is stopped and a new image is generated based on the current VM running processes. The rest of the HTTP calls are executed very fast, in less than 1.5 second. Secondly, in the heterogeneous procedure, the user performs a migration of an instance from FIWARE to VMWare based system. Figure 3 demonstrates the heterogeneous VM migration and the HTTP calls as described in Section III C. We used an image of Centos 7 (896.6 MB) and an image of Ubuntu 12.04 (458.8 MB) and we deploy two small flavors VMs (as in previously).

OpenStack and VMWare based platforms. The fear of vendor lock-in is often cited as a major obstacle to Cloud service adoption. We discussed the technical parameters to be tuned depending on different types of virtualization engines that are based on different hypervisors and supported disk formats. The migration toolkit takes a backup of the VM that contains all the information needed and deploys an identical VM containing the same configuration. The toolkit provides features such as interoperability between OpenStack and VMWare based Clouds as it supports migration of running instances and reinstantiation while it supports a variety of images (QCOW2, AMI, VMDK and other). The experimental analysis shows fast VM migration as we achieve two milliseconds response time for each HTTP call to OpenStack and VMWare, excluding the image download and upload phases. In future, we will focus on the process migration that abstracts applications from the underlying OS and could bridge the heterogeneity gaps. Process migration will reduce more the total migration times since it will transfer only the selected process along with related sub-processes instead of the whole VM. REFERENCES [1]

Figure 3: VM migration times for different service HTTP calls in heterogeneous systems (OpenStack and VMWare)

Similar to previous experiment, Figure 3 demonstrates the time required to execute the various calls in a heterogeneous Cloud environment. The x-axis includes the HTTP calls as described in Section III D and the y-axis is the HTTP response time in milliseconds from each Cloud provider. Here the size of the snapshot is bigger than the size of the initial image. Also, the service executes most of the API calls related with configurations but the time required for downloading and uploading increases significantly the total time of migration, actions that depend to the image size and on the bandwidth speed. Other than this, both experiments show that the service executes most of the API calls related with configurations in reasonable time frames. To conclude, in this experiment we presented the (a) homogeneous VM migration executed between two OpenStack platforms and (b) heterogeneous between OpenStack and VMWare. We observe that 7 out of 9 calls are executed at around 2 seconds, while the rest 2 calls that are related with download and upload process are slow (since this is considered as resonable since it includes data transfer of two VMs of 896.6MB and 458.8MB respectively). We can sumarize that the migration times are directly proportional to the VM sizes (because the download and upload phases are slow) and are slightly affected by the bandwidth speed and the physical distance of the datacenters. VII. CONCLUSIONS This work presented a heterogeneous VM migration tool that allows VM transferring between OpenStack as well as

T. Fifield, D. Fleming, A. Gentle, L. Hochstein, J. Proulx, E. Toews, and J. Topjian. (2014) OpenStack Operations Guide. O’Reilly Media, Inc., 1st edition. [2] L. Vakanas, S. Sotiriadis and E. Petrakis (2015) Implementing the Cloud Software to Data approach for OpenStack environments, Adaptive Resource Management and Scheduling for Cloud Computing, Held in conjunction with PODC-2015, Donostia-San Sebastián, Spain, on July 20th, 2015 [3] D. Petcu.Consuming resources and services from multiple Clouds . J . Grid Comput. , 12(2):321–345, June 2014. 
 [4] D. Petcu. 2013. Multi-Cloud: expectations and current approaches. In Proceedings of the 2013 international workshop on Multi-Cloud applications and federated Clouds (MultiCloud '13). ACM, New York, NY, USA, 1-6. [5] S. Sotiriadis, N. Bessis, A. Anjum, and R. Buyya. An inter-Cloud metascheduling (icms) simulation framework: Architecture and evaluation. IEEE Transactions on Services Computing. [6] K. Nagin, D. Hadas, Z. Dubitzky, A. Glikson, I. Loy, B. Rochwerger, and L. Schour. 2011. Inter-Cloud mobility of virtual machines. In Proceedings of the 4th Annual International Conference on Systems and Storage (SYSTOR '11). ACM, New York, NY, USA, , Article 3 , 12 pages. [7] M. Armbrust, A. Fox, R. Griffith, A. D. Joseph, R. Katz, A. Konwinski, G. Lee, D. Patterson, A. Rabkin, I. Stoica, and M. Zaharia. 2010. A view of Cloud computing. Commun. ACM 53, 4 (April 2010), 50-58. [8] Ankita Desai, Rachana Oza, Pratik Sharma, Bhautik Patel (2013) Hypervisor: A Survey on Concepts and Taxonomy, International Journal of Innovative Technology and Exploring Engineering (IJITEE) ISSN: 2278-3075, Volume-2, Issue-3, February 2013 [9] OpenStack Opensource Cloud Computing System, Available in https://www.openstack.org, Accesed: April 28, 2017 [10] VMWare Documentation, Available in: https://www.vmware.com/support/pubs/, Accessed: April 28,2017

APPENDIX The VM migration toolkit is available as open source software in https://github.com/dkargatzis/cloud-migration.