Roystonea: A Cloud Computing System with

Roystonea: A Cloud Computing System with Pluggable Component Architecture Chao-En Yen

Pangfeng Liu

Jan-Jan Wu

Graduate Institute of Networking and Multimedia, National Taiwan University, Taipei, Taiwan, Email: [email protected]

Department of Computer Science and Information Engineering, Graduate Institute of Networking and Multimedia, National Taiwan University, Taipei, Taiwan, Email: [email protected]

Institute of Information Science, Academia Sinica, Taipei, Taiwan, Email: [email protected]

Abstract—A Cloud computing system provides infrastructure layer services to users by managing virtualized infrastructure resources. The infrastructure resources include CPU, hypervisor, storage, and networking. Each category of infrastructure resources is a subsystem in a cloud computing system. The cloud computing system coordinates infrastructure subsystems to provide services to users. Most current cloud computing systems lacks pluggability in their infrastructure subsystems and decision algorithms, which restricts the development of infrastructure subsystems and decision algorithms in cloud computing system. A cloud computing system should have the flexibility to switch from one infrastructure subsystem to another, and one decision algorithm to another with ease. This paper describes Roystonea, a hierarchical distributed cloud computing system with pluggable component architecture. The component pluggability ability gives administrators the flexibility to use the most appropriate subsystem as they wish. The component pluggability of Roystonea is based on a specifically designed interfaces among Roystonea controlling system and infrastructure subsystems components. The component pluggability also encourages the development of infrastructure subsystems in cloud computing. Roystonea provides a testbed for designing decision algorithms used in cloud computing system. The decision algorithms are totally isolated from other components in Roystonea architecture, so the designers of the decision algorithms can focus on algorithm design without worrying about how his algorithm will interact with other Roystonea components. We believed that component pluggability will be one of the most important issues in the research of cloud computing system.

I. I NTRODUCTION Cloud computing system provides infrastructure layer services to users by managing virtualized infrastructure resources. The infrastructure resources in a data center include CPU hypervisor, storage, and networking. Each category of infrastructure resources is a subsystem in a cloud computing system. The cloud computing system coordinates infrastructure subsystems to provide services to users. Cloud computing service is usually provided as a lease of virtual machine, and efficient management of virtual machine depends on the close cooperation among the infrastructure subsystem managers.

An infrastructure subsystem manager needs to handle a large number of physical machines, and communicate with other subsystem managers and the cloud computing system. For example, when a user requests virtual machines for computation from a cloud computing system, the cloud computing system first searches for available physical machines that can host the required virtual machines. Then the cloud computing system instructs the hypervisor manager to create virtual machines on the target physical machines. At the same time, the cloud computing system instructs the storage subsystem to provide virtual machine image for creating the virtual machines. The storage subsystem manages storage of a data center, and it also provides a distributed file system among physical machines. The storage subsystem is in charge of managing the virtual machine image files of the virtual machines the cloud computing system provides. These image files must be retrieved from disk in order to create virtual machines. The storage subsystem also provides a distributed file system that enables virtual machine to migrate among physical machines. The migration mechanism helps balance the workloads among physical machines. After the virtual machines are created, the network subsystem manager maps virtual network interface to appropriate physical network interface. This network mapping establishes the network topology requested by users. The virtual network topology provides a virtual local network for users and allows users to access their virtual machines. This example illustrate that various infrastructure subsystem managers – hypervisor, storage, and network, must closely cooperate to provide services in a cloud computing system. Most current cloud computing systems lacks pluggability of their infrastructure subsystem managers. By pluggability we mean the ability to switch from one infrastructure subsystem manager to another. Most current cloud computing systems use their favorite infrastructure subsystem manager as well as their decision algorithms, and do not provide an interface so that users can build and use their own subsystem manager and decision algorithms.

The lack of pluggability restricts the development of subsystem manager and decision algorithms in cloud computing systems. For example, if we want to develop power consolidation algorithm for virtual machine provisioning, we must be able to plug our new power consolidation scheduler into a cloud computing system. Currently this replacement of scheduler demands serious efforts in modifying the source code of current cloud computing systems. If we want to experiment our new scheduler in multiple cloud computing systems, we have to go through this tedious modification multiple times, which is time consuming and error prone. The lack of pluggability also makes it impossible for a cloud computing system administrator to choose an infrastructure subsystem manager to suit his needs. Namely the administrator is stuck with the infrastructure subsystem managers in his current cloud computing system. The cloud computing system should have the flexibility to switch from one set of infrastructure subsystem and decision algorithms to another with ease. This paper describes Roystonea, a hierarchical distributed cloud computing system with pluggable component architecture. The component pluggability ability gives administrators the flexibility to use appropriate subsystem as they wish. The component pluggability ability also encourages the development of infrastructure subsystems in cloud computing. We believed that component pluggability will be one of the most important issues in the research of cloud computing system. The component pluggability of Roystonea is based on a specifically designed interfaces among Roystonea controlling system and infrastructure subsystems components. Roystonea controls its subsystems by a set of interfaces, which serve as bridges between the cloud computing system and subsystem components. We first carefully define every task a cloud computing system must perform. By this clear definition of task functionality we are able to define the relation between cooperative infrastructure subsystems and Roystonea controlling system. Each infrastructure subsystem only needs to cooperate with those components that it interacts with, not the whole cloud computing system. Roystonea also provides a testbed for designing decision algorithms used in subsystems. For example, we may design deployment algorithm used to deploy virtual machines to physical machines, or power conservation algorithm that packs running virtual machines together, or image file managing algorithms that speed up the process of virtual machine image retrieval. These algorithms are the crucial to the performance of infrastructure subsystem. The rest of the paper is organized as follows. Section II describes related works. Section III explains the design principles of Roystonea. Section IV describes the architecture of Roystonea. Section V shows our experience of Roystonea. Section VI gives concluding remarks and introduces further research issues of Roystonea.

II. R ELATED W ORK Cloud computing systems is designed to manage infrastructures in data centers. Eucalyptus [1] is an open-source cloud computing system that manages computational and storage infrastructure. Eucalyptus provides a platform that is distributed, flexible, compatible and open to experimental instrumentation and study [2], [1]. Eucalyptus also implements an Infrastructure as a Service (IaaS) platform that is accessible via an API compatible with Amazon EC2 [3] and Amazon S3 [4]. OpenNebula [5] is a virtual infrastructure manager that organizations can use to deploy and manage virtual machines, either on local resources or external public clouds. OpenNebula automates disk image preparation and network configuration. OpenNebula also proposes the concept of an independent platform that disregards the underlying virtualization layer (e.g., Xen [6], KVM [7]). OpenNebula also proposed Haizea [8], a resource lease manager that acts as a scheduling back end for OpenNebula, providing leasing capabilities such as advance reservations and resource preemption [8], [5]. Nimbus is an open-source toolkit focused on providing Infrastructure as a Service (IaaS) capabilities to the scientific community. Nimbus focus on enabling providers of resources to build private or community IaaS clouds [9], [10], [11]. There are many virtualization mechanisms that provide computer virtualization ability to enable multi-tenancy, and to isolate virtual machines in a cloud computing system. Xen [12], [6] is an open source hypervisor for the Intel x86 CPU, and it was publicly released in 2003. Xen allows several guest operating systems to run on a single host machine concurrently. Xen provides para-virtualization [12] mode to speed up the performance of the guest operating systems. Kernel-based Virtual Machine (KVM) [7] is a popular Linux kernel virtualization infrastructure that only supports full virtualization. Virtual machines run as user processes, and guest operating systems require hardware assistance to run correctly in privileged mode. A distributed file system manage the physical disks of all servers as a single file system, which can be shared by different physical machines during live migration [13]. GlusterFS [14] is an open source, distributed file system capable of scaling to several petabytes and handling thousands of clients. GlusterFS aggregates various storage servers into one large parallel network file system [14]. Ceph [15] is a distributed file system that provides reliability and scalability. Ceph also provides a new approach to handle data and metadata by replacing file allocation tables with generating functions [15]. Hadoop Distributed File System (HDFS) [16] is the primary storage system used by Hadoop applications. HDFS creates multiple replicas of data blocks and distributes them on compute nodes throughout a cluster to enable reliable, rapid computations [16]. It is crucial for a cloud computing system to monitor system resources efficiently and accurately so that it can manage the resources. Ganglia [17] is a scalable hierarchical distributed

monitoring system for clusters. Ganglia relies on a multicastbased protocol to monitor the status of machines [17]. Nagios [18] is a network monitoring software application. Nagios also provides alert mechanism for system administrators. Network management is a fundamental requirement of a cloud computing system. Iptables [19] is a command line program for configuring Linux IPv4 packet filtering rule set. Iptables can add, remove, or modify the rules of the packet filter. Iptables routes network packages through multiple levels of network interface. Open vSwitch [20] is a multilayer open source software virtual switch. Open vSwitch can operate both as a software switch running within a hypervisor, and as the control stack for switching hardware. Open vSwitch has been ported to multiple virtualization platforms [20]. III. D ESIGN P RINCIPLE There are three guiding principles in the design of Roystonea – pluggable components interfaces and a hierarchical architecture based on logical racks, and virtual machine based cloud services. A. Pluggable Component Interface Roystonea is a cloud computing system with pluggable component design. The component pluggability is based on interfaces between components. 1) Pluggable Component: The component pluggability of Roystonea is based on a specifically designed interfaces between Roystonea controlling system and infrastructure subsystems components. Roystonea controls its subsystems by a set of interfaces, which serve as bridges between the cloud computing system and subsystem components. We first carefully define every task a cloud computing system must perform. By this clear definition of task functionality we are able to define the relation between cooperative infrastructure subsystems and Roystonea controlling system. Each infrastructure subsystem only needs to cooperate with those component that it interacts with, not the whole cloud computing system. Roystonea also provides a testbed for designing decision algorithms used in subsystems. For example, we may design deployment algorithm used to deploy virtual machines to physical machines, or power conservation algorithm that packs running virtual machines together, or image file managing algorithms that speed up the process of virtual machine image retrieval. These algorithms are the crucial to the performance of infrastructure subsystem. The pluggability of Roystonea gives a system administrator the flexibility to choose the most suitable components so as to provide the most appropriate functionalities under different circumstances. A good cloud computing system should allow system administrators to use any infrastructure subsystem component that he thinks most fit. There are various options of infrastructure subsystems, and different subsystem can provide different functionalities. For example, KVM [7] is much easier to install than Xen [6] is, but Xen has better performance in terms of speed of para-virtualization [12].

Another circumstance that affects the choice of components is the number of servers in a data center. For instance, NFS [21] is suitable for small scale clusters, such as machines within a single rack. On the other hand, NFS has very poor performance when the number of servers is hundred or thousand, where a distributed file system should be deployed. Therefore, we do need the flexibility in choosing infrastructure subsystem for a cloud computing system. 2) Component Interface: Interface is the key idea behind the pluggability of Roystonea. For the ease of replacing components, Roystonea uses interfaces to isolate the works of infrastructure subsystem from cloud computing system, so the interface works as the bridge between the cloud computing system and the user specified components. Each infrastructure subsystem does not need to communicate with every part of the cloud computing system – it only needs to communicate with the corresponding interfaces. For example, without an interfaces a storage subsystem needs to communicate with the cloud computing system for managing virtual machine image files, it also needs to communicates with the monitoring subsystem for the information of available storage resource, it also needs to communicate with the user to provide storage services. Now with the help of interfaces each specifically declared interfaces is responsible for each of the task above – managing virtual machine image files, getting the information of available storage resources, and interacting with users. As the result, the communication between cloud computing system and its subsystems become much easier to handle. We separate the works in Roystonea into three major functional units – Virtual Machine Manager, Subsystem Manager, and Coordinator. Each of them contains several pluggable components, as shown in Figure 1. We will describe the three and other components in Section IV later. These components communicate with others through interfaces. The interface of every pluggable component is precisely defined so that one can easily develop a Roystonea subsystem component by just following the interface.

Fig. 1.

Roystonea interface

B. Logical Rack Hierarchy The hierarchy design of Roystonea is based on logical racks. The concept of logical rack is based on the network topology of physical machines, and physical machines connected to the same switch form a logical rack. There are two advantages of our logical rack based hierarchical design. First the hierarchical architecture design isolates heavy communication traffic within a logical rack. If we place a set of virtual machines that need to communicate with each other frequently into a logical rack, the traffic among them will only be within the switch. Second, a logical rack is the basic unit for monitoring system and forming shared file system in Roystonea, which allows simple implementation and delivers good performance, as we will explain in the rest of this section. The distributed monitoring system in Roystonea monitors the status of physical machines within a logical rack as a basic unit. In contrast Ganglia [17], a scalable distributed monitoring system monitors physical machines, is based on a hierarchy of physical machine locations. We believe that physical machines at the same switch, not within the same rack, should form the basic unit for monitoring. For example, every node in the basic unit of Ganglia broadcasts its status to one another, so the system can query any node within the basic unit to find out the information of every node. If the basic unit spans across different switches, the broadcast traffics will be enormous. Therefore we believe that the basic unit of monitoring should be a logical rack. Roystonea also uses a logical rack as a basic unit to build shared file systems, as shown in Figure 2. For ease of migration, the virtual machine image files should be stored in a shared file system. There are two reasons of using logical racks in while building shared file system. First, the shared file system are only built within a logical rack, so the shared file system does not span across different switches. This will improve I/O performance of the shared file systems since if a shared file system spans across different switches, the network traffic among these different switches will degrade the performance of the shared file system. Second, despite that a shared file system has only limited storage, we can still place large scale data distributively in different virtual machines across different logical racks. Since Roystonea has a storage subsystem that communicates with different logical racks, the virtual machines can still communicate with each other, and the maximum amount of the data that could be stored in Roystonea will not be limited by the capacity of a single shared file system. Therefore, Roystonea does support parallel programming models that handle large data set, e.g., MapReduce, with ease. C. Hierarchical Architecture of Roystonea Roystonea has a three-level hierarchy – a cloud consists of several clusters, a cluster consists of logical racks, and a logical rack consists of physical machines connected to the same switch. Please refer to Figure 3 for an illustration. A node controller manages a physical machine. A rack controller controls the node controllers in the same logical rack. A cluster

Fig. 2.

Roystonea storage subsystem design

controller controls the rack controllers in a cluster. A cloud controller, which is one of the cluster controllers, controls all cluster controller.

Fig. 3.

The hierarchical architecture of Roystonea

1) Node Controller: Roystonea provides an interfaces between node controller and infrastructure subsystems. A node controller receives instructions from rack controller, and relay them to infrastructure subsystems for execution. 2) Rack Controller: A rack controller manages a logical rack in Roystonea. A rack controller receives instructions from cluster controller, and relays them to node controller for execution. Roystonea provides two interfaces – one for a rack controller to communicate with other rack controllers, and one for a rack controller to communicate with its cluster controller. 3) Cluster Controller: A cluster controller controls a cluster by managing the rack controllers of a cluster. Each cluster controller maintains resource status and they do not need to synchronize resource information with each other. Each cluster controller also maintains its own user accounts, and the cloud controller ensures that all users in the cloud will have unique user names. Roystonea also provides an interfaces for cluster controllers to communicate with each other. A user can use resources in different clusters by communicating with their cluster controllers. 4) Cloud Controller: A cloud controller is one of the cluster controllers in a cloud. A cloud controller only needs to maintain a mapping from a user name to the cluster this user belongs to. When a user requests resources, the cloud controller will look up this mapping table, and relay the

request to the cluster controller of the cluster this user belongs to. Since the cloud controller guarantees that every user name is unique cluster controllers do not need to synchronize user account information with each other. D. Virtual Machine Based Services Roystonea provides the following services based on virtual machines – Hadoop [22], HBase [23], MPI [24], Apache [25], and NIS [26]. We pack a service into a virtual machine prototype. When a user requests a service, Roystonea deploys virtual machines that provides the service the user requested, and configures the virtual machines for the user. When a user requests storage for a large amount of data, Roystonea sets up virtual machines and stores the data in these virtual machines distributively. By using virtual machines as storage those programming models that requires petabytes of data, for example Hadoop [22], can run on Roystonea.

Fig. 5.

Virtual machine management

We use an example to illustrate the interaction between user, the coordinator, the virtual machine manager and the hypervisor subsystem manager, and the hypervisor.

IV. A RCHITECTURE Roystonea is a hierarchical distributed cloud computing system with pluggable component architecture as shown in Figure 4. There are three major components in Roystonea – virtual machine manager, subsystem manager, and coordinator. Each of these components has several pluggable subsystem components. The virtual machine manager is responsible for creating, terminating, or consolidating virtual machines. The subsystem manager manage subsystems and controls the hardware of data center. The coordinator controls and integrates virtual machine manager and the subsystem manager. The coordinator has a database subsystem that keeps track of resource status, user accounts, etc., of Roystonea.

Fig. 4.

The components of Roystonea

A. Virtual Machine Manager The virtual machine manager receives requests from coordinator for the following three tasks. The first task is apply user specified decision algorithms to place virtual machines. The second task is to balance the workloads among physical machines. The third task is to consolidate the workloads of virtual machines. The virtual machine manager controls the hypervisor subsystem manager through subsystem manager interface, and the hypervisor controls virtual machines. That is, the hypervisor on each physical machine executes the command from the hypervisor subsystem. Please refer to Figure 5 for an illustration.

1) A user requests virtual machines through Roystonea web interface. The request is written into the database subsystem through Roystonea web interface. 2) The coordinator retrieves the request from the database subsystem, then instructs the virtual machine manager to create virtual machines. 3) The virtual machine manager applies a user specified placement algorithm to determine the physical machines for deploying the virtual machines. 4) The decision algorithm requests information from the database subsystem for the information it needs for making the decision. 5) The decision algorithm determines the physical machines for deploying virtual machines. 6) The virtual machine manager instructs the hypervisor subsystem manager to create virtual machines. 7) The hypervisor subsystem informs the hypervisor, e.g., Xen in Figure 5, to create virtual machines. The virtual machine manager provides a convenient test bed for designing and testing decision algorithms in Roystonea. Many Roystonea subsystems needs decision algorithms for scheduling resources. For example, the virtual machine manager needs a decision algorithm that given the request for virtual machines and the status of physical machines, computes the most suitable set of physical machines for deploying the virtual machines. Those decision algorithms are totally isolated from other components in Roystonea architecture, so the designers of the decision algorithms can focus on algorithm design without worrying about how his algorithm will work with other Roystonea components. The virtual machine management is events-driven for efficiency. For example, the monitoring system does not need to check resource status frequently, since the virtual machine manager is the only subsystem that could change the resource status in Roystonea. As a result the monitoring system only needs to update resource status when the virtual machine management has taken actions.

B. Subsystem Manager The subsystem manager provides interfaces for controlling actual hypervisor, network, system monitor, and storage subsystems used in Roystonea. Since the subsystem manager interact with actual subsystem through clear defined interfaces, we can replace component with ease and achieve pluggability, which is the most important goal in Roystonea. 1) Hypervisor Subsystem: The hypervisor subsystem manages CPUs and memory in order to provide computing services in a cloud computing system. These computing servers are provided as virtual machines, and the hypervisor subsystem creates virtual machines, manages virtual machines created, and provides them to users. We define the interaction between the virtual machine manager and the hypervisor subsystem manager as follows – the virtual machine manager makes decisions, which are implemented by the hypervisor subsystem manager. That is, the virtual machine manager and the hypervisor subsystem manager cooperates to manage virtual machines in Roystonea. The virtual machine manager decides the physical machines for deploying virtual machines, creates virtual machines, migrates virtual machines, and shuts down virtual machines. These virtual machine controlling instructions given to the hypervisor subsystem manager, and then relayed to the hypervisor on physical machines. Figure 5 illustrates the cooperation of the virtual machine manager and the hypervisor subsystem. The design goal of our current hypervisor subsystem is to keep the role of hypervisor as simple as possible. We believe that the hypervisor subsystem just needs to execute the instructions given by the virtual machine manager, and leave the deployment decisions to the virtual machine manager. This greatly simply the role of hypervisor, which was designed to create and run virtual machines only. In contrast many cloud computing systems [2], [5], [11], tie the deployment work to the embedded hypervisor, which makes it difficult to implement new deployment algorithms since the decision is tied with hypervisor. 2) Storage Subsystem: The storage subsystem manages storage resource in a cloud computing system. Roystonea separates the storage subsystem into two parts – storage subsystem manager and the physical storage system on each physical machine. a) Storage Subsystem Manager: The storage subsystem manager performs three tasks – managing storage resource on every physical machine, keeping track of file system metadata, and reclaiming space from files that are no longer used. •

•

The storage subsystem manager must manage storage resource on every physical machine. Nowadays most storage systems are distributed and virtualized, i.e., they consist of a large number of physical storage nodes that collectively provide a virtual file system. The storage subsystem manager must manage this file system efficiently and effectively. The storage subsystem manager must keep track of file system metadata, which other subsystems may query. For

example, when a hypervisor needs to attach a image file to a virtual machine, it needs to know where to find the image file, and it is the responsibility of the storage subsystem to give the exact location of the file. • The storage subsystem must reclaim space from unused files. When a lease of a virtual machine expires, the image file of that virtual machine is no longer in use, and we may reclaim the storage of these files. b) Physical Storage System: The physical storage system on physical machine is in charge of storing files, providing storage resource information, and tolerating faults by selfrecovering. We may choose from many possible file systems, each of them is capable of provide storage resources on the physical machines, to be the physical storage system. The physical storage system should be independent of the tasks of the storage subsystem manager, so we can plug in any storage system into Roystonea with ease. However, fault-tolerance is crucial in cloud computing system, because fault-tolerance provides high availability (HA) services, which is a key factor in cloud computing. As a result the choice of physical storage system must be consistent with the goal of high availability. The physical storage system must provide a shared file system in cloud computing system because it enables a hypervisor to migrate a virtual machine among physical machines. We would like to migrate virtual machines because it balances the load of physical machines. The migration also achieves power conservation by consolidate virtual machines so that the number of active physical machines can be reduced. Roystonea use a hybrid approach to migrate virtual machines without the performance penalty if the image files are stored in a single distributed file system. For every rack Roystonea builds a distributed file system among all machines in the same rack, so that the file system can be shared among all machines within this rack. That is, storage of machines in the same rack are integrated as a distributed shared file system. As a result virtual machines in the same rack can migrate to any other machines within the rack, because they can see the same shared file system. The migration enables the ability of load balance, or power consolidation. Figure 2 illustrates how this hybrid approach works in Roystonea. 3) Networking Subsystem: The network subsystem provides access to virtual machines for users by maintaining public IPs of virtual machines. However, public IP is a very limited resource for a cloud computing system. In order to use limited public IPs effectively, Roystonea maps the ports of virtual machines under private IP to ports of front-end machines of the data center. Users can then access their virtual machines by setting up an SSH connection through the ports of a cloud computing system front end. 4) Monitoring Subsystem: The monitoring subsystem constantly monitors the status of physical and virtual machines. The status information include the usage of CPU, memory, and disk, etc. The monitoring subsystem keeps track of these information in a database within a data center, and Roystonea depends on this database to manage the entire data center. For example, when the monitoring subsystem discover that a

C. Coordinator The coordinator receives user requests from Roystonea web interface and then informs necessary Roystonea components to serve the requests. The coordinator serves the requests by communicating with both the virtual machine manager and subsystem manager, and maintaining a information database. For example, when the coordinator receives a user request for creating a virtual machine, it first requests the virtual machine manager for a virtual machine, then it request the storage manager for the image file for the virtual machine being requested. The coordinator also informs the monitoring manager to monitor the virtual machine just created. The coordinator also maintains an information database for a data center. The information database provides a query interface for other components to gather information they need. 1) Database Subsystem: The database subsystem maintains informations of a data center because a cloud computing system relies on these information to manage the data center effectively. For example, the database subsystem keeps track of user accounts, user credential, user history, and machine status for both physical and virtual machines – all are useful information for managing a data center. The database subsystem helps other components manage a data center by providing a query interface for other components. Since all Roystonea components are pluggable, every component can query information from the database subsystem via a pre-defined interface. 2) Virtual Machine Prototype Database: Roystonea provides the following services based on virtual machines – Hadoop [22], HBase [23], MPI [24], Apache [25], and NIS [26]. When a user request for one of the these services, Roystonea deploys and configures that type of virtual machine for the user. These specifically prepared virtual machines with applications in them are called prototype virtual machine images. The virtual machine prototype database manager manages prototype virtual machine image files. The virtual machine prototype database works with the storage subsystem to provide storage for prototype virtual machine images, which Roystonea will use to create virtual machines. The virtual machine prototype database also allows users to upload their own virtual machine image files into Roystonea, like Amazon EC2 [3] does.

V. E XPERIENCE Roystonea has been developed for more than one year, and the first Roystonea system became operational at National Taiwan University in September, 2010. This Roystonea system has over 70 registered users working on various scientific disciplines. This Roystonea system has been providing Hadoop cluster [22], HBase cluster [23], MPI computation cluster [24], Apache web services [25], and storage service, to researches in National Taiwan University [27], [28], [29]. The applications running on Roystonea include large-scale image object retrieval [27], graph-based learning for largescale image datasets [28], and large-scale ADL analysis [29]. In particular, Roystonea provides a cloud platform for the integrated project – “The Elder Health Care” of the College of Electrical Engineering and Computer Science, National Taiwan University. Our Roystonea system was developed and now runs on a cluster donated by TrendMicro Inc. [30]. The cluster has fifteen servers. Each server has two quad core CPUs, twenty four GB memory and three 1.5 TB hard disks. Since September 2010 Roystonea has hosted over 1500 virtual machines providing service, such as Hadoop [22], HBase [23], MPI [24], Apache [25], NIS [26], and storage services. In average Roystonea hosted about forty virtual machines, which use 180GB of memory and 7.5TB of storage, per day. Figure 6, 7, and 8 illustrate the monthly resource usage history of Roystonea development platform since September, 2010. The number of Virtual Machine 350 Number 300

250

Number

virtual machine has crashed, it informs Roystonea about this crash event, and Roystonea will create and deploy a backup virtual machine. By doing so the cloud computing system could maintain high availability of its services. The performance of the monitoring subsystem must be scalable. That is, the system must maintain consistent performance even when the number of machines being monitored increases. In a large data center the number of physical machine could be tremendous, so it crucial to have a scalable and distributed monitoring system for such data centers.

200

150

100

50

0 Sep

Oct

Nov

Dec

Jan

Feb

Mar

Apr

May

June

Month

Fig. 6. The number of leased virtual machine of Roystonea development platform

VI. C ONCLUSIONS

AND

F UTURE W ORK

A cloud computing system coordinates infrastructure subsystems to provide services to users. The services is usually provided as a lease of virtual machine, and efficient management of virtual machine depends on the close cooperation among the infrastructure subsystem managers. Most current cloud computing systems lacks pluggability of their infrastructure subsystems and decision algorithms. The lack of pluggability also makes it impossible for a cloud computing system administrator to choose an infrastructure subsystem manager to suit his needs. Most current cloud

Memory usage

design of new policy in the placement of image file. We also would like to set up more Roystonea sites in National Taiwan University and collect feedbacks from users and system administrators.

180000 Memory 160000 140000

GBytes*hours

120000 100000

R EFERENCES

80000

[1] “Eucalyptus project,” http://open.eucalyptus.com/. [2] D. Nurmi, R. Wolski, C. Grzegorczyk, G. Obertelli, S. Soman, L. Youseff, and D. Zagorodnov, “The eucalyptus open-source cloud-computing system,” in Proceedings of the 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid. Washington, DC, USA: IEEE Computer Society, 2009, pp. 124–131. [3] “Amazon elastic compute cloud,” http://aws.amazon.com/ec2/. [4] “Amazon simple storage service,” http://aws.amazon.com/s3/. [5] “Opennebula project,” http://opennebula.org/. [6] “Xen,” http://www.xen.org/. [7] “Kvm,” http://www.linux-kvm.org/. [8] B. Sotomayor, R. Montero, I. Llorente, and I. Foster, “Virtual infrastructure management in private and hybrid clouds,” Internet Computing, IEEE, vol. 13, no. 5, pp. 14–22, 2009. [9] K. Keahey and T. Freeman, “Science clouds: Early experiences in cloud computing for scientific applications,” in Proceedings of Cloud Computing and Its Applications. Chicago, Illinois, USA, 2008. [10] K. Keahey, M. Tsugawa, A. Matsunaga, and J. Fortes, “Sky computing,” Internet Computing, IEEE, vol. 13, no. 5, pp. 43–51, October 2009. [11] “Nimbus project,” http://www.nimbusproject.org/. [12] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield, “Xen and the art of virtualization,” in Proceedings of the nineteenth ACM symposium on Operating systems principles. ACM, 2003, pp. 164–177. [13] C. Clark, K. Fraser, S. Hand, J. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield, “Live migration of virtual machines,” in Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation - Volume 2. USENIX Association, 2005, pp. 273–286. [14] “Glusterfs,” http://www.gluster.org/. [15] S. Weil, S. Brandt, E. Miller, D. Long, and C. Maltzahn, “Ceph: a scalable, high-performance distributed file system,” in Proceedings of the 7th symposium on Operating systems design and implementation. Berkeley, CA, USA: USENIX Association, 2006, pp. 307–320. [16] “Hadoop distributed file system,” http://hadoop.apache.org/hdfs/. [17] M. Massie, B. N. Chun, and D. Culler, “The ganglia distributed monitoring system: Design, implementation, and experience,” Parallel Computing, vol. 30, no. 7, pp. 817–840, 2004. [18] “Nagios,” http://www.nagios.org/. [19] “iptables,” http://www.netfilter.org/projects/iptables/index.html. [20] “Open vswitch,” http://openvswitch.org/. [21] D. Hitz, J. Lau, and M. Malcolm, “File system design for an nfs file server appliance,” in Proceedings of the USENIX Winter 1994 Technical Conference on USENIX Winter 1994 Technical Conference. Berkeley, CA, USA: USENIX Association, 1994, pp. 19–42. [22] “Hadoop,” http://hadoop.apache.org/. [23] “Hbase,” http://hbase.apache.org/. [24] “Mpi,” http://www.mcs.anl.gov/research/projects/mpi/. [25] “Apache,” http://httpd.apache.org/. [26] “Nis,” http://www.linux-nis.org/. [27] Y. Kuo, H. Lin, W. Cheng, Y. Yang, and W. Hsu, “Unsupervised auxiliary visual words discovery for large-scale image object retrieval,” in Proceedings of the IEEE Conference on Computer Vision and Patern Recognition. IEEE, 2011. [28] W. Lee, G. Wu, L. Hsieh, and W. Hsu, “Multi-layer graph-based semisupervised learning for large-scale image datasets using mapreduce,” in Proceedings of the 34th Annual ACM SIGIR Conference. ACM, 2011. [29] Y. Huang, Y. Ho, C. Lu, and L. Fu, “A cloud-based accessible architecture for large-scale adl analysis services,” in Proceedings of the 4th International Conference on Cloud Computing. IEEE, 2011. [30] “Trendmicro,” http://us.trendmicro.com/us/home/indexnight.html. [31] T. Richardson, Q. Stafford-Fraser, K. R. Wood, and A. Hopper, “Virtual network computing,” Internet Computing, IEEE, vol. 2, no. 1, pp. 33–38, January 1998. [32] “Virtual private network,” http://en.wikipedia.org/wiki/Virtual private network. [33] “A core mpls ip vpn architecture,” http://tools.ietf.org/html/rfc2917.

60000 40000 20000 0 Sep

Oct

Nov

Dec

Jan

Feb

Mar

Apr

May

June

Month

Fig. 7.

Memory usage history for our experiment environment Disk usage

7000 Disk 6000

TB*hours

5000

4000

3000

2000

1000

0 Sep

Oct

Nov

Dec

Jan

Feb

Mar

Apr

May

June

Month

Fig. 8.

Disk usage history for our experiment environment

computing systems use their favorite infrastructure subsystem manager as well as their decision algorithms. The cloud computing system should have the flexibility to switch from one set of infrastructure subsystem and decision algorithms to another with ease. We developed Roystonea, a hierarchical distributed cloud computing system with pluggable component architecture. The component pluggability ability gives administrators the flexibility to use appropriate subsystem as they wish. The component pluggability ability also encourages the development of infrastructure subsystems in cloud computing. We believed that component pluggability will be one of the most important issues in the research of cloud computing system. Roystonea has been developed for more than one year, and the first Roystonea system became operational at National Taiwan University in September, 2010. This Roystonea system has been providing Hadoop cluster [22], HBase cluster [23], MPI computation cluster [24], Apache web services [25], and storage service, to more than 70 registered users working on various scientific disciplines, at National Taiwan University [27], [28], [29]. The future works of Roystonea include adding new features and improve the current subsystems. We would like to add new features like auto-scaling capability of virtual machine based web-service, remote control capability of virtual machines through VNC [31] softwares and VPN [32], [33] connection. Also we would like to improve our subsystems, like the