Virtual Cluster Development Environment for grids R. Kumar1 , Thamarai Selvi Somasundaram1 , R. Rajiv 1, P. Balakrishnan1 , Rajendar1, D. Surendran1 , E. Mahendran1 , Rajesh Britto1 , Madusudhanan1 1
Department of Information Technology Anna University, MIT Campus Chromepet, Chennai – 600044, India. Email :
[email protected],
[email protected]
Abstract: - Virtualization is a promising technology that has attracted much attention, particularly in grid community. Recent advances using virtualization technologies enable multiplexing the physical resources by means of virtual machines resulting in better resource utilization. We have proposed Virtual Cluster Development Environment (VCDE) to dynamically form virtual cluster on demand providing the gr id execution environment using Virtual machines. The formation of grid environment to execute any MPI or sequential application has been automated by VCDE. The user can simply submit the job along with the requirements of hardware and software need for the execution. The VCDE provides easy access to grid by reducing the burden of GT installation and other configurations. It is important to note that there is no human intervention in the preparation of virtual cluster as well as setting up the grid environme nt. The Proposed VCDE uses virtualization technologies and it helps the user to make use of grid as easy as possible. Keyword: Virtualization, Grid, Clusters Back ground Recent advances in virtualization technologies [3] are lead by the efforts of projects like Xen [7], VMware [8]. In virtualized nodes, the physical resources are multiplexed among Virtual Machines (VMs). The Virtual Machine Monitor (VMM) is the software that manages the physical node. Thus, the guest-OS running inside the VM receives a slice of the physical node. Research challenges for virtualization have being studied for HPC and Grid [1][2]. Virtual machine monitors must offer primitives to assign single resources to each VM according to demand. These mechanisms must be exposed to standardized interfaces through a pinning framework for managing resources in VM environments. Similar to the pinning mechanism of Xen to assign processors in Symmetric Multiprocessing (SMP) architectures to VMs. Additionally, the mechanism should be extended to every type of resource that could be virtualized by VMM or other virtualization approach. A coarsegrain resource management of virtualized resources could potentially have more drawbacks than benefits when sharing an under-utilized physical node between VMs. Virtualization can enhance the functionality and ease the management of current and future Grids by enabling on demand creation of services and virtual clusters with customized environments, QoS provisioning and policy-based resource allocation. Virtualization is a promising technology that has attracted much interest and attention, particularly in the Grid community. Virtualization can add many desirable features to the functionality of current and future Grids: • On-demand creation of services and virtual clusters with pre-determined characteristics • QoS support for Grid jobs processing
• Execution environment isolation • Improved resource utilization due to statistical multiplexing of different Grid jobs demands on the shared hardware. In this work, we try to automate dynamic cluster formation on the fly with out any user intervention and achieve fundamental cluster management functions, with minimum impact on the grid middleware itself. Related work In this section we present the work rela ted to virtual enabled systems. K. Keahey et al [12] [13] [16] introduce the concept virtual workspace (VW), an approach which aims the provisioning of a customized and controllable remote execution environment for Grid resource isolation purposes. The underlying technology that supports the prototype are virtual machines for the hardware virtualization (Xen and VMware) and Globus Toolkit (GT). The interactions of resource provisioning follow VW descriptions. The descriptions define the VM state (running, shutdown, paused, etc) and the main properties that a required VM must complain such RAM size, disk size, network, etc. Currently, there is not discussion about the performance impact of multiple VM running at the same time. Borja Sotomayor et al [14] [17] argued for a model in which a virtual workspace associated with a well-defined resource allocation, in particular its availability, can be procured by negotiating an agreement with the resource provider via for example WSAgreement The Xenoserver project [6] is building a distributed infrastructure based on the popular and efficient Xen virtual machine [7]. The In-Vigo project [9] made substantial progress in this direction while the Virtuoso [10] project has explored networking issues arising from use of VMs. Timothy Freeman et al [15], as virtual machines (VMs) are used for a wider range of applications, we encounter the need to integrate virtual machine provisioning models into the current site resource management infrastructure as seamlessly as possible. Our research work differs from other previous works is that it attempts to automate the grid fabric setup in order to execute the grid jobs in a virtualized grid environment. We have proposed Virtual Cluster Development Environment (VCDE) to dynamically form virtual cluster on demand providing the grid execution environment using Xen. A Virtual Cluster Service (VCS) has been deployed in Globus Toolkit to submit the job along with the users’ requirements to VCDE. Virtual Cluster Development Environment (VCDE) The overall Virtual Cluster Development Environment architecture is depicted in the Figure 1. The two custom services Virtual Cluster Service and Virtual Information Service have been developed and deployed in Globus Toolkit 4.0.1. The purpose of Virtual Cluster Service (VCS) is to contact the VCDE Server to create the virtual cluster. The Virtual Information Service (VIS) is used to aggregate all the physical and virtual resource details. Through the Job Submission Portal the user has to access the Virtual Cluster Service Client and give their requirements before that user is authenticated and subsequently users authorization is performed. This Client Code invokes the Virtual Cluster Service (VCS), the VCS in turn invokes the VCDE Server. The User has to mention their job requirements in job description file which is in
XML format. The job description file consists of users hardware and software requirements. Hardware requirements such as type of Operating System, size of memory, ramsize and node count software requirements such as Jdk1.5, Mpich-1.2.6 version, etc., The Administrator/User can view the job and resource details using the Virtual Information Service (VIS). The VIS service contacts the VCDE Server and updates the job and resource details in temporary cache in XML format.
JOB SUBMISSION PORTAL
GLOBUS CONTAINER
NETWORK MANAGER
USER POOL
JOB STATUS SERVICE
VIRTUAL CLUSTER SERVER
IP POOL
SECURITY SERVER HOST POOL
RESOURCE AGGREGATOR
SCHEDULER MATCH MAKER
DISPATCHER JOB POOL
VIRTUAL CLUSTER MANAGER
VIRTUAL MACHINE CREATOR VIRTUAL COMPUTE NODE 1 COMPUTENODE 1
EXECUTOR MODULE
VIRTUAL MACHINE CREATOR VIRTUAL HEAD NODE
TRANSFER MODULE
VIRTUAL MACHINE CREATOR
VIRTUALCLUSTER
COMPUTENODE 2
VIRTUAL COMPUTE NODE n
CLUSTER HEAD NODE
VIRTUAL INFORMATION SERVICE
VIRTUAL CLUSTER DEVELOPMENT ENVIRONMET
VIRTUAL CLUSTER SERVICE
COMPUTENODE n
Figure.1. Virtual Cluster Development Environment Architecture Virtual Cluster Server is the central or core part of the Virtual Cluster Development Environment. It is the entry point for all the services. The Virtual Cluster Server maintain the details of resource, user, job and also it maintains the Dispatcher Module, Network Manager , Resource Aggregator Module, Security Server Module, Executor Module ,Transfer Manager Module. Resource Aggregator module fetches all the physical resource information from the VM Creator Resource Info Server and these informations are updated to the Host Pool. The Physical resource information updated from Resource Info Server is dynamic resource available at that time. The Host Pool maintains the Cluster Head node and Compute node Logical Volume Partition, Logical Volume (LV) disk space total and free, Ram Size total and free, Kernel type, Gateway, Broadcast Address, Network Address, Netmask etc. This Dispatcher Module is invoked when the job is submitted to the Virtual Cluster Server. The Dispatcher module segregates the user’s hardware, software
requirements and creates the job template for actual job to be run and updates in the job pool with job id and invokes the match making module. The matched hosts are identified and compute the rank for the matched resources. The Match Making Module access the Host Pool and compares the users Hardware requirements with Physical Resource Information available in the Host Pool and generates the matched host list. Using the matched host list it computes the rank for the matched resources. The rank computed based on the resource which has highest Ram Size Free and Logical Volume Disk Space Free. The ranked resource is then send to the dispatcher module. The Host Pool is responsible for maintaining the physical cluster head node and compute node resource information such as Host Id, Host Name, Logical Volume Partition, Logical Volume Disk Space Free and Total, Ram Size Free and Total, Kernel Version, Status of Xen Hypervisor, Ram Disk, Network address, Gateway, Netmask and Broadcast Address and also it maintains the virtual cluster resource details after the virtual cluster installation is over. The User Pool ma intains the list of authorized users. It has the facility to allow which users are allowed to create the virtual execution environment. It also has the provision to limit the no of jobs for individual users to limit the traffic and for giving opportunity to other users in the grid. The Job Pool maintains the Job Queue. The Job from the job queue and their requirements is selected and sent to the dispatcher. The Job Queue maintains the job id, virtual cluster head node name, job details and their current status. The Job Status service in the VCDE Server access the Job Pool display the status of virtual cluster and job. The scheduler module we have followed the model of First Come First Serve (FCFS) algorithm. It receives the matched hosts with their ranks and order the resources based on their rank. It generates the resource list for virtual machine creation from the ordered resources based on the users node count and generated resource list .The generated resource list is send to the Dispatcher Module. Virtual Cluster Manager Module (VCM) we implemented the Round-Robin Algorithm. Based on the users node count VCM creates virtual cluster with one head node and others as compute nodes. Using this algorithm we are creating the virtual machines in a circular manner. From the scheduled list generated from scheduler module the first host in the resource list as Head Node and remaining as Compute Node. The main purpose of Security Server Module is to perform mutual authentication dynamically. When the Virtual Cluster installation and configuration is completed, the service running in the virtual cluster head send the signing policy files and the users identity to this security server and this server will updates the identity in the Physical Cluster Head Node grid-map file automatically. This security server send the physical cluster head node signing policy file and the users identity to the service running in the virtual cluster head node and it is updated. Executor Module is responsible for once the Virtual Cluster formation is over the VCM module fetches the job details from the job pool and create RSL file and invokes the Transfer Module to transfer for file transferring. The Executor module contacts the Virtual Cluster Head Node Job Managed Factory Service and submits the job and periodically updates the job status in the Job Pool. GRAM component of the virtual cluster is initiated as well as virtual cluster is created. It also monitors the job executio n. It tracks the information about the successful creation of the virtual cluster from the
VMCreator. After the mutual authentication process is over the transfer module is invoked. The Transfer Module uses the gridftp file transfer mechanism and transfers the Job Executable, Input files, Output Files and the RSL file to the Virtual Cluster Head Node in secure manner. After the job completion in the virtual cluster head node the output file is transferred automatically to the physical cluster head node users home directory. Virtual Machine Creator (VM Creator) consists of two major components viz., Resource Info Server Module and Virtual Machine Creator Module. The Resource Info Server module fetches the details periodically using linux utilites and Xen Hypervisor command such as Hostname,Type of Operating System,Kernel Version,Type of Kernel,Ram Disk,Root Device,Logical Volume Device,Logical Volume Disk Space Free and Total, Ram Size Free and Total,Xen Hypervisor status, Broadcast Address, Netmask, Network Address and Gateway and contacts the VCDE server and updates VCDE server. The Virtual Machine Creator creates the virtual machines based on the message received from the Virtual Cluster Manager in the VCDE Server Module. If the message received from the VCM is HEAD node, it starts to create the Virtual Cluster with Head Node Configuration or else if the message received from the Virtual Cluster Manager is CLIENT it starts to create the compute node with minimal required software. Virtual Machine Automation, during the virtual Machine automation process first we create physical volume group in the Logical Volume Device. In the physical volume group we create the logical volume group. Based on the users requirements we create the root device, swap device with required memory. Using the users requirements we create the configuration file for virtual machine creation. In the configuration file we have to specify the Kernel Path, Ram Disk, Memory, Name of the Virtual machine, Root Device, Hostname, IP Address, Subnet Mask, Gateway, Broadcast Address. For the cluster setup, the fundamental server setup such as DNS, NFS, NIS and DHCP have been installed and configured automatically in the verge of Virtual Cluster Formation. The Domain Name Service (DNS) server is configured automatically in the Virtual Cluster Head Node. During the DNS configuration the compute node hostnames are added in the /etc/hosts file. This is useful so that a host can look up an IP Address locally before going out to query the DNS Server. The Network File System (NFS) server configuration involves the two step process. The first file is to create the /etc/exports file. This file defines which parts of your server's disk get shared with the rest of your network and the rules by which they get shared. The second step is to start the NFS server processes that read the /etc/exports file. The Network Information Service (NIS) facilitates the sharing of critical data stored in flat files among systems on a network. Files such as /etc/passwd and /etc/group which ideally would remain uniform across all hosts are shared via NIS. The Network Information Service is really just a simple database that clients can query. We write the system level bootable script to start all the services when the system boots itself. Based on the user's requirements the required softwares are installed automatically. We enable the remote shell without password configuration between the computing nodes. While a virtual machine monitor such as Xen Machines can create a virtual machine, that machine must be able to connect to a network accessible by a computational grid. Unlike a process running on the underlying physical machine, the
virtual machine appears to the network to be one or more new network interface cards. Virtual Networking Service is responsible for assigning new internet protocol address and host name for a dynamically created VM. With Virtual IP address newly created virtual nodes can able to communicate with each other. It is also possible to supply an IP address dynamically from the host’s network via Dynamic Host Configuration Protocol (DHCP). Experimental Setup In our testbed we form the five node physical cluster of one head node with four compute nodes. All the fives nodes are Xen 3.0 VMM enabled. In the head node we install the Globus Toolkit4.0, 1, Jdk1.5, Mpich-1.2.6, Jakarta-tomcat-5.0.27, Junit-3.8.1, Apache-ant-1.6.4 with our VCDE Server and VCDE Scheduler. In the compute nodes we install Torque-1.2.0, Mpich-1.2.6 with our VCDE VM Creator. We store the images in NFS directory of Head Node so it is sharable by compute nodes in the physical cluster. We deployed our Virtual Cluster Service and Virtual Information Service in the Head Node of the Globus Toolkit. The main purpose of the Globus Toolkit here for authenticating and authorizing the user and invoking the Virtual Cluster Server for creating Virtual Cluster based On Demand. The installation and configuration of Globus toolkit is done automatically by invoking our script. The pre-requisite softwares for the installation of GT4 is also done automatically. There is no human intervention needed when your installation and configuration during grid fabric setup. We have configured the simple CA without human intervention and created the certificates for host, globus and the third party user who is requesting the virtual cluster. Finally we integrated the Globus Toolkit with Local Resource Management System (PBS). We configure the RSH configuration in Head and compute nodes for password less communication. Image Constituents Node Type Head Node
Image Size 2.0 GB file system image
Compute Node
1.0 GB file system image
VM Image Constituents Fedora Core 4 GT4.0.1 Binary Installer Jdk1.6 Apache Ant 1.6 PostgreSQL 7.4 Torque-1.2.0 Mpich-1.2.6 Junit-3.8.1 Jakarta-tomcat-5.0.27 FASTA Application and Nucleotide sequence database Fedora core 4 Mpich-1.2.6 Torque-1.2.0
Experimental Results In our grid testbed, we have created and tested the two node, three node and five node virtual clusters. For the creation of virtual machines in head node and compute nodes takes time duration of 60 seconds. In the head node all the pre-requisites with GT4 binary installer require 1 minute. For compute node software installation takes 60 seconds. Totally it takes 3 minutes to create the virtual cluster, one head node and 4 compute nodes including perquisite software for globus tool kit installation, globus installation bundle, local scheduler plus networking setup for the cluster.
Fig. 2. Virtual Cluster Formation Time Conclusion The VCDE (Virtual Cluster Development Environment) has been designed and developed for creating virtual clusters automatically to satisfy the requirements of the users. There is no human intervention in the process of the creating the virtual execution environment. The complete automation takes more time, so in the near future, the performance of the VCDE will be improved. References 1. Foster, I., C. Kesselman, J. Nick, and S. Tuecke, The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration. 2002: Open Grid Service Infrastructure WG, Global Grid Forum. 2. Foster, I., C. Kesselman, and S. Tuecke, The Anatomy of the Grid: Enabling Scalable Virtual Organizations. International Journal of Supercomputer Applications, 2001. 15(3): p. 200-222. 3. Goldberg, R., Survey of Virtual Machine Research. IEEE Computer, 1974. 7(6): p. 3445. 4. Keahey, K., I. Foster, T. Freeman, X. Zhang, and D. Galron, Virtual Workspaces in the Grid. ANL/MCS-P1231-0205, 2005.
5. Figueiredo, R., P. Dinda, and J. Fortes. A Case for Grid Computing on Virtual Machines. in 23rd International Conference on Distributed Computing Systems. 2003. 6. Reed, D., I. Pratt, P. Menage, S. Early, and N. Stratford. Xenoservers: Accountable Execution of Untrusted Programs. in 7th Workshop on Hot Topics in Operating Systems. 1999. Rio Rico, AZ: IEEE Computer Society Press. 7. Barham, P., B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebar, I. Pratt, and A. Warfield. Xen and the Art of Virtualization. in ACM Symposium on Operating Systems Principles (SOSP). 8. Sugerman, J., G. Venkitachalan, and B.H. Lim. Virtualizing I/O devices on VMware workstation's hosted virtual machine monitor. in USENIX Annual Technical Conference. 2001. 9. Adabala, S., V. Chadha, P. Chawla, R. Figueiredo, J. Fortes, I. Krsul, A. Matsunaga, M. Tsugawa, J. Zhang, M. Zhao, L. Zhu, and X. Zhu, From Virtualized Resources to Virtual Computing Grids: The In-VIGO System. Future Generation Computer Systems, 2004. 10. Sundararaj, A. and P. Dinda. Towards Virtual Networks for Virtual Machine Grid Computing. in 3rd USENIX Conference on Virtual Machine Technology. 2004. 11. Jiang, X. and D. Xu, VIOLIN: Virtual Internetworking on OverLay INfrastructure. Department of Computer Sciences Technical Report CSD TR 03-027, Purdue University, 2003. 12. Keahey, K., I. Foster, T. Freeman, X. Zhang, and D. Galron. Virtual Workspaces in the Grid. in Europar. 2005. Lisbon, Portugal. 13. Keahey, K., I. Foster, T. Freeman, and X. Zhang, Virtual Workspaces: Achieving Quality of Service and Quality of Life in the Grid, Scientific Progamming Journal, 2005. 14. I.Foster, T. Freeman, K.Keahey, D.Scheftner, B.Sotomayor, X.Zhang: Virtual Clusters for Grid Communities. CCGRID 2006, Singapore (2006) 15. T. Freeman, K. Keahey, Flying Low: Simple Leases with Workspace Pilot, Euro-Par 2008 16. Keahey, K., T. Freeman, J. Lauret, D. Olson, Virtual Workspaces for Scientific Applications, SciDAC 2007 Conference, Boston, MA. June 2007 17. Sotomayor, B. Masters paper, A Resource Management Model for VM-Based Virtual Workspaces ,University of Chicago, February 2007 18. Bradshaw, R., N. Desai, T. Freeman, K. Keahey, A Scalable Approach To Deploying And Managing Appliances, TeraGrid 2007, Madison, WI. June 2007
