tegrity Superdome servers running Linux and HP-UX on the high end. The HP Virtual Server Environment, built on the HP-UX Workload. Manager, is a software ...
Design of an Economics-Based Software Infrastructure for Secure Utility Computing on Supercomputing Clusters Gregory A. Koenig and William Yurcik National Center for Supercomputing Applications (NCSA) University of Illinois at Urbana-Champaign {koenig,byurcik}@ncsa.uiuc.edu
Abstract Recent advances in technology have created a modern computational environment in which high performance commodity superclusters are interconnected by high bandwidth Internet connections, thus establishing a worldwide computational grid with large aggregate computational capabilities. Several projects attempt to build on this infrastructure by providing mechanisms that allow grid resources to be allocated on demand in response to sudden bursts of work. These projects fall under a model of computing called utility computing. In this paper, we describe state of the art efforts within the field of utility computing as well as our work to create a utility computing environment across a federation of clusters within our organization.
1
Introduction
Over the past decade the performance of consumer-grade microprocessors has improved to match performance levels previously available only from proprietary microprocessors. Similarly, low-latency highbandwidth system area networks have been developed with performance surpassed only by the interconnection networks found in the most high-end supercomputers. These changes have created an environment for modern high-performance computing in which low cost commodity components, usually deployed in the form of “clusters” of off-the-shelf PC hardware, replace expensive proprietary systems without significant loss of performance. One effect of this trend towards
low cost, high-performance computer hardware is that the number of computers capable of delivering hundreds of gigaflops of performance has risen dramatically as universities and companies, and even individual departments, are able to afford computational capability that was previously available only to elite supercomputing centers. Simultaneously, advances in wide area networking technologies have resulted in nationwide networks with the capability of delivering gigabit-per-second bandwidths to the desktop. Additionally, the reliability of these networks has increased greatly. The result of these advances is that the Internet has become a tool used by millions of people each day to access a rich variety of content on the World Wide Web. Further, this Internet infrastructure has been leveraged to create a worldwide computational grid which allows high-performance computing resources at multiple geographically distributed locations to be joined into a unified resource capable of being directed towards solving individual problems. That is, grid computing allows users to easily create “virtual organizations” that connect resources owned by multiple organizations into a single synthesized entity. For example, a grid application could collect data from scientific devices at two different sites, perform a computation with the data on a supercomputer at a third site, and display the results of the computation with visualization tools and equipment at a fourth site. Overall, these advances have led to entirely new models of computing which leverage flexible software capable of running efficiently within dynamic computing environments. One such model is utility computing, which attempts to address the challenge to an organization of meeting fluctuating demands for computing resources. Because an organization’s demand for computing resources can vary drastically over time, maintaining sufficient resources to meet peak requirements can be costly. Conversely, if the organization cuts costs by maintaining only minimal computing resources, there may not be sufficient resources to meet peak requirements. With utility computing, resources are made available to users as needed, similar to how electricity or water is made available as needed by turning on a switch or a faucet. Like traditional utilities such as electricity or water, computational resources may be provided on demand by centralized service providers. To this
end, vendors such as IBM have suggested a model for utility computing in which the vendor maintains large installations of clusters that can provide computational cycles to multiple subscribers as needed. The intended goal of utility computing is to lower the overall costs of Information Technology incurred by each subscriber while at the same time not sacrificing access to the quantity of resources necessary to meet peak demands. We believe that the utility computing model offers a compelling way to match high-performance computing resources, possibly spread across multiple sites on the Internet, with the users of these resources. In our vision, individual organizations will continue to own and maintain computational resources that are targeted toward the unique goals of the organization. We expect that the workloads of many of these resources will often be bursty with periods of high utilization punctuated by periods of low utilization. Our aim is to increase the overall utilization of these resources by applying utility computing concepts to the problem of efficiently allocating computing resources. This could entail several ideas applicable to both single clusters and to multiple cooperating clusters. In the context of a single cluster, the resources within the cluster could be scheduled on the basis of how much a user is willing to pay to have access the cluster on demand. For example, if the cluster is mostly idle, a user might be allowed to run a job on the cluster at a very low price1 . Later, if a second user who has a tight deadline for their job wishes to use the cluster, their job might be given higher priority access to the cluster if the user is willing to pay a higher price for immediate access to the resources. That is, the first user’s job might be suspended in order that the second user’s job could be executed in time to meet the user’s deadline. In the context of multiple cooperating clusters, resources across a federation of clusters might be scheduled on the basis of how costly it is for a particular cluster to accept additional work. When a user submits a job, each cluster could return a corresponding “bid” representing its price for completing the job. If a particular cluster is mostly idle, it 1 Here, “price” may be represented in actual monetary terms or in terms of “service units” which the user expends at some fixed rate to use the machine.
would return a relatively low bid; conversely, if the cluster is heavily loaded, it would return a higher bid. The user, seeking to minimize their expense, would select the most favorable bid and choose that cluster for executing the job. One can even envision a scenario in which the computational resources belonging to multiple organizations might barter CPU cycles with each other to improve job turnaround times during periods of heavy utilization. That is, if Organization A’s resources are heavily utilized during a period of peak usage and Organization B’s resources are lightly utilized, B might barter with A, allowing some of A’s jobs to spill over onto B’s lightly utilized resources with the expectation that sometime in the future when B’s resources are heavily utilized and A’s resources are lightly utilized, A returns the favor by allowing some of B’s jobs to spill over onto A’s lightly utilized resources. Overall, we believe that ideas such as these can be used to create a market economy for computational cycles in which users with specific requirements for access to computational resources are efficiently matched with the resources that can most cost-effectively address the user’s requirements. As users submit jobs into a federation of clusters, the user seeks to minimize the amount of money they spend on running the jobs. Simultaneously, as multiple independent resources bid on jobs, each resource seeks to maximize its profit. We expect that the end result of creating this type of market economy of compute cycles will be an improvement of the overall utilization of all resources within the federation. This paper describes our work towards creating a utility computing environment on a cluster maintained at the Technology Research, Education, and Commercialization Center (TRECC) [6]. The workload on the cluster consists of a wide variety of tasks ranging from running education and research oriented software to the support of the unique problems presented by entrepreneurs. We believe that this workload is well suited to utility computing because it requires the resource to have the ability to be re-targeted rapidly to adapt to these changing workloads. For example, educational users may be given free or low-cost access to the TRECC facilities, but entrepreneurs who are willing to pay more to access the facilities, with the expectation of faster turnaround,
should be given higher priority. Further, the TRECC cluster is one of several clusters to be configured in this manner. Our longterm goal is to create a utility computing environment that expands past the boundary of the TRECC cluster to encompass the clusters owned by other organizations collaborating in this endeavor. The remainder of this paper is organized as follows. Section 2 describes the key enabling technologies that are driving the transition to utility computing. Next, Section 3 discusses current work related to utility computing that is being done in both commercial and academic settings while Section 4 offers a critical analysis of this work. Section 5 describes our design for a Web services based system for utility computing which is based on ideas taken from several other systems but recasts them into a framework more suited towards our goals. Finally, Section 6 offers our conclusions and describes our plan for future work in this area.
in a high-performance cluster, a single large computationally-intensive problem is decomposed into a number of smaller problems that can be solved simultaneously. These smaller problems are distributed among machines in the cluster and intermediate results as well as control and coordination messages are passed among cluster nodes via software middleware such as the Message Passing Interface [14, 19]. At the end of the computation, the results of the sub-problems are combined to provide a solution to the overall problem. The key idea behind cluster computing is that commodity components are now powerful enough to be leveraged successfully for solving problems that until recently could only be solved feasibly by large, expensive supercomputers. The result of this is that access to large amounts of computational power is now possible for nearly any organization with a reasonable budget.
2.2
2
Enabling Technologies
The trend towards utility computing is being driven by several key enabling technologies. In general, these enabling technologies center around standards-based system software running on commodity hardware and communicating via pervasive communication infrastructures. Specifically, in this section we describe clusters of commodity hardware, Web services, and grid computing.
2.1
Clusters of Commodity Hardware
In cluster computing, commodity off-the-shelf (COTS) computer systems are connected together with high-performance interconnection networks such as Myrinet [10] and InfiniBand [23] to provide solutions for high-availability or high-performance computing. For example, in a high-availability cluster, one or more redundant systems mirror the activity of a primary system so that a failure in any single component of the cluster will not cause a failure of the overall cluster or a lack of availability for the services provided by the cluster. Alternatively,
Web Services
During the first several years of development of the World Wide Web, the focus was on creating a human-centric infrastructure in which people directly accessed information stored on Web pages. Due to the success of the Web and the resulting large volumes of information contained in Web pages, developers began a transition to a software-centric infrastructure in which programs were able to access and process Web content, perhaps drawing information from several sources. Several resulting technologies have been developed to facilitate such programmatic access to Web content, the most recent of which is Web services [11]. Web services center around four related key pieces: • WSDL – Web services should be self-describing, and WSDL (Web Services Description Language) is the mechanism by which this is accomplished. To use WSDL to describe a Web service, a programmer creates an XML description of the template necessary for clients to bind to the service. Information in this template includes the Service, a collection of related endpoints; Port, a definition for a single endpoint including a network address; Binding, the concrete protocol and data format specification for a given port; Port
Type, the set of operations supported by an endpoint; Operation, a description of the operations supported by the service; Message, a description of the message data being communicated; and Types, a description of the data types used in service messages, described in terms of a type definition system. In short, the WSDL description of a Web service includes details of the functions that may be invoked along with the parameters and return values for these functions. • UDDI – Web services should also provide a mechanism by which they can be discovered by other Web services. UDDI (Universal Description, Discovery, and Integration) is such a mechanism. Using UDDI, a programmer registers a Web service in a directory allowing searching via several methods. White Pages lookups allow a service to be located based on specific information such as the name of the service. Yellow Pages lookups allow a service to be located based on a general description of the type of service. Finally, Green Pages lookups allow a service to be located based on technical information about the Web service. • XML-RPC – XML-RPC is a Remote Procedure Calls mechanism that allows method or function calls to be made across a network. When a caller uses XML-RPC, data are encapsulated in XML wrappers and sent to a remote machine via an HTTP POST request. This request causes a method or function to be invoked on the remote machine. After the method or function completes, the results of the invocation are returned to the caller in XML-encoded HTTP replies. • SOAP – SOAP, Simple Object Access Protocol, is similar to XMLRPC in that it allows remote method or function invocation. However, unlike XML-RPC, SOAP provides a service description grammar which allows details of a given method or function, such as the parameters required, to be discovered automatically. Because of this, SOAP is more complex than XML-RPC. The important idea behind Web services is that they are programming language independent and composable, that is, capable of being
glued together to create Web applications that provide a rich software infrastructure that may not have been envisioned by any of the developers of the individual Web services.
2.3
Grid Computing
Grid computing [17, 18] is a model of computation in which several geographically distributed resources are united to solve a single problem that requires large numbers of processor cycles or access to large amounts of data. Grid computing has been used successfully in several projects that bring together resources on a national and even worldwide scale. In some ways, grid computing can be thought of as a way of creating “clusters of clusters” in which computational resources such as high-performance clusters and visualization engines are brought together across organizational and authentication boundaries, although the ultimate goals of grid computing reach far beyond building clusters of clusters. The fundamental result of grid computing is that it is no longer necessary for a single organization to own all of the resources that members of the organization require. Instead, each organization can invest in resources that are directly aligned with the organization’s goals and unite these resources with resources owned by other organizations as necessary to form “virtual organizations” that exist for the duration of a single computational job. Grid computing is typically deployed in the form of the Globus Toolkit [16], a software “toolkit” of building blocks that can be used individually or as a whole to build grid software. Components of the toolkit include services for resource allocation (i.e., submitting jobs on remote supercomputers), remote access to files, and grid directory information (i.e., locating resources on the grid). These components are underscored by a public-key security infrastructure. More recently, design of the Globus Toolkit has been driven by the Open Grid Services Architecture (OGSA) [15]. OGSA primarily defines a software infrastructure for Grid services by building on the concept of Web services, but redirecting the technology towards problems unique to grid computing. For example, Web services do not include support for “state” within a service, which means that each request made on
a Web service must be completed as a single atomic transaction. Or, stated another way, multiple related requests made by the same client to the same Web service are treated by the service as independent requests due to the fact that Web services are stateless. In the context of grid computing, however, having a service maintain state is very desirable. To that end, Grid services provide a mechanism by which the service can maintain state across multiple requests made by the same client.
3
Related Work
This section describes related work in the field of utility computing. Efforts in both the commercial and academic sectors are discussed.
3.1
Commercial Efforts
In commercial settings, utility computing currently involves a large amount of hype and few useful solutions. In general, customers are interested in the utility computing promises of reduced capital expenses, faster technology development and deployment times, and faster rollouts of new information technology services. Utility computing solutions provided by vendors, however, fulfill these promises in widelyvarying ways with some vendors offering solutions that provide a large number of useful technologies and other vendors providing a relatively few unsatisfying technologies. Overall, each vendor seems to define utility computing in a slightly different way, with each vendor offering solutions centered around their own particular areas of expertise. That is, vendors who produce only hardware tend to view utility computing as a hardware-centric problem; similarly, vendors who produce only software tend to view it as a problem that can be solved entirely through sophisticated software. The vendor with perhaps the best all-around strategy for utility computing is IBM. IBM’s strategy, called “E-Business On Demand,” [2] leverages experience in IBM’s Business Consulting Services along with a wide range of hardware and software solutions. For years, IBM has
built an infrastructure through which companies can outsource some or all of their Information Technology operations. More recently, this infrastructure is being re-targeted to allow customers to remotely access computing capacity as needed and pay only for the capacity that is used, ramping-up or dialing-down the amount of processing power as demand for resources fluctuates. From a hardware perspective, IBM’s utility computing solutions encompass a wide spectrum of hardware platforms ranging from their xSeries Intel-based Linux and Windows servers at the low end, to their iSeries and pSeries Power-based machines running Linux, AIX, and OS/400 at the midrange, and extending to their zSeries mainframes at the high end. These hardware platforms are matched with a wide variety of system-level software that facilitate utility computing. For example, IBM’s VM operating system for its zSeries mainframes allows a single machine to be divided into a number of virtual machines that operate independently and are indistinguishable from true physical machines. Similarly, IBM’s Virtual Server Service software extends the virtual machine concept to the entire range of eSeries machines, allowing clusters of Intel-based Linux machines to be divided into multiple sub-clusters and allocated to several customers on demand. Overall, IBM’s E-Business on Demand seems to offer a compelling combination of virtualization and adaptability of resources, key features of a successful utility computing solution. Hewlett-Packard (HP) also offers a very compelling utility computing strategy called “Adaptive Enterprise” [1]. Succinctly, HP believes that all enterprises are under increasing pressure to be agile and adaptable, necessitating the ability to uncover customer needs and quickly change existing IT infrastructures to meet these needs. Adaptive Enterprise addresses these goals by incorporating a spectrum of HP’s services, hardware, and software. HP offers hardware solutions ranging from x86- and Itanium-based servers running Linux on the low end, to HP Integrity Superdome servers running Linux and HP-UX on the high end. The HP Virtual Server Environment, built on the HP-UX Workload Manager, is a software infrastructure that allocates virtual resources on machines in response to application demands. This software leads to HP’s Instant Capacity On Demand (iCOD) solution which allows
customers to pay for processors in a machine only when they need to use them to satisfy bursts of work. For example, with iCOD on an eight-processor Superdome server, a customer may purchase the right to activate only two processors. Later, in response to a sudden heavy burst of work, the customer may elect to activate more processors to handle the additional workload. iCOD ensures that the customer is billed only for the capacity used. Overseeing the entire range of HP utility computing solutions is HP OpenView, a software package that allows an entire enterprise’s hardware and software environment to be managed from a single console. Sun Microsystems has historically championed the mantra that “the network is the computer,” so it comes as no surprise that their utility computing strategy, called “N1 Grid” (after the goal of “managing N computers as 1”), focuses heavily on a network-centric view of computational resources [5]. The stated goal of their utility computing solution is to “intelligently match IT resources to meet business demand on a pay-for-use basis.” To accomplish this goal, Sun leverages a variety of systems software running on Sun hardware. First, Sun’s Solaris operating system has the capability of being partitioned into a number of “containers” which are essentially virtual machines that can be allocated to independent tasks on demand. At a higher level, the Sun Grid Engine provides a unifying batch subsystem which can be used to create pools of computational resources across an enterprise’s machines. Finally, Sun’s Java Enterprise System provides a programming language and virtual machine environment for creating and deploying software which can run on hardware from multiple vendors regardless of the underlying processor architecture. One area where Java has found large success is in the creation of Web services. Taken together, Sun uses these technologies to create solutions for providing Capacity On Demand and Temporary Capacity On Demand in a fashion similar to HP’s iCOD. Taking a software-only approach, Microsoft addresses utility computing with its “Dynamic Systems Initiative” (DSI) [3]. DSI is primarily Microsoft’s XML initiatives rolled together under the umbrella of Web services running under it’s .NET environment, a language-independent runtime system similar to Sun’s Java. As part of DSI, Microsoft offers
the System Definition Model (SDM), an XML-based language used to describe environment-specific information related to applications, operating systems, and management software. Overall, Microsoft’s approach suffers from the fact that it requires a large-scale replacement of existing operating systems and management software to realize the goals of utility computing. Such a large-reaching replacement of existing software infrastructures is undesirable. A final company that we feel is a key player in the utility computing field is Cluster Resources, Inc. Although Cluster Resources does not consider itself a utility computing vendor per se, the company’s software solutions provide mechanisms for scheduling cluster resources within a single cluster or across multiple clusters, and these solutions provide a key component of an overall utility computing solution. Cluster Resources’ Maui Scheduler is an advanced job scheduler for use on clusters and supercomputers. It supports a large range of scheduling policies, dynamic priorities, the ability to schedule reservations on resources to guarantee that a given batch job will be completed by a fixed deadline. Maui Silver builds on the Maui scheduler by allowing jobs to be scheduled and reservations to be made across multiple clusters. Finally, Maui Gold is an open source accounting and allocation system that allows resources to be shared at a cost measured in terms of computational units or dollars. Using Gold, jobs can be charged for their use of resources, and this charge can be set on a per-resource basis. Overall, the software suite provided by Cluster Resources presents a unified framework through which resources can be dynamically allocated and the use of these resources can be billed accordingly. Clearly, this result is very related to the goals of utility computing. Table 1 summarizes the vendor-driven efforts to develop utility computing solutions. Overall, current vendor efforts in utility computing focus on their definition and vision for utility computing based primarily on each vendor’s existing areas of expertise. This has led to a large amount of hype surrounding utility computing, and it is not clear at this time how much of this hype will become realized solutions. However, it seems likely that some useful solutions will be produced with vendor-driven utility computing, similar to how useful software resulted from the hype surrounding client-server computing a decade ago.
Company IBM
Utility Computing Strategy E-Business On Demand
Hewlett-Packard (HP)
Adaptive Enterprise
Sun Microsystems
N1 Grid
Microsoft
Dynamic Services Initiative
Cluster Inc.
Resources,
Maui Scheduler
Summary Wide range of hardware platforms ranging from mainframes down to rackmounted x86 and Itanium Linux clusters. Virtualization of resources accomplished through Virtual Machine (VM) operating system or Virtual Server Service, allowing machines to be divided into sub-machines or sub-clusters and allocated to customers on demand. Wide range of hardware platforms ranging from Superdome SMP machines down to rackmounted x86 and Itanium Linux clusters. Virtualization of resources accomplished through Virtual Server Environment, allowing individual processors within a single machine to be activated and deactivated on demand to address bursts of work. Customers purchase only the capacity they actually require. Virtualization of hardware resources (limited to Sun hardware only) accomplished through the use of Solaris “containers.” Sun Grid Engine provides a batch environment across multiple independent machines. Java Enterprise System provides a processor-independent environment for software deployment, particularly useful for creating Web services. A software-only approach wrapping Microsoft’s XML initiatives in .NET Web services. Requires widescale adoption of Microsoft software in order to make work. Not a traditional utility computing vendor. Maui and Maui Silver schedulers can schedule single clusters of machines or create cross-cluster reservations. Maui Gold accounting and allocation system allows resources to be shared at a cost, measured in credits of computational units or dollars.
Table 1: Summary of commercial utility computing efforts
3.2
Academic and Research Efforts
Where vendor-driven efforts in utility computing are motivated by the desire to produce solutions by which customers can decrease IT expenses, academic and research efforts are typically motivated by the desire to maximize utilization of computational resources such as supercomputing clusters. Further, vendor-driven efforts focus on providing general all-encompassing solutions whereas academic and research efforts are usually focused on providing a solution for a particular pointed problem. For this reason, some academic and research efforts which have goals related to utility computing do not classify themselves as utility computing projects; nevertheless, we consider them as well as research efforts which specifically address utility computing. The Purdue University PUNCH project [22] is a platform for Webbased computing that allows users to access and run programs over the Internet. Using standard Web browsers, users can access applications and data existing on resources that are located at different sites and which may be managed by multiple organizations. To accomplish this, PUNCH leverages several of the enabling technologies described in Section 2 such as the OpenPBS batch system [4] and Condor (described next), front-ended by Globus services, to provide access to clusters of compute power. An underlying virtual filesystem, implemented in terms of NFS and Ufo [9], provides a single consistent namespace for file access across the entire PUNCH system. Finally, each application’s interface is displayed on the user’s remote terminal through the use of VNC [13]. PUNCH uses these technologies together in a very compelling way to create a synthesized environment for serving applications to end users. While the PUNCH solution is not directly in line with our goals, we believe that the technologies involved will be an important part of our utility computing goals with the TRECC cluster. The University of Wisconsin Condor system [25] is a specialized workload management system for compute-intensive jobs. Like with traditional batch systems, users submit serial or parallel jobs into a Condor “flock” and the Condor scheduler chooses when and where to run the jobs based on a job placement policy. Unlike traditional batch systems, Condor is focused more on the goal of providing “high-throughput com-
puting,” access to as many compute cycles as possible over an extended period of time (e.g., months). To this end, Condor was originally designed to run on user desktop systems, allowing jobs to access the otherwise wasted CPU cycles of machines that are idle when the console user is not using the machine. Condor provides mechanisms through which an executing job can be checkpointed and moved to a different machine when the console user returns. More recent versions of Condor address issues related to traditional batch environments. Overall, the Condor system presents technology that is useful to utility computing, particularly with its checkpoint and restart mechanisms. Often, ondemand access to computational resources involves suspending a low priority job to allow a higher priority job to have control of a set of resources. Condor’s checkpointing mechanism seems to offer a good way of accomplishing this. The Monash University (Australia) Nimrod system [8] is a software application for parallelizing and distributing large computational experiments based on the exploration of a range of parameterized scenarios. Using Nimrod, it is possible to specify and generate a parametric experiment and then control the execution of the code across distributed computers. Nimrod/G extends the base Nimrod system by incorporating Globus components to discover computational resources and then remotely access the resources to launch jobs. The interesting contribution from the standpoint of utility computing is that the Nimrod developers have experimented with the use of economy-based models for scheduling and specifying completion deadlines for computational resources. The Cluster-On-Demand (COD) project [12] at Duke University is directly attempting to achieve utility computing results. COD uses remote boot technology to dynamically reconfigure nodes in a cluster based on user-specified configuration templates. Using COD, a user is allocated a set of cluster nodes and is then allowed to specify an operating system image from a set of images stored in a database. This operating system image is used to boot the user’s nodes. Through this mechanism, the user is given dedicated and nearly unrestricted access to the allocated machines, allowing the user to run a wide variety of software, including even systems-level software, which is infeasible
under clusters that are allocated via traditional batch systems. The University of Illinois Faucets system [21] is the project that is perhaps the closest match to our goal of providing utility computing on the TRECC cluster. Faucets consists of a cluster batch scheduler along with a set of daemon processes which implement an economybased model for allocating cluster resources and for bartering these resources across multiple clusters. The system can allocate resources for traditional serial or parallel jobs as well as for “adaptive jobs” that use the Charm++ runtime [20] to grow and shrink to a varied number of physical processors. Building on this dynamic scheme for managing resources, the system creates a market economy for computational cycles by having each cluster within a federation of clusters issue a “bid” for each new batch job that is submitted to the system. These bids are based on the costs incurred by each cluster to run the job, with the expectation that lightly-utilized clusters will return a relatively low bid while highly-utilized clusters will return a higher bid. Another factor influencing each cluster’s bid for a job is a Quality of Service specification associated with the job. This QoS specification describes details specific to the job such as the minimum and maximum number of processors required by the job, deadline for when the job must be completed, and agreed-upon payoff amounts for completing the job before the deadline and after the deadline. Thus, jobs will tend towards execution on the more lightly loaded clusters within the federation with the expectation that the overall utilization of all clusters in the federation will be improved. Further, a heavily utilized cluster owned by one organization could barter with another organization’s cluster during a period of heavy utilization, allowing some of the heavily utilized cluster’s jobs to execute on the second cluster, with the expectation that the first cluster would “return the favor” at some point in the future by allowing jobs to be executed on it. Overall, the Faucets system presents a large number of very interesting ideas which we feel are important for utility computing. However, we believe that there are two ways in which Faucets is not directly suitable for our use. First, although Faucets can theoretically be used with any cluster scheduler, the system has been designed to be used primarily with the Faucets scheduler. In some cases, replacing the scheduler in an existing cluster
may not be entirely feasible. Further, the Faucets scheduler is under active development and as such experiences periods of varying stability. Second, Faucets defines its own protocols for communication among the system’s daemons. We believe that a successful utility computing solution should use more open standards such as Web services in order that technologies such as Globus can more easily be leveraged by the system. Table 2 summarizes the academic and research efforts to create utility computing solutions. These projects are perhaps more varied than the vendor efforts described above, however in contrast to the hype and buzzwords surrounding vendor-driven approaches, research approaches often address the specific requirements of particular problems. Further, research approaches often use a wider variety of pre-existing software since they usually do not have affinity to a particular vendor’s suite of software. For these reasons, we believe that research efforts will heavily guide our efforts to create a utility computing environment at the TRECC facility.
4
Critical Analysis of Current Approaches to Utility Computing
In the preceeding section, we examined a number of commercial and research approaches to utility computing. One clear observation from this examination is that the definitions and goals of utility computing vary widely from vendor to vendor. In this section, we present a critical analysis of current utility computing efforts with regard to our goal of creating a utility computing environment at the TRECC facility. A prerequisite for analyzing the suitability of current approaches to utility computing is that we must establish a definition of what we consider utility computing. Therefore, we present such a definition now. We define utility computing as the dynamic allocation of computational resources based on economic factors, with the goal of improving overall resource utilization by establishing a market economy for computational cycles. By this definition, utility computing involves much more than outsourcing applications in advance, dynamically activating processors
Project Purdue University PUNCH
University of Wisconsin Condor
Monash University Nimrod/G
Duke University Cluster-On-Demand University of Illinois Faucets
Summary Provides Web-browser based access to applications and data to users located anywhere on the Internet. Leverages several of the enabling technologies described in Section 2 to provide on-demand access to compute resources with application display redirected to a user’s local desktop. Provides a framework for “high-throughput computing,” accessing as many computational cycles as possible over an extended period of time (e.g., months). Condor jobs may be checkpointed and restarted to move them around across the resources dedicated to a Condor flock. An application for parallelizing and distributing large computational experiments based on the exploration of a range of parameterized scenarios. Applies economic models in conjunction with Globus services to broker access to distributed computational resources. A framework allowing cluster nodes to be dynamically booted into an operating system image that is selected by the user from a set of candidate images. A utility computing system consisting of a cluster batch scheduler along with a set of daemons which implement an economy-based model for allocating resources and bartering for CPU cycles across resources. The system can allocate resources for traditional serial or parallel jobs as well as “adaptive jobs” that can grow and shrink to a varied number of physical processors.
Table 2: Summary of research and academic utility computing efforts
in an existing multi-processor machine, or implementing solutions for self-healing, self-managing, self-recovery, or self-* [24]. Rather, our definition builds on the premiss that the steadily declining cost of computing hardware will enable most organizations to own increasingly powerful computational resources, and further, that the workloads of many of these resources will often be bursty with periods of high utilization punctuated by periods of low utilization. By allocating these resources on the basis of schemes derived from basic economic principles, a market economy for computational cycles can be created which can be used to increase the overall utilization of all resources. Based on our definition of utility computing, we turn a critical eye towards the utility computing solutions described in Section 3. Our analysis focuses on four specific points: 1. Cost is an important criterion for any utility computing solution. Since we define utility computing in terms of resource allocation based on economic factors, it is not surprising that cost plays an important role. As mentioned above, hardware costs have been steadily decreasing to the point where companies can afford to purchase computational capacity that was previously available only to elite supercomputing centers. The real cost of owning computational resources is found in the costs associated with managing these resources. For these reasons, we believe that successful utility computing solutions must focus not on the strategy of replacing costly hardware within an organization, but rather on the strategy of providing occasional access to computational resources to augment an organization’s existing infrastructure while at the same time keeping the maintenance costs incurred by the organization low. A second way in which cost plays a role in utility computing solutions is in the way in which computational resources are priced. For example, what is the base unit cost by which customers are charged for accessing utility computing resources? This question is key in providing a pricing scheme for utility computing and also in enabling customers to determine how many resources they need to purchase for their workload.
2. Ensuring the security of customer data is important. By definition, utility computing involves executing some or all of an application on computational resources that are not owned by the organization submitting the job. The data associated with such applications has the potential to be exposed as it passes over the network and is cached in the memory and storage subsystems of remote machines. Unfortunately, due to the fact that remote resources owned by another organization are under the complete control of that organization, preventing all sources of information leakage is a very challenging problem. We believe that the only hope for a solution to this problem is in current efforts related to Trusted Computing [7] which create a protected portion of a remote machine’s core memory and shield this protected region from access by even the machine’s operating system. 3. Utility computing solutions must be heterogeneous. In this regard, many of the utility computing solutions provided by vendors today fall short because they tend to revolve around each vendor’s hardware and software platforms. As discussed above, we believe that utility computing will most likely involve tying together resources owned by several different organizations, and the expectation follows that these resources are likely to involve hardware and software from a wide variety of vendors. To that end, a successful utility computing solution must address issues related to heterogeneity. Further, this prevents vendor lock-in where a customer becomes dependent on hardware and software from a single vendor due to the fact that switching to the utility computing solution from a different vendor becomes prohibitively expensive. Another point where heterogeneity plays a role in utility computing is in the underlying protocols used to implement components for doing resource allocation, accounting, authentication, and other related tasks. Several existing software packages, some of which are described in previous sections of this paper, already provide proven robust solutions for many of these components. We believe that is important to re-use existing solutions whenever possible. To this end, by designing a utility computing solution around open
standards such as Web services, we hope to leverage the wide variety of existing software packages to speed implementation and deployment of our solution. 4. Application software must be suitable for execution in a distributed environment. In many cases, traditional enterprise applications are not suitable for running in a utility computing environment because these applications often do not consist of multiple discrete components that can be divided among several computational resources. Further, in the case of applications that can be broken into multiple pieces, the application must be able to tolerate the latency involved in having its pieces spread across a wide area network. For example, if two halves of an application are distributed across resources in a utility computing environment, the two halves may be able to communicate internally at low latency but might encounter a a latency of tens or hundreds of milliseconds when communications take place between the halves. A successful utility computing solution must address these types of issues. Based on these criteria, in the next section we describe a proposed solution for creating a utility computing environment with the TRECC facility resources. We believe that our solution will be successful for a number of reasons. First, as we have previously described, under our model of utility computing, organizations will continue to own and maintain their own resources. Our utility computing solution will exist as a software layer above the existing infrastructure. In this way, our solution provides an “added value” to the existing infrastructure; organizations have little to lose and a great deal to gain by deploying our solution. Second, in contrast to vendor-driven utility computing solutions which are required to solve all facets of the utility computing problem simultaneously (e.g., authentication, resource allocation, remote access to files, security), we have the luxury of solving subsets of the problem at a time. Another way of thinking about this distinction is that vendors must work in a “top-down” fashion, starting from an overall design which leads to an implementation. In contrast, we can work in a “bottom-up” fashion starting from the problems at hand and working up to an implementation. This fact leads to the third way that
we believe our solution will be successful. Because of the bottom-up approach we intend to take, we will automatically create a heterogeneous solution which encompasses all of the hardware and software from multiple vendors within the TRECC environment. Finally, because the workloads on the TRECC cluster are heavily skewed towards scientific workloads, they are very good candidates for being distributed across multiple resources.
5
A Web Services Model for Utility Computing
The economy-driven model for utility computing that we have outlined in previous sections of this paper requires four fundamental building blocks. These building blocks are centered around the areas of (1) authentication, (2) resource allocation and scheduling, (3) file storage and remote access, and (4) accounting and charging for resource use. This section describes these four facets of the model in more detail and our overall design for tying them together into a synthesized utility computing solution for the TRECC cluster. The first building block required for deploying a utility computing solution at TRECC is authentication. The primary challenge is to deploy a solution that allows authentication credentials to be created and destroyed easily and quickly. It is expected that in several cases access to the TRECC cluster will be provided to users for a day or two at a time. For this reason, the ability to create new user accounts and have them immediately active on the cluster is critical. Due to the use of a market economy for cycles as described in previous sections, each user on the cluster must have unique authentication credentials so the number of service units spent by each individual can be tracked; the use of traditional “guest” accounts is sub-optimal in this scenario. The second building block required for deploying a utility computing solution at TRECC is resource scheduling. As described previously, resource scheduling under our model of utility computing leverages heavily upon features not found in entry-level batch systems. Examples of such features include the ability to obtain reservations for resources
(used to guarantee that a given job will complete within a set timeframe) or the ability to preempt the execution of a low-priority job in favor of a higher-priority job (which requires the ability to checkpoint, migrate, and restart jobs). To satisfy this goal, we intend to examine the use of the Maui scheduler, the Faucets scheduler, and the Condor scheduler (described in Section 3). The third building block required for deploying a utility computing solution at TRECC is file storage and access to remote data. For the preliminary stages of our project, we intend to simply require users to stage their files on one of three RAID arrays physically connected to the TRECC cluster head node and made available to the compute nodes by NFS. Future stages of the project will allow access to user data across the network by using a Web-based distributed filesystem such as Ufo. The fourth building block required for deploying a utility computing solution at TRECC is accounting and charging for resource use. Because our model for utility computing hinges on the economy-driven allocation of resources, the ability to account for usage of resource is essential. As users make bids for resources and resources complete jobs on behalf of users, each user’s account must be debited accordingly. In the preliminary stages of our project, we envision using a simple database application to keep track of each user’s balance of service units. In later stages of the project, we may evaluate the use of products such as Maui Gold for doing accounting and chargeback. To tie these four building blocks together, we will develop a lightweight software glue layer using Web services. We believe that Web services are an ideal glue technology due to the fact that they can be used to provide a simple front-end to most other software technologies. Further, Web services provide direct interoperability with Grid services that are being developed by the Globus Project. By building an infrastructure around Web services, we can pick and choose Grid services components from the Globus Toolkit as appropriate instead of duplicating existing efforts. Finally, and perhaps most importantly, Web services are composable, which means that we can implement our utility computing solution in phases and then compose the pieces into a larger solution in later stages as the project unfolds. The following
two examples illustrate this point. Figure 1 depicts our vision for the first phase of a utility computing project on the TRECC cluster. In the figure, two users (shown as “1” in the figure) communicate via client software with our Web service glue software (shown as “2” in the figure). This communication first involves the users identifying themselves to the cluster authentication mechanism. In the first phase of our project, authentication is carried out through a simple username and password pair, the same username and password that the users use to log into the cluster interactively. After authentication, communication between the user’s client and the web service involves the negotiation of a bid for running a job on the cluster. To negotiate this bid, the Web service must take several factors into consideration. First, the job criteria (number of processors, deadline, etc.) specified by the user must be considered. Second, the overall utilization of the cluster, as reported by the cluster scheduler (shown as “3” in the figure), must be determined. Finally, the number of service credits held by the user in the accounting system (shown as “4” in the figure) must be examined to ensure sufficient funds. After negotiation, the Web service submits the jobs into the scheduler in order of their negotiated priorities. The scheduler, in turn, allocates cluster resources to the jobs. When each job completes, the Web service again contacts the accounting system to deduct the appropriate number of credits from the user’s account. Future phases of the project will leverage on the composability of Web services to build a solution that encompasses multiple clusters at distributed sites. In this phase, each cluster will behave independently as described in the first phase. A Web service, acting as a broker, will contact the Web services that front-end each cluster to negotiate bids across the set of clusters.
6
Conclusion
In this paper, we have provided a survey of existing efforts in the field of utility computing and have analyzed them with a critical eye towards their applicability to our project of creating a utility computing envi-
ronment at the TRECC facility. Future efforts on this endeavor are to begin building the software infrastructure described in Section 5 of this paper.
References [1] HP Grid & Utility Computing homepage. http://devresource.hp.com/drc/topics/utility comp.jsp. [2] IBM On Demand Business http://www.ibm.com/services/ondemand/.
homepage.
[3] Microsoft Dynamic Systems Initiative homepage. http://www.microsoft.com/windowsserversystem/dsi/dsioverview.mspx. [4] Portable batch system homepage. http://www.openpbs.org/. [5] Sun N1 grid homepage. http://wwws.sun.com/software/solutions/n1/. [6] TRECC homepage. http://www.trecc.org/. [7] Trusted computing group http://www.trustedcomputinggroup.org/home.
Figure 1: A depiction of the first phase of deployment of our utility computing project
homepage.
[8] D. Abramson, R. Sosic, J. Giddy, and M. Cope. The laboratory bench: Distributed computing for parametised simulations. 1994 Parallel Computing and Transputers Conference, pages 17– 27, November 1994. [9] Albert Alexandrov, Maximilian Ibel, Klaus E. Schauser, and Chris Scheiman. Extending the operating system at the user level: the ufo global file system. USENIX’97, 1997. [10] N. J. Boden, D. Cohen, R. E. Felderman, A. E. Kulawik, C. L. Seitz, J. N. Seizovic, and Wen-King K. Su. Myrinet — A gigabitper-second local-area-network. IEEE Micro, 15(1):29–36, February 1995. [11] Ethan Cerami. Web Services Essentials. O’Reilly, February 2002.
[12] Jeff Chase, Laura Grit, David Irwin, Justin Moore, and Sara Sprenkle. Dynamic virtual clusters in a grid site manager. Twelfth International Symposium on High Performance Distributed Computing (HPDC-12), June 2003. [13] T. Richardson et al. Virtual network computing. IEEE Internet Computing, 2(1):33–38, January - February 1998. [14] Message Passing Interface Forum. MPI: A message-passing interface standard. Technical Report UT-CS-94-230, Department of Computer Science, University of Tennessee, April 1994. [15] I. Foster, C. Kesselman, J. Nick, and S. Tuecke. The physiology of the grid: An open grid services architecture for distributed systems integration. Open Grid Services Infrastructure Working Group, June 2002. [16] Ian Foster and Carl Kesselman. The globus toolkit. In Ian Foster and Carl Kesselman, editors, The Grid: Blueprint for a New Computing Infrastructure, pages 259–278. Morgan-Kaufmann, San Francisco, CA, 1999. [17] Ian Foster and Carl Kesselman, editors. The Grid: Blueprint for a Future Computing Infrastructure. Morgan-Kaufmann, July 1999. [18] Ian Foster and Carl Kesselman, editors. The Grid 2: Blueprint for a Future Computing Infrastructure. Morgan-Kaufmann, January 2004. [19] William Gropp, Ewing Lusk, Nathan Doss, and Anthony Skjellum. High-performance, portable implementation of the MPI Message Passing Interface Standard. Parallel Computing, 22(6):789–828, September 1996. [20] L. V. Kale and Sanjeev Krishnan. Charm++: Parallel programming with message-driven objects. In Gregory V. Wilson and Paul Lu, editors, Parallel Programming using C++, pages 175– 213. MIT Press, 1996.
[21] Laxmikant V. Kal´e, Sameer Kumar, Jayant DeSouza, Mani Potnuru, and Sindhura Bandhakavi. Faucets: Efficient resource allocation on the computational grid. Technical Report 03-01, Parallel Programming Laboratory, Department of Computer Science, University of Illinois at Urbana-Champaign, March 2003. [22] Nirav H. Kapadia, Jose A. B. Fortes, Mark S. Lundstrom, and Dolors Royo. Punch: A computing portal for the virtual university. International Journal of Engineering Education, 17(2), March April 2001. [23] Gregory F. Pfister. An introduction to the infiniband architecture. In Hai Jin, Toni Cortes, and Rajkumar Buyya, editors, High Performance Mass Storage and Parallel I/O: Technologies and Applications. IEEE/Wiley Press, New York, 2001. [24] John D. Strunk and Gregory R. Ganger. A human organization analogy for self-* systems. First Workshop on Algorithms and Architecture for Self-Managing Systems, June 2003. [25] Douglas Thain, Todd Tannenbaum, and Miron Livny. Condor and the grid. In Fran Berman, Geoffrey Fox, and Tony Hey, editors, Grid Computing: Making the Global Infrastructure a Reality. John Wiley & Sons Inc., December 2002.
