Peer-to-peer grid computing with the OurGrid ... - Semantic Scholar

Peer-to-peer grid computing with the OurGrid Community Nazareno Andrade1 , Lauro Costa1 , Guilherme Germ´oglio1 , Walfredo Cirne1 1

Laborat´orio de Sistemas Distribu´ıdos – Universidade Federal de Campina Grande Av. Apr´ıgio Veloso, 882 Bloco CO, Bodocong´o – 58.109-970, Campina Grande, PB, Brasil {nazareno,lauro,guiga,walfredo}@dsc.ufcg.edu.br

Abstract. For a number of research and commercial computational problems, it is possible to use as much computing power as available to speed the resolution of the problem through parallel processing. Grid computing has done much in the direction of enabling users to use the computing power of resources across administrative boundaries for solving this kind of problem. However, not much has been done to solve the precedent problem of gaining access to resources spread across several institutions. We have addressed this issue in the OurGrid Toolkit developing the OurGrid Community, a peer-to-peer network for sharing computational power. The goal of this system is to provide easy access to large amounts of computational resources for anyone who needs them. All participants contribute idle resources to form a shared pool from which all can benefit. To motivate the contribution to this pool, the OurGrid Community uses an allocation mechanism that rewards the peers that donate more to the system. This paper describes the OurGrid Community and its first deployment in a grid across Brazil called Pau´a, which is presently being used by several Brazilian research institutes.

1. Introduction Grid computing is the coordination of large collections of resources to provide a platform for the execution of resource-intensive applications. It enables, for example, the aggregation of high-performance resources of several institutions as a single platform to speed up the execution of applications such as data mining, image processing and simulations of drug effectiveness. Several grid computing efforts have made enormous progress on the problems of using large collections of widely distributed resources to run applications in a secure way [Czajkowski et al., 1998, Litzkow et al., 1988]. However, the problem of how this collection of resources is formed in the first place does not have automatic and well adopted solutions. Today, to federate resources from different institutions, there needs to be a multilateral negotiation between all institutions. After that, for a user to obtain access to the grid he needs to negotiate policies of use with its coordinators. We claim that these mechanisms are neither scalable nor flexible enough to allow the creation of arbitrarily-large grids. The OurGrid Community is a peer-to-peer system that addresses the issue of grid assembly for applications that can benefit from best-effort resource allocation. We have taken the approach often used in peer-to-peer systems to design our software as a simple system, stressing ease of deployment in order to ease make its adoption. Doing that, we aim to provide today grids that are useful, scalable and flexible to a significant portion of grid computing users. The key trade-off in the design of the OurGrid Community is that it provides no guarantees on the quality of service obtained from its resources. Each peer in the community is an institution that owns a number of resources and occasionally needs more computing power than these resources can provide. Whenever

a peer needs more power, it requests resources to the community. Whenever it has idle resources, it allocates them to one of the requesters. As there are no guarantees about the quality of service obtained from the idle resources donated to the community, not all applications are suitable for OurGrid. Namely, our focus has been on providing the complete functionality needed to run Bag-of-Tasks applications on it. Bag-of-Tasks applications are those divisible in tasks that don’t need to communicate among them to proceed computation [Cirne et al., 2003]. This kind of application is useful for a variety of problems, such as computer imaging, Monte Carlo simulations, parameter sweep, data mining and biophysics simulations. More complicated classes of applications could be run using the volatile resources obtained from OurGrid, as long as there are applications layers above the resource obtaining one that abstract the failures and heterogeneity from the applications. An important notice is that providing the ability for all peers to access the resources of each other does not solve the whole problem of assembling grids. If all peers have an equal share of the resources shared, why would they bother to contribute to the community? Measurement studies of file-sharing peer-to-peer systems have shown that in these systems most peers freeride: they just consume the resources from the community, without contributing any [Adar and Huberman, 2000, Ripeanu and Foster, 2002, Saroiu et al., 2002]. In the OurGrid case, freeriding is particularly undesired, as the benefit obtained from the community by its users is directly related to the amount of resources available. To encourage resource contribution to the network, OurGrid uses a resource allocation mechanism called Network of Favours [Andrade et al., 2004a]. The Network of Favours is an autonomous reputation scheme that rewards peers that contribute more. This way, there is an incentive for each peer to contribute as much as possible to the system. The OurGrid Community is one of the three pieces that compose the OurGrid Toolkit for BoTs: the OurGrid Community, MyGrid and SWAN. The OurGrid Toolkit is a complete solution for running BoTs applications on grids. Its purpose is to provide to the user (i) abstractions to deal with the grid, (ii) application scheduling, (iii) resource obtaining and (iv) security. From these, the OurGrid Community is responsible for the resource obtaining functionality. All of these functionalities were developed with the design principles of obtaining a solution that is extensible, encompassing and easy to deploy. The toolkit aims to be a tool to foster and support research on grid computing but, more importantly, it aims to be a tool that supports a real community of users that rely on it. It is currently in its version 3.0.2, being used by a number of institutions in Brazil. Finally, it is free and open source, and can be downloaded in www.ourgrid.org. In the rest of this paper we go in detail through the architecture and deployment of the OurGrid Community. In Section 2, we describe the OurGrid Toolkit and the role of the OurGrid Community in it. How the OurGrid Community works is further detailed in Section 3. Section 4 discusses how the software continues to be developed and its current deployment status, and Section 5 illustrates this status with experimental data. Finally, in Section 6, we present related work.

2. The OurGrid Toolkit The three major components of the OurGrid toolkit and their interaction are shown in Figure 1. MyGrid is the user broker when dealing with the grid, the OurGrid Community is responsible for assembling grid to be used by MyGrid instances and SWAN is the component that guarantees resource access is done in a secure way.

Figure 1: Overview of the components of the OurGrid Toolkit.

As the user broker, MyGrid is responsible for providing the user with high-level abstractions that allow him to deal with the grid. The key abstractions provided are tasks, jobs, and grid machines. A job is a collection of tasks, which are the units of work that can be executed in parallel. Each resource in the grid is called a grid machine. The user is responsible for providing a description of his jobs, and an entry point for the OurGrid Community in the form of the URL for an OurGrid Peer. MyGrid uses this information to request grid machines to the peer and to schedule the user application on the resources it gains access to. Jobs, tasks and grid machines have attributes. This allows the user to specify the requirements of jobs and tasks and MyGrid to match tasks to grid machines. The matching can also incorporate scheduling heuristics to improve the performance of different types of applications. MyGrid was the first component of the OurGrid Toolkit to be developed, and it allowed the user to run his applications on all resources he had direct access to. However, it did not help the user to gain access to more resources. Obtaining access to resource in other administrative domains consisted of human negotiations with known administrators. This was the main motivation for the development of the OurGrid Community. With the Community component, users can federate the resources they have access to, creating a shared resource pool. An instance of the OurGrid Community is composed of a set of peers. Each peer represents a site. We call users in the same site local users and all other ones community users. Each local user uses a MyGrid instance to interact with the local OurGrid Peer. Each MyGrid requests grid machines to the peer based on the jobs its user has submitted, and the peer is then responsible for assembling a set of grid machine that satisfy the requests. If a peer does not have all the resources needed to satisfy the requests in its site, it then requests resources to the other peers in the OurGrid community. Each peer is responsible to allocate both the local resources and those obtained from the community among its local users. When a peer provides access to his resources to unknown community users, several security issues emerge. The resource owners need to certify that community users cannot damage the resources made available or use them for malicious purposes. To address this issue, we have developed SWAN, a sandboxing technology that prevents the grid tasks running on a machine from damaging it and from accessing the network. All processes initiated by grid users run in virtual machines isolated from the operating system that runs in the grid machine. These virtual machines have controlled access to the

resources of the grid machine, and cannot access the network interfaces. Note that this works very well with BoT applications, as they don’t need to communicate during their computation. Each task is initiated by the OurGrid peer in a virtual machine, and once its results are collected and returned, the virtual machine is destroyed.

3. The OurGrid Community The purpose of the OurGrid Community is to federate computational resources in a peerto-peer network. As most users in a same site have access to the same set of resources, we have designed the network so that each peer is not a user, but a site. For example, there is just one OurGrid peer running for the Laborat´orio de Sistemas Distribu´ıdos (Distributed Systems Lab) in our University. Each peer has direct access to a set of resources. For a peer to access a resource, there needs to be an implementation of an interface called GridMachine that provides access to that resource. GridMachine defines the minimum set of operations needed to run a task on the resource. Through it the rest of the system can abstract the complexity of dealing with different types of resources. Table 1 shows the five operations defined in the interface, and their semantics. Note that the operations are the simplest possible set of operations that allow one to run a task on a grid machine. This minimizes the effort needed to add new implementations of this interface and allows implementations to be possible to types of resources where more complex operations would not be available. Operation ping() run() putFile() getFile() kill()

Semantics Check the availability of the grid machine. Starts a process the grid machine. Stores data in the grid machine. Retrieves data from the grid machine. Kills a process in the grid machine.

Table 1: Semantics of the operations defined in the GridMachine interface.

The set of local resources that the peer has direct access to is set by its administrator, using the same abstractions created by MyGrid. Currently, OurGrid supports direct access to Linux and Windows machines running an OurGrid Java agent and access to machines in which there is no agent but just a remote login through SSH or RSH. Whenever there are requests from local or community users, the peer tries to satisfy them with the available resources. Each grid machine has a set of attributes that describe it to the users and application scheduler and is used to match the machine with requests. So as to never degrade the benefit the users obtain from their local resources, requests from local users always have priority over those from community users. Whenever the local resources are not enough to satisfy a local request, the peer asks for resources to all other peers in the community. Both the local and obtained resources are allocated to the local users accordingly to a configurable user policy. If there is no request from the local users, the available resources are considered idle, and the Network of Favours is used to decide to whom they should be allocated. The Network of Favours is a resource allocation scheme based on reputation. Peers that donate more resources to the community have a better reputation on it, and are prioritized when they request resources. The whole scheme is autonomous, in the sense that the reputation of other peers is computed by each peer taking into account only information that is locally available. It therefore has much lighter infrastructure requirements

than similar approaches based on global reputation assessments or economic schemes: for example it does not need certified identities, trusted third parties, or e-banking systems. Moreover, it does not require users to estimate the computation requirements that their applications will need. The Network of Favours is discussed in detail in other works by Andrade et al. [Andrade et al., 2004a, Andrade et al., 2004b].

4. Current Status The OurGrid Toolkit, and therefore the OurGrid Community, is currently in its 3.0.2 version, and can be downloaded from www.ourgrid.org. It is free and open source, released under GPL. All functionality needed for an open deployment of the OurGrid Community is already in place, with the exception of SWAN. Although already developed, this component is still being integrated in the toolkit. We currently plan to release it together with the version 3.1 of the software, in the first semester of 2005. Without SWAN, only trusted parties can use our current deployment of the OurGrid Community, and it is not open for anyone to join. The present deployment, called Pau´a, is currently composed of 6 institutions across Brazil, with around 100 non-dedicated resources resources made available. It is presently used for running simulations of the effectiveness of drugs on Brazilian variants of the HIV, to simulate heuristics for scheduling in grid and supercomputer models, and for running data-mining applications. Future applications being developed to use the OurGrid Community include the planning of use of water in the drought-prone Northeast of Brazil. The OurGrid Toolkit is developed by a community of about 40 people spread across a number of Brazilian Universities and research institutions, where contributions are always welcome. As examples of components developed in different institutions we can cite the SWAN component, which was developed by HP Brazil, the OurGrid GridMachine implementation for Windows machines, developed by Instituto Eldorado, MyGrid Scheduler and other GridMachine implementations, developed by UFCG and the toolkit security, developed by Instituto Atlˆantico.

5. An Example of Use To illustrate the behavior of an actual deployed OurGrid Community infrastructure, we present the execution of two applications. These applications have run with resources from two communities: Pau´a and OurGridTestBed. The MyGrid instance used to submit each application received urls to two local OurGrid Peers. The Pau´a community is built on a stable software version, while the OurGridTestBed community is an experimental environment for the OurGrid Toolkit software development. The resources in the latter are machines from several laboratories across our university campus and development partners, containing around 50 machines through 5 peers. The two applications have different purposes. A synthetic application that does not have GridMachine requirements has been executed in order to illustrate the resources available in the communities. In order to show a real-world example, the second application is a real application from a user in our laboratory. It simulates scheduling heuristics for supercomputers. This application can run only on linux machines, and therefore doesn’t use all available resources. To illustrate the resources available in the community over a period of time, in the execution of the synthetic application we have obtained from MyGrid the number of machines every 10 seconds. Figure 2 presents the variation of the number of machines along

Figure 2: Number of machines used over time.

the execution time period. This period comprehends a morning (8 AM to 12 AM). The maximum number of machines was 63 and the mean was 47. This application submitted a job composed by 1000 tasks. Each task takes 566.484 seconds to run on a Pentium IV 2.8 GHz with 1GB or RAM (the fastest machine available) and job completion occurred 4 hours after its submission. Therefore, we have executed it 39.50 times faster than we would have done in hour fastest single machine. The data related to second application was obtained from the log of MyGrid. This application asked for linux machines and submitted jobs from 2 PM to 4 AM of the next day. During the afternoon, the application obtained an average of 29 machines. In the remaining time, it achieved a maximum of 56 machines with average of 47 while the communities did not have any other user. Note that during the evening the average number of machines was not the maximum available. This happened because several machines in the OurGridTestBed community presented misconfiguration on their storage permissions, making it impossible to execute tasks on them. When this happens, MyGrid ignores machines that present such errors. The application has submitted 12 jobs of 100 tasks with an average duration of 395.76 seconds per tasks on a Pentium IV 2.8 GHz with 1GB. The jobs obtained an average duration of 1265.27 seconds rendering a speedup of 31.28.

6. Related Work Providing incentives for resource provision to the grid is an economic problem, and therefore market-based approaches have been proposed to address this issue [Buyya and Vazhkudai, 2001, Buyya et al., 2000, Newhouse et al., 2004, Lai et al., 2004]. However, to the best of our knowledge, none of these proposals is yet being widely used. The reason for that appears to be threefold. First, grid economies require heavy infrastructure, including for example e-caching and commonly trusted certification authorities, and such infrastructure is typically not widely available. Second, smooth running of grid economies may require some technologies that are not fully developed yet, such as third-party auditing of computational services. Third, dealing with the grid as a market implies that users must be able to estimate the application demands in order to submit their job. Experience with supercomputer users suggests that users have a hard time trying to do this [Lee et al., 2004]. To circumvent deployment difficulties imposed by the grid economy, OurGrid

takes the peer-to-peer systems approach. Most current peer-to-peer systems share files and storage rather than computing power, but other proposals for computational peerto-peer grids do exist. Butt et al. have proposed organizing flock of Condor pools in a peer-to-peer network [Butt et al., 2003] and XtremWeb [Fedak et al., 2001] envisions the creation of a peer-to-peer network of collaborators to use a shared volunteer computing platform. Triana [Taylor et al., 2003] and P3 [Oliveira et al., 2002] also provide grid computing platforms based on a peer-to-peer approach. None of these proposals address the issue of incentives for providers of resources. Although peer-to-peer systems can be an effective and robust way of sharing resources, if there is a non-zero cost of donation, and the system does not discriminate between freeriders and collaborators, then peers have an economic incentive to become freeriders, thus reducing the resources available for donation in the community. We have implemented the Network of Favours in OurGrid aiming to provide a peer-to-peer sharing system for computational power that motivates resource sharing by rewarding the provision of resources.

7. Conclusion In this work, we have described the OurGrid Community, a peer-to-peer community for sharing computational resources that is part of the OurGrid Toolkit. The toolkit design guidelines dictate that all of its components must be extensible, encompassing and easy to deploy. The OurGrid Community is extensible and encompassing through its GridMachine interface. It makes possible the transparent addition of new types of resources to the system, and being a simple set of operations, it allows implementations of the interface to be possible to virtually any resource that can receive data and run a computation. With this interface it is also possible to wrap other grid software, such as Globus [Czajkowski et al., 1998] or Condor [Litzkow et al., 1988], in order to provide interoperability. The OurGrid community is easy to deploy as it does not depend on any infrastructure that is not widely available for its deployment. It doesn’t need, for example, a trusted electronic bank or a trusted certification authority. These three goals aim to make possible for users to rely on using the OurGrid Community in production today. We expect that such use of the OurGrid Community allow us to learn from real grid usage, fostering both a wider adoption of such technologies and the research on relevant problems for end users.

8. Acknowledgments This work presents software developed by a team from which the authors are only a small part. We would like to acknowledge and thank the OurGrid Project team and everyone that has contributed to the Toolkit evolution, either with code or use. In particular, Filipe Marques, Danilo Penna, Alexandre Duarte, Flavio Santos, Aliandro Lima e Nigini Ab´ılio were fundamental to the current release of the OurGrid Community. This work was developed in collaboration with HP Brazil R&D and is partially funded by CNPq.

References Adar, E. and Huberman, B. (2000). Free Riding on Gnutella. First Monday, 5(10). Andrade, N., Brasileiro, F., Cirne, W., and Mowbray, M. (2004a). Discouraging Free Riding in a Peer-to-Peer CPU-sharing Grid. In Proc. 13th IEEE Symposium on High Performance Distributed Computing (HPDC’04).

Andrade, N., Mowbray, M., Cirne, W., and Brasileiro, F. (2004b). When Can an Autonomous Reputation Scheme Discourage Free-riding in a Peer-to-Peer System? In Proc. 4th Workshop on Global and Peer-to-Peer Computing(GP2PC). Butt, A. R., Zhang, R., and Hu, Y. C. (2003). A Self-Organizing Flock of Condors. In Proc. Supercomputing 2003. Buyya, R., Abramson, D., and Giddy, J. (2000). An Economy Driven Resource Management Architecture for Global Computational Power Grids. In International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA 2000), Las Vegas, USA. Buyya, R. and Vazhkudai, S. (2001). Compute Power Market: Towards a market-oriented grid. In The First IEEE/ACM International Symposium on Cluster Computing and the Grid (CCGrid 2001), Beijing, China. IEEE Computer Society Press. Cirne, W., Brasileiro, F., Sauv´e, J., Andrade, N., Paranhos, D., Santos-Neto, E., Medeiros, R., and Silva, F. (2003). Grid computing for Bag-of-Tasks applications. In Proc. I3E2003. Czajkowski, K., Foster, I., Karonis, N., Kesselman, C., Martin, S., Smith, W., and Tuecke, S. (1998). A Resource Management Architecture for Metacomputing Systems. In IPPS/SPDP’98 Workshop on Job Scheduling Strategies for Parallel Processing, pages 62–82. Fedak, G., Germain, C., N’eri, V., and Cappello, F. (2001). Xtremweb: A generic global computing system. In IEEE International Symposium on Cluster Computing and the Grid. Lai, K., Huberman, B. A., and Fine, L. (2004). Tycoon: A distributed marketbased resource allocation system. Technical Report cs.DC/0404013, HP Labs. http://arxiv.org/abs/cs.DC/0404013. Lee, C. B., Schwartzman, Y., Hardy, J., and Snavely, A. (2004). Are user runtime estimates inherently inaccurate? In 10th Workshop on Job Scheduling Strategies for Parallel Processing. Litzkow, M., Livny, M., and Mutka, M. (1988). Condor - a hunter of idle workstations. In Proc. 8th International Conference of Distributed Computing Systems. Newhouse, S., MacLaren, J., and Keahey, K. (2002–2004). Grid Economic Services Architecture Working Group home page. http://www.doc.ic.ac.uk/˜sjn5/GGF/gesawg.html. Oliveira, L., Lopes, L., and Silva, F. (2002). P3 (Parallel Peer to Peer): An Internet Parallel Programming Environment. In Web Engineering and Peer-to-Peer Computing Workshop, LNCS 2376, pages 274–288, Pisa, Italy. Ripeanu, M. and Foster, I. (2002). Mapping the Gnutella network: Macroscopic properties of large-scale peer-to-peer systems. In First International Workshop on Peer-toPeer Systems (IPTPS). Saroiu, S., Gummadi, P. K., and Gribble, S. D. (2002). A Measurement Study of Peerto-Peer File Sharing Systems. In Proc. Multimedia Computing and Networking 2002 (MMCN ’02), San Jose CA, USA. Taylor, I., Shields, M., Wang, I., and Philp, R. (2003). Distributed P2P Computing within Triana: A Galaxy Visualization Test Case. In Proc. IPDPS’2003.