A Self-Configurable Desktop Grid System On-Demand - IEEE Xplore

2012 Seventh International Conference on P2P, Parallel, Grid, Cloud and Internet Computing

A Self-Configurable Desktop Grid System On-demand 1

Walid Saad1,Heithem Abbes1,2

2

Christophe Cérin2,Mohamed Jemni1

LIPN/UMR 7030— CNRS/Université Paris 13, 99, avenue Jean-Baptiste Clément, 93430 Villetaneuse, F RANCE {heithem.abbes,christophe.cerin}@lipn.univ-paris13.fr

Research Unit UTIC ESSTT/Université de Tunis, 5, Av. Taha Hussein, B.P. 56, Bab Mnara,Tunis,T UNISIA [email protected], [email protected]

Abstract—Desktop Grids have been used intensively to deploy applications at low cost. Applications types are classified into high throughput computing and high throughput data management. Desktop Grids such as Condor, BOINC, XtremWeb and OurGrid provide a wide range of high throughput computing systems. On the other hand, several data management systems such as Bitdew, GatorShare, Stork, GridFTP, have been developed to handle a huge amount of data. Users are restricted to exploit Desktop Grid resources with a specific computing system and a specific data management system. To address this limitation, we propose in this work an extension of the BonjourGrid middleware, to support different data management systems by exploiting existing data protocols and middleware. With BonjourGrid, users of the same infrastructure can select, each one, the desired Desktop Grid middleware among the most popular ones to deploy and execute their applications. Now, users can select in addition to the computing system, the desired data manager protocol in a transparent and decentralized manner. Experimentation, using BitDew and Stork, on the Grid’5000 platform, demonstrate that the new release of BonjourGrid provides with performance and at a low overhead. Index Terms—Desktop Grids, BonjourGrid, Data Management, Decentralization

data-intensive applications. Very popular examples for this trend are EGEE [25], PPDG [27], Archer [13](Grid appliance [21], [26]), and SETI@Home [24]. Hence data management becomes one of the major challenge of Desktop Grids. In this context, we believe that data management requirements can be classified into:

I. I NTRODUCTION Desktop Grid or Voluntary Computing systems forms one of the biggest computing platforms in using idles resources over Internet or over LAN’s networks. Desktop Grids have been successfully used for solving scientific applications around the world at low cost. Desktop Grids offer a variety of middleware; the most popular are BOINC [6], CONDOR [9], OURGRID [7] and XTREMWEB [11]. However, Desktop Grid computing systems suffer from problems rooted in the architecture itself. First, current implementations follow the old-fashioned master-worker paradigm. Obviously, vulnerability to failures and permanent administrative monitoring are the disadvantages of client-server architectures. To bypass this, a novel system was proposed, called BonjourGrid, able to orchestrate, in a decentralized manner, multiple instances of Institutional Desktop Grid middleware using BOINC, CONDOR and XTREMWEB in order to make a meta-grid middleware. On the other hand, the data requirements of e-Science applications increase dramatically with the emergence of

Grid Computing researchers have developed data management systems such as Stork [15], SRM [16], JuxMem [8], Freeloader [20]. These systems have demonstrated their robustness and reliability in Grid environments. In addition, Bitdew [12] and GatorShare [22] provide a programmable framework for data management in Desktop Grids. In this paper we argue that these systems are easily integrated into Desktop Grids computing system such as CONDOR, BOINC and XTREMWEB. In this way we propose a new self-configurable Desktop Grid system on demand. Our solution focuses on an extension of the BonjourGrid meta-middleware for supporting different data management systems by exploiting existing data protocols and middleware. Now, users can select in addition to computing system, the desired data manager protocol in a transparent and decentralized manner. The paper is organized as follows. In section 2 we outline the scientific issues in Data Desktop Grid computing and we present an overview of the BonjourGrid Meta-Middleware. Section 3 gives a detailed description of our data management approach. Section 4 illustrates

978-0-7695-4841-8/12 $26.00 © 2012 IEEE DOI 10.1109/3PGCIC.2012.6

•

•

196

Data placement activities on Grid environment. Users get difficulties to implement easily their applications. In fact, to deploy an application, user should place manually, using simple script, the data from its site to the computation platform. To accomplish this task, the user must necessarily have a knowledge about the data transfer protocols such as FTP, SRM tools [16], Globus GridFTP [5], and GridTorrent [23]; An efficient high-throughput data management system to support data-intensive applications. In most cases, Desktop Grid middleware uses a centralized mechanism for distributing and managing data. This procedure influences negatively on performance, scalability and fault tolerance of the platform.

(idle, worker or master) when changes occur as well as information about its local load or its use cost, in order to provide useful metrics for the selection of participants. Under these assumptions, the master node can select a subset of workers nodes according to the selection criteria. The master and the set of selected workers build the Computing Element (CE) which will execute and manage the user application. When the execution of an application of a CE terminates, its master becomes free, returns in the idle state and it releases all workers before returning to the idle state too. Then, the nodes can participate to others projects. The key idea of BonjourGrid is to make a meta-grid middleware relying existing Institutional Desktop Grid middleware, and to orchestrate and coordinate multiple instances, i.e multiple CEs, through a publish/subscribe system. Each CE will be owned by the user who has started the master on his machine. Then this CE is responsible for the execution of one or many applications for the same user. As shown on Figure 1, in the user level, a user A (resp. B) deploys his application on his machine and the execution seems to be local. Level 1 (middleware layer) shows that, actually, a CE with 4 (resp. 5) workers has been dynamically created, specifically for the user A (resp. B). Level 0 shows that all machines are interconnected and under the availability of any user. BonjourGrid is based on three fundamental parts (Figure 2) : a) A fully decentralized resources discovery layer, based on Bonjour protocol [19]; b) A CE, using a Desktop Grid middleware such as XtremWeb, Condor or BOINC, which executes and manages the various tasks of applications; c) A fully decentralized protocol of coordination between a) and b) to manage and control all resources, services and CEs .

the implementation process. Then, in section 5, we show the experimental setup and analyze the obtained results. Finally, we present related works in section 6 and we conclude the paper in section 7. II. B ACKGROUND AND M OTIVATIONS In this section we overview challenges and scientific issues in data Desktop Grid computing and we remind the main concepts of the BonjourGrid system and its principal components. A. Scientific issues in Data Desktop Grid computing There are many scientific and enterprise applications that deal with a huge amount of data for instance in parallel visualization. In order to process large data-sets, these applications typically need a high performance computing infrastructure. However, since resources in a desktop grid are typically accessed through wide area network links, the bottleneck comes with the bandwidth limitation. The issue is to imagine architectures that are able to mask (in part) the bandwidth limitation. In our case we choose a decoupling between the source and the computation by the mean of a remote cache and a local cache. Our schema is simple enough to be integrated into major desktop grid platforms. It does not capitalize on the data reuse pattern that can be exploited to achieve better performance. Data re-utilization can be either among tasks of the same particular job or among a succession of job executions of the same or related applications. This issue is left to the data middleware that we plug in our system. Our work is to coordinate computing and data platforms into a unified system. According to Fedak et al. in chapter 11 of [10], in Desktop Grid environments, basic data-management tasks such as reliably storing large data-sets are very difficult to accomplish, first because of the volatility of nodes. Second, data privacy and security must be enforced on Desktop Grids because we deal with untrusted computers. The data protection mechanism may add non-trivial overhead when processing large volumes of data. Third, since the resources are geographically distributed, the design of a scalable data-intensive solution on these systems is an issue. In this work, we focus on the latter issue, the other ones are left as part of either the computing or data exchange systems.

Figure 1.

B. Overview of BonjourGrid System The principal goal of BonjourGrid system [2], [1], [4] is to create, dynamically and in a decentralized way, a specific execution environment for each user to execute any type of applications without any system administrator intervention. Each user, behind a desktop machine in his office, can submit an application. BonjourGrid deploys a master (coordinator), locally on the user machine, and requests for participants (workers). Negotiations to select them should now take place. Using a publish/subscribe infrastructure, each machine publishes its state

BonjourGrid abstract layers

In addition, we have proposed in [3] a fault tolerant mechanism for BonjourGrid to tolerate the faults of coordinators. The basic idea is to create, dynamically, k replicas coordinators for each application. We adopt a passive replication technique (where k is a configuration parameter depending on the size of the grid). If a main coordinator fails, a secondary coordinator is resumed. BonjourGrid tries to keep, for each application, k replicas of the coordinator and if there are enough free available

197

machines in the system. Thus, BonjourGrid allocates dynamically a new machine to replace a coordinator ’leaving’ the system and if the application is not yet completed.

B. Local Cache Our local cache is based either on BitDew or GatorShare. To manage data-intensive applications, BonjourGrid will dynamically create a local cache service. According to the choice of the user, the service starts the data manager (BitDew or GatorShare). Thus, the local cache asks the remote cache (Stork) for the data location in order to put it into its local repository. Then, BonjourGrid starts the chosen computing middleware to run the application. Simultaneously, workers launch a data request from the local cache and start computations. C. Integration with BonjourGrid

Figure 2.

In BonjourGrid [2], each machine can have one of three states: idle, worker or coordinator. Thanks to Bonjour protocol each state is associated to a service; IdleService for free machines, WorkerService for slaves and CoordinatorService for the coordinator. When a machine changes its state, it publishes the corresponding service to notify its new situation, after disabling the previous state. BonjourGrid is based on a completely decentralized protocol that allows coordination between nodes in a transparent manner for the user. For this reason, we must have a self-configuration mechanism and an automatic launching of all components without any intervention from users. Thus, we have added to the CoordinatorService a new field called DataID. This field is the signature generated by the local cache from a compressed file that contains the necessary data for the application. The file is already placed in the remote cache of the coordinator machine. Finally, workers contact automatically the local cache by requesting the file specified by the received DataID.

Abstraction layers of BonjourGrid meta-middleware

III. D ATA M ANAGEMENT A PPROACH The idea consists in creating, dynamically and on demand for each user application, a decentralized data manager in addition to the computing system. The proposed solution supports existing applications without any change in their implementation. The data manager will be initiated and supervised by the BonjourGrid’s coordinator node in parallel with the chosen computing middleware (BOINC , XtremWeb or Condor). Thus, BonjourGrid becomes a Meta-Data Grids that orchestrates multiple simultaneous instances of data managers and computational systems. To realize this approach, we compose the data management framework in two caches that interact continuously in order to maintain the availability of data throughout the execution of an application. Each cache has a set of services. We propose a Remote Cache for data placement activities, in order to reserve the disk space for application and to transfer the required data from the user site to BonjourGrid. Once data are placed on the platform, a second cache called Local Cache is automatically launched to publish and to distribute data over workers.

IV. I MPLEMENTATION In this section we describe the implementation of the data manager modules. Our goal is to create a selfconfigurable computing platform. We try to perform all the deployment steps in a transparent way for the user. This includes data placement on BonjourGrid, data distribution to workers nodes. The philosophy of BonjourGrid is to use existing middleware and data management systems so we have designed an API for interaction between all the components.

A. Remote Cache The literature for data management tools used in large scale distributed systems is abundant and we choose Stork [15] as the remote cache. Stork is a reliable data placement tool for institutional grids. It offers a variety of protocols that may access to different data servers (GridFTP, SRM, SRB, HTPP, FTP,etc.). So, we aim at creating dynamically and on demand for each user a Stork service that may retrieve data from distant data servers and place them on the BonjourGrid’s coordinator node. The user specifies the URL of the data that will be transmitted. This address must contain the storage server and the transfer protocol to use. It is also necessary to specify the required disk space size to reserve and the address of the server where the results (files) will be archived.

A. High level API for inter-middleware communication In order to use simultaneously different data management framework in the computing element layer of BonjourGrid, we have developed a Python API to offer an interface for (see Figure 3): 1) Communication between the job scheduler and the data manager; 2) Inter-caches interaction (between data manager caches); 3) The management of data in each cache (creation, download, distribution, reserve and release disk space, etc.).

198

than 200 machines of the Orsay’s cluster. We set a Xen image of BonjourGrid containing all necessary packages. On each node, we deploy one virtual machine (Debian Linux operation system, 512MB RAM, 32-bit platform, 1Gbps). In the experiment, we select randomly the virtual host where the data manager and middleware will be deployed. In the other hand, we set a specific image of Grenoble’s cluster as a remote data repository. A. Usage scenario To perform experiments, we used bag-of-tasks applications that run data intensive jobs. For each application, input data files are placed in advance on a machine of Grenoble’s cluster. From the Orsay’s front-end machine, we submit the applications with different input data sizes. Now, we focus on a first usage scenario that uses the current BonjourGrid middleware based on Condor middleware for the computing element. In this scenario input data are put by Stork on the coordinator node. Then, Condor File Transfer Manager (CFTM) ensures distribution of data into workers hosts. The second scenario evaluates the new design of BonjourGrid, which simultaneously manages a computing system and data manager protocol. At this stage, data distribution will be carried out by Bitdew. For each scenario, we used 4 applications with parallel jobs, varying from 50 to 200 tasks, 50 to 200 machines on the Orsay’s machines, and input data size applications varying from 100 MB to 1 GB (see table I).

Figure 3. Abstraction layers of BonjourGrid meta-middleware with Data Manager

B. User interaction with BonjourGrid User interaction with BonjourGrid is depicted on figure 4. With the purpose of making the process of applications deployment easier, we have also developed a program which takes as parameters (1) the user configuration file and (2) the list of machines that will be used as coordinators or workers. In the configuration file, users can specify the information about the application (binaries files, job number, workers), the computing system (Condor/Boinc/Xtremweb), the remote cache information (input/output URL data, disk space to reserve with stork, etc.), the local cache setting (system: BitDew/GatorShare, transfer protocol: HTTP/FTP/Bittorrent, replication, fault tolerance etc.). All these settings are specified in a template file (config.cfg).

Table I A PPLICATIONS USED FOR THE EXPERIMENTATION Applications Input Data Size (in MB)

100

300

500

1000

Nodes Number

50..200

50..200

50..200

50..200

B. Results analysis

Figure 4.

In this section, according to the different scenarios mentioned above, we present the evaluation of performance of the new BonjourGrid. Our experiments are performed out to show that BonjourGrid can manage master-worker application (bag-oftasks) with intensive data using Xen virtual machines. We first remind that in the ’old’ BonjourGrid design, coordinator creates and submits jobs to the workers. Before a job will be executed, worker has to download data from the coordinator host. However, with the new data management approach we will follow a different strategy. In fact, coordinator has to first schedule input data to workers nodes by calling the Bitdew framework. Once an input data is in place, the worker starts job execution. The application file is a binary which is required to calculate the size of the input data specified as a parameter. At the end of computation, each job will create a result file containing the size of the output data. Then output data files are transferred to the coordinator with Condor’s transfer mechanism. The metrics we use are as follows:

User interaction with BonjourGrid

V. E XPERIMENTATION AND VALIDATION In this section, we investigate the performance and scalability issues of BonjourGrid with a data manager in performing master-worker computations. The experimentation are conducted on Grid’5000 testbed using more

199

1) Total Completion Time: The results are shown in figure 5. These figures illustrate the total completion time for the 4 applications (see table I) using different number of nodes. For each application, the total completion time contains the time exhausted by the remote cache (Stork) to download input data from user site (Grenoble) to BonjourGrid coordinator node (Orsay), augmented with the building time of a computing element (Condor pool), augmented with the data distribution time generated by the local cache daemon (BitDew), augmented with the job execution time. These figures show that if the data size is less than 350 MB (applications 1 and 2), the ’old’ BonjourGrid gives a completion time equal to the new release (with data manager). Moreover, if the data size exceeds 500 MB (applications 3 and 4), the scenario with a data manager gives the best results. As seen in the table II, computations using data manager features give better completion times for applications composed of a large input data. In fact, the BonjourGrid completion time varies depending on data size and on the number of workers which is not the case with the new approach, since the curve is nearly linear. This fact is good for the scalability of BonjourGrid with a data management mechanism.

(a) BonjourGrid performance with 50 nodes

(b) BonjourGrid performance with 100 nodes

(c) BonjourGrid performance with 200 nodes Figure 5. BonjourGrid performance on a Bag-of-tasks application with different input data sizes executed on 50 to 200 nodes. The two curves present the turnaround time of BonjourGrid without or with Data Manager. In both cases, respectively, input data distribution is performed by Condor Transfer File Manager or by Bitdew using FTP protocol.

Table II T IME TO COMPLETE JOBS WITH 1GB OF DATA Nodes

BonjourGrid Time (in sec.)

BonjourGrid with Data manager Time (in sec.)

50 100 200

840 2100 3900

480 900 1620

cluster and BonjourGrid on the Orsay’s cluster). Table III S TORK TIME FOR REMOTE DATA SCHEDULING : RESERVE DISK SPACE AND TRANSFER INPUT DATA FROM G RENOBLE ’ S CLUSTER TO B ONJOUR G RID SITE (O RSAY ’ S CLUSTER ).

2) Breakdown of total completion time: For further details, in this section we explain the breakdown of the total completion time as mentioned above (see figure 6). As introduced previously, the total completion time is divided into : (1) the time needed by Stork to download input data from user site to execution site (BonjourGrid); (2) the time needed to create and to deploy a computing element (in the BonjourGrid new release, this time will include the time required by BitDew for distributing input data to workers nodes); (3) Jobs execution time with Condor (in the BonjourGrid old release, this time will include the time required by the Condor Transfer Manager for putting input data to workers nodes). • Time using Stork: In the experiments, application tasks depend on one data. In table III, we show the time spent by Stork to place input data of each application. In both scenarios, results show that Stork gives the same time to schedule data which depends on the size of data to be transferred. To schedule data, Stork first starts by reserving disk space required by data, checks to identify which protocol to use in the transfer for both source and destination hosts (Grenoble’s

Applications Input Data Size (in MB)

100

300

500

1000

Stork Time (in sec)

9

44

80

104

In the above example (Listing 1 and Listing 2), we request for Stork to perform the transfer from edel29.grenoble.grid5000.fr to vmgdx-125.orsay.grid5000.fr using the HTTP protocol. The Stork server is started on vmgdx-125.orsay.grid5000.fr host (10.148.0.4) and listens on the 5000 port. In the same host, the Stork client (the submit host) submits the data job by launching the stork_submit command. Listing 1.

Stork Data Submit File

[ p r o t o c o l =" h t t p " ; s r c u r l =" h t t p :// edel − 29. g r e n o b l e . g r i d 5 0 0 0 . f r / app4 . t a r " ; d e s t u r l =" f i l e : /home/ s t o r k /app4 . t a r " ; e r r ="/home/ s t o r k /new . e r r " ; output ="/home/ s t o r k /new . out " ; daptype =" t r a n s f e r " ; ]

200

Listing 2.

transfer every 500 milliseconds. As a consequence, the Bitdew server is potentially unreachable and workers becomes unable to download input data. To bypass the overloading of the Bitdew server, we set an elegant way to solve this issue in using a pipelined transfer of data. Hence, the construction of the computing element will be divided into several stages which depend on the number of workers nodes. • Job Execution Time: Figures 6 show that for all applications the job execution time is less important with the data manager. We prove this result by the data placement strategy. As we discussed above, input data are scheduled on workers nodes before jobs submission, so the job scheduler (Condor scheduler) becomes alleviate and it can schedule, in parallel, all jobs. In contrast, without data manager the scheduler is responsible, simultaneously, for distributing input data and for scheduling jobs. For that reason the scheduler follows a pipelined strategy to schedule jobs.

Stork Server Log

[ protocol = " http " ; s t a t u s =" r e q u e s t r e c e i v e d " ; submithost = " < 1 0 . 1 4 8 . 0 . 4 : 4 9 2 2 3 > " ; dapid = 3 ; timestamp=absTime ("2012 − 01 − 17 T21 : 3 1 : 1 3 − 0 0 0 0 " ) ; remoteuser =" stork@vmgdx − 125. orsay . g r i d 5 0 0 0 . f r " ; output ="/home/ s t o r k /new . out " ; daptype =" t r a n s f e r " ; e r r ="/home/ s t o r k /new . e r r " ; executehost =" "; u s e p r o t o c o l = 0 ; pid = − 1; s r c u r l =" h t t p :// edel − 29. g r e n o b l e . g r i d 5 0 0 0 . f r / app1 . t a r " ; d e s t u r l =" f i l e : /home/ s t o r k /app1 . t a r " ; owner =" s t o r k " ] •

Computing element construction and data distribution time:

Figures 6 show the needed time to create a computing element using 50, 100 and 200 nodes. This time illustrates the real-time for the initialization of different daemons on BonjourGrid platform, computing system (Condor manager and Condor worker) and data management protocols (Bitdew server and Bitdew client). With ‘old’ BonjourGrid case, the result proves that when we use a large number of workers nodes, the time to deploy a computing element increases slightly. In contrast, we find that with data manager, BonjourGrid gives an overhead on all applications. The overhead varies from 117 to 399 seconds. Table IV demonstrates that the overhead is maximal in application number 4 (with 1GB of data). But, when the size of data is small (100Mb and 300 Mb), both cases give nearly the same performance. So, overhead time in the data manager case varies depending on data size and on the number of workers which is an expected result. However we have quantified the overhead. To explain the sources of the overhead, we note that data distribution is performed concurrently with the computing element creation. Indeed, we noticed an additional time to launch the Bitdew Framework. This time is divided into the initialization of Server, putting input data into data repository, setting the signature of file, the time of initialization of Bitdew client on workers nodes and the time to download input data from Server. In addition, let us mention one important aspect of Bitdew’s work. It should be notice that in most cases Bitdew is overloaded and workers nodes cannot download input data. Thus, experience has shown that the number of workers and the size of data affect the response time of Bitdew’s data scheduler. For instance, when we use more than 100 nodes and more than 100Mb of data, we have found that only 65 workers can download data from Bitdew’s data repository. We explain this phenomenon by: 1) It is possible that the Bitdew server looses connections to the workers since it uses the same port to receive all requests of data; 2) the back-off technique adopted by Bitdew’s Data Transfer (DT). In fact, by analyzing the logs of the Bitdew server, we have found that DT heartbeat is configured to monitor

VI. R ELATED WORKS Grid Appliance [21] is a self-configuring Desktop Grid that is used to create a high-throughput computing pool based on Condor middleware. It provides a virtual infrastructure both within a local-area and across wide-area networks via a peer-to-peer virtual network called IPOP. After downloading and installing the virtual machine containing the appliance, participants can submit and execute long-running applications using the command line interface. The Grid Appliance forms the core infrastructure of Archer project [13]. But, the auto-configuration to construct a computing system is insufficient to deal with data-intensive applications. In Archer applications, data is transferred to nodes by a shared WAN NFS. With limited bandwidth, users can not submit jobs with large input data. To address this problem, GatorShare [22] was integrated with Grid Appliance to enable high-throughput data distribution. However, the originally of BonjourGrid compared to Grid Appliance is that it supports multiple middleware (Condor, BOINC and XtremWeb) and data management can be carried out by Bitdew and GatorShare, while Grid Appliance supports only Condor and GatoShare. In addition, with BonjourGrid virtual machine, users can build a local Desktop Grid environment1 . The main efforts to enable self-configuration on Desktop Grid environment where initiated by a flexible tools that address the issues of development, testing and deployment of master-worker applications. Representative example includes PyMW [14] and nuBOINC [17]. PyMW is a python module targets no expert Desktop Grid users by hiding complexity of job submission and results collection. PyMW is designed to provide a unified interface to BOINC and Condor computing environment. The nuBOINC is a set of BOINC extensions that offer a job submission user interface. It allows users, without any 1 http://grid6.cck.rnu.tn/BonjourGrid/

201

Table IV B REAKDOWN T IME TO COMPLETE JOBS WITH 1GB OF DATA Time (in min)

BonjourGrid

BonjourGrid with Data manager

Nodes

CE Construction

Job Execution

CE Construction and Data Distribution

Job Execution

50 100 200

281 646 1166

568 1625 2893

430 833 1565

29 42 63

programming knowledge, to define each job input files and applications parameters required to execute the jobs. Let us note that these tools can be integrated with the BonjourGrid system to provide a fully flexible platform.

is a key point and we will envision the integration of our work as an option of the SlapOS system. A CKNOWLEDGMENT Experiments presented in this paper were carried out using the Grid’5000 experimental testbed, being developed under the INRIA ALADDIN development action with support from CNRS, RENATER and several Universities as well as other funding bodies (see https://www.grid5000.fr).

VII. C ONCLUSION AND F UTURE W ORKS In this work we have proposed a configurable Desktop Grid system on-demand. We have introduced a new release of the BonjourGrid meta-middleware in which multiple computing systems and data management frameworks are orchestrated in a transparent and decentralized manner. With different case studies, we have evaluated the performance and scalability of BonjourGrid with data manager functionality over a large number of machines. Experimentation conducted on Grid’5000 testbed show that BonjourGrid new release can manage data-intensive applications at a low overhead induced by the decentralization of the data distribution. At some stage of the experimentation we have noticed that BitDew (the local cache) can be overloaded if the number of attached workers increases dramatically. Thanks to the flexibility of BonjourGrid we bypass this problem by using a pipelined data distribution. We conclude that BonjourGrid is capable to manage, at the same time, a computing system associated with a data manager using more than 200 nodes in the Grid5000 testbed. Several issues must be taken into account in our future works. The first issue is to evaluate the data management approach with parallel applications with data dependencies. The Desktop Grid synthesis [10] mentions that Desktop Grid technologies may help in coupling Clouds and Grids, in particular for data exchange. SlapOS [18] is an open source Distributed Cloud Computing system created by The Nexedi company. It combines grid computing and Enterprise Resource Modeling (ERP) to provide a new form of distributed cloud such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS) and Software as a Service (SaaS). In [18], we have explained how the Desktop Grid paradigm is reused for the SlapOS system which is a provisioning and billing system for the cloud. SlapOS2 is part of a 2.3M euros FUI project in which we are working on the coordination of servers. The scientific questions and issues are to isolate problems in clouds that could be solved with grid technologies. Data management

R EFERENCES [1] Heithem Abbes, Christophe Cerin, and Mohamed Jemni. Bonjourgrid as a descentralized scheduler. IEEE APSCC, December 2008. [2] Heithem Abbes, Christophe Cerin, and Mohamed Jemni. Bonjourgrid: Orchestration of multi-instances of grid middlewares on institutional desktop grids. Parallel and Distributed Processing Symposium, International, 0:1–8, 2009. [3] Heithem Abbes, Christophe Cérin, Mohamed Jemni, and Walid Saad. Fault tolerance based on the publish-subscribe paradigm for the bonjourgrid middleware. Proceedings of the 2010 11th IEEE/ACM International Conference on Grid Computing, Brussels, Belgium, October 25-29, 2010, pages 57–64, 2010. [4] Heithem Abbes, Christophe Cérin, Mohamed Jemni, and Walid Saad. Toward a meta-grid middleware. Journal of Internet Technology, Volume 11 No1, 2010. [5] William Allcock, John Bresnahan, Rajkumar Kettimuthu, Michael Link, Catalin Dumitrescu, Ioan Raicu, and Ian Foster. The Globus Striped GridFTP Framework and Server. Proceedings of the 2005 ACM/IEEE conference on Supercomputing, 0, 2005. [6] David P. Anderson. Boinc: A system for public-resource computing and storage. Grid Computing, IEEE/ACM International Workshop on, 0:4–10, 2004. [7] Nazareno Andrade, Walfredo Cirne, Francisco Vilar Brasileiro, and Paulo Roisenberg. Ourgrid: An approach to easily assemble grids with equitable resource sharing. Job Scheduling Strategies for Parallel Processing, 9th International Workshop, JSSPP 2003, Seattle, WA, USA, June 24, 2003, Revised Papers, 2862:61–86, 2003. [8] Gabriel Antoniu, Luc Bougé, and Mathieu Jan. JuxMem: An Adaptive Supportive Platform for Data Sharing on the Grid. Scalable Computing: Practice and Experience, 6(33):45–55, November 2005. [9] Ali R. Butt, Rongmei Zhang, and Y. Charlie Hu. A self-organizing flock of condors. J. Parallel Distrib. Comput., 66(1):145–161, 2006. [10] Christophe Cérin and Gilles Fedak. Desktop grid Computing. ISBN10: 1439862141 ISBN-13: 978-1439862148, 2012. [11] G. Fedak, C. Germain, V. Neri, and F. Cappello. XtremWeb: a generic global computing system. Cluster Computing and the Grid, 2001. Proceedings. First IEEE/ACM International Symposium on, pages 582–587, 2001. [12] Gilles Fedak, Haiwu He, and Franck Cappello. BitDew: A data management and distribution service with multi-protocol file transfer and metadata abstraction. Journal of Network and Computer Applications, 32(5):961–975, September 2009. [13] R. J. Figueiredo, P. O. Boykin, J. A. B. Fortes, T. Li, J.-K. Peir, D. Wolinsky, L. K. John, D. R. Kaeli, D. J. Lilja, S. A. McKee, G. Memik, A. Roy, and G. S. Tyson. Archer: A Community Distributed Computing Infrastructure for Computer Architecture Research and Education. Collaborative Computing: Networking, Applications and Worksharing, 10:70–84, 2009.

2 http://www.slapos.org/

202

[14] Eric M. Heien, Yusuke Takata, Kenichi Hagihara, and Adam Kornafeld. Pymw - a python module for desktop grid and volunteer computing. Proceedings of the 2009 IEEE International Symposium on Parallel&Distributed Processing, pages 1–7, 2009. [15] Tevfik Kosar and Miron Livny. Stork: Making Data Placement a First Class Citizen in the Grid. Distributed Computing Systems, International Conference on, 0:342–349, 2004. [16] A. Shoshani, A. Sim, and J. Gu. Storage Resource Managers: Middleware Components for Grid Storage. In Nineteenth IEEE Symposium on Mass Storage Systems, 2002. [17] João Nuno Silva, Luís Veiga, and Paulo Ferreira. nuboinc: Boinc extensions for community cycle sharing. Proceedings of the 2008 Second IEEE International Conference on Self-Adaptive and Self-Organizing Systems Workshops, pages 248–253, 2008. [18] Jean-Paul Smets-Solanes, Christophe Cérin, and Romain Courteaud. Slapos: A multi-purpose distributed cloud operating system based on an erp billing model. IEEE SCC, pages 765–766, 2011. [19] Daniel Steinberg and Stuart Cheshire. Zero configuration networking: The definitive guide, first edition. o’reilly media,inc. December 2005. [20] Sudharshan S. Vazhkudai, Xiaosong Ma, Vincent W. Freeh, Jonathan W. Strickland, and et al. Freeloader: Scavenging desktop storage resources for scientific data. In proceeding of supercomputing, 2005. [21] D. I. Wolinsky and R. J. Figueiredo. Simplifying resource sharing in voluntary grid computing with the grid appliance. Parallel and Distributed Processing, 2008. IPDPS 2008. IEEE International Symposium on, pages 1–8, April 2008. [22] Jiangyan Xu and Renato J. O. Figueiredo. Gatorshare: a file system framework for high-throughput data management. Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, HPDC 2010, Chicago, Illinois,USA, June 21-25, 2010, pages 776–786, 2010. [23] Antonis Zissimos, Katerina Doka, Antony Chazapis, and Nectarios Koziris. GridTorrent: Optimizing data transfers in the Grid with collaborative sharing. In: Proc. of the 11th Panhellenic Conference on Informatics (PCI’07), Patras, Greece,. [24] Seti@home. http://setiathome.ssl.berkeley.edu. [25] Enabling grids for e-science in europe. http://www.eu-egee.org. [26] Grid appliance. http://www.grid-appliance.org. [27] Particle physics data grid. http://ppdg.net.

Execution Job CE Construction / Distribution Data

900 800

Time (Second)

700 600 500 400 300 200

100 MB

300 MB 500 MB #Application Data Size in MB

BonjourGrid

DataManager

DataManager

BonjourGrid

BonjourGrid

DataManager

BonjourGrid

0

DataManager

100

1000 MB

(a) Breakdown of Total Completion Time with 50 nodes Execution Job CE Construction / Distribution Data

2,500

Time (Second)

2,000 1,500 1,000

100 MB

300 MB 500 MB #Applications Data Size in MB

BonjourGrid

DataManager

DataManager

BonjourGrid

DataManager

BonjourGrid

DataManager

0

BonjourGrid

500

1000 MB

(b) Breakdown of Total Completion Time with 100 nodes Execution Job CE Construction / Distribution Data

4,500 4,000

Time (Second)

3,500 3,000 2,500 2,000 1,500 1,000

100 MB

300 MB 500 MB #Application Data Size in MB

BonjourGrid

DataManager

DataManager

BonjourGrid

DataManager

BonjourGrid

DataManager

0

BonjourGrid

500

1000 MB

(c) Breakdown of Total Completion Time with 200 nodes Figure 6. Breakdown of total completion time per application in time to deploy a computing element, time to distribute input data and time to execute jobs.

203