On using Virtual Machines for Adaptive Distributed Query Processing ...

COMPUTING SCIENCE On using Virtual Machines for Adaptive Distributed Query Processing in Grids A. Mukherjee and P. Watson.

TECHNICAL REPORT SERIES No. CS-TR-1084

April, 2008

TECHNICAL REPORT SERIES No. CS-TR-1084

April, 2008

On using Virtual Machines for Adaptive Distributed Query Processing in Grids Arijit Mukerjee and Paul Watson Abstract Grid computing has made it possible for users to perform computationally expensive applications on dynamically acquired distributed resources. Users can combine data and analysis components distributed over the globe to build new, complex applications. Distributed query processing is now an established way of structuring computation and analysis over structured data, and well-used Grid tools like OGSADAI and OGSA-DQP provide respectively a common interface to heterogeneous databases and a way of exploiting distributed resources. Unfortunately, the potential benefits are often undermined when databases and computational services are distributed over a wide area resulting in significant computation and communication costs. This paper describes a system that addresses this by allowing key components including query operators, computational services and databases to be dynamically deployed in virtual machines over a Grid. Databases are replicated within virtual machines which are automatically deployed by the system which monitors the performance over a set of queries using a feedback mechanism. Results of internetscale experiments are presented to show the performance benefits.

© 2008 University of Newcastle upon Tyne. Printed and published by the University of Newcastle upon Tyne, Computing Science, Claremont Tower, Claremont Road, Newcastle upon Tyne, NE1 7RU, England.

Bibliographical details MUKHERJEE, A., WATSON, P., On using Virtual Machines for Adaptive Distributed Query Processing in Grids [By] A. Mukherjee, P. Watson Newcastle upon Tyne: University of Newcastle upon Tyne: Computing Science, 2008. (University of Newcastle upon Tyne, Computing Science, Technical Report Series, No. CS-TR-1084)

Added entries UNIVERSITY OF NEWCASTLE UPON TYNE Computing Science. Technical Report Series. CS-TR-1084

Abstract Grid computing has made it possible for users to perform computationally expensive applications on dynamically acquired distributed resources. Users can combine data and analysis components distributed over the globe to build new, complex applications. Distributed query processing is now an established way of structuring computation and analysis over structured data, and well-used Grid tools like OGSA-DAI and OGSA-DQP provide respectively a common interface to heterogeneous databases and a way of exploiting distributed resources. Unfortunately, the potential benefits are often undermined when databases and computational services are distributed over a wide area resulting in significant computation and communication costs. This paper describes a system that addresses this by allowing key components including query operators, computational services and databases to be dynamically deployed in virtual machines over a Grid. Databases are replicated within virtual machines which are automatically deployed by the system which monitors the performance over a set of queries using a feedback mechanism. Results of internet-scale experiments are presented to show the performance benefits.

About the author Arijit joined the Computing Science department as a Research Associate in the MyGrid project in May 2002. He did his Masters in Computer Science and Engineering from Jadavpur University, India in 1997, and since then worked in the software industry. He joined Tata Consultancy Services (the largest software consultancy in South East Asia) in 1997, and worked in the Networks Group in various projects for the telecommunication companies like Bell Atlantic, Nokia, Siemens. His major projects were related to Residential Broadband (Bell Atlantic), Intelligent Network (Nokia), Corporate GSM (Siemens/Opuswave). In 2000, he joined Verizon Communications Inc. (formerly Bell Atlantic) in the US as a software engineer and worked there on Verizon's Data Network Management System till May 2002.Currently, Arijit works in the MyGrid project as well as the Distributed Query Processing of OGSA-DAI. He is also doing his PhD under Prof. Paul Watson. Paul Watson is Professor of Computer Science and Director of the North East Regional e-Science Centre. He graduated in 1983 with a BSc (I) in Computer Engineering from Manchester University, followed by a PhD in 1986. In the 80s, as a Lecturer at Manchester University, he was a designer of the Alvey Flagship and Esprit EDS systems. From 1990-5 he worked for ICL as a system designer of the Goldrush MegaServer parallel database server, which was released as a product in 1994. In August 1995 he moved to Newcastle University, where he has been an investigator on research projects worth over £13M. His research interests are in scalable information management. This includes parallel database servers, data-intensive e-science and grid computing. In total, he has over thirty refereed publications, and three patents. Professor Watson is a Chartered Engineer, a Fellow of the British Computer Society, and a member of the UK Computing Research Committee.

Suggested keywords VIRTUAL MACHINES, DISTRIBUTED QUERY PROCESSING, ADAPTIVE, GRID

On using Virtual Machines for Adaptive Distributed Query Processing in Grids Arijit Mukherjee School of Computing Science Newcastle University Newcastle Upon Tyne, NE1 7RU, UK [email protected]

Abstract Grid computing has made it possible for users to perform computationally expensive applications on dynamically acquired distributed resources. Users can combine data and analysis components distributed over the globe to build new, complex applications. Distributed query processing is now an established way of structuring computation and analysis over structured data, and well-used Grid tools like OGSADAI and OGSA-DQP provide respectively a common interface to heterogeneous databases and a way of exploiting distributed resources. Unfortunately, the potential benefits are often undermined when databases and computational services are distributed over a wide area resulting in significant computation and communication costs. This paper describes a system that addresses this by allowing key components including query operators, computational services and databases to be dynamically deployed in virtual machines over a Grid. Databases are replicated within virtual machines which are automatically deployed by the system which monitors the performance over a set of queries using a feedback mechanism. Results of internet-scale experiments are presented to show the performance benefits.

1. Introduction Virtual organisations built over Grids[14] allow collaborative sharing of computational and data resources over a wide area network. To support this, several tools, such as Globus[7], Condor[27] and SunGridEngine[6] allow the construction of distributed applications over grid-based resources. Many grid applications need to collate structured data from several databases distributed across a wide geographical area [23, 26]. Distributed query processing techniques offer a convenient way for combining and filtering data from multiple repositories and applying one or more analysis operations to the resulting dataset[19]. Achieving good performance for such data-centric ap-

Paul Watson School of Computing Science Newcastle University Newcastle Upon Tyne, NE1 7RU, UK [email protected]

plications offers key challenges, especially within the Grid setting. In the case of widely distributed data sources, a high data transmission cost between the data source and analysis resources affects the performance for long-running computations retrieving large amounts of data. Within a service-oriented setting, the communication takes place through the exchange of SOAP messages and the overhead increases for larger amounts of data resulting in larger service invocation cost[15][12]. Serialisation of data leads to an expansion of about 10 times the size of the equivalent binary representation[13]. This causes significant additional cost for data transmission during distributed query processing[20]. One approach to avoid this network overhead is by collocating the analysis services with the data [29], although a high data transfer cost may still be incurred in case of experiments returning a large data-set to the user. For experiments which do not rely on the most recently updated data, such as analysing data from bio-informatics databases such as EMBL [2], an alternative approach is to employ data replication, which requires a lengthy administrative process. When multiple databases and analysis services are involved, an automatic way of deploying database snapshots and computational services is needed; achieving this is the focus of this paper. The approach described in this paper utilises virtual machine techniques to allow on demand deployment of computational services and database snapshots on available computational resources in order to benefit the query processing activities by extending OGSA-DQP[8] with the dynamic deployment features supported by DynaSOAr[28]. The resulting DQP framework thus is able to reconfigure itself at run-time based on the performance of the system by collocating several computational entities, deploying multiple copies of databases and computational services as and when required. The virtual machine techniques adopted in this approach also account for specialised services requiring a specific environment or fine tuning with the hardware, which are common in many e-Science experiments. The paper is organised as follows: Section 2 gives a

background of OGSA-DQP and DynaSOAr. The architecture of the extended DQP system with support for dynamic deployment is outlined in Section 3. In Section 4, several internet-scale experiments are described and their results analysed, followed by the related work in Section 5 and conclusions in Section 6.

2. Background OGSA-DQP[8][10], which builds on OGSA-DAI[3], is a well known service-oriented distributed query processing system for a grid environment. Based on earlier work on parallel query processing techniques discussed in [25], OGSA-DQP provides a framework that supports queries over standardised “Grid Data Service” (GDS) and other analysis services made available over the grid thereby combining data access with data analysis, and at the same time, adapts techniques from parallel databases to provide implicit parallelism for complex data-intensive requests. Two major services, namely the Grid Distributed Query Service (GDQS or the coordinator) and the Query Evaluation Service (QES or the evaluator), constitute the OGSADQP framework. The coordinator extends OGSA-DAI, and is deployed as an OGSA-DAI data service exposing a data service resource[4]. It supports querying over a set of OGSA-DAI data services, each wrapping a database on some computational node. During the initialisation phase, a global schema is constructed over the resources to be integrated and then queries expressed in SQL may be submitted. The coordinator compiles the query to generate a parallel execution plan in the form of a data flow graph, which is evaluated in a pipeline fashion with elements deployed on the distributed resources. A query execution engine comprised of a collection of evaluators is created at run-time for processing each query. Each evaluator is a WS-I compliant Web Service[9] encapsulating the physical algebra operators for query processing and is responsible for processing partitions of a query execution plan by communicating with other evaluators and third-party web services. This version of OGSA-DQP is fairly static in the sense that it utilises the resources that are known during the initialisation phase and hence, there is no notion of reconfiguration based on the performance of executing queries. Such configuration increases the probability of higher data access costs when databases are remote and higher data transmission costs between widely distributed computational services. To overcome these issues by allowing the framework to reconfigure itself when necessary, OGSA-DQP was extended to incorporate the dynamic deployment features provided by DynaSOAr[28] which enables dynamic deployment of services and virtual machines containing specialised services and database snapshots on available computational nodes. It offers a service-only view for dis-

tributed application design, where, when no existing deployments of the required service can meet the computational requirements, a copy, if available, is deployed dynamically on a suitable host in order to exploit the computational resources exposed by it. Dynamic deployment can be exploited within DQP in a variety of scenarios, such as collocation of key components to minimise the cost of data transport, induced parallelism during the analysis, availability etc. In this paper, we discuss two scenarios where virtualization techniques have been successfully used. Data Caching - Many e-Science experiments such as analysing the data from bio-informatics databases such as EMBL [2] are not particularly reliant on the most up-to-date data. Virtual machine techniques supported by DynaSOAr can be exploited in such scenarios by deploying a virtual machine with a snapshot of the database as an OGSA-DAI service closer to the computational entities. Contrary to the concept of moving the computation closer to the data, this resembles data replication, with the exception that unlike conventional data replication techniques, the requirement of a database management system on the target host and the intervention of a database administrator are not required. Based on the current performance collected using a feedback methodology, the framework reconfigures itself by deploying one or more database snapshots within the local network. A separate background process however is required to periodically synchronise the snapshot with the actual data set that was important for the application. Services with special requirements - Scientific workflows often involve specialised services such as Blast[1], which requires a special environment combining a set of libraries and a database, all of which together can be encapsulated within a virtual machine, and deployed as and when required. Further, scientific applications are often tuned with the host for optimal performance depending on the processor architecture, available memory, disk space etc. The tuning process is manual, time-consuming and is impossible to automate. A flexible alternative approach may be to package the application or service in a virtual machine, tune it and store in a repository for on-demand deployment on available resources which match the requirements.

3. Towards a Dynamic Distributed Query Processor Demand driven deployment of services and databases within OGSA-DQP is achieved by extending the framework to incorporate some DynaSOAr components such as the Host Provider, Software Repository and Service Registry, with the decision-making process for dynamic deployment embedded within the coordinator. Loose coupling with the DQP framework allow these new services to be shared with other services or frameworks requiring a software reposi-

tory or a UDDI registry. Fig. 1 shows an overview of the extended OGSA-DQP architecture. To allow dynamic deployment of services, the computational entities are exposed as HostProviders which make themselves known to the registry. This makes the query evaluation and analysis services optional on the computational nodes as the HostProvider is capable of deploying the necessary services at the run time. The deployable services are uploaded to the Software Repository and registered at the registry with a reference to the location of the repository as part of the registry entry.

Figure 1. Basic DQP Architecture

The DQP system is initialised with a configuration document specifying the supported data and analysis resources based on which the schema and metadata are imported from the data/analysis services. Additionally, an estimate of the network latency time between the available resources and the actual data sources is computed by sending and receiving a pre-calculated packet, which is later used during the query optimisation phase to schedule scan and operation call operators to specific hosts based on connectivity. Further, during this phase, the coordinator takes a proactive decision to deploy any analysis service specified in the configuration document if it is available in the repository. This ensures that the optimiser is able to consider multiple instances of the analysis service thereby parallelising the operation call operator with a view to maximise the performance. During the query compilation/optimisation phase, the coordinator takes further decisions regarding prospective deployment of the evaluation services on available nodes with a view to collocate various entities. A scheduler responsible for parallelising the physical plan into several partitions and assigning operators to the available computational resources first introduces intra-operator parallelism to some physical operators such as join and operation call by introducing exchange operators. It then assigns operators to specific computational resources by considering the computational metadata that describes the characteristics of each node, and the most recent network latency information

for each available node. It uses certain heuristics during this allocation phase which ensures that the partitions with data accessing operators are assigned to evaluators on the same node as the data, or a node closest to it, resulting in a collocation of the data and the evaluation engine. A similar approach is taken for the operation call operator which encapsulates the invocation to the analysis service resulting in further collocation of entities. As all available resources are considered in this phase, a requirement may arise for the dynamic deployment of services on some nodes. The dynamic deployment framework sends a deployment request to the corresponding nodes, which download and deploy the required services, and update the registry about the new deployments, so that these instances can be considered for subsequent queries without any further need for dynamic deployment. Thus, unlike the original OGSA-DQP where a tightly coupled set of resources are used with the data access, evaluation and analysis services pre-deployed on them, the extended version tries to exploit the dynamic deployment features by using only a collection of data hosts, and computational resources allowing dynamic service deployment. Each node is effectively “created” by combining a number of components as and when required. The DQP system, in this case, has the option of choosing the services to deploy, either based on the user preferences, or the quality of service parameters, or even predefined service level agreements. The dynamic version of DQP thus incorporates a flexibility to the framework which was absent in the earlier versions.

3.1. Using Virtualization Virtualization is used here as an alternative to data replication in scenarios where frequent queries are submitted against the same dataset and some analysis is performed on the result, a common case in many scientific domains where the most updated data is not a mandatory requirement. In this extended version of DQP, a snapshot of the corresponding dataset is deployed in a virtual machine(VM) and stored as any other deployable service in the repository, so that it can be deployed on demand on hosts with a suitable environment for hosting the VMs. All the dynamic deployment options, such as collocation and proactive service deployment for parallelism are exploited during the query compilation/optimization phase, but for caching the data or deploying the data closer to the computation, a different methodology of feedback is used. Several performance measurements, such as number of tuples transferred, total cost of transferring the data, the total cost of the actual evaluation process etc. are collected from each of the participating evaluation services after the completion of each query evaluation process and are analysed by a background

process inside the coordinator. It correlates the total execution cost for the query with the cost of transporting the data from remote nodes and calculates the trend incorporating all the previous data received. Based on this analysis, if it is found that the data transmission cost is the major contributor to the total query execution cost and is following an increasing trend, and is more than the cost of deploying the database locally, the process of reconfiguring the DQP system by deploying the VM on a local node is initiated if a snapshot of the database is available. It is also possible to configure the process in such a way that the consumer specifies the maximum cost (in terms of the maximum amount of time required to invoke a service, or the maximum amount of time required to transfer partial results from the evaluators) that is acceptable during the evaluation of a query. In a scenario where a remote provider charges the consumer based on the amount of data being transmitted, the reconfiguration may be triggered when this cost at the provider site surpasses the maximum allowable cost specified by the consumer. Fig. 2 outlines the process of collecting performance feedback and reconfiguring the corresponding data resource.

to this particular instance or session of DQP are processed within this new DQP data resource. A new consumer can however initiate a new DQP session, with the same configuration as the original, in which case the original resources will be used, unless a reconfiguration is called for. The cost of the second reconfiguration may be minimal as it may be possible to reuse the previous deployment of the virtual machine.

In Fig. 2(a), the coordinator after receiving the query from the client (in step 1) performs the compilation/optimisation of the query and dynamically deploys necessary services on participating nodes (step 2). The query execution process follows a sequence of sending the query partitions to respective evaluation services (step 3), evaluating each partition by communicating with other evaluation and analysis services (step 4, 5, 6 and 7) and finally sending the complete result to the coordinator and hence the client (step 8). Once the query execution is complete, the coordinator sends a request to all participating nodes requesting for the performance data (step 9). The feedback from all the nodes are analysed, and based on the analysis, if the criteria for deployment is satisfied, a virtual machine which encapsulates the data source, data access and analysis services, is deployed within the local network (Fig. 2(b)) which creates a new configuration for the DQP data resource. During the reconfiguration phase, the existing resource which was processing the earlier queries remains alive, and a new data resource is created, which after the configuration is completed, replaces the former. Once the VM is instantiated properly, the schema from the newly deployed data resource and other services must be imported by the DQP data resource. The endpoint of the data service which was pointing to the actual remote database is overwritten with the endpoint of the new data service on the VM. Any instance of the evaluation service or the HostProvider service on that remote node are excluded as well, as all these services are locally available after the VM initialisation. Once this is done successfully, the former DQP data resource is replaced with the new one, and all further queries submitted

(a) Query evaluation and feedback collection

N0

GDQS Data Resource OQLQueryStatement Activity

8

Query Evaluation Service

print exchange

4 exchange Query Evaluation Service

7

reduce op_call

6

5

join

Analysis Service

9

exchange join

OGSA−DAI DSR 1

2

exchange

3

reduce

Data Source

N1

exchange Query Evaluation Service

scan

reduce 6

1

8

op_call

Analysis Service

join

exchange

exchange

OGSA−DAI DSR 2 Data Source

reduce

Query Evaluation Service

reduce scan

op_call 6

N2

join

Analysis Service

exchange join

OGSA−DAI DSR 3 exchange Data Source

Client

reduce scan

N3

11111111111111111111111111111111111111111111111111 00000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000 11111111111111111111111111111111111111111111111111 N0 00000000000000000000000000000000000000000000000000 11111111111111111111111111111111111111111111111111 00000000000000000000000000000000000000000000000000 11111111111111111111111111111111111111111111111111 00000000000000000000000000000000000000000000000000 11111111111111111111111111111111111111111111111111 00000000000000000000000000000000000000000000000000 11111111111111111111111111111111111111111111111111 00000000000000000000000000000000000000000000000000 11111111111111111111111111111111111111111111111111 00000000000000000000000000000000000000000000000000 11111111111111111111111111111111111111111111111111 00000000000000000000000000000000000000000000000000 11111111111111111111111111111111111111111111111111 00000000000000000000000000000000000000000000000000 11111111111111111111111111111111111111111111111111 00000000000000000000000000000000000000000000000000 11111111111111111111111111111111111111111111111111 00000000000000000000000000000000000000000000000000 11111111111111111111111111111111111111111111111111 00000000000000000000000000000000000000000000000000 11111111111111111111111111111111111111111111111111 00000000000000000000000000000000000000000000000000 11111111111111111111111111111111111111111111111111 00000000000000000000000000000000000000000000000000 11111111111111111111111111111111111111111111111111 00000000000000000000000000000000000000000000000000 11111111111111111111111111111111111111111111111111 00000000000000000000000000000000000000000000000000 11111111111111111111111111111111111111111111111111 00000000000000000000000000000000000000000000000000 11111111111111111111111111111111111111111111111111 VMW−1 00000000000000000000000000000000000000000000000000 11111111111111111111111111111111111111111111111111 Local Network

GDQS Data Resource OQLQueryStatement Activity

7

Query Evaluation Service

print

exchange

3

exchange

Query Evaluation Service

6

reduce

op_call

5

4

join

Analysis Service

2

exchange

join

OGSA−DAI DSR 1

exchange reduce

Data Source

scan

exchange

Query Evaluation Service

1

reduce op_call

Analysis Service

join

5 OGSA−DAI DSR 2

exchange

exchange

Data Source

reduce

Query Evaluation Service

reduce scan

op_call 5

N2

Analysis Service

join exchange join

8

OGSA−DAI DSR 3 exchange Data Source

Client

reduce scan

N3

(b) Reconfiguring the data resource by deployment of a VM

Figure 2. Reconfiguration in dynamic OGSADQP

4. Experiment A extensive experimental setup for evaluating the dynamic DQP framework consisting of five test data resources and the HostProvider service was created on a set of Linux machines within the Newcastle University GIGA cluster each of them being a four-processor Intelr XeonT M CPU 2.80GHz system, with 2GB memory. The GDQS was deployed on a desktop running on Windows XP (Service Pack

2) machine - a four-processor Intelr Pentium(R)T M CPU 3.0GHz with 2GB memory. The same machine which hosted the GDQS also hosted the Software Repository and the Service Registry. A copy of the analysis service was deployed on a Linux (one-processor Intelr XeonT M CPU 2.40GHz system with 1GB memory) system at the Edinburgh Parallel Computing Centre (EPCC). To compare the results with a real-world situation, an exactly similar DQP framework with databases, data access, and HostProvider services was set up on a set of Planetlab[5] nodes, with compute resources located at geographically remote locations, such as in Toronto, Tokyo, Berkeley, and several European cities. Each database was replicated on five nodes to avoid experimental problems due to node failures. The local network was a high speed 100Mbps ethernet, and the connection with the PlanetLab network being through the JANET [24], a high speed gigabit ethernet between the universities in the UK and GREN [11], the Global Research and Educational Network. Experiments were designed to fetch data out of each table with varying cardinalities, optionally perform a join with data from another remote database and perform the analysis on each tuple using the analysis service. Results were collected in order to compare the performance of the various setups with static and dynamic configurations on local and remote PlanetLab nodes. Two virtual machines were used to investigate the deployment of a database snapshot containing a copy of the ProteinSequence and ProteinProperty databases respectively with the necessary OGSA-DAI services. The evaluation and analysis services were also pre-deployed to complete the virtual machine packaging. The registration of these virtual machines with the registry resulted in the announcement of the availability of all the packaged services and the data resources, allowing a consumer to discover them.

4.1. Comparing a Virtual Machine Performance To compare the performance of virtual machines with respect to physical resources, a set of experiments involving the same distributed query retrieving data of different cardinalities from the database and invoking an analysis service on each tuple before returning the result to the consumer were performed on various setups. The results of these experiments are compared in the graphs Fig 3. It can be seen from the results that the performance of the query when executed within a virtual machine is comparable to the performance of the same query when executed on nodes within the local network. The cost of invoking the analysis service, even when the service was hosted at EPCC, Edinburgh, was extremely high compared to the cost when the service was local (either on a VM or a real host). The costs were even

higher when the participating nodes were remote, such as in the case of the PlanetLab nodes.

Figure 3. Execution comparison on EPCC, local and VM nodes

4.2. Deploying a Database Snapshot Locally To investigate the possibility of using virtual machines for deploying database snapshots, a set of queries each accessing large amounts of data with varying cardinalities from the databases were used repeatedly, so that each query retrieved larger and larger amounts of data and transmitted them over the network to other services involved in the processing. Each experiment was performed with the following configurations: 1. a setup where the databases were hosted on PlanetLab nodes and the local giga cluster nodes were used as available computational resources supporting dynamic deployment, resulting in remote data access. 2. a setup where the databases, data services and the dynamic deployment framework were all deployed on PlanetLab nodes, resulting in local data access, but larger cost of transporting the data. 3. a setup where the databases, data services and the dynamic deployment framework were all deployed on the local giga cluster nodes, resulting in an entirely local query execution setup. 4. a setup where the performance feedback model was enabled and the DQP system was allowed to deploy a database snapshot by deploying the corresponding virtual machine within the local network. Performance data were collected using the feedback model at the end of each query by the coordinator entity.

In the setup where reconfiguration was enabled, the performance results were analysed based which, the reconfiguration process was triggered if required, which led to a local deployment of the database snapshot. Further queries were directed to the newly configured DQP data resource which utilised the new configuration in order to benefit from the ad-hoc virtual organisation created by the new resource deployment. It can be seen in the graphs in Fig. 4(a) that the cost of transporting the data was considerably lower after a reconfiguration was completed by the coordinator. During the query processing, once the increasing trend in the cost of data transport was detected for both nodes hosting the ProteinSequence and ProteinProperty databases respectively, the coordinator deployed a snapshot for both of them on the local network. After the reconfiguration, the transport cost for both nodes reduced dramatically, as a result of which the overall execution cost was also reduced (shown in Fig. 4(b)). Comparing transport cost for ProteinSequence and ProteinProperties 400000

ProteinSequence-PL ProteinProperties-PL ProteinSequence-SW ProteinProperties-SW

350000

Cost in milliseconds

300000

250000

200000

An interesting observation was made in another set of experiments (Fig. 5) with a select-project-join-operation call query where an analysis service was invoked on each retrieved tuple after performing a join. The ProteinProperty database was deployed locally first as the coordinator detected an increasing trend in the transport cost from this resource. This resulted in a momentary higher cost of data transport for the node hosting the ProteinSequence database. The reason behind this can be explained by considering the location of the participating nodes and the evaluation process in DQP. All the participating nodes in DQP exchange data between themselves during evaluation. The join operation in this query was parallelised on two nodes hosting the two relevant databases. Initially, both nodes were from the PlanetLab domain and located in the USA, which meant that they were closer in terms of network latency than when the ProteinProperty database was deployed locally at Newcastle. When the first reconfiguration took place, the network latency between the new node on the local network and the PlanetLab node hosting the ProteinSequence database increased as they became farther apart in terms of network connectivity resulting in an increased data transmission cost between them thereby affecting the overall query execution cost(Figure 5(b)). The DQP coordinator subsequently reconfigured the ProteinSequence database by deploying the snapshot on a local node, and the transport cost as well as the overall query execution cost reduced to what is normally observed within the local network.

150000

5. Related Work

100000

50000

0 0

5000

10000

15000

20000

25000

30000

35000

40000

Number of tuples

(a) Comparing data transport cost from the datanodes Comparing total execution cost in different setups for a select-project-join qu 500000

execution-PL execution-SW

450000

Cost in milliseconds

400000 350000 300000 250000 200000 150000 100000 50000 0

0

5000

10000

15000

20000

25000

30000

35000

40000

Number of tuples

(b) Execution cost of a query for a remote and switching setup

Figure 4. Comparing query performance in remote and switching setup

Virtualization technologies are now being used in a wider scale within the context of Grid computing. Several contemporary work point to the useability of these techniques to provide support for ad-hoc virtual organisational models, adaptive models to handle resource volatility etc. In [17] the authors speak about creating Virtual Workspace over a cluster of virtual machines for individual consumers. In Virtuoso[18] and Violin[16], work has been done with regards to internetworking issues arising from a VM setting. In [22], the authors have been able to move a running VM from one host to another without interrupting the processing. All of these are relevant to the concept of using virtualization techniques within the Grid context. Our work differ from them in the sense that we propose an approach of creating a uniform service-oriented layer which abstracts the underlying resources and allows the creation of an ondemand ad-hoc virtual grid, transparent to the consumer by enabling on-demand deployment of different types of components such as web services, virtual machines, stored procedures, .NET services together in one framework. Further, the use of virtual machines within DynaSOAr and DQP provide an alternative view of collocating data and computation

Comparing transport cost for ProteinSequence and ProteinProperties 300000

ProteinSequence-PL ProteinProperties-PL ProteinSequence-SW ProteinProperties-SW

Cost in milliseconds

250000

200000

150000

6. Conclusion

100000

50000

0 0

5000

10000

15000

20000

25000

30000

35000

40000

Number of tuples

(a) Comparing data transport cost from the datanodes Comparing total execution cost in different setups for a select-project-join qu 500000

execution-SW execution-PL

450000 400000

Cost in milliseconds

in particular can be used within a distributed query processing framework to provide improvement in query processing performance by introducing a variant of data caching and deploying services requiring special environments locally to avoid increasing cost of data transport.

350000 300000 250000 200000 150000 100000 50000 0 0

5000

10000

15000

20000

25000

30000

35000

40000

Number of tuples

(b) Execution cost on a remote and switching setup

Figure 5. Performance of a select-project-joinoperation call query

by deploying database snapshots when the system automatically predicts this will be of benefit. Dynamic deployment of services raises issues such as deciding on whether to deploy a service or a virtual machine on a new node or to use an existing, but possibly overloaded, deployment. The GridSHED[21] project for job scheduling has been investigating this area, and the results are being utilised to design a better scheduling model for DynaSOAr. The logical separation of Service Provisioning and Resource Provisioning in DynaSOAr opens up the possibility of using the Software Marketplace as introduced in [28], where, based on the preferences such as preference to a particular provider or quality of service specifications (such as cost, throughput, reliability etc) from the consumer, the DQP system may be able to select a certain version of the analysis or evaluation service from a list of all the available services. In summary, this paper is the first to show how dynamic deployment in general, and virtualization techniques

This paper presents an overview of the work done in order to exploit the possibilities of dynamic service deployment and the use of virtualization techniques within the context of distributed query processing on the Grid. It can be seen from the experimental results that distributed query processing can significantly benefit from the dynamic deployment and virtualization mechanisms of DynaSOAr by considering several scenarios, such as deploying database snapshots and services requiring special environments locally. The approach is extensible and includes scope of creating software market places by choosing the computational resources from a pool based on the cost and (or) the quality of service provided by the host. The work on DynaSOAr is continuing to investigate different aspects of the system, such as resource allocation, load balancing, efficient and robust deployment of virtual machines, and future work involves looking into the utility of the framework for adaptive and fault-tolerant distributed query processing systems and evaluating various transport technologies for transferring the service code.

Acknowledgment We wish to acknowledge the support from our colleagues in OGSA-DAI, OGSA-DQP and DAIT projects at Manchester University and EPCC. We wish to thank Jim Smith for his helpful discussions. We would also like to thank the UK Engineering and Physical Sciences Research Council, and the DTI for the GridSHED and DAIT projects, and the partners of the CRISP project.

References [1] Basic Local Alignment Search Tool. http://www. ncbi.nlm.nih.gov/BLAST. [2] EMBL Nucleotide Sequence Database. http://www. ebi.ac.uk/embl/. [3] OGSA-DAI. http://www.ogsadai.org.uk/. [4] OGSA-DAI Glossary of Terms. http: //www.ogsadai.org.uk/documentation/ ogsadai-wsrf-2.2/doc/reference/ glossary.html. [5] PlanetLab - An Open Platform for developing, deploying, and accessing planetary-scale services. http://www. planet-lab.org.

[6] Sun Grid Engine. http://www.sun.com/software/ gridware. [7] The Globus Toolkit. http://www.globus.org/ toolkit. [8] What is OGSA-DQP? http://www.ogsadai.org. uk/about/ogsa-dqp/. [9] WS-I Basic Profile Version 1.0. http://www. ws-i.org/Profiles/BasicProfile-1. 0-2004-04-16.html. [10] M. N. Alpdemir, A. Mukherjee, N. W. Paton, P. Watson, A. A. A. Fernandes, A. Gounaris, and J. Smith. Servicebased Distributed Querying on the Grid. In First International Conference on Service Oriented Computing (ICSOC 2003), LNCS 2910, pages 467–482. Springer-Verlag, 2003. [11] S. Banerjee, T. Griffin, and M. Pias. The interdomain connectivity of planetlab nodes. In Passive and Active Network Measurement, 5th International Workshop, PAM 2004, volume 3015 of Lecture Notes in Computer Science, pages 73– 82. Springer, 2004. [12] F. E. Bustamante, G. Eisenhauer, K. Schwan, and P. Widener. Efficient wire formats for high performance computing. In Supercomputing ’00: Proceedings of the 2000 ACM/IEEE conference on Supercomputing (CDROM), page 39, Washington, DC, USA, 2000. IEEE Computer Society. [13] K. Chiu, M. Govindaraju, and R. Bramley. Investigating the Limits of SOAP Performance for Scientific Computing. In HPDC ’02: Proceedings of the 11th IEEE International Symposium on High Performance Distributed Computing, page 246, Washington, DC, USA, 2002. IEEE Computer Society. [14] I. Foster and C. Kesselman, editors. The Grid: Blueprint for a New Computing Infrastructure. Morgan Kaufmann, 2003. [15] M. Govindaraju, A. Slominski, V. Choppella, R. Bramley, and D. Gannon. Requirements for and evaluation of RMI protocols for scientific computing. In Supercomputing ’00: Proceedings of the 2000 ACM/IEEE conference on Supercomputing (CDROM), page 61, Washington, DC, USA, 2000. IEEE Computer Society. [16] X. JIANG and D. XU. Violin: Virtual Internetworking on Overlay Infrastructure. In Tech. Rep. CSD TR 03-027, Department of Computer Sciences, Purdue University, 2003. [17] K. Keahey, I. T. Foster, T. Freeman, X. Zhang, and D. Galron. Virtual Workspaces in the Grid. In Euro-Par 2005, Parallel Processing, 11th International Euro-Par Conference, volume 3648 of Lecture Notes in Computer Science, pages 421–431. Springer, 2005. [18] B. Lin and P. A. Dinda. VSched: Mixing Batch And Interactive Virtual Machines Using Periodic Real-time Scheduling. In SC ’05: Proceedings of the 2005 ACM/IEEE conference on Supercomputing, page 8, Washington, DC, USA, 2005. IEEE Computer Society. [19] M. E. Martone, A. Gupta, and M. H. Ellisman. eneuroscience: challenges and triumphs in integrating distributed data from molecules to brains. Nature Neuroscience, 7:467–472, 2004. [20] A. Mukherjee and P. Watson. Adding Dynamism To OGSADQP: Incorporating The DynaSOAr Framework In Distributed Query Processing. In Euro-Par 2006 Workshops:

[21]

[22]

[23]

[24] [25]

[26]

[27]

[28]

[29]

Parallel Processing, EuroPar 2006, volume LNCS 4375, pages 22–33. Sringer Verlag, 2006. J. Palmer and I. Mitrani. Optimal Server Allocation in Reconfigurable Clusters with Multiple Job Types. In Computational Science and its Applications (ICCSA 2004), Assisi, Italy, 2004. P. Ruth, J. Rhee, D. Xu, R. Kennell, and S. Goasguen. Autonomic Live Adaptation of Virtual Computational Environments in a Multi-Domain Infrastructure. In Autonomic Computing, 2006. ICAC 2006. IEEE International Conference on, pages 5–14, 2006. M. Schaller. Reclustering of high energy physics data. In Statistical and Scientific Database Management, pages 194– 203, 1999. I. L. Smith. Joint academic network (JANET). Computer Networks and ISDN Systems, 16(1-2):101–105, 1988. J. Smith, A. Gounaris, P. Watson, N. W. Paton, A. A. A. Fernandes, and R. Sakellariou. Distributed Query Processing on the Grid. In GRID ’02: Proceedings of the Third International Workshop on Grid Computing, pages 279–290. Springer-Verlag, 2002. A. S. Szalay, P. Z. Kunszt, A. Thakar, J. Gray, D. Slutz, and R. J. Brunner. Designing and mining multi-terabyte astronomy archives: the Sloan Digital Sky Survey. SIGMOD Rec., 29(2):451–462, 2000. T. Tannenbaum, D. Wright, K. Miller, and M. Livny. Condor – A Distributed Job Scheduler. In T. Sterling, editor, Beowulf Cluster Computing with Linux. MIT Press, October 2001. P. Watson, C. Fowler, C. Kubicek, A. Mukherjee, J. Colquhoun, M. Hewitt, and S. Parastatidis. Dynamically deploying Web services on a grid using Dynasoar. In Ninth IEEE International Symposium on Object and ComponentOriented Real-Time Distributed Computing, ISORC. IEEE, 2006. P. Watson and P. Lee. The NU-Grid Persistent Object Computation Server. In 1st European Grid Workshop, Poznan, Poland, 2000.