Optimizing SAS tasks for Grid Architectures

Astronomical Data Analysis Software and Systems XV ASP Conference Series, Vol. 351, 2006 C. Gabriel, C. Arviset, D. Ponz and E. Solano, eds.

Optimizing SAS tasks for Grid Architectures Aitor Ibarra, Daniel Tapiador, Fernando F´elix, Carlos Gabriel, Christophe Arviset, John Hoar European Space Agency, ESAC, Madrid, Spain Salim Ansari European Space Agency, ESTEC, Noordwijk, The Netherlands Abstract. We present the first prototype of a XMM-Newton pipeline processing task parallelized at a CCD level, which can be run in a GRID system. By using the GridWay application, the processing of the XMM-Newton data is distributed across the Virtual Organization constituted by three different research centres (ESAC, ESTEC, Complutense Madrid University).

1.

Introduction

The advent of Virtual Observatories (Williams 2004) will allow the astronomers around the world the access to an unprecedented amount of scientific data. The processing of those data will be carried out, in most cases, by using the computational resources supplied by the Data Provider to the scientific community. Due to the large amount of data available, it is crucial that the Data Provider could manage their hardware resources in an optimal way, if they do not want the data processing to slow down. In this context, GRID technology offers the capability of managing not only the user queries retrieving data from the archive, but also the online processing of that data. By using GRID architectures, this online processing can be a bottleneck, depending on the data software reduction used. The Science Archives hosted at the European Space Astronomy Centre (ESAC) and particularly the XMM-Newton Science Archive (XSA), give the scientific community the possibility to process the data on-the-fly1 , by using the latest version of the XMM-Newton Science Analysis Software, SAS (Gabriel 2004). Beneath this data archive, the GRID tool controls the available computational resources in order to optimize the data retrieval carried out by the users. SAS tasks were developed to be run in a single processor computer by reducing the data CCD by CCD sequentially (see Figure 1). Therefore they are currently not optimized to be used in a GRID distributed architecture. 1

Archive Inter-Operability Tool for XMM-Newton: http://xsa.esac.esa.int:8080/aio/doc/index.html

520

Optimizing SAS tasks for Grid Architectures

Figure 1.

521

XMM-Newton SAS EPIC PN on-the-fly data processing

In this paper, we present the first prototype of a SAS task parallelized, at a CCD level, which can be run in a GRID system. One of the X-ray cameras on board the XMM-Newton, EPIC-PN (Str¨ uder 2001) is composed of 12 CCD. By means of the GRID Virtual Observatory infrastructure, the parallelized SAS task through the Grid Architecture, is able to process XMM-Newton data on-the-fly, and in most cases, the data processing is speeded up. 2.

Grid Computing Paradigm

Grid Computing has emerged as an important new field in computer science, distinguished from conventional distributed computing by its focus on large-scale resource sharing along different organizations or control domains (Foster 2001). Therefore, Grid computing tries to address the problems derived from the fact that scientific and engineering organizations usually need to collaborate. So as to accomplish certain goals that would otherwise be unreachable. The most recent definition (Foster 2002) is a three point checklist which says that a Grid is a system that: 1. Coordinates resources that are not subject to a centralized control... 2. ... by using standard, open, general-purpose protocols and interfaces... 3. ... to deliver nontrivial qualities of service. 3.

Grid Architecture

A typical Grid architecture comprises basically the local resources (local jobmanagers, etc), a Grid middleware that makes it uniform to access to these local resources and Grid tools or portals that provides a user friendly interface. As can be seen in Figure 2, the Grid middleware we have used is the Globus Toolkit2 , which in fact has become the ”de facto” standard for Grid computing 2

The Globus Toolkit. www.globus.org

522

Ibarra et al.

middleware. It provides services for allocating resources and executing jobs on different local jobmanagers (GRAM), by transferring data (GridFTP) and getting both static and dynamic information about the Grid (MDS), all this with a high level of security. Since accessing Globus Toolkit services is rather complicated and requires a high level of expertise, the use of a more user friendly Grid tool than the Grid middleware is highly recommended. The tool used for the demonstration is GridWay3 . GridWay is a user-oriented workload management tool (meta-scheduler or Grid job manager) that provides an unattended and efficient execution of jobs (in a submit & forget fashion) on heterogeneous, dynamic and loosely coupled Grids.

Figure 2. Grid architecture used in the testbed.

Figure 3. Virtual Organization resources (ESACESTEC-UCM).

The testbed built for running the jobs is shown in Figure 3. As we see, the Grid comprises different local resources (heterogeneity) that belong to looselycoupled organizations. Our Virtual Organization is composed by: • The European Space Astronomy Centre (ESAC). Located in Madrid, Spain, • The European Space Research and Technology Centre (ESTEC). Located in Noordwijk, The Netherlands • the Complutense University of Madrid (UCM). Located in Madrid, Spain. Those resources can be added or removed dynamically (and quickly) without the final user noticing. Therefore, if a host/cluster is down the user could eventually notice a delay on the time spent on the execution, but the jobs would always run properly. 4.

XMM-Newton SAS results

With this study, we have been able to analyze EPIC PN data using GRID technologies. The GridWay solution is able to automatically distribute the execution of the data reduction of the twelve CCD independently across the entire Virtual Organization infrastructure. 3

The GridWay Tool. www.gridway.org

Optimizing SAS tasks for Grid Architectures

523

To show the availability of our system, we have analyzed EPIC PN observations retrieving the data from the XMM-Newton Archive. The data is retrieved from the nearest available cluster to the Archive (in this case, ESAC cluster), and then the pre-processing phase is executed (see Fig. 1). When this phase has finished, the ccd-processing phase is executed (all the data files corresponding to each CCD have been already packed and sent to the node chosen by the Grid Scheduler). Finally, when all the jobs have been processed, the post-processing phase is executed and the final data files are produced. As a result of our benchmarks, we have seen that when analyzing small size observations (ie. due to shorter Exposure Times), the sequential SAS task (executed in a single node) takes less time than the Grid SAS task due to the time spent in the data transfer. Instead, for large size observations, the parallelized SAS task requires less time in the overall data processing. Despite the geographical distribution of the Virtual Organization, with an increase of elapsed time in data transfers, our results show that it is possible and convenient under certain conditions to use a Grid Architecture for the EPIC PN data reduction. 5.

Conclusions

We have presented the first parallelized SAS task prototype that can be executed in a Grid Architecture by using Globus and GridWay solutions. The conclusions of the work done are enumerated below, but basically they are the same as those which arise when developing any Grid application. • Collaboration between loosely coupled organizations. • Reliability. Some resources of the Virtual Organization may be down but the tasks can be still executed without any user interaction. • Speed-up. We take advantage of the massive parallel CPU capacity we get from the Grid. • The network bandwidth is extremely important in Grid computing applications. References Foster, I. 2002, in GRIDToday, vol.1 no. 6. Foster, I. 2001, in Lecture notes in Computer Science, vol. 2150. Gabriel, C. et al. 2004, in ASP Conf. Ser., Vol. 314, ADASS XIII, ed. F. Ochsenbein, M. Allen, & D. Egret (San Francisco: ASP), 759 Str¨ uder, L. et al. 2001, in A&A, 365, L18 Williams, R. et al. 2004. VO Architecture Overview, V. 1.0, IVOA WG. (www.ivoa.net/Documents/Notes/IVOArch/IVOArch-20040615.html)