The NERC Cluster Grid Dan Bretherton, Jon Blower and Keith Haines Reading e-Science Centre, ESSC, University of Reading, RG6 6AL
http://www.resc.rdg.ac.uk
UK e-Science All Hands Meeting, 10th - 13th September 2007
1. Introduction
http://www.nerc-essc.ac.uk
6. G-Rex Portal Screen Shots: browsing services and monitoring progress
A Grid is a collection of computational resources that are connected in a way that facilitates sharing across administrative domains. Shared resources can include High Performance Computing (HPC) and High Throughput Computing (HTC) clusters, data storage facilities and scientific instruments. Grids facilitate the operation of Virtual Organisations that cross geographical and institutional boundaries. The NERC Cluster Grid aims to facilitate collaboration between Natural Environment Research Council (NERC) institutions by making it easier to share their HPC and HTC cluster resources. The following key features are required for the NERC Cluster Grid: o Load and performance monitoring o Minimal data footprint on remote clusters o Easy job submission and control o Security This poster explains how these features are being provided, and describes how the Grid is being used by NERC scientists.
2. Current status of the NERC Cluster Grid
7. Ganglia Screen Shots
Job submission and control facilities are being provided by means of the Grid Remote eXecution (G-Rex) software being developed at the Reading e-Science Centre (ReSC). G-Rex is the successor to Styx Grid Services [1,2,3] (SGS). G-Rex is light-weight middleware that allows applications residing on remote computational resources to be launched and controlled as if they are running on the user's own computer. The NEMO ocean model has been deployed as a G-Rex service on HPC clusters at the Environmental Systems Science Centre (ESSC), the Proudman Oceanographic Laboratory (POL) and the British Antarctic Survey (BAS). Users with a G-Rex account can run NEMO on any of these clusters without making significant changes to their model setup and scripts etc. There are plans to use G-Rex as the core grid middleware in two current NERC e-science projects: GCEP ("Grid Coupled Ensemble Prediction") and GCOMS ("Global Coastal Ocean Modelling System"). This will involve deploying the HadCM3 coupled climate model and the POLCOMS ocean model as G-Rex services on a wide range of compute resources around the country, including clusters in NERC Cluster Grid. The Ganglia grid monitoring system (http://ganglia.sourceforge.net/) has been installed at ReSC. The Ganglia Web Frontend for the NERC Cluster Grid shows load and performance statistics for the clusters at ESSC, POL and BAS. It can be viewed at the following URL: http://www.resc.reading.ac.uk/ganglia/ The ReSC G-Rex Portal describes the available services at ESSC and allows users to browse information about previous and current jobs, including their status, configuration, input files and output files. The portal can be viewed at this URL: http://lovejoy.nerc-essc.ac.uk:9080/G-Rex/welcome.html
3. Introduction to G-Rex G-Rex is implemented in Java using the Spring application framework (http://www.springframework.org/). It can be used on any Java supported platform. The G-Rex server is a Web application that allows existing binary executables to be wrapped and exposed as remote services. It runs inside a servlet container such as Apache Tomcat (http://tomcat.apache.org/). G-Rex services are accessed using the G-Rex client program, grexrun, which behaves as if the remote service were actually running on the user's own computer. The main advantages of G-Rex are: • Output files are continuously transferred back to the user as they are being created, allowing the progress of the job to be easily monitored • Output files are deleted from the server when they are no longer needed, preventing the unnecessary accumulation of data and reducing the overall data footprint of the service • Work-flows can be created using simple shell scripts [1], and existing scripts can be converted to use grexrun without significant modifications • G-Rex is very easy to install and use
4. Cluster Grid services are run just like local programs Once deployed as G-Rex services, programs can be run from anywhere on the Internet from the command line. Let's say that we have exposed a program called model as a G-Rex service on the ESSC cluster. This program takes two parameters, the names of the input and output files. The model service can be invoked by a user at another institution with the following command (with ID and PW replaced by the G-Rex user name and password of the authorised user): grexrun.sh http://ID:
[email protected]:8080/G-Rex/model IN OUT The G-Rex software performs the following tasks: • Creates submit file for job management system (Sun Grid Engine for this particular cluster) • Creates new instance of the model service on the G-Rex server • Uploads the input file to the server • Runs the model service instance • Downloads the output file as it is being produced All of this is performed automatically, removing the need for the user to manually create a job submit file and upload and download files. The grexrun.sh command waits for the job to finish on the cluster. This feature makes it easy to write shell scripts to implement work-flows involving one or more G-Rex services and local applications. As a simple example, a long run can be broken up into manageable stages using a script with a loop that re-submits the next stage as soon as the previous stage has finished.
8. Security and Administration Ganglia No firewall ports have been opened for the NERC Cluster Grid Web Frontend. The Ganglia Meta Daemon at ESSC obtains its information about the remote clusters through SSH tunnels. The only requirement is that the Grid administrator has SSH access to shell accounts on all the clusters in the Grid. Alternatively, a cluster administrator may prefer to open a single firewall port used by Ganglia for exchanging information. The cluster administrator can control which remote institutions are able to receive Ganglia data. G-Rex Apache Tomcat's authentication procedures are not used for any of the NERC Cluster Grid deployments of G-Rex. Instead, G-Rex uses a method based on the Acegi security framework (http://www.acegisecurity.org/) with HTTP digest authentication. Access to G-Rex services is governed by user accounts and groups defined in the server configuration file. It is not necessary to open any firewall ports in order to deploy G-Rex services. A cluster user with a shell account can deploy G-Rex services on the cluster by running Apache Tomcat in his or her name. The services can be made available to authorised users at other institutions via SSH tunnels. As an alternative to giving shell accounts to remote users and letting them set up tunnels, a cluster administrator may prefer to allow access to a set of G-Rex services from certain institutions by opening a single firewall port. In this scenario the cluster administrator has control over both who uses the cluster and what they can do. No ports need to be opened for G-Rex clients, i.e. G-Rex service users at the remote institutions.
5. The NEMO ocean model NEMO ("Nucleus for European Modelling of the Ocean" - http://www.lodyc.jussieu.fr/NEMO/) is the standard model for operational ocean forecasting in France (http://www.mercator.eu.org/), and will be used in the next generation of Hadley Centre climate models as well as for operational use within the National Centre for Ocean Forecasting (NCOF - http://www.ncof.gov.uk/) in the UK. The main challenges associated with deploying NEMO on a remote cluster are: • Model setup includes a complicated directory structure and a large set of interdependent scripts • Large volume of output needs to be transferred back to user at regular intervals These are the main parameters of a typical one-degree model run for a one year period: • Most efficient with 20 processors • Usually takes 6-7 hours of CPU time on the ESSC cluster • Every 5 minutes the model outputs approximately 120 MB of data in 180 separate files • The output for a one year run is roughly 9 GB in size, a total of 13140 separate files • 50-year runs are common. The model is re-submitted as a new job after each year.
References [1] J. Blower, A. Harrison, K. Haines, Styx Grid Services: Lightweight, easy-to-use middleware for scientific workflows, Lecture Notes in Computer Science, 3993 996-1003 (2006). [2] J. Blower, K. Haines, E. Llewellin, Data streaming, workflow and firewall-friendly Grid Services with Styx, Proceedings of the UK e-Science All Hands Meeting, September 2005 [3] J. Blower, K. Haines, Building simple, easy-to-use Grids with Styx Grid Services and SSH, Proceedings of the UK e-Science All Hands Meeting, September 2006
