The Herschel Data Processing System

Astronomical Data Analysis Software and Systems XV ASP Conference Series, Vol. 351, 2006 C. Gabriel, C. Arviset, D. Ponz and E. Solano, eds.

The Herschel Data Processing System S. Ott, J. Bakker, J. Brumfitt, N. de Candussio, L. Dwedari, A.M. Heras, S. Leeks, A.P. Marston, J.-J. Mathieu, J. Pizarro, H. Siddiqui European Space Agency B. Ali, B. Latter, P. Morris, J. Rector, B. Schulz NASA Herschel Science Center J. Corrales Garcia Consejo Superior de Investigaciones Cientficas W. De Meester, R. Huygen, B. Vandenbussche Katholieke Universiteit Leuven S. Guest Rutherford Appleton Laboratory J. Kemp Imperial College London D. Kester, R. Shipman, P. Zaal SRON Netherlands Institute for Space Research A. Lorenzani Osservatorio Astrofisico di Arcetri E. Sturm, M. Wetzstein, E. Wieprecht Max-Planck-Institut f¨ ur extraterrestrische Physik Abstract. The Herschel Space Observatory is the fourth “cornerstone” mission in the ESA science programme. Its payload consists of the three instruments HIFI, PACS and SPIRE. Herschel will offer three years of routine observations for the entire scientific community. The Herschel Data Processing System is expected to provide a comprehensive facility for astronomical data reduction, for calibration and engineering purposes, and to generate products for the Herschel archive. We will present an overview of this exciting state-of-the-art software development project.

516

The Herschel Data Processing System 1.

517

Overview of the Herschel mission

The Herschel Space Observatory will perform photometry and spectroscopy in approximately the 57-670 µm range. It will have a radiatively cooled 3.5m diameter telescope, and a science payload complement of three instruments (HIFI, PACS and SPIRE) housed inside a superfluid helium cryostat. Herschel will be operated as an observatory facility offering three years of routine observations. It will be available for the entire scientific community. Herschel is being implemented together with the Planck mission as a single programme and sharing a common Ariane 5 launcher. The launch towards the operational orbit around L2 is planned for 2007. 2.

History and raison d’ ˆ etre of the Herschel Data Processing System

At the beginning of the software development for Herschel it was considered sufficient to provide astronomers with raw data and the software tools to carry out a basic data reduction, and no data products were to be generated and delivered. However, in the last few years, the expectations of the astronomical community on what to expect from observatories and their data processing systems, data products, and archives have evolved: Scientifically useful products must be provided suitable for access by “Virtual Observatory” (VO) tools. Following intensive discussions between ESA and the PI teams, it was agreed that systematic pipeline processing of all data must be carried out and that an integrated easy-to-use, well tested, documented and freely distributable data processing system will be made available to the community. To avoid long delays before the know-how and the data processing tools developed at the instrument centres are made available to the general community, the “Instrument Calibration”, “Trend Analysis” and “Product Generation and Verification” systems are implemented as part of the general Data Processing system that will be provided to the astronomical community. Already now the Herschel Data Processing system is used during the instrument level tests for instrument characterisation and calibration. Since the astronomer will perceive and use Herschel as an observatory, commonality across the three instruments for tools and data products is of importance. 3.

Overview of the Herschel Data Processing System

The Herschel Data Processing system is a part of the Herschel Common Science System (HCSS). This software is jointly developed by ESA, the three instrument teams and the NASA Herschel Science Center. It is coded in Java and Jython for platform independency and provides the basis for the following subsystems: • Interactive Analysis for the astronomical community • Interactive Analysis for calibration scientists • Quality Control Processing (QCP) • Standard Product Generation (SPG) • Quick Look Assessment (QLA) • Trend Analysis (TA)

518

Ott et al.

A suite of Java libraries provide the means to perform numerical analysis, to plot and display data and to execute specialised tasks such as the streaming and step-by-step execution of raw telemetry and refined products. Care is taken that the data reduction algorithms can be used for both streaming and sequential processing. In addition, this suite provides utility classes to contain and manipulate complex data, to write out data to relational and object-oriented databases and, in various formats, to a local file system. Other components permit the dynamic loading of new Java and Jython classes or provide graphical user interface utilities. Jython is used to provide scripting capabilities. It has a powerful feature in being able to access Java classes directly. Occasionally Jython “wrapper” scripts are used to simplify the library interface presented to the general user. WWW services will be used to access external non-Java packages. The user interface for Herschel Data Processing is a tool called “JIDE”, a GUIbased Jython interpreter. It is an extension of the standard Jython shell, and loads a customised Jython environment, keeps a history of successfully executed Jython statements, implements a set of basic editing functions and provides access to the context-sensitive online help. The documentation system is based on the DocBook XML format. DocBook is a markup language that is specifically suited for technical documentation. It was chosen as it provides a rich set of tags to describe the content of especially software documentation. The documents are generated by manual editing, or are directly generated from the code, or by a combination of both. The documentation is fully integrated into the online help system, and available in both HTML and PDF. 4.

Development Methodology

The development is distributed over several sites. Common development by ESA, the three PI teams and the NASA Herschel Science Center optimises the use of available resources by simplifying maintenance, documentation and testing. Having only one system instead of many different ones also helps to provide the best possible user support by the Herschel Science Centre. Several working groups responsible for management, architecture, look-and-feel and documentation coordinate the development. As for the HCSS, we use the concepts of “smooth transition”, “iterative development” and “continuous integration”: smooth transition means to incrementally develop one system which, in subsequent states of realisation, supports • the Instrument Level Tests in the laboratories of the PIs • the Integrated System Tests in industry with all instruments in the spacecraft • the in-orbit mission phases • the post-mission active archiving phase • the exploitation of the Herschel legacy archive This methodology avoids a tedious and error prone conversion of checkout to operational procedures and leads to a much more thoroughly tested Science Ground Segment at launch.

The Herschel Data Processing System

519

iterative development means that the software is developed in several iterations. Each iteration lasts approximately six months and covers — for the identified goals — the full software development cycle from requirements capture to testing and documentation. This methodology allows for a fast reaction on user feedback and new or changed requirements. continuous integration means that the software is built and tested via its testharnesses each night. This helps to identify integration problems much faster and problems can be localised quickly. The Herschel Data Processing system is open to contributions; the key programmes and astronomical community are encouraged to feed back their data products and the tools and algorithms developed to produce them. 5.

Managerial and Technical Challenges

In addition to a tight schedule — as the system has to be launch–ready in 16 months — the two main challenges are the longer decision process and the communication overhead imposed by a distributed development and to agree on work priorities: As the instruments are in different development phases they have different immediate needs. Furthermore one has to balance the current experts’ needs vs. future astronomers’ needs. As Java and Jython are “young” languages that are still under development not all functionality of classical languages are available and have to be provided by coding or by including of additional external libraries. As the system is still under development, APIs change and existing scripts have to be modified. Also the switch from a procedural to object-oriented thinking and coding style needs some adaptation. Finally, the already existing code of the three instruments will have to be adapted to ensure the same look and feel within the Herschel Data Processing System. References Pilbratt, G.L., Herschel mission: status and observing opportunities, 2004, Proc. of SPIE, ed. J.C. Mather, 5487, 401-412 Poglitsch, A., Waelkens, C., Bauer, O.H., et al., The photodetector array camera and spectrometer (PACS) for the Herschel Space Observatory, 2004, Proc. of SPIE, ed. J.C. Mather, 5487, 425-436 Griffin, M.J., Swinyard, B.M., Vigroux, L., The Herschel-SPIRE instrument, 2004, Proc. of SPIE, ed. J.C. Mather, 5487, 413-424 de Graauw, Th. & Helmich, F.P., Herschel-HIFI: The Heterodyne Instrument for the Far-Infrared, 2001, Proc. of “The Promise of the Herschel Space Observatory”, ESA SP-460, 45-51