Abstract. We have developed a Python-based regression testing system ... by many tasks; 2) the test routines shall be capable of automatic execution; 3) all test ...
Astronomical Data Analysis Software and Systems XI ASP Conference Series, Vol. 281, 2002 D. A. Bohlender, D. Durand, and T. H. Handley eds.
A Python-Based IRAF Regression Testing System H. Bushouse, B. Simon, H. Shukla, E. Wyckoff Space Telescope Science Institute, 3700 San Martin Drive, Baltimore, MD 21218, USA Abstract. We have developed a Python-based regression testing system for tasks that run within the IRAF environment. The system uses XML format configuration files to control test instances for each task. The configuration file specifies the task to be tested, the IRAF parameter file to use with the task, as well as the names and types of output files to be compared upon task completion. Python methods are used to perform the file comparisons, including comparisons of ASCII, binary, and FITS files. Tasks can be tested individually or an entire directory tree can be traversed, running tests for all configuration files that are found.
1.
Introduction
Our development of a regression testing system for IRAF-based software tasks began with establishing the following goals for the system. We wanted a system that would reveal if changes of computer platform, operating system, or task code resulted in unexpected changes in task results. We also wanted to be able to perform complete testing of an entire IRAF package tree while also having the capability for a programmer to test a single task that was being worked on. Finally, we also wanted a system that could be run automatically as part of our code revision check-in process. 2.
System Requirements
To help determine the necessary requirements for the system, we compiled a set of “use cases,” which are a collection of example IRAF/STSDAS/TABLES tasks selected to represent all possible combinations of input and output data types that would need to be accomodated. The various output data types include STDOUT, images (in OIF, GEIS, and FITS formats), tables (in ASCII, STSDAS, and FITS formats), and task parameters. Many tasks produce multiple outputs of a combination of types. Therefore the system must allow for the specification and testing of many output files and types, as well as the ability to apply different comparison types, for a single test instance. Furthermore, the output from some tasks overwrites its input. Hence a mechanism must be available to produce a temporary copy of an input file in order to preserve the original each time the task is run. Based on the desired goals and the range of use cases that needed to be accommodated, we determined the following requirements for the regression test 129
130
Bushouse et al.
system: 1) tasks shall be tested from within the STScI PyRAF environment in order to make use of parameter parsing and IRAF environment variables needed by many tasks; 2) the test routines shall be capable of automatic execution; 3) all test and comparison routines shall have a uniform convention for indicating success or failure; 4) successful and unsuccessful results shall be explicitly indicated; 5) regression tests shall be capable of being invoked at various levels of complexity and thoroughness for each task; 6) each test case shall have a mechanism available for executing arbitrary Python and PyRAF commands either before or after an individual task is run; and 7) output file comparators shall be capable of ignoring expected differences. 3.
System Design and Features
The regression test system operates by retrieving information from an XMLformat configuration file for each task to be tested, executes the task using a specified IRAF parameter file to control the task, and then comparing the specified output file(s) against existing reference files. The system consists of four main Python modules encompassing: 1) overall flow control, 2) configuration file parsing, 3) task execution, and 4) output file comparison. 3.1.
Flow Control
The test system will automatically “walk” down a specified directory tree, searching for XML format test configuration files. Upon finding a configuration file, the file is opened and the contents parsed. If the test level identifier in the configuration file (see below) matches that of the user-requested testing level, control is passed to the task execution module for running the task. After the task has been executed flow is handed over to the output file comparison module. Upon completion of the file comparisons, the search continues for additional configuration files. 3.2.
Configuration Files
The regression test configuration files are in XML format and contain all the necessary information for running a single test of one task. The list of available XML tags and their functions are as follows. title: String for identifying the test in processing log information. taskname: IRAF/PyRAF task to be executed. pfile: Name of the IRAF-style parameter file to use when running the task. level: Test level indicator, used to identify the thoroughness of a particular test configuration. If equal to or lower than the level requested by the user, the test is run. pre-exec: An optional list of Python/PyRAF commands to execute before running the task. This can be useful for loading needed IRAF packages, copying or deleting files, and so on.
A Python-Based IRAF Regression Testing System
131
output: Specifies a list of one or more output file information sets. Each set consists of: file: Output file produced by test run. If set to “STDOUT” the task execution module will trap STDOUT to a file. reference: Name of the reference output file. comparator: Type of comparison to be performed between “file” and “reference.” Currently available comparison types are ASCII, binary, FITS image, and FITS table. ignorekeys: An optional list of header keyword values to be ignored. This is useful for keywords that are expected to change value, such as file names and file creation dates. post-exec: An optional list of Python/PyRAF commands to execute after running the task. This is useful for manipulation of output files. 4.
Operational Features • Execution: The regression test system can be run from either the host operating system level or from within the PyRAF environment. • Command-line Arguments: The user can specify 1) the starting level of an entire directory tree to be searched or the name of a particular configuration file to be executed, 2) the name of a file to which the output log information will be written, and 3) the desired levels of tests to be performed. • Test Files: A minimum of 4 files are needed to test a given task: a task configuration file (XML format), a parameter file, an input data file, and an output reference file. Additional configuration, parameter, input, and output files can be provided as necessary for testing tasks that use or produce multiple inputs and outputs, as well as for setting up different test configurations for a given task. • Test Organization: As implemented at STScI, we have built duplicate IRAF, STSDAS, and TABLES directory trees that mirror the structure of the source code trees for these packages, but contain only the file necessary for running regression tests. The testing tree is under configuration control. • Automation: Because the regression test system is completely data driven, it can be run in a fully automated and unattended fashion if desired. At STScI we currently have the test system setup to run every night from a Unix “cron” job. The daily logs are examined for any reported problems.
