Instrument Science Report WFC3 2000-019
IR Data Pipeline Using Fowler Sampling I. Dashevsky, C. Hanley, M. Robberto November 15, 2001
ABSTRACT We have created a data reduction pipeline to characterize data from infrared detectors, which are tested by the Goddard Space Flight Center Detector Characterization Laboratory for the purpose of choosing a flight detector for the WFC3 infrared channel. The details of the pipeline are presented in this report.
Introduction The Wide Field Camera 3 (WFC3) will be installed on HST during Servicing Mission 4. It consists of two ultraviolet 2K x 4K CCDs and an infrared 1K x 1K Mercury Cadmium Telluride array. The infrared (IR) array is being developed by Rockwell Science Center. A pipeline is required to process data with as little user interaction as possible. The primary purpose of the data is to test the capabilities and requirements of the engineering and flight IR detectors. The test data is taken at the Goddard Space Flight Center Detector Characterization Laboratory (DCL). The STScI and DCL share responsibilities to characterize the detectors. This report describes the data pipeline developed and used at STScI. The data pipeline is written in the IRAF command language (CL). It is available in the wfc3.wfc3ir package, which is maintained by Christopher Hanley (
[email protected]) and currently resides in the Science UNIX Cluster directory /data/clavius2/WFC3/IRAF. The name of the procedural script is irpipeline.cl. The purpose of the pipeline is to automate the process of reducing the DCL data and calculating basic statistics for the image and reference pixels. The following section, “Pipeline and Masking Procedures”, describes the data reduction and masking steps. The section, “Pipeline Parameters”, lists the irpipeline input parameters and includes an example on calling and using the pipeline.
Copyright© 1999 The Association of Universities for Research in Astronomy, Inc. All Rights Reserved.
Instrument Science Report WFC3 2000-019
Pipeline and Masking Procedures Irpipeline Input/Output Figure 1 shows the flow diagram for the irpipeline procedural script. The parameters are listed in Table 3. The input is a FITS file containing “raw” data from the DCL, which is typically a 1024 x 12288 pixel image consisting of twelve 1024 x 1024 pixel images. However, the pipeline is capable of processing any type of Fowler sampling data. Fowler sampling data consists of a reset followed by multiple initial readouts, then an exposure, completed by an equal number of final readouts. The output from irpipeline is a multiple extension FITS file. Each extension, starting with the 0th extension (i.e., [0]), contains a 1024 x 1024 pixel image. The number of extensions depends on the size of the input image and the number of readout frames specified by the user. Each image in the output file is the difference of the nth final and nth initial readouts. Also, the output image may be masked (optional), which is described in the subsection “Procedure to Mask Data”. The output file keeps the same root name as the input file with the suffix “_cln_s” added. This suffix indicates that the quadrants are sorted and rotated as per DCL specifications. There are also output text tables with statistics, which are described in the subsection, “Statistics Output”. Steps Before Masking Figure 2 (a) shows an example of a DCL FITS image: the bottom half, starting from pixel [1,1] (or [0,0] in IDL), consists of six initial readouts and the top half consists of six final readouts. The irpipeline procedure copies each 1024 x 1024 pixel region into a separate, temporary image. The next step is to subtract the nth final readout from the nth initial readout, which are shown in Figure 2 (b). The resulting image, called a “first difference”, is shown in Figure 2 (c). Then, the quadrants are rotated and reassembled, as shown in Figure 3. In Figure 3 (a), the rows or horizontal pixels in quadrants 1 and 2 are exchanged with the corresponding columns or vertical pixels. Also, quadrants 2, 3 are rotated 180 degrees and quadrant 4 is left unchanged. The quadrants in Figure 3 (a) are reassembled as depicted in Figure 3 (b). Procedure to Mask Data In irpipeline the user can specify whether to mask the output image. To perform masking irpipeline calls another wfc3.wfc3ir procedure named applymask. It masks the input image by using a mask file as a template for bad pixels (see subsection “Creating a Bad Pixel Mask”). If an image pixel position corresponds to a bad pixel in the mask file then the pixel in the input image is replaced with the bad pixel value. If it is not a bad pixel then the input image pixel is left unchanged. The statistics calculated by irpipeline are constrained by limits to throw out the pixels equal to the bad pixel value.
2
Instrument Science Report WFC3 2000-019
Figure 1: Data flow schematic for irpipeline.
Copy each 1024 x 1024 pixel region into a temporary image.
IRPIPELINE DATA FLOW
Subtract nth final read from nth initial read.
Rotate and flip quadrants.
Output unmasked images.
Relocate the quadrants.
NO Mask bad pixels?
NO
YES
YES
Output unmasked images and statistics files.
Run APPLYMASK.
Calculate statistics?
Calculate statistics?
NO
Output masked images.
YES Output masked images and statistics files.
3
Instrument Science Report WFC3 2000-019 Figure 2: (a) Example of a FITS file retrieved from DCL, which is 1024 x 12288 pixels; (b) the 1st initial and 1st final readouts; (c) a first difference image output from irpipeline, which is the difference of the frames shown in (b). 1st Final Readout
Six final readouts, taken after exposing detector to a source for a given duration.
1st First Difference
Six initial readouts, taken prior to exposing detector.
1st Initial Readout
(a)
(b)
4
(c)
Instrument Science Report WFC3 2000-019 The bad pixel value is a parameter read from the mask file header by irpipeline. A user may specify the bad pixel value when creating the mask file or by editing the mask using the IRAF procedure imreplace in the standard package images.imutil. Three keywords are added to the masked images: BPM or Bad Pixel Mask, which is the mask file name, MASKED, which indicates whether an image was masked and is equal to T (true) or F (false), and MASKVAL, which gives the bad pixel value. These keywords are added to every header extension of the “cln_s” file. Creating a Bad Pixel Mask The mask is created using a separate procedure in the wfc3.wfc3ir package called irmask. The input to irmask is a list of “cln_s”, unmasked files, such as short and long duration dark images. The output is a bad pixel mask. There are two mask pixel values: 1 for a good pixel and -32767 (default) for a bad pixel, which was chosen for signed integer data. The bad pixel value is an irmask input parameter specified by the user. It should be outside the mean pixel distribution, otherwise, you will not be able to distinguish between the good and bad pixels in the masked image. Each input image is masked using a sigma clipping algorithm and the masks are added to create a final mask file. The minimum value of the good pixels is greater than mean - m x stddev and the maximum value is less than mean + m x stddev, where stddev is the variation of the pixel distribution and m is any real number, which is a user input parameter. Pixels in the wings of the distribution, which are set by these limits, are considered bad pixels. These are replaced by the bad pixel value in two steps: first the pixels less than or equal to the minimum good pixel value, then the pixels greater than or equal to the maximum good pixel value are masked. Finally, the remaining pixels, corresponding to good pixels, are replaced by 1. After creating the mask, it is useful to check the number of good and bad pixels using the IRAF procedure imstat in the images.imutil package, since the fraction of good pixels is given at the end of the irmask processing. The number of good pixels may be checked by setting the lower and upper limits equal to 1. Similarly, the number of bad pixels may be checked (see “Example”). Currently, reference pixels are not masked. All the reference pixels in the mask file have a value of 1. Statistics Output Other output of irpipeline are text tables listing the mean, median, and standard deviation of the image section in each quadrant and also of the corresponding reference pixels. The columns in the text tables are described in Table 1. For each quadrant, the statistics
5
Instrument Science Report WFC3 2000-019 are calculated separately for the image portion and may be optionally masked to exclude bad pixels. Also, for each quadrant, statistics are calculated separately for the reference pixels in the horizontal and vertical direction, which are not masked. Table 1. The output text tables column names and descriptions. The N is replaced by 1, 2, 3 or 4, depending on the quadrant number as shown in Figure 3 (b). Table for Inboard Reference Pixels
Tables for Outboard Reference Pixels
Column Name
Description
Column Name
Description
ImageName
Name of image and extension.
ImageName
Name of image and extension.
qNmean
Mean of a 507 x 507 image region for the Nth quadrant.
qNmean
Mean of a 507 x 507 image region for the Nth quadrant.
qNmed
Median of a 507 x 507 image region for the Nth quadrant.
qNmed
Median of a 507 x 507 image region for the Nth quadrant.
qNstd
Standard deviation of a 507 x 507 image region for the Nth quadrant.
qNstd
Standard deviation of a 507 x 507 image region for the Nth quadrant.
qNhmean
Mean of the four inboard horizontal reference pixel lines for the Nth quadrant.
oqNhmean
Mean of the horizontal outboard reference pixels of the same capacitance for the Nth quadrant.
qNhmed
Median of the four inboard horizontal reference pixel lines for the Nth quadrant.
oqNhmed
Median of the horizontal outboard reference pixels of the same capacitance for the Nth quadrant.
qNhstd
Standard deviation of the four inboard horizontal reference pixel lines for the Nth quadrant.
oqNhstd
Standard deviation of the horizontal outboard reference pixels of the same capacitance for the Nth quadrant.
qNvmean
Mean of the four inboard vertical reference pixel lines for the Nth quadrant.
oqNvmean
Mean of the vertical outboard reference pixels of the same capacitance for the Nth quadrant.
qNvmed
Median of the four inboard vertical reference pixel lines for the Nth quadrant.
oqNvmed
Median of the vertical outboard reference pixels of the same capacitance for the Nth quadrant.
qNvstd
Standard deviation of the four inboard vertical reference pixel lines for the Nth quadrant.
oqNvstd
Standard deviation of the vertical outboard reference pixels of the same capacitance for the Nth quadrant.
A total of five text files are output. Four of these files (e.g., refpix_01.txt, refpix_02.txt, refpix_03.txt and refpix_04.txt) are for the outermost row/column of reference pixels, which use four different outboard capacitors. One of the files (e.g., refpix_I.txt) is for the inner row/column of reference pixels, which have the same internal capacitance. The pixel regions associated with each output text table are given in Table 2.
6
Instrument Science Report WFC3 2000-019 Table 2. The pixel regions used to calculate the statistics in the output text tables. For the outboard reference pixels, every 4th pixel is used in the specified pixel region. Pixel Region Used for Statistics ([x1:x2,y1:y2]) Descriptive Region
Image region in all text files. Internal horizontal reference pixels in refpix_I.txt. Internal vertical reference pixels in refpix_I.txt.
Quadrant 1
Quadrant 2
Quadrant 3
Quadrant 4
[6:512,6:512]
[6:512,513:1019]
[513:1019,6:512]
[513:1019,513:1019]
[6:512,2:5]
[2:5,513:1019]
[1020:1023,6:512]
[1020:1023,513:1019]
[2:5,6:512]
[6:512,1020:1023]
[513:1019,2:5]
[513:1019,1020:1023]
Outboard reference pixels in refpix_01.txt.
horizontal
[5:513,1]
[5:513,1024]
[512:1020,1]
[512:1020,1024]
vertical
[1,5:512]
[1,512:1020]
[1024,5:513]
[1024,512:1020]
Outboard reference pixels in refpix_02.txt.
horizontal
[6:514,1]
[6:514,1024]
[511:1019,1]
[511:1019,1024]
vertical
[1,6:514]
[1,511:1019]
[1024,6:514]
[1024,511:1019]
Outboard reference pixels in refpix_03.txt.
horizontal
[7:515,1]
[7:515,1024]
[510:1018,1]
[510:1018,1024]
vertical
[1,7:515]
[1,510:1018]
[1024,7:515]
[1024,510:1018]
Outboard reference pixels in refpix_04.txt.
horizontal
[8:516,1]
[8:516,1024]
[509:1017,1]
[509:1017,1024]
vertical
[1,8:516]
[1,509:1017]
[1024,8:516]
[1024,509:1017]
Figure 3: quadrants are rotated and reassembled as per the DCL quadrant naming convention. Pixel [1024,1024]
(a)
(b)
1
2
2
4
4
3
1
3
rotate 180 deg. exchange rows & columns
Pixel [1,1]
7
Instrument Science Report WFC3 2000-019
Pipeline Parameters The parameters specified by the user are given in Table 3. Table 3. The user specified input parameters of irpipeline. Parameter
Value Type
Description
inlist
String value, e.g., “@list” or “filename.fits”.
File name that contains a list of raw DCL images or a single image name. The list must be preceded by the “@” sign.
stat
Boolean value (yes/no).
Generate image statistics?
mask
Boolean value (yes/no).
Mask output images?
maskfile
String value of mask FITS file, e.g., “mask.fits”.
Name of the bad pixel mask FITS file used to mask all the input files.
skip
Integer value, e.g., 0.
Number of beginning first differences to skip.
endskip
Integer value, e.g., 0.
Number of end first differences to skip.
outfile
String value., e.g., “refpix_I.txt”.
Name for inboard reference pixel statistics output file.
outfile2
String value, e.g., “refpix_0”.
Root name for outboard reference pixel statistics output files.
delimg
Boolean value (yes/no).
Delete the input files?
iminglist
Leave empty.
For internal use only.
Example To use the current wfc3 procedures include the following lines in your login.cl or loginuser.cl file: task wfc3.pkg = "/data/clavius2/WFC3/IRAF/wfc3.cl" set wfc3scripts = "/data/clavius2/WFC3/IRAF/" and start IRAF. Make sure to check the parameters of the scripts used in the “Example”, since these may have changed. 1. In IRAF load the wfc3.wfc3ir packages: cl> wfc3 wf> wfc3ir 2. Go to a directory that contains raw DCL images and list the files. wf> cd /data/spartan3/WFC3/FPA15/DEMO/ wf> ls AU08_dk150_1hr01.fits AU08_dk150_1hr02.fits
8
Instrument Science Report WFC3 2000-019 AU08_dk150_1hr03.fits AU08_dk150_1sr01.fits AU08_dk150_1sr02.fits AU08_dk150_1sr03.fits The files AU08_dk150_1hr0*.fits and AU08_dk150_1sr0*.fits are long (1 hour) and short (1 second) duration dark images obtained with Fowler sampling, similar to Figure 2 (a). These dark images will be used to create the mask file. 3. Reduce the dark images using irpipeline, skipping all but one of the first differences: wf> unlearn irpipeline wf> irpipeline *.fits stat=no mask=no skip=3 endskip=2 4. Use the “cln_s” darks to create a bad pixel mask called FPA15_150K_mask.fits with irmask: wf> ls *cln_s.fits > list wf> unlearn irmask wf> irmask @list maskfile=”FPA15_150K_mask.fits” FPA15_150K_mask.fits: fraction of good pixels = 0.994 Check that the mask file has the correct number of good pixels: wf> imstat FPA15_150K_mask.fits lower = 1 upper = 1 #
IMAGE
NPIX
MEAN
FPA15_150K_mask.fits 1042211
STDDEV
1.
0.
MIN
MAX
1.
1.
wf> = 1042211./(1024.*1024.) which is equal to the fraction of good pixels. Now, check that there are no extraneous pixel values in the mask: wf> imstat FPA15_150K_mask.fits lower=-32767 upper=-32767 #
IMAGE
FPA15_150K_mask.fits
NPIX 6365
MEAN
STDDEV
-32767.
0.
MIN
MAX
-32767. -32767.
wf> = (1042211.+6365.)/(1024.*1024.) which is equal to 1, as required. Delete the unmasked dark images: wf> imdel *cln_s.fits 5. Run irpipeline to reduce and mask AU08_dk150_1hr01.fits for all 6 first differences: wf> unlearn irpipeline wf> irpipeline “AU08_dk150_1hr01.fits” mask=yes maskfile=”FPA15_150K_mask.fits” You may verify the values in the refpix*.txt files using imstat, e.g.,
9
Instrument Science Report WFC3 2000-019 wf> imstat AU08_dk150_1hr01_cln_s.fits[4][6:512,6:512] lower=-32766 \ upper=INDEF fields=”mean,midpt,stddev” The mean, midpt, and stddev values should equal the values in the q1mean, q1med, and q1std columns for the ImageName AU08_dk150_1hr01_cln_s.fits[4], in the refix_I.txt file. Similarly, you may verify the statistics calculations for the reference pixels by setting the lower limit to INDEF.
Version History Table 4. The table below summarizes the version history of the irpipeline script. Date
History
Version
21 Sept. 2001
Created by Christopher Hanley
1.0
24 Oct. 2001
Modified to include masking.
2.0
Acknowledgement We would like to Howard Bushouse (STScI) for his invaluable review, as well as Ed Cheng (GSFC/DCL), Bob Hill (GSFC/DCL), and the staff of the Goddard Space Flight Center Detector Characterization Laboratory for all their assistance.
10
