FEBRUARY 2007
CAO ET AL.
169
An Improved Algorithm for the Operational Calibration of the High-Resolution Infrared Radiation Sounder CHANGYONG CAO NOAA/NESDIS/Center for Satellite Applications and Research, Camp Springs, Maryland
KENNETH JARVA Science and Technology Corporation, Greenbelt, Maryland
PUBU CIREN QSS Group, Inc., Lanham, Maryland (Manuscript received 24 May 2006, in final form 17 November 2006) ABSTRACT Radiance data from the High-Resolution Infrared Radiation Sounder (HIRS) have been used routinely in both direct radiance assimilation for numerical weather prediction and climate change detection studies. The operational HIRS calibration algorithm is critical for producing accurate radiance to meet the user’s needs, and it has significant impacts on products at all levels. Since the HIRS does not calibrate every scan line, the calibration coefficients between calibration cycles have to be interpolated based on a number of assumptions. In the more than 25-yr history of operational HIRS calibration, several interpolation methods have been used and, unfortunately, depending on which method is used, these algorithms can produce HIRS level 1b radiance data with significant differences. By analyzing the relationship between the instrument self-emission and gain change during filter temperature fluctuations, in this paper a significant flaw in the previous operational calibration algorithm (version 3) is identified. This caused calibration errors greater than 0.5 K and periodically degraded the HIRS radiance data quality of NOAA-15, -16, and -17 between 1998 and 2005. A new HIRS calibration algorithm (version 4) is introduced to improve the calibration accuracy, along with better indicators for instrument noise in the level 1b data. The new algorithm has been validated in parallel tests before it became operational at NOAA/National Environmental Satellite Data and Information Service (NESDIS). Test results show that significant improvements in calibration accuracy can be achieved especially for NOAA-15/HIRS. Several areas of further calibration improvements are also identified. The new algorithm has been used for all operational satellites at NOAA/NESDIS since 28 April 2005.
1. Introduction Radiances observed by the High-Resolution Infrared Radiation Sounder (HIRS) are assimilated directly in operational numerical weather prediction, and are used, in conjunction with measurements from other instruments, to calculate the atmosphere’s vertical temperature and moisture profiles, outgoing longwave radiation (OLR), and upper-tropospheric humidity
Corresponding author address: Dr. Changyong Cao, NOAA/ NESDIS/Center for Satellite Applications and Research, 5200 Auth Rd., WWB, Rm. 712, Camp Springs, MD 20746. E-mail:
[email protected] DOI: 10.1175/JTECH2037.1
JTECH2037
(UTH). Other retrievals include ocean surface temperatures, total atmospheric ozone levels, precipitable water, cloud height and coverage, and land surface temperature. The HIRS calibration algorithm directly affects the radiance accuracy and thus indirectly affects the product quality at all levels. This is especially true for climate change detection studies using the HIRS data, where a calibration algorithm change may have significant impacts on their analyses and results. HIRS is one of the primary instruments for operational atmospheric sounding carried on National Oceanic and Atmospheric Administration (NOAA)’s polar orbiting satellite series for more than two decades. It is
170
JOURNAL OF ATMOSPHERIC AND OCEANIC TECHNOLOGY
a traditional cross-track line scanning radiometer that measures scene radiance in the infrared and visible spectrum. Among the 20 spectral channels, there are 12 longwave channels (669–1529 cm⫺1), 7 shortwave channels (2188–2657 cm⫺1), and 1 visible channel (0.69 m), with a single telescope and a rotating filter wheel consisting of 20 individual spectral filters. An elliptical scan mirror is stepped 56 times in increments of 1.8° to provide cross-track scanning. The instantaneous field of view (IFOV) for the HIRS/3 (on NOAA-15, -16, and -17) is 1.4° for the longwave and 1.3° for the shortwave channels, providing a nominal footprint of ⬃20 km at nadir. To facilitate cloud clearing, the IFOV for HIRS/4 (NOAA-18 and beyond) instrument has been reduced by half (down to a field of view of 0.7°, or ⬃10 km nadir footprint), although the number of cross-track samples and step size remain the same as that of HIRS/3. The gaps between IFOVs therefore have increased as a result. The calibration of the HIRS infrared channels makes use of views of the onboard warm blackbody and cold space. This provides a two-point calibration in which the calibration slope and intercept for each channel can be computed and used to convert instrument output counts to radiance. However, there are several complications in the operational HIRS calibration that affect the calibration accuracy. HIRS calibrates only once every 40 scan lines, or one calibration cycle in every 256 s. As a result, the calibration coefficients between the calibration cycles must be interpolated. In the more than 25-yr history of operational HIRS calibration, several interpolation methods have been used and, unfortunately, depending on which method is used, the derived calibration coefficients can produce different radiance values for the level 1b data. In this study, the operational HIRS calibration algorithms versions 2 and 3 are evaluated. The fundamental problems in the previous algorithm (version 3), and the theoretical basis of the new algorithm are discussed. The details in the improvements of the new algorithm, its validation, and related data format change are introduced. The new algorithm has been implemented at NOAA/NESDIS for the calibration of HIRS on all operational NOAA satellites since 28 April 2005. The purpose of this paper is to provide a better understanding of the rationale of the new algorithm, and its effects on the products and data users. It also serves as the primary reference for those who are interested in the independent implementation of this algorithm for either operational use or for the time series analysis of historical HIRS data.
VOLUME 24
2. Previous operational HIRS calibration algorithms In the operational ground processing system at NOAA/NESDIS, the calibration of HIRS uses a twopoint calibration at the calibration cycles: the first calibration point is the average of 48 observations of space and the radiance of space; while the other is the average of 48 observations of the warm blackbody calibration target and its radiance. Note that there are actually 56 observations made during both the space and warm target scans; however, since the first eight samples of the space view are contaminated by the earth as the instrument slews to the space position, these are excluded from use in the calibration. For the sake of consistency, the corresponding blackbody-view samples are also excluded. The remaining samples are checked against the gross limit boundaries specified in the calibration parameter input data set (CPIDS) database. Any that exceed those limits are excluded from further consideration. The kinetic or bulk temperature of the blackbody is computed from readings of the platinum resistance thermometers (PRTs; there are four for HIRS/3 and five for HIRS/4) embedded in the blackbody base. The temperature is then converted to radiance by use of the Planck function. The slope and intercept of the line connecting the space and blackbodyview points at the HIRS calibration cycles are referred to as the raw calibration coefficients. The slope of the calibration line for a channel is the inverse of the instrument gain for that channel. For a typical two-point calibration with space view and blackbody view, the calibration slope for a given infrared channel is computed as: S ⫽ 共Rbb ⫺ Rsp兲Ⲑ共Cbb ⫺ Csp兲,
共1.1兲
where S is the calibration slope; Rbb is the blackbody radiance, computed based on the blackbody PRT temperature, spectral response function for the channel, and the Planck function; Rsp is the space-view radiance, which is zero; Csp is the average of the 48 space-view counts during a calibration cycle; and Cbb is the average of the 48 blackbody-view counts. Traditionally, the equation for calculating the earthview radiance takes the form of Re ⫽ I ⫹ Ce ⫻ S,
共1.2兲
I ⫽ ⫺Csp ⫻ S,
共1.3兲
where Re is earth-view radiance, I is the intercept, and Ce is the earth-view count. Note that despite a nonlinear equation in quadratic term is shown in the KLM user’s guide, the actual equation used in the operations has been the linear Eq. (1.2) in the history of HIRS and
FEBRUARY 2007
CAO ET AL.
will remain in this form for reasons discussed in section 4. An important data block for HIRS is the superswath, which starts with a calibration cycle, and consists of a space view followed by a warm blackbody view (56 samples each), and then 38 earth-view scan lines with 56 samples per scan line. Note that the early HIRS models such as HIRS/2 on NOAA-14 and earlier spacecrafts had additional 56 samples of the cold blackbody view between the space and warm blackbody views (with only 37 earth-view scan lines). In fact, HIRS is one of the few radiometers designed with two onboard blackbodies to better handle nonlinearity for improved calibration accuracy. Unfortunately, the data from the cold blackbody was not usable due to large temperature gradients and possible thermal interactions with the surround. As a result, the cold blackbody view was eliminated in the HIRS/3 and HIRS/4 series. In retrospect, the cold blackbody could have served as a redundant blackbody if it had the same temperature as the warm blackbody, which could potentially reduce blackbody calibration uncertainty for climate studies. Nevertheless, hereafter in this paper, the blackbody refers to the HIRS onboard warm blackbody. When the scan mirror dwells on space or the blackbody during a calibration cycle, earth-view data are unavailable; this results in the characteristic gaps between superswaths in HIRS data (Fig. 1). Similarly, during the 38 scans of earth, the instrument cannot be calibrated, and the calibration coefficients must be interpolated between the adjacent calibration cycles. A simple calibration algorithm was used for HIRS/2 series of the instrument (prior to NOAA-15/HIRS). In this algorithm, constant calibration coefficients from the last calibration cycle were applied to the first half of the earth scan lines within a superswath, while the remaining half of the earth scan lines used calibration coefficients derived from the next calibration cycle. This was a relatively robust algorithm for handling the peculiar calibration cycles of HIRS. Unfortunately, this algorithm occasionally caused artificial jumps in the earth view radiances when switching calibration coefficients in the middle of a superswath. This was especially noticeable when the calibration coefficients experienced rapid changes from one calibration cycle to the next. In 1998, an experimental HIRS calibration algorithm was developed by L. McMillin, with FORTRAN programs made available (Kidwell 1998, appendix M) for calibrating NOAA-12/HIRS. This algorithm addresses the issue of the jump at the center of a superswath, and also takes into account of the environmental temperature changes through a superswath. In this algorithm,
171
FIG. 1. The concept of HIRS superswath and calibration cycles. At each calibration cycle there are 56 space-view (white lines) and blackbody-view (gray lines) samples, which occur every 40 scan lines. Within each superswath, there are 38 earth-view scan lines. Sample image of channel 13 shown here; other channels are similar.
the calibration coefficients are first computed using two separate methods: 1) the measured values at the calibration cycles, and 2) the predicted values based on pre-established correlation between the instrument temperatures and the calibration coefficients based on past datasets. Then the differences between these two set of values are resolved at the calibration cycles, before interpolation is applied to the earth-view data. A fundamental problem with this approach is that the correlation between instrument temperatures and calibration coefficients is treated as static for a given instrument, which is invalid as discussed in sections 3 and 4. The notion that the calibration coefficients can be predicted for the lifetime of the instrument based on a preestablished fixed set of coefficients is not well founded, because the contribution of each instrument component to the background radiation is dynamic and can change over time. In addition, the documentation for this algorithm is sketchy. Neither the algorithm for computing the correlation coefficients, nor the physical mechanism by which the instrument component temperature (such as the electronics temperature) relates to the slope and intercepts can be found in the documentation. Although this algorithm is provided in the
172
JOURNAL OF ATMOSPHERIC AND OCEANIC TECHNOLOGY
NOAA polar orbiter user’s guide for interested users, it was never implemented in the operations. With the launch of NOAA-15 in 1998, a new operational algorithm (referred to as HIRS algorithm version 3) was developed at NOAA/NESDIS for NOAAKLM/HIRS (or HIRS/3) and has been used in the operations (Goodrum et al. 2000). In this algorithm, it was assumed that the HIRS instrument gain (inverse of calibration slope) does not change appreciably within any 24-h period. Therefore, it was believed that a 24-h average slope could be used to calibrate all the data during the period. In addition, it was believed that the secondary mirror baffle temperature contributed significantly to the observed radiation and thus affected the intercepts for the earth-view scan lines between two calibration cycles. Therefore, in this algorithm the correlation between the secondary mirror baffle temperature and the intercept was computed once every 24 h and used to adjust the intercepts using the following equations, I⬘n ⫽ Iln ⫹ Itn,
共2.1兲
where I⬘n is the intercept used for earth-view scan line n (n ⫽ 2 to 39, given space view ⫽ 0 and blackbody view ⫽ 1), Iln is the linear interpolation of the intercept between the two closest calibration cycles, and Itn is the adjustment to the intercept based on the secondary mirror baffle temperature. Below Iln and Itn are computed as
冋
Itn ⫽ b1 共Tn ⫺ Tk⫺1兲 ⫺ n
冋
Ilc ⫽ I⬘c共k⫺1兲 ⫹ n
I⬘c共k兲 ⫺ I⬘c共k⫺1兲 40
册
Tk ⫺ Tk⫺1 , 40
册
(2.2)
, 共2.3兲
where I⬘c(k) is the intercept at calibration cycle k, recomputed using the 24-h average slope and space-view average; n is the scan-line number within the calibration cycle (n ⫽ 2 to 39, when n ⫽ 0, Tn ⫽ Tk⫺1; n ⫽ 40, Tn ⫽ Tk), Tk is the secondary mirror baffle temperature at calibration cycle k, Tn is the secondary mirror baffle temperature for scan line n, and b1 is the slope of the linear regression line. HIRS calibration algorithm 3 can cause significant calibration biases, because the assumption of a stable instrument gain is occasionally invalid. Similar problems have also been reported by the Met Office when comparing radiances produced by NESDIS and their Advanced Television Infrared Observation Satellite (TIROS) Operational Vertical Sounder (ATOVS) and Advanced Very High Resolution Radiometer (AVHRR) Processing Package (AAPP; P. Brunel 2002, personal communication). Although during normal op-
VOLUME 24
FIG. 2. The concept of background radiation or instrument selfemission for an infrared radiometer, assuming that the target radiation (R target ) and instrument self-emission (R self ) are linearly combined when reaching the detector (Rdetector). For HIRS, Rself is on the order of 98%, while Rtarget is approximately 2% of Rdetector.
erations, the instrument gain of HIRS is stable within a typical 24-h period, it can change in response to the temperature of the aft-optics, including the temperature of the filter wheel and its surround. This is because the background radiation or instrument self-emission (Fig. 2) reaching the detector is many times larger than the incoming scene radiation (ITT 1996, 1998). A small change in the background radiation can significantly change the calibration slope. This is both demonstrated theoretically and has also been observed in the operational calibration of HIRS as discussed below. It should be noted that despite the similarities in design between the HIRS and the Geostationary Operational Environmental Satellites (GOES) sounder, both are filter wheel radiometers with similar channels built by the same manufacturer, there are significant differences. In particular, the HIRS has warm filters (⬃292 K) that are not temperature controlled, while the GOES sounder filter is cooled at ⬃210 K. The warm filter in HIRS has a significant impact on the background radiation falling on the detector and contributes more than 95% of the signal at the detector. The HIRS filter wheel has a nominal orbital variation of ⬃0.1 K. The filter wheel temperature is not measured directly but is inferred from the filter wheel housing temperature, which has a temperature range specification of 273.15 to 333.15 K. Despite the usual narrow orbital temperature variation, the filter wheel temperature can increase at times due to friction in the filter wheel bearing, which is typically accompanied by increased filter wheel jitter, higher-than-normal filter motor current and, sometimes, the filter wheel being out of sync. To alleviate this problem, the filter housing heater may be turned on to draw lubricants into the bearing but, unfortunately, this also further increases the temperature of the filter and its surround by several degrees. This temperature increase significantly changes the background flux reaching the detector. Since the HgCdTe
FEBRUARY 2007
CAO ET AL.
173
FIG. 4. The 24-h average slope can significantly deviate from the current slope when the filter temperature changes, which causes calibration biases in earth view data. Sample orbit: NSS.HIRX. NK.D02129.S2211.E2340.B2072930.WI. FIG. 3. The effect of nonlinearity and self-emission for HIRS. Since self-emission dominates, the instrument gain is stable at point a despite changes in target temperature. However, if the self-emission changes as the filter temperature increases, the instrument gain will shift to a⬘ .
detector has a nonlinear response, the significant increase in the radiation reaching the detector causes the response to shift to a different portion on the nonlinear response curve (Fig. 3), which has a different gain (becomes less responsive at higher temperatures). When this occurs, the 24-h average slope, which could be a slope computed as much as 24 h ago, will become outdated. The opposite occurs when the filter temperature drops from a high temperature to a more normal temperature. As a result, using the 24-h average slope can cause calibration biases (Fig. 4). Since the calibrated earth-view radiance is approximately a linear function of the slope, if there is a 1% percent bias in the slope, it will translate to 1% bias in the earth-view radiance (or as much as 1 K for the longwave channels). In the operations, biases on the order of 2% have been observed in some channels of HIRS on NOAA-15 during such events (Cao et al. 2005). Since 2002, the NOAA-15 HIRS filter temperature fluctuates on the order of a few degrees several times a month (Fig. 5), during which the radiance data became highly biased and has degraded the data quality and undermined users’ confidence in the HIRS data. Another known effect related to temperature change in interference filters such as those used for HIRS is the spectral shift toward the longer wavelength as the filter temperature increases. This shift is more pronounced in the shortwave channels than in the longwave. However, we believe that this is a secondary effect compared to the slope problem discussed above, because this spectral shift is relatively small, on the order of 0.1 cm⫺1
K⫺1 for channels near 15 m and 0.5 cm⫺1 K⫺1 for channels around 3.9 m, according to measurements of HIRS filter witness samples by National Institute of Standards and Technology (NIST; Cao et al. 2004). It is true that some HIRS channels are located on a steep slope of a spectral radiance curve (such as the longwave CO2 channels), which makes the measurement very sensitive to spectral shifts. However, there are HIRS channels located in the flat regions of the spectral radiance curve with much reduced sensitivity to spectral shift (such as the surface channels). Therefore, spectral shift–induced biases in HIRS are channel dependent. Also, since the longwave and shortwave channels are located on difference regions of the atmospheric ab-
FIG. 5. The (lower curve) filter wheel housing and (upper curve) filter wheel motor temperature fluctuated frequently for NOAA15/HIRS from April 2002 to June 2003, which caused discrepancies between the 24-h average slope and the actual slope.
174
JOURNAL OF ATMOSPHERIC AND OCEANIC TECHNOLOGY
FIG. 6. Relation between secondary mirror baffle temperature and channel 2 intercept for one sample orbit for NOAA-17/HIRS
sorption band with different slopes on the spectral radiance curve (3.7–15 m), the bias in the longwave and shortwave due to spectral shift would go in opposite directions. But this is not what we observed when the filter temperature increases. Since the effect of spectral shift is relatively small compared to the slope problem, and its correction is channel dependent, the HIRS algorithm version 4 does not address this issue. The effect of spectral shift will be examined in HIRS time series analysis in future studies. It is now believed that the algorithm version 3 for calculating the intercepts is also flawed in that it overestimates the importance of the secondary mirror baffle temperature on background flux. Since the secondary mirror is highly reflective and not very emissive, its temperature fluctuation may not change the background flux on the detector sufficiently to cause significant changes in the intercepts. In the HIRS algorithm 3, the secondary mirror temperature is in effect used to empirically represent the entire change in the background flux of the instrument, which may not be appropriate. Figure 6 shows the typical relationship between the intercept and secondary mirror baffle temperature for one orbit of NOAA-17/HIRS data. It is clear that this relation is elliptical instead of linear in shape. There appears to be a different relation during the day than during night. Therefore, the linear correlation assumption is inaccurate at best. Also, the correlation is derived from global data under a variety of conditions, and its applicability to the interpolation between two particular calibration cycles may not be justified. In addition, it is arguable whether the secondary mirror baffle temperature is a reliable indicator of background flux because 1) the secondary mirror assembly has a small thermal inertia and its temperature fluctuates more than any of the other components of the instrument, and 2) the intercept is determined by
VOLUME 24
the radiation integrated over the entire optical chain from the primary mirror to the detector, not just the secondary mirror. It is possible that the correlation between the intercept and the fore-optics temperature could be determined with a thermal model incorporating the temperatures of the various components, which might then be represented by additional terms in Eq. (1). Unfortunately, these issues cannot be fully resolved in the operational calibration of HIRS, because in the current operational processing, only a small portion of the data (about three superswaths) is accessible at a time. Ideally, data from the entire orbit, the orbit before and after, would be used in estimating the intercepts. The improved algorithm presented in the next section does not fully address this issue because of the operational constrains. Further studies are required to resolve these issues and to improve the calibration accuracy of HIRS for climate applications.
3. The new HIRS calibration algorithm (version 4) The HIRS calibration algorithm version 4 was developed mainly to resolve the issue with the calibration slopes in the algorithm version 3, along with several other improvements discussed later. We believe that the assumption of a constant instrument gain over a 24-h period is not well founded, and that it does not account for the impact of instrument background flux changes on the instrument gain. This leads to a very different approach to calculating and applying the instrument gain and slopes in the new algorithm. As a general principle, the calibration coefficients for earth views should be derived from the calibration events closest in time to those views so that the instrument gain at the time of earth observation can be known with some certainty. In addition, each calibration has uncertainties because all the samples contain noise. In the new algorithm these problems are addressed by using a three-point running average slope that is computed from nearby calibration events. In particular, the slope used for all the earth views of a given superswath is the average of the raw slopes from the calibration cycle of that superswath and those from the superswaths that come immediately before and after. The calibration intercept is then recomputed using the three-point running average slope and the space-view counts. It is acknowledged that this new algorithm does not fully address the issue with the intercepts, given the operational constrains. However, a software switch is added to the intercept adjustment so that the dependence on the telescope temperature can be turned on or off, thus providing alternatives in computing the intercepts. In
FEBRUARY 2007
175
CAO ET AL.
the current operational data processing, the switch is in the on position so that the intercepts are calculated based on secondary mirror baffle temperature for backward compatibility.
a. The raw slopes of a calibration cycle As discussed in previous sections, 56 observations of space followed by 56 observations of the blackbody constitute a calibration cycle. Such a cycle occurs every 40 scan lines or 256 s (Fig. 1). For each calibration cycle, the ground processing computes the “raw” slopes and intercepts for the 19 infrared channels and stores them with the blackbody-view data in the HIRS level 1b file. The meaning of the stored parameters remains the same as in the previous algorithm version 3. However, because of the increased significance of the individual raw slopes and intercepts in the new algorithm, the following changes have been made to improve the quality control on data used for the calculation. First, samples that are out of the gross count limit range are removed, and the number of samples left is recorded. In case all samples are out of the gross limit range, the slope cannot be computed for this calibration cycle (same as in the version 3 algorithm). The gross limits have two functions: 1) they are used as thresholds to ensure that the data range is within ⫾4095 to avoid errors. 2) They can be adjusted to handle saturation problems. Typically, the count values are several hundreds away from ⫾4095. However, when the instrument background flux increases, some channels may have space-view counts increase to near or above 4095. In such cases, setting the gross limits to ⫾4094 will effectively screen out saturated values. This was successfully used for NOAA-18/HIRS to screen out saturated channel-1 data shortly after launch. Second, the mean and standard deviation of the surviving populations are computed and the standard deviation of each is compared to the noise equivalent delta counts (NEDCs), which are equal to the noise equivalent delta radiance (NEDN; an instrument design specification) divided by the 24-h average slope of the given channel. Channels for which the standard deviation exceeds NEDC are considered out of the instrument specification, and scan lines from subblocks with one or both of the space and calibration target views out of spec are flagged accordingly (see section 3e). The counts are subjected to a 3-sigma filter in which elements of the population more than three standard deviations from the mean are eliminated, and the survivors are used to recompute a new mean and standard deviation. The median is also computed. If the median is more than one standard deviation from the mean, the count population is considered anomalous and is flagged accord-
ingly. Channels that pass the NEDC check are also subjected to the 3-sigma filter. Their mean and standard deviation are recomputed from the survivors and the mean is used in the calculation of the raw slopes. It should be noted that the NEDC tests in the new algorithm represent a major improvement over previous versions, because NEDN/NEDC is one of the most important indicators of instrument performance. Previously users had no simple way of knowing whether the instrument met noise specifications for a given orbit. Now users can simply check the NEDC flags for the blackbody and space views.
b. Calibration for a normal superswath 1) RUNNING
AVERAGE SLOPE CALCULATION
For a regular superswath in the middle of an orbit, the calibration slopes for the 38 earth-view scan lines of the current superswath is the running average of three values: the raw slopes of calibration cycles k ⫺ 1, k, and k ⫹ 1 (Fig. 1); that is, S⬘k:k⫹1 ⫽ 共Sk⫺1 ⫹ Sk ⫹ Sk⫹1兲ⲐM,
共3.1兲
where S⬘k:k⫹1 is the running average of the three slopes, to be used for the 38 earth-view scan lines in the current superswath; that is, the superswath between k and k ⫹ 1. Also Sk⫺1, Sk, and Sk⫹1 are the raw slope for the calibration cycles k ⫺ 1, k, and k ⫹ 1, respectively; k is the calibration cycle number; and M is the number of quality checked raw slopes used for averaging (M ⫽ 1, . . . , 3). There are several reasons for using a running average for the slopes. First, as a matter of a basic calibration principle, this will provide updated slopes that are computed near the time of earth observations, compared to the 24-h average slopes (potentially outdated) used in the version 3 algorithm. Second, using a three calibration cycle average reduces fluctuations in the individual raw calibration coefficients due to instrument noise. The assumption here is that the background flux will not change significantly during an 8-min period (or approximately two calibration cycles). This is a significant improvement over the algorithm version 3 where it was assumed that such fluctuations would not occur in a 24-h period.
2) INTERCEPT After the three-point average slopes are computed, the intercepts at the time of the space view are recomputed using the average slopes using Eq. (3.2) below. The recomputed intercepts and the three-point average slopes are stored with the space-view scan-line data: I⬘k ⫽ ⫺S⬘k:k⫹1 ⫻ Csp,
共3.2兲
176
JOURNAL OF ATMOSPHERIC AND OCEANIC TECHNOLOGY
where I⬘k is the recomputed intercept for the calibration cycle k, S⬘k:k⫹1 is the average slope for the current superswath, and Csp is the space-view count average for the calibration cycle k. The three-point average slopes are then used for each of the earth scan lines of the superswath. The intercepts, however, are interpolated using a slightly modified version of Eq. (2.1): I⬘n ⫽ Iln ⫹  ⫻ Itn,
共3.3兲
where I⬘n is the intercept for earth-view scan line n (n ⫽ 2 to 39), Iln is the linear interpolation of the intercept between the two calibration cycles, and Itn is the correction to the intercept based on the secondary mirror baffle temperature. Also,  is a switch with values of either 0 or 1 (stored in the CPIDS). When  ⫽ 0, Itn is effectively turned off, and the interpolation for the intercepts becomes linear. The above equation for computing intercepts is essentially unchanged from algorithm version 3, except that the correction based on the secondary mirror baffle temperature can now be turned on or off by changing .
c. Calibration for a partial superswath A partial superswath is one in which some scan lines within a superswath are missing. They typically occur at the beginning and end of orbital datasets. In such cases, the number of earth-view scan lines in the superswath could be any number between 2 and 39. Most HIRS orbits have partial superswaths. Partial superswaths also occur when there are data gaps. If for some reason, the calibration data at the beginning of a superswath k or at the beginning of superswath k ⫹ 1 is unusable (e.g., if the space views were saturated), then the remaining data should be treated as a partial superswath in data processing. Since the raw slopes and intercepts are missing on one side of the partial superswath, it is generally recognized that the calibration coefficients for the partial superswaths cannot be estimated as reliably as those for a normal superswath. However, it should be noted that a 6-min data overlap (more than a full superswath) of succeeding orbital datasets for the current NOAA polar-orbiting satellites was intended in part to cover the gaps in the HIRS coverage. In most cases, users requesting a continuous datastream (i.e., a sequence of consecutive orbits) should be able to safely discard any partial superswaths at the beginning or end of an orbit, because the discarded data are actually available in the adjacent orbit in a full superswath. If the HIRS orbital datasets were always processed in the same sequence in
VOLUME 24
which they are created, the last set of raw calibration coefficients from an orbit could be used to interpolate the calibration coefficients at the beginning of the next. However, not all HIRS datasets are processed sequentially (the so-called blind orbits come in out of the normal order) and the following alternative method is used. 1) The calibration slope of a partial superswath is the average value of the quality checked slopes from the nearest one or (preferably) two adjacent calibration cycles. In case none is acceptable, the most recent 24-h average, which is stored in an auxiliary file [the HIRS Calibration File (HCF)], is used. When this occurs, the data may become anomalous, as indicated in the flags (Table 1). 2) The calibration intercepts for a partial superswath are computed using the same method as the previous algorithm version 3. 3) For independent direct read-out receiving stations, b1 in Eq. (2.3) computed using global data (which is currently derived from 24 h of global data) may not be appropriate for interpolating the intercepts for the locally received data. Since the correlation between the secondary mirror baffle temperature and the intercept vary over an orbit, the b1 values can be better estimated using local direct read-out data and separating day and night. Finally, changes in data processing will take place in the upcoming Initial Joint Polar-orbiting Operational Satellite System (IJPS), a cooperative effort between NOAA and European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT). In the IJPS era, the HIRS level-0 data are designed to come in sequence, and the occurrence of partial superswaths will be greatly reduced.
d. Moon in space-view detection and correction The moon occasionally contaminates the HIRS view of space during calibration. This can happen during several consecutive orbits in a month. When this occurs, although it only corrupts the space views of one calibration cycle, it can degrade the calibration coefficients for the earth view in the superswaths nearby if untreated. Moon detection and correction has been a concern for all NOAA instruments. The previous moon detection/correction algorithm for HIRS used a simple threshold technique. To determine whether a particular space view was contaminated by the moon, the average of the measurements of space (in counts) for channel 19 is compared to the 24-h average of the channel 19 (3.7 m) space counts (stored in the HCF), and if the difference was greater than a threshold value (in CPIDS),
FEBRUARY 2007
177
CAO ET AL. TABLE 1. Major changes in the HIRS level 1b quality indicators (effective 28 Apr 2005). NOAA-N definition
Bit 28 in quality indicator bit field
Bit 6 in scan-line quality flags (calibration problem code)
Bit 4 in the (channel by channel) calibration quality flag
Bit 3 in the (channel by channel) calibration quality flag
Bit 2
Used as a quality summary flag to indicate calibration anomaly detected (same as the original KLM definition) Scan contains marginal calibration in some of the IR channels. This is triggered if any of the bits 4, 3, or 0 below in the calibration quality flags is set for any IR channel. Defined as anomalous space or blackbody view data, or abs(mean – median) ⬎NEDC. For earth view, this is set if either the space or blackbody view bit 4 is set. Cal slope from the HCF (HIRS calibration file). Triggered if the slope cannot be computed and has to use 24-h average slope as replacement. Blackbody views failed NEDC test
NOAA-KLM definition Calibration anomaly detected
Bit 14: Scan line was calibrated using fewer than the preferred number of scan lines
User software change needed to interpret the bits correctly.
Bit 3: All bad PRTs for this line
User software change needed to interpret the bits correctly. When triggered, it may indicate major problems with the calibration. User software change needed to find out whether instrument performance meets the noise specification. However, high noise does not necessarily trigger bit 6 above. User software change may be needed to find out whether instrument performance meets the noise specification. However, high noise does not necessarily trigger bit 6 above. Occurs when the HCF file is missing Software change maybe needed to check the correct channel
Bit 2 marginal BB view counts
Space view failed NEDC test
Bit 1 marginal space view counts
Bit 0
Some QC tests could not be applied Words 2–20: channels 1, 17, 2, 3, 13, 4, 18, 11, 19, 7, 8, 20, 10, 14, 6, 5, 15, 12, 16, 9
Bit 0 marginal PRT temperature Channel order discrepancy existed between document and data in previous version
the contaminated counts were replaced by the average from the most recent calibration cycle and then used in calculating the raw calibration coefficients. A new method for determining whether a calibration view is contaminated by the moon has been implemented for Advanced Microwave Sounding Unit A (AMSU-A; Kigawa and Mo 2002) and is intended to be used for HIRS as well. In this algorithm the separation angle between a vector pointing in the direction of the space view and a vector pointing in the direction of the moon is computed. If the angle is less than a limiting threshold whose value is stored in the CPIDS database,
No impact on users. When triggered, user software should further check bit 6 below to find the cause. User software change needed to interpret the bits correctly. If triggered, need to further check bits 4, 3, and 0 below for each channel.
Bit 4: All bad space view counts for scan line
Bit 1
Channel order
Impact on users
the view or views are considered to be contaminated, and the HIRS space-view data will be excluded for the purposes of calibration. Once the moon is detected in calibration cycle k, the slope S(k) becomes invalid and is removed from Eq. (3.1). The slope becomes the average of two slopes. When the space view is contaminated by the moon, the intercepts for the calibration cycle k are computed using the alternative equation below, which relies on blackbody-view counts and PRT temperatures: I⬘k ⫽ Rbb ⫺ Cbb ⫻ S⬘k:k⫹1,
共3.5兲
178
JOURNAL OF ATMOSPHERIC AND OCEANIC TECHNOLOGY
FIG. 7. The algorithm successfully predicted and corrected the calibration coefficients for the moon event for NOAA-16/HIRS space view at 0709:57 UTC 10 Mar 2006 (NSS.HIRX.NL.D06069. S0618.E0810.B2816263.WI).
where Rbb is the radiance computed based on the PRT data, and Cbb is the blackbody-view count. It is assumed that the instrument gain did not change within the last two calibration cycles, and the PRT measurements of the blackbody temperature have no anomalies. However, because of technical difficulties in the operational implementations, the new moon detection and correction algorithm has not been implemented for HIRS. Instead a temporary moon detection algorithm has been put into operation at the time when this paper was written. In this temporary algorithm, the spaceview count is predicted based on the average slope and the blackbody-view count values. Then the predicted and measured space-view counts are compared and moon contamination is identified using a threshold value. This temporary moon detection algorithm was made operational in October 2005. Preliminary analysis on sample NOAA-16/HIRS data shows that the algorithm worked correctly as expected when the moon was in space view (Fig. 7).
e. Quality check of the calibration coefficients Regardless whether the moon is detected in the space view, the average slopes will be checked to ensure that their values are within the normal range. The criteria is that the three raw slopes used in calculating the average slopes for a superswath have to agree within a certain percent (currently set to 2% using an adjustable variable PDIFAVE, an item stored in the CPIDS). If not, the one farthest from the mean value is removed and the mean recomputed. The mean slopes are then checked against their 24-h orbital averages. If any deviates from the 24-h average value by more than a threshold (PDIF24HR ⫽ 10%,
VOLUME 24
which is adjustable and stored in the CPIDS database) for a channel, it is considered an anomaly and the 24-h average value is used and such occurrences are indicated in the level 1b data (Table 1). The average slope is then used to recompute the intercepts for the current and the next superswaths. If the space-view calibration point is not contaminated, the intercept is computed with the average slope and space-view counts. Otherwise, the blackbody-view counts and radiance are used to calculate the intercept [Eq. (3.5)]. These recomputed intercepts then serve as the anchor points for interpolating intercepts for the intervening earth scans. The operational software has been designed to work with or without the HCF; that is, with or without a knowledge of the 24-h average calibration slopes. Obviously, if the HCF is missing or it has less than 24 h of data (e.g., immediately after the instrument activation for a new satellite), then some of the quality checks cannot be performed and the scans produced under those conditions are flagged accordingly (Table 1). The version 4 algorithm is developed based on several assumptions. Each assumption has been checked against NOAA KLM and N data. Instrument performance will continue to be monitored and should the need arise, the threshold values discussed earlier will be adjusted accordingly. The main assumption is that the calibration slope for each of the 19 IR channels will not change more than 10% within any consecutive 24-h period. Historical data suggest that the 24-h variation in the slopes during normal operations is less than 2%. The slopes are most responsive to changes in the filter wheel temperature. In the extreme case of NOAA-15/ HIRS, a ⬃6% change in the slope has been observed in some channels when the filter wheel temperature changed a few degrees during a 24-h period.
f. Data storage and format changes in the level 1b file As discussed previously, the raw slopes and intercepts for each channel are stored along with the blackbody scan data in the level 1b datasets. This remains the same as the previous versions of level 1b data. However, the average slopes and recomputed intercepts are stored along with the space-view data, which is different from the previous version where it was zero-filled at these locations. The new scheme allows better charting and trending of the calibration coefficients. It also facilitates the comparison of the raw and average calibration coefficients. Note that the HIRS level 1b data contains two sets of coefficients: primary and secondary. For earth-view scans, the recomputed average slopes for the current superswath and the interpolated intercepts are stored as primary calibration coefficients. The
FEBRUARY 2007
CAO ET AL.
FIG. 8. Slope comparison between version 3 and 4 algorithms (parallel test for sample orbit: NSS.HIRX.NK.D05075.S1902. E2037.B3556061.GC).
secondary coefficients are used for testing and the contents may be subject to change over time. They are currently set as follows: the slope is identical to that of the primary coefficients, but the intercept contains the linear intercept Iln, instead of I⬘n in Eq. (3.3). This allows an analyst to compare theses two sets of coefficients and evaluate the effect of the alternative algorithms in calculating the intercepts. The 24-h average slopes and the b1 coefficients in Eq. (2.3) are stored in the level 1b header, which are documented in the NOAA-N user’s guide (http://www2.ncdc.noaa.gov/docs/klm/).
4. HIRS calibration algorithm version 4 calibrated radiance data comparison The new algorithm was validated by comparing results from the version 3 and the new algorithm version 4. Figure 8 is a comparison of the slopes from a random sample NOAA-15 HIRS orbit on day 75 in 2005. Several observations become obvious in the comparison. First, for the earth view, the 24-h average slopes (thick line on top) are off from the raw slope values (plus signs) at the calibration cycles by ⬃1%. For channel 2 (14.71 m) at scene temperature ⬃250 K, this translates to an error about 0.5 K. Second, the three-point running average slope from the new algorithm closely track the raw slopes at the calibration cycles (diamonds), which vary slightly over the orbit. The threepoint averages also tend to smooth the data. Third, at the calibration cycles, there is a small difference in the slopes between the version 4 and version 3 algorithms (plus versus diamond signs). This small difference is not a concern because it is caused by the fact that the software is running on new hardware that has a better precision for floating-point numbers; a minor error in the
179
FIG. 9. Calibration biases between the new and old algorithms for selected longwave channels (biases for shortwave channels smaller). Nadir pixel brightness temperature difference for sample orbit: NSS.HIRX.NK.D05075.S1902.E2037.B3556061.GC.
computation of the blackbody temperature in the version 3 algorithm was corrected; and an updated set of Planck constants were used in the new algorithm [Planck constants used: C1 ⫽ 1.191 042 7 ⫻ 10⫺5 mW (m2 sr cm⫺4)⫺1; C2 ⫽ 1.438 775 2 (K cm). Source: 2002 CODATA http://physics.nist.gov/cuu/Constants/index. html]. Calibration biases in the earth-view data due to the differences in the slopes are found to be on the order of 0.5 K at nadir (Fig. 9). This case clearly shows that the new algorithm can improve the calibration accuracy, because the calibration in the version 3 algorithm departed from the blackbody calibration point. The error becomes smaller for an HIRS instrument during its normal performance period (stable filter wheel temperature), where the typical earth-view brightness temperature differences between version 3 and 4 algorithms are less than 0.2 K. It was also found that large brightness temperature difference (⬎1 K) occur for the shortwave channels (such as the 3.7-m channel) for very cold scenes, such as in the polar regions, where brightness temperatures can be 200 K or less. This is because at this temperature, the signal in the 3.7-m channel is smaller than the noise, which is on the order of 0.001 mW (m2 sr cm⫺1)⫺1. Differences on the order of 0.0005 mW (m2 sr cm⫺1)⫺1 at this temperature can cause differences in brightness temperature of several degrees. In this case, temperature difference is not as meaningful as radiance difference. The differences in the intercepts computed with and without regard to the secondary mirror baffle temperature (i.e., using the primary versus the secondary coefficients) were also evaluated. Recall that the intercept
180
JOURNAL OF ATMOSPHERIC AND OCEANIC TECHNOLOGY
in the primary coefficients is computed in the same way as in the algorithm version 3, using the secondary mirror baffle temperature for correction. The intercepts in the secondary coefficients, on the other hand, are linear interpolations between calibration cycles. The difference in the intercepts and thus the radiances is essentially due to the temperature swings of the secondary mirror. In this study, the differences are found to be as much as 0.5 K, although the biases are highly localized within a superswath. We believe that the correction using the secondary mirror baffle temperature introduces errors as discussed previously. However, the effect of the bias may not be significant for products because this localized bias occurs only when the secondary mirror baffle temperature has large swings such as in the terminator regions. The Met Office has also independently implemented the HIRS4 calibration algorithm. As of 18 March 2005, it has been reported that their implementation produced calibration coefficients that closely matched those of the NESDIS level 1b secondary coefficients (N. Atkinson 2005, personal communication). In other words, the radiance data generated by the Met Office are now practically identical to the radiances generated at NOAA/NESDIS using the linear intercepts. The discrepancies between the AAPP and NESDIS HIRS radiance data discussed earlier have now been resolved. The new HIRS calibration algorithm version 4 was implemented in the NOAA/NESDIS operational ground data processing system. After a few issues related to the level 1b flags (Table 1) and data format changes were clarified and resolved with the users, the algorithm became operational on 28 April 2005. It has been used for the operational production of HIRS radiance data for NOAA-15, -16, -17, and -18 since then. There are a few remaining issues with the HIRS calibration that require further study. First, in the postlaunch on-orbit verification of NOAA-18/HIRS, it was found that the longwave channels do not meet the noise specification. Extensive analyses show that the noise is originated from the aft optics possibly due to a loose part, and it is unrelated to the HIRS calibration algorithm change. Second, the nonlinearity correction has not been used for the calibration of HIRS, since the effect is relatively small, and contradictions exist in the prelaunch nonlinearity test results (same nonlinearity found in linear channels). For example, according to an analysis by the Massachusetts Institute of Technology (MIT) Lincoln laboratory on the NOAA15/HIRS/H302 prelaunch thermal vacuum data, only channels 7–10 showed any appreciable nonlinearity, between 0.06% and 0.12% (E. Wack 1998, personal communication), which are at or below the instrument noise level. Nev-
VOLUME 24
ertheless, further analysis on HIRS nonlinearity is required. Third, the temporary lunar detection module must be monitored closely. It will be eventually replaced by the new algorithm and validated prior to being made operational.
5. Summary In the more than 25-yr history of operational HIRS calibration, several algorithms have been used to generate HIRS radiance data and, unfortunately, these algorithms can produce HIRS level 1b radiance data with significant differences. Based on analysis of the gain stability of the HIRS instrument in relation to the instrument self-emission during filter temperature fluctuations, a flaw in the version 3 algorithm has been identified. This flaw can cause calibration errors on the order of 1 K or more, and periodically degraded the quality of the HIRS products from NOAA-15, -16, and -17 between 1998 and 2005. A new HIRS calibration algorithm (version 4) is introduced to improve the calibration accuracy, along with better quality indicators for instrument noise in the level 1b data. The new algorithm introduced fundamental changes in computing the slopes, with alternative methods for computing the intercepts. The new algorithm was validated in parallel tests before it became operational at NOAA/NESDIS. Test results show that significant improvements in calibration accuracy have been achieved, especially for NOAA-15/HIRS. The algorithm has been used in the operational production of HIRS radiance data since 28 April 2005. Acknowledgments. The authors wish to thank Drs. Jerry Sullivan, Robert Iacovazzi, Thomas Kleespies, Fuzhong Weng, Mitch Goldberg, and Al Powell for a critical reading of the manuscript and their valuable suggestions and comments. This study is partially funded by NOAA/NESDIS/OSD. The manuscript contents are solely the opinions of the author(s) and do not constitute a statement of policy, decision, or position on behalf of NOAA or the U.S. government.
REFERENCES Cao, C., M. Weinreb, and S. Kaplan, 2004: Verification of the HIRS spectral response functions for more accurate atmospheric sounding. Proc. Conf. on Characterization and Radiometric Calibration for Remote Sensing, Logan, UT, Utah State University, CD-ROM. ——, H. Xu, J. Sullivan, L. McMillin, P. Ciren, and Y. Hou, 2005: Inter-satellite calibration of the High Resolution Infrared
FEBRUARY 2007
CAO ET AL.
Radiation Sounders on NOAA-15, -16, and -17 from Simultaneous Nadir Overpass Observations. J. Atmos. Oceanic Technol., 22, 381–395. Goodrum, G., K. B. Kidwell, and W. Winston, 2000: NOAA KLM User’s Guide. Department of Commerce, Washington, DC. [Available online at http://www2.ncdc.noaa.gov/docs/klm/.] ITT, 1996: High Resolution Infrared Radiation Sounder HIRS/4 Technical Description. ITT Aerospace/Communications Division, Contract NAS5-30384, Doc. 8168948, 183 pp. ——, 1998: High Resolution Infrared Radiation Sounder HIRS/3
181
Instrument Manual and Alignment/Calibration Handbook & Optical Data. ITT Aerospace/Communications Division, Contract NAS5-30384, Doc. 8162669, 706 pp. Kidwell, K., cited 1998: Appendix M: New HIRS calibration procedure (NOAA-12 only). NOAA Polar Orbiter data user’s guide, NOAA/NESDIS. [Available online at http://www2. ncdc.noaa.gov/docs/podug/html/m/app_m.htm.] Kigawa, S., and T. Mo, 2002: An algorithm for correction of lunar contamination in AMSU-A data. NOAA Tech. Rep., NESDIS 111, U.S. Department of Commerce, 30 pp.
