Landsat TM Satellite Image Restoration Using

02-060.qxd 12/11/03 4:01 PM Page 91

Landsat TM Satellite Image Restoration Using Kalman Filters D. Arbel, E. Cohen, M. Citroen, D.G. Blumberg, and N.S. Kopeika

Abstract The quality of satellite images propagating through the atmosphere is affected by phenomena such as scattering and absorption of light, and turbulence, which degrade the image by blurring it and reducing its contrast. The atmospheric Wiener filter, which corrects for turbulence blur, aerosol blur, and path radiance simultaneously, is implemented in the digital restoration of Landsat Thematic Mapper (TM) imagery. Digital restoration results for Landsat TM imagery using the atmospheric Wiener filter were presented in the past. Here, a new approach for digital restoration of Landsat TM imagery is presented by implementing a Kalman filter as an atmospheric filter, which corrects for turbulence blur, aerosol blur, and path radiance simultaneously. Turbulence MTF is calculated from meteorological data. Aerosol MTF is consistent with optical depth. The product of the two yields atmospheric MTF, which is implemented in both the atmospheric Wiener and Kalman filters. Restoration improves both resolvable detail and contrast. Restorations are quite apparent even under clear weather conditions. Although aerosol MTF is dominant, slightly better results are obtained when the shape of atmospheric MTF includes turbulence, in addition to that of aerosol MTF, as shown by the use of criteria for restoration success. In general, the Kalman restoration is superior.

Introduction Satellite images obtained by imaging vertically through the atmosphere are blurred, distorted, and exhibit poorer contrast relative to ideal images (i.e., images without degradation). During recent years, there has been an effort to develop methods of restoration using the atmospheric optical transfer function in order to compensate for image degradation. Images propagating through the atmosphere are attenuated by absorption and large angle scattering by aerosols. They are also blurred by small angle scattering caused by aerosols, and by optical turbulence (Kopeika, 1998a). In the remote sensing community essentially all atmospheric blur in satellite imagery is attributed to small-angle light scatter by aerosols and is called the adjacency effect because it causes photons to be imaged in pixels adjacent to those in which they ought to have been imaged. A detailed summary of much of the adjacency effect literature, including numerical calculations, experimental results, Monte Carlo simulations, resolution measurements, image correction methods based on aerosol scatter blur, etc., is found in Kopeika et al. (1998a). In part of the propagation community, however, image blur is often attributed solely to turbulence. This assumption often leads to questionable results which do not correlate well with

Department of Electrical and Computer Engineering and Department of Geography and Environmental Development, Ben-Gurion University of the Negev, P.O.B. 653, Beer Sheva, 84105, Israel ([email protected]; blumberg@bgumail. bgu.ac.il; [email protected]). P H OTO G R A M M E T R I C E N G I N E E R I N G & R E M OT E S E N S I N G

turbulence theory, as described in Kopeika et al. (1998b). It is our view that image blur through the atmosphere involves both turbulence and small-angle forward scatter of light by aerosols (Kopeika, 1998a; Kopeika et al., 1998b). Interestingly enough, much of the coherence studies of turbulence and aerosol light scatter were carried out by the same researchers, whose pioneering works are also well known in each community. The aerosol modulation transfer function (MTF) contributions of Fante, de Wolf, Ishimaru, Lutomirski, etc., who are well known for their works on turbulence, are summarized in Kopeika et al. (1998b) and Kopeika (1998b). Those references also contain numerous examples of the problems arising in works by more recent members of the turbulence community when they ignore blur contributions caused by small-angle forward scatter of light by aerosols. Here, we consider the opposite phenomenon, i.e., the inadvisability of ignoring turbulence blur in the remote sensing community, which generally assumes all atmospheric blur is caused by aerosols (adjacency effect). A question to be considered is whether image restoration should be based on aerosol MTF alone or, instead, on the product of aerosol and turbulence MTF. Results are compared here. The influence of aerosol and optical turbulence strength on laser beam widening in the atmosphere is described in Zilberman et al. (2001) and Kopeika et al. (2001). This is equivalent to point spread functions at various elevations. It was mentioned there that the beam widening caused by atmospheric aerosols is significant at higher levels of the atmosphere (up to 20 km). On the other hand, those measurements indicate that turbulence is dominant at the lower level of the atmosphere, and then decreases sharply as the altitude increases. Little information about higher levels of the atmosphere is available yet. During recent years, there has been an effort to develop methods of restoration and filtering of images while using atmospheric optical transfer functions (MTF and phase transfer function) in order to compensate for image blur and distortions. Use of the standard Wiener filter for correction of atmospheric blur is often not effective because, although aerosol MTF is rather deterministic, turbulence MTF is random. The atmospheric Wiener filter (Sadot et al., 1995; Kopeika, 1998a) is one method for overcoming turbulence jitter (fluctuations). The atmospheric Wiener filter, which corrects for turbulence blur, aerosol blur, and path radiance simultaneously, is implemented here in digital restoration of Landsat Thematic Mapper (TM) imagery. Digital restoration results for Landsat TM imagery using the atmospheric Wiener filter were presented in the past (Sadot et al., 1995; Arbel et al., 1998; Arbel et al., 1999; Arbel and Kopeika, 2000). Here, a new approach for Photogrammetric Engineering & Remote Sensing Vol. 70, No. 1, January 2004, pp. 91–100. 0099-1112/04/7001–0091/$3.00/0 © 2004 American Society for Photogrammetry and Remote Sensing January 2004

91

02-060.qxd 12/11/03 4:01 PM Page 92

digital restoration of Landsat TM imagery is presented by implementing a Kalman filter as an atmospheric filter, which corrects for turbulence blur, aerosol blur, and path radiance simultaneously. Turbulence MTF is calculated from meteorological data. Aerosol MTF is consistent with optical depth. The product of the two yields atmospheric MTF. Here we present restoration results for both the atmospheric Wiener filter and the Kalman filter using the atmospheric MTF, and compare the two.

Principle of the Method Atmospheric Wiener Filter In this approach, the atmospheric MTF variance, which derives essentially from turbulence, is treated as an additional noise source because noise by definition is random. Aerosol MTF, on the other hand, because it is fairly constant as long as atmospheric conditions do not vary too much, contributes primarily to an average atmospheric MTF. Turbulence MTF changes with time due to its tilt jitter characteristic. These tilts are random and their temporal power spectra are usually limited to several tens up to a few hundred Hertz under ordinary atmospheric conditions. The image distortions caused by overall atmospheric MTF can thus be regarded as the sum of a deterministic and random filter (Guan and Ward, 1989a; Guan and Ward, 1989b). The deterministic filter includes aerosol and average turbulence MTFs, whereas the random filter includes the noise components of the imaging system, including both turbulence MTF variance and the hardware noise (Sadot et al., 1995; Kopeika, 1998a). The atmospheric Wiener filter removes turbulence blur, aerosol light scatter blur, and path radiance simultaneously. The first two are important in order to resolve small detail and pinpoint various locations (Sadot et al., 1995; Kopeika 1998a). Path radiance removal is important in order to see objects in their natural contrast. This is important when comparing reflectances or emittances at different wavelengths in multispectral imaging, or when comparing reflectances or emittances of the same scene at different times. Because atmospheric path radiance changes with time and wavelength, it distorts such comparisons and its removal makes such comparisons possible. Although other techniques have been suggested for atmospheric path radiance correction (Switzer et al., 1983; Kowalik et al., 1983; Schowengerdt, 1983; Jensen, 1996; Vermote and Tanre, 1997), none correct for blur. The atmospheric Wiener filter causes the system MTF to be significantly broadened after image restoration. Such broadening to higher spatial frequencies permits resolution of smaller detail. In addition, system MTF at higher spatial frequencies is increased vertically to yield higher contrast. Path radiance removal thus permits better comparison of small object-plane elements at different wavelengths and times. Artificial contrast stretching techniques can also improve contrast, but they distort the inherent natural contrast which is revealed by the atmospheric Wiener filter. Further processing can be applied if desired. Kalman Filter In 1960, R. E. Kalman published his famous paper describing a recursive solution to the discrete-data linear filtering problem (Kalman, 1960). Since that time, due in large part to advances in digital computing, the Kalman filter has been the subject of extensive research and application. Originally, Kalman filtering was designed as an optimal Bayesian technique to estimate state variables at a time t t from indirect and noisy measurements at time t, assuming as known the statistical correlations between variables and time. Kalman was the first to propose the application of state variables to the Wiener filtering problem. Kalman filters can also 92

January 2004

be used to estimate variables in a static (i.e., time-independent) system, if the mathematical model is suitably segmented. Kalman filtering is a useful technique for estimating or updating the previous estimate of a system’s state by using indirect measurements of the state variables, and using the covariance information about how measurements of a particular aspect of a system are correlated to the actual state of the system. The Kalman filter estimates a process by using a form of feedback control: the filter estimates the process state at some time and obtains feedback in the form of (noisy) measurements. As such, the equations for the Kalman filter fall into two groups: time update equations and measurement update equations. The time update equations are responsible for projecting forward (in time) the current state and error covariance estimates to obtain a priori estimates for the next time step. The measurement update equations obtain an improved a posteriori estimate. The Kalman filter is a set of mathematical equations that provides an efficient computational (recursive) solution of the least-squares method. The filter is very powerful in several aspects: it supports estimations of the past, present, and even future states, and it can do so even when the precise nature of the modeled system is unknown. The digital implementation of Kalman filters ought to be much more accurate than that of Wiener filters, or at least similar. Although Kalman filters need much more time for calculations, with present computers time cost is negligible if better results are obtained. On the other hand, if the calculations last too long then by decreasing the number of calculations using advanced operations a reduced filter may be achieved. Thus, the time parameter for restoration would not play a role anymore and, consequently, the restoration results of a Kalman filter may be much more accurate and last almost the same time as the case of a Wiener filter. The Reduced Kalman Filter In this paper we use a method which expands Kalman filters from one dimension (1D) into two dimensions (2D) (Woods and Radewan, 1977; Woods and Ingle, 1981). The reduced Kalman filter (Woods and Radewan, 1977) is implemented here for Landsat TM satellite image restoration. Our main problem was numerous calculations for a standard image size (i.e., 256- by 256-pixel size). There is a need for approximations in order to reduce some of the calculations. This is based on image correlation characteristics. The reduced filter estimates the next value (pixel) in the image based on the estimated nearest neighbors. 1D Kalman Filter For a space invariant and linear image acquisition system, without a noise dependent signal, the recorded image is presented in the following model (Woods and Radewan, 1977; Woods and Ingle, 1981; Rosenfeld and Kak, 1982; Biemond et al., 1983; Angwin and Kaufman, 1989; Zhang and Steenaart, 1989): r(m, n) hij s(m i, n j) v(m, n), (1) (i,j)∈R

where h(x, y) is 2D pulse (point) response, s(m, n) is the original image, v(m, n) is random noise, and R is the finite range of h. The image restoration problem is the estimation of s from r, where the information is only partial. In general, the quality of the estimation depends on model correctness. In most cases, the system response, h, is unknown. Therefore, there is a need to estimate it. The dynamic system for which a Kalman filter estimates its state vector can be easily defined (Woods and Radewan, 1977). According to this model, the filter equations are divided into two parts; extrapolation (time update) and measurement update. These are the basic equations for 2D Kalman filter development (Woods and Radewan, 1977). P H OTO G R A M M E T R I C E N G I N E E R I N G & R E M OT E S E N S I N G

02-060.qxd 12/11/03 4:01 PM Page 93

2D Kalman Filter Assuming that the image size is N by N pixels created by raster scanning, in this recursive method it is possible to relate each pixel in the image to past pixels or future pixels. The specific pixel is in the present, a present pixel. To set the dynamic system equations, we assume previous information (Rosenfeld and Kak, 1982) that the real image structure can be described by a 2D independent auto regressive (AR) model with white process noise u(m, n) and zero mean. The model equation is s(m, n)

ckl s(m k, n l) u(m, n)

(2)

(k,l)∈R⊕

where ckl is the prediction coefficients, s(m, n) is the present pixel estimated in the original image from past pixels in the range R⊕, and the range R⊕ is the pixels belonging to the non-symmetric half plain (NSHP), which were defined as past. When the range is limited to an M by M pixel size, then Figure 1. Points’ spatial distribution in the state vector. R⊕ {(m k, n l) : (1 k M, 0 l M) ∪ (M k 0, 1 l M)}

(3)

where M is empirically defined according to the image correlation. A state vector can be defined on the basis of raster scanning, in which the 2D problem is transferred into a 1D representation: i.e., s(n, m) [s(m, n), s(m 1, n), …, s(1, n); s(N, n 1), …, s(1, n 1); …; s(N, n M), …, s(m M, n M)]T.

(4)

The vector length is M(N 1). According to the vectorial representation, the state equations can be developed as in 1D vectorial representation (Woods and Radewan, 1977; Woods and Ingle, 1981). The calculation load (stress) can be calculated by the following. For an N by N image size, with a prediction model of size M by M, where N is much greater than M, the vector length is M(N 1); the matrix dimension in the Kalman filter would be approximately NM. Therefore, the calculations in the filter are nearly O(M 3, N 3). Kalman Filtering in the Measurement Update Stage The prediction stage in the filter can be implemented with O(M 2) operations over previous estimations. On the other hand, the update stage includes calculations for each one of the O(MN) terms of the state vector. For the case of N much greater than M, the calculation number can be reduced by updating only those terms in the state vector which are located adjacent to the measurement. The assumption is that the influence of the measurement is insignificant for those terms which are located far from the measured pixel. For convenience, we refer to the update area as the prediction range, R⊕; in other words, we update O(M 2) state variables instead of O(MN). According to the reduced range, a reduced state vector is defined. This vector fits the raster scanning which is appropriate for estimation order finding; i.e., s1(n, m) [s(m, n), …, s(m M, n); s(m M, n 1), …,

s(m M, n M)] .

(5)

Those points in the state vector s, which are not included in s1 are included in s2. Spatial distribution of the points is described in Figure 1. P H OTO G R A M M E T R I C E N G I N E E R I N G & R E M OT E S E N S I N G

s(m, n) sT1 (n, m), sT2 (n, m).

(6)

The same dynamic state equation with no reduction can also be defined (Woods and Radewan, 1977). There is a way to represent the Kalman filter equations based on the basic dynamic system state equations. There will be two parts of equations for the prediction stage and for the measurement update stage (Woods and Radewan, 1977). Scalar Representation for the Filters’ Representation The scalar form of the Prediction stage is R(m,n) (m, n; k, l) cop R(m1,n) (m o, n p; k, l), b a op

(k, l) ∈S(m,n) ⊕ ,

(7)

(m, n; m, n) cklRb(m1,n)(m, n; m k, n l) 2w, (8) R(m,n) b kl

where S(m,n) ⊕ is the range of the state vector s(m, n). The appropriate scalar equations for the Measurement update stage are (m, n; i, j) R(m,n) b (i, j) , K (m,n) 1 (m,n) Rb (m, n; m, n) 2v

(i, j) ∈R(m,n) ⊕ ,

(9)

sˆ(m,n) (i, j) sˆ(m,n) (i, j) a b (m, n), K (m,n)(m i, n j)r(m, n) sˆ(m,n) b

(10)

for (i, j) ∈R(m,n) ⊕ . R(m,n) (i, j; k, l) R(m,n) (i, j; k, l) a b (i, j; k, l), K(m,n)(m i, n j)R(m,n) b

s(m M, n 1); …; s(m M, n M), …, T

The full state vector has the following format:

(11)

(m,n) for (i, j) ∈R(m,n) ⊕ and (k, l) ∈S⊕ . In the scalar representation, the number of calculations in the Kalman filter is about O(M 3N) calculations. By reducing the range of (k, l) instead of the range S⊕ to a smaller one with a size order of R⊕, the calculations for each stage are reduced to O(M 4).

January 2004

93

02-060.qxd 12/11/03 4:01 PM Page 94

Image Estimation for Degradations Caused by the Systems’ Transmission in Addition to Noise If the image is not ideal, then h 1 in Equation 1 and acts as a FIR filter with a size of order M by M or less. The response for a point, h, is hT1 [h0,0, …, hM,0, …, hM,1, …; hM,M, …, hM,M].

(12)

Now we can write the equations in the scalar form considering h. The scalar prediction equations remain the same. On the other hand, the equations of the update stage change and have a new representation: i.e., hklR(m,n) (m k, n l; m i, n j) b kl

K (m,n)(i, j) , (m k, n l; m i, n j) 2v hklhijR(m,n) b kl ij

(i, j) ∈R(m,n) ⊕ ,

(13)

(i, j) sˆ(m,n) (i, j) K (m,n)(m i, n j) sˆ(m,n) a b

r(m, n) hkl sˆ(m,n) (m k, n l) , b R⊕

(14)

for (i, j) ∈R(m,n) ⊕ . R(m,n) (i, j; k, l) R(m,n) (i, j; k, l) K (m,n)(m i, n j) a b hmo,np R(m,n) (m o, n p; k, l), b

(15)

op

(m,n) for (i, j) ∈R(m,n) ⊕ and (k, l) ∈S⊕ . The prediction coefficient calculation is described well in Clarkson (1993). Therefore, no additional calculations are presented here.

Experiment Image and Data Acquisition The utility of satellite imagery depends largely on how the image is processed. It is possible to distinguish certain features from others based on their spectral characteristics or how strongly a feature shows up at one wavelength compared to another. The applications are endless. In such applications it is desirable to both restore the image (deblur it) and also to get rid of the atmospheric path radiance because it decreases contrast differently at different wavelengths and under different weather conditions. The atmospheric Wiener filter deblurs and gets rid of path radiance simultaneously (Kopeika, 1998a). The required input includes average atmospheric MTF whose correct shape is critical (Kopeika, 1998a). This shape changes noticeably if either turbulence or aerosol MTF is neglected. In 1999 Landsat 7 was launched. Since then, Landsat 7 has been orbiting the globe and covers the same area (including Israel) every 16 days. In order to use meteorological data for Landsat image restoration, ground-based measurements were obtained from meteorological ground stations located at several locations in the image scene, which was recorded in the image. Those measurements provide atmospheric data such as the optical depth (thickness) and other meteorological data used for turbulence characterization (Kopeika, 1998a). The aerosol optical depth is measured over a spectral range, which corresponds to the Landsat TM spectral bands. Here, we present a scene recorded on 07 August 1999 and use the meteorological data recorded at the same time. The aerosol optical depth for the scene is based on vertical measurements, which were conducted at the ground station. Usu-

94

January 2004

ally, the ground station recorded Landsat images when the weather or meteorological conditions were good. This results in lack of degradation effects on the recorded image and, therefore, there was little to restore (deblur). Nevertheless, although these images were recorded under good meteorological conditions, there was still enough to restore as presented here in examples below. Kalman Filter for Satellite Imagery The Kalman filter is adaptive with variable amplification K(m, n). This filter has a convergence point, which is defined when the amplification reaches its steady state. Despite the feedback of the filter, if the amplification does not reach the steady state, then the filter does not converge. This means that there is a false selection of the filters’ parameters such as transfer function, noise variance, object model coefficients, or simply incorrect initial values. The amplification K(m, n) is independent from information (the recorded image values). Therefore, it is recommended that several initial executions be run to test the duration of convergence, and in the next executions to refer to the amplification as a constant. Such a process may significantly decrease the restoration time. In spite of little knowledge and information about image parameters such as noise and correlation coefficients, restoration with Kalman filters yield good results as presented in the examples below. Criteria for Restoration Success The criteria for estimation and restoration success of satellite images can be divided into several categories. There are relative criteria, which are a comparison between different restoration results obtained from the original image, in contrast to self-criteria, where the statistical characteristics of each image are individually inspected. These include criteria dealing with the image in the spatial domain as well as criteria for images in the spatial frequency domain. To find a way to determine which of the restoration results yielded the best restoration result, a psychophysical experiment was conducted. Fifty observers with experience in image processing were exposed to a random set of satellite image pairs, including combinations of different images, and also several combinations of the same images. The database of all images which were used in the experiment included the original images, the restored images using the Kalman filter, the restored images using the atmospheric Wiener filter, and also different restorations of the same image based on different aerosol and turbulence MTFs (the meteorological parameters were changed). All images were presented at the same contrast level. The observers selected visually the best image from each pair of images and skipped over those images which looked alike or similar. Then, several examination criteria were examined. Quantitative criteria such as edge response were not always accurate because of dependency on the image details. The given satellite image had high resolution but it was recorded from about a 750-km altitude. The image was recorded over a wide area with many details, which sometimes does not leave enough pixels for both black-and-white strips to form an edge. Therefore, this information was not enough to obtain a reliably correct edge response. Several regions in the image may include fields or tilled lands. In that case, edge response may indicate the quality of the restoration results and, also, may serve as a criterion for deblurring success. In our experiment we used edge response as a tool for measuring the restoration results and for comparison between restoration results of Kalman filters and Wiener atmospheric filters. Spectral criteria based on the spectrum of the image or its gray levels may serve as enhancement indicators. In most cases the histogram of the restored image is richer than that of

P H OTO G R A M M E T R I C E N G I N E E R I N G & R E M OT E S E N S I N G

02-060.qxd 12/11/03 4:01 PM Page 95

the original image, meaning that there are many more graylevel bins in the restored image histogram than in the original image. Therefore, the restored image is much more detailed. When an ideal image is not obtainable, mean-squareerror criteria are not satisfied. In most cases there is only the recorded image, which is already blurred. In some cases, there were a series of images of the same scene with different meteorological conditions. An image with good meteorological conditions (with optical depth of about 0.4 or less) may be referred to as an ideal image and the restored images with optical depth of 0.29 and more may be successfully compared to the ideal image.

Results Limitations The optical depth value depends on the ground station location. Therefore, the optical depth for restoration can be a mean value for the whole scene or it may vary in each segment of the image according to differences in weather conditions. Consequently, restoration results may be adversely affected by such inaccuracy in optical depth values which determine high frequency aerosol MTF asymptotes. When the optical depth a has high values (a greater than 2.0), too many photons may have been lost for the image to be restored successfully. Because of the attenuation, the surface image may be noise-limited rather than blur-limited (Kopeika, 1998a). On the other hand, in spite of low optical depth values (less than 0.6) in the case of high spectral band values

(from IR up to thermal), not enough details and especially few color levels were recorded in the original images, which left little to restore. Restored Images Figure 2 presents an original Landsat 7 Enhanced Thematic Mapper Plus (ETM) image of the city of Beer-Sheva, Israel, and its surroundings. It was recorded from a 750-km satellite altitude with an optical depth of 0.29. The Image was recorded using an ETM sensor with a 15-m ground resolution, and its wavelength range is 520 to 900 nm (Panchromatic band). Image size is 1265 by 1001 pixels. The original image is of 256- by 256-pixel size, which is a frame (large segment) defined as a ROI (region of interest) in an image of 1265- by 1001-pixel size (Figure 2). Each segment was processed individually and, therefore, the following restoration results present some segments from the original image. In addition, calculations of MTFs and restorations require time according to image size. Therefore, restoration of all segments as a series of images may be faster than the restoration of the original scene. Nevertheless, it is more accurate to restore the split image, segment by segment, than the entire full size image. The only known optical depth value is at the ground station where it was measured. The given scene includes only one ground station, which is located in the city of Beer-Sheva. There is also another ground station, which is located in Sede-Boqer, 30 km southeast of Beer-Sheva, and supplies meteorological data. Those measurements are useful as a reference for the restorations in other segments where

Figure 2. Original image of Beer-Sheva, Israel, from the Landsat 7 satellite.


January 2004

95

02-060.qxd 12/11/03 4:01 PM Page 96

Figure 3. Atmospheric MTF (calculated aerosol MTF with turbulence MTF) with a 0.29 and cutoff spatial frequency 6.88 cycles/mrad.

different meteorological conditions may exist. It is justified to cut a frame, including the ground station surroundings, as closely as possible and restore it with the atmospheric MTF, which is based on the meteorological measurements. Then, by using relative values of the meteorological data, other segments with different meteorological conditions can be restored. Figure 3 shows an example of atmospheric MTF (presented as a continuous black line) calculated from all the gathered data, for both aerosol (presented as a bright gray line) and turbulence (presented as the upper gray line) MTFs. Note that the high spatial frequency asymptotic value of aerosol MTF approximately equals atmospheric transmission according to optical depth. Turbulence MTF is estimated from meteorological surface data (Kopeika, 1998a). Figures 4b, 5b, and 6c show the restored images using the atmospheric Wiener filter, and Figures 4c, 5c, and 6d show the restored images using the Kalman filter. Both filters were based on the aerosol MTF only. Figures 4d, 5d, and 6e show the restored images using the atmospheric Wiener filter, and Figures 4e, 5e, and 6f show the restored images using the Kalman filter. Here, both filters were based on the atmospheric MTF, which is obtained by using aerosol MTF with turbulence MTF. Each MTF has other characteristics defined by the following: cutoff spatial frequency where the asymptote begins, asymptote level (usually calculated according to the optical depth), and turbulence effects, which give rise to a high spatial frequency slope instead of the horizontal asymptote. The overall atmospheric MTF is dominated by the aerosol MTF, as expected for satellite imagery. Figure 6b presents an original air photograph of BeerSheva, Israel. It was recorded from an aircraft flying height of about 10 km above ground level (AGL) at a 0.5-m per pixel resolution. Figure 6b was zoomed out in order to fit the size of the scene in the air photograph to that of the Landsat 7 ETM image, which is presented in Figures 6a and 6c through 6f. The air photograph is more detailed than the Landsat ETM image and has high resolution compared to that of the Landsat ETM image. Therefore, the air photograph can assist in visual comparison between the restoration results. One could verify the appearance of new details by comparing the same ROIs in both the restored image and the air photograph. For example, the best restoration result for Figure 6a is Figure 6f, when 96

January 2004

Figure 4. (a) Original image recorded with a 0.29. Wavelength range is 520 to 900 nm. (b) Restored image after atmospheric Wiener filter using aerosol MTF only. (c) Restored image after Kalman filter using aerosol MTF only. (d) Restored image after atmospheric Wiener filter using aerosol and turbulence MTFs. (e) Restored image after Kalman filter using aerosol and turbulence MTFs.

using the Kalman filter based on both aerosol and turbulence Streets and buildings sharpen noticeably, and the same details also appear in the air photograph, whereas in other restoration results it is barely seen. Figure 7a shows the original image with three ROIs (regions of interest). Figures 7b, 7e, and 7h show the zoomed ROIs of the original image. Figures 7c, 7f, and 7i show the zoomed ROIs of the restored image in Figure 4d using the atmospheric Wiener filter. Figures 7d, 7g, and 7j show the zoomed ROIs of the restored image in Figure 4e using the Kalman filter. In both types of restorations, the restored image is more detailed. For example, roads, buildings, rural fields, wadies etc. were deblurred and became more accurate than in the original image. In uniform areas, such as the wilderness areas, MTFs.


02-060.qxd 12/11/03 4:01 PM Page 97

Figure 5. (a) Original image recorded with a 0.29. Wavelength range is 520 to 900 nm. (b) Restored image after atmospheric Wiener filter using aerosol MTF only. (c) Restored image after Kalman filter using aerosol MTF only. (d) Restored image after atmospheric Wiener filter using aerosol and turbulence MTFs. (e) Restored image after Kalman filter using aerosol and turbulence MTFs.

degradation may appear. It is important to see that turbulence has some effect on the recorded image, combined with the aerosol effects. The high spatial frequency asymptote in Figure 3 agrees well with atmospheric transmission according to the optical depth value. Criteria for Restoration Results The first stage of analyzing the restoration results was a psychophysical experiment. The psychophysical experiment results were processed in real time. It was found that after 27 observers there was a convergence in the results. For example, Figure 6 was presented as part of the experiment; 62 percent of the observers found an improvement in Figure 6c when compared to Figure 6a, 93 percent found an improvement in Figure 6d when compared to Figure 6a, P H OTO G R A M M E T R I C E N G I N E E R I N G & R E M OT E S E N S I N G

Figure 6. (a) Original image recorded with a 0.29. Wavelength range is 520 to 900 nm. (b) Air photograph of Beer-Sheva, Israel. (c) Restored image after atmospheric Wiener filter using aerosol MTF only. (d) Restored image after Kalman filter using aerosol MTF only. (e) Restored image after atmospheric Wiener filter using aerosol and turbulence MTFs. (f) Restored image after Kalman filter using aerosol and turbulence MTFs.

68 percent found an improvement in Figure 6e when compared to Figure 6a, 96 percent found an improvement in Figure 6f when compared to Figure 6a, 90 percent found an improvement in Figure 6d when compared to Figure 6c, 93 percent found an improvement in Figure 6f when compared to Figure 6e, 72 percent found an improvement in Figure 6e when compared to Figure 6c, 83 percent found an improvement in Figure 6f when compared to Figure 6d, and 88 percent found an improvement in Figure 6d when compared to Figure 6e. When a pair of images included the same image, the observers found a similarity of about 44 percent in the images. Figures 4 and 5 have fewer small details such as buildings and roads. Therefore, the results of the perception experiment were less impressive but still indicated the same tendency. January 2004

97

02-060.qxd 12/11/03 4:01 PM Page 98

Figure 7. (a) ROIs (regions of interest) in Figure 4. The ROIs are (b) Original ROI 1 (upper left corner), (c) Restored ROI 1 after atmospheric Wiener filter using aerosol and turbulence MTFs, (d) Restored ROI 1 after Kalman filter using aerosol and turbulence MTFs, (e) Original ROI 2 (upper middle), (f) Restored ROI 2 after atmospheric Wiener filter using aerosol and turbulence MTFs, (g) Restored ROI 2 after Kalman filter using aerosol and turbulence MTFs, (h) Original ROI 3 (lower middle), (i) Restored ROI 3 after atmospheric Wiener filter using aerosol and turbulence MTFs, and (j) Restored ROI 3 after Kalman filter using aerosol and turbulence MTFs. For restoration results obtained using the atmospheric Wiener filter (in Figures 4, 5, and 6), an average of about 70 percent of the observers found an improvement when compared to the original image. For restoration results obtained using the Kalman filter, an average of about 94 percent of the observers found an improvement when compared to the original image. For restoration results obtained with the aerosol MTF only, an average of about 78 percent of the observers found an improvement when compared to the original image. For restoration results obtained with both the aerosol and the turbulence MTF, an average of about 89 percent of the observers found an improvement when compared to the original image. According to the perception (psychophysical) experiment, it is clearly understood here that both the Kalman filter and the atmospheric Wiener filter yielded an improvement in the restored images. Best restoration results were displayed by 98

January 2004

the Kalman filter, especially when the Kalman filter included both aerosol and turbulence MTFs. Most criteria were tested by comparing the original image and the restored one, and by comparing one restored image to another (for filter comparison). By testing the results, each restoration has its quantitative value, which defines the quality of the restoration process. All Quantitative values were normalized to one scale table, with which to classify the quality of the restoration. In order to compare between all the restoration criteria, there was a need for a scale transformation for each criterion. The relative quality values begin at 0 percent for the worst case and exceed 100 percent for the best. For example, the image quality measure (IQM) concept (Norman and Brian, 1992) was applied in the same way to all restored images obtained through different restoration techniques. The results for the National Imagery Interpretability P H OTO G R A M M E T R I C E N G I N E E R I N G & R E M OT E S E N S I N G

02-060.qxd 12/11/03 4:01 PM Page 99

Rating Scale (NIIRS) criteria (Leachtenauer et al., 1997) are considerable and very promising. Therefore, by this criteria there was improvement in all restored images. According to the NIIRS criteria, the atmospheric Wiener filter yields up to 7 percent improvement and, finally, the Kalman filter yields an improvement of 20 percent and more. The NIIRS criteria supports the arguments that restoration based on atmospheric (aerosol and turbulence) MTF is better than restoration based on aerosol MTF alone, and that the Kalman filter is much better than the atmospheric Wiener filter in restoring satellite images. Criteria such as NIIRS, edge response, and MTF indicate an average restoration success in the range of 2 percent (when using the atmospheric Wiener filter based on the aerosol MTF alone) and up to 8 percent (when using Kalman filter based on both aerosol and turbulence MTFs). The image has a low optical depth value (0.29). As a result, improvement may be relatively small. Because of the low optical depth, turbulence is not negligible. Atmospheric MTF also takes the turbulence effect into account and, as a result, there is a relative increase in improvement. Statistical criteria such as MSE are not applicable here.

Conclusions Satellite imagery is usually considered from the standpoint of aerosol blur alone, with turbulence neglected entirely. It is seen that the best restoration results from the human perception standpoint occur when turbulence is also considered. Inclusion of the turbulence MTF causes the average atmospheric MTF to decrease at high spatial frequencies instead of flattening out. Here, the presented images were recorded in clear weather. The vertical imaging path through the atmosphere is very long. Both the good weather conditions and the high satellite elevation resulted in diminished effects of turbulence on the original images. At high satellite elevations the recorded images may not be influenced by turbulence as much as by aerosols. On the other hand, if the atmospheric weather conditions at lower levels are worse than those at higher levels of the atmosphere, then turbulence effects may be included in the recorded image and, therefore, the restoration process should take into account degradations caused by both aerosols and turbulence. Atmospheric MTF modeling is used to restore even clear weather Landsat TM images so that atmospheric blur and path radiance are removed simultaneously, resulting in sharper, clearer images of smaller resolvable detail and improved contrast. Best restorations occur when aerosol MTF is modeled as developed previously using the practical instrumentationbased aerosol MTF (Sadot and Kopeika, 1994; Sadot et al., 1995). In general, the worse the atmospheric degradation, the more impressive the restoration using the proper atmospheric MTF shape, which is affected primarily by the aerosol MTF. It is seen here that aerosol blur is much more dominant than turbulence blur, and that the proper atmospheric MTF shape, which includes the unique aerosol MTF shape, is critical to obtaining good restoration. The good correlation between high spatial frequency aerosol MTF asymptote and atmospheric transmission according to aerosol optical depth are consistent with aerosol MTF modeling. Here we demonstrate how to obtain qualitative satellite imagery restoration results using both the atmospheric Wiener filter (Sadot et al., 1995) and a new form of the Kalman filter. Kalman filtering requires much more time for calculations than does Wiener filtering, but with present computers time cost is negligible if it leads to better results. The calculation load depends on the mathematical approach, which depends on previous information about the noise and the solution. The quality of the restoration depends on the correctness of the models of the involved signals P H OTO G R A M M E T R I C E N G I N E E R I N G & R E M OT E S E N S I N G

(Equation 1) and on the accuracy of the assumed previous information. The restoration quality describes the proximity of the estimation to the original signal. The problem is that the restoration quality of an image is inconclusive and depends on the definition and the goals of the restoration process. For Example, mean-square-error (MSE) criteria do not supply an optimal result from the point of view of the human eye, but can be suitable for automatic identification of objects in the image. Usually, the problem is the accuracy of the previous information. According to the MSE criteria, a Wiener filter is consistent in time and gives solutions for stationary signals. On the other hand, for a Kalman filter no stationary signals are needed. A Kalman filter is causal and not consistent in time. In the case of image restoration, the assumed previous information is the same for both the Wiener and Kalman filters because the power spectrum of the original image needed for a Wiener filter and the prediction coefficients of the auto regressive (AR) model of the original image are both direct known transformations of the autocorrelation. The accuracy of the information in both cases depends on the information accuracy about the autocorrelation of the unknown original image. Proximity of the autocorrelation is usually achieved by the measured image itself or by another image with no blur or noise. The previous information about the noise is similar when the noise is Gausian. In general, the information about the noise is known or can be measured from the measured image itself. The reduced 2D Kalman filter for image restoration is implemented here. A solution for the load in calculations is based on state reduction in the measurement update stage. Each pixel in the process is estimated with its nearest neighbors, which were estimated earlier. Each new pixel (after the measurement) updates the previous estimated pixels in its neighborhood. The size of the nearest neighborhood of the previous pixels is defined according to the range size of the linear prediction model, R⊕+, based on the correlation in the original image. According to empirical tests, this size is a few pixels in each direction. The scalar representation of the filter allows additional states’ reduction because no multiplication is needed here, unlike that in the matrix representation. It is important to know that the systems’ transmission h is not always given (known). Therefore, there is a need to identify it from the measurements. Because causality is not relevant in an image, smoothing may be efficient instead of filtering. In other words, instead of using measurements from the past over a distance M in each direction in the non-symmetric half plain (according to the raster scanning), we can use the measurements over a distance M in all directions. The size M, which is defined according to real image correlation characteristics, is constant. Therefore, fixed lag smoothing can be tested for image restoration. According to some restoration results, part of which are presented here in Figures 4, 5, 6, and 7, Kalman filters yield better restoration results (deblurring and denoising the recorded images) than do atmospheric Wiener filters. This conclusion is strongly based on the perception experiment which served here as the criteria for restoration success. A Kalman filter is much more complex than the atmospheric Wiener filter. In the ideal case, if and when the estimation process is accurate, the Wiener filter should present the best restoration results. However, there is no chance to achieve a perfect estimation. In the estimation process there may be some missing parameters and/or at least a few approximations. Therefore, a Kalman filter should present much more accurate restoration results. If we take into account the calculation time and complexity, one may choose the atmospheric Wiener filter for restoration. If some atmospheric parameters (such as the optical depth, temperature, wind speed, humidity, and C2n if known) needed for the degradation definition are January 2004

99

02-060.qxd 12/11/03 4:01 PM Page 100

neither available nor accurate, then a Kalman filter may yield solutions which would compensate for this lack of information, unlike the atmospheric Wiener filter. Inclusion of turbulence blurs also leads to slightly improved restoration. It is clearly understood here that image segmentation is required when a wide area is being recorded in a satellite image. The necessity of image segmentation depends on the range from the ground station where the measurements take place, and on variation in weather conditions at different locations which affect the final calculated values of the atmosphere effects, such as optical depth, turbulence parameters, etc.

Acknowledgments The authors are grateful to A. Karnieli from the Remote Sensing Laboratory at the J. Blaustein Institute for Desert Research in Sede-Boqer; and the Department of Geography and Environmental Development, from Ben-Gurion University of the Negev, Israel; for providing the ground station meteorological data based on their measurements. This work is supported partially by the Paul Ivanier Center for Robotics and Production Management at Ben-Gurion University of the Negev.

References Angwin, D.L., and H. Kaufman, 1989. Image restoration using reduced order models, Signal Processing, 16:21–28. Arbel, D., O. Moldovan, R. Jacobson, and N.S. Kopeika, 1998. Imaging vertically through the atmosphere: restoration of satellite images based on atmospheric MTF evaluation, Proceedings, Propagation and Imaging through the Atmosphere II, 19–24 July, San Diego, California (SPIE, Bellingham, Washington), 3433:250–261. Arbel, D., A. Sagiv, M. Kuznivzki, and N.S. Kopeika, 1999. Satellite image restoration filter comparison, Proceedings, Propagation and Imaging through the Atmosphere III, 18–23 July, Denver, Colorado (SPIE, Bellingham, Washington), 3763:187–198. Arbel, D., and N.S. Kopeika, 2000. Criteria for satellite image restoration success, Proceedings, Advanced Signal Processing Algorithms, Architectures, and Implementations X, 30 July– 04 August, San Diego, California (SPIE, Bellingham, Washington), 4116:417–428. Biemond, J., R. Jelle, and J.J. Gerbrands, 1983. A fast Kalman filter for images degraded by both blur and noise, IEEE Transactions on Acustics, Speech, Signal Processing, 31:1248–1256. Clarkson, P.M., 1993. Optimal and Adaptive Signal Processing, CRC Press, Inc., Boca Raton, Florida, 560 p. Guan, L., and R.K. Ward, 1989a. Restoration of randomly blurred images by the Wiener filter, IEEE Transactions on Acustics, Speech, Signal Processing, 12:589–592. , 1989b. Deblurring random time-varying blur, Journal of Optical Society of America A, 6:1727–1737. Jensen, J.R., 1996. Introductory Digital Image Processing, A Remote Sensing Perspective, Second Edition, Prentice-Hall, Englewood Cliffs, New Jersey, 318 p. Kalman, R.E., 1960. A new approach to linear filtering and prediction problems, Transaction of the ASME, Journal of Basic Engineering, 82(Series D):35–45. Kopeika, N. S., 1998a. A System Engineering Approach to Imaging, SPIE, Bellingham, Washington, 679 p. , 1998b. Blur in imaging through the atmosphere: A system engineering approach to imaging, Proceedings, Propagation and Imaging through the Atmosphere II, 19–24 July, San Diego, California (SPIE, Bellingham, Washington), 3214:320–331. Kopeika, N.S., I. Dror, and D. Sadot, 1998a. The causes of atmospheric blur: comment on Atmospheric scattering effect on spatial resolu-

100

January 2004

tion of imaging systems, Journal of Optical Society of America A, 15:3097–3106. Kopeika, N.S., D. Sadot, and I. Dror, 1998b. Aerosol light scatter vs. turbulence effects in image blur, Proceedings, Optics in Atmospheric Propagation and Adaptive Systems II, 19–24 July, San Diego, California (SPIE, Bellingham, Washington), 3433:44–51. Kopeika, N.S., A. Zilberman, and Y. Sorani, 2001. Measured profiles of aerosols and turbulence for elevations of 2–20 km and consequences on widening of laser beams, Proceedings, Optical Pulse and Beam Propagation, 20–26 January, San Jose, California (SPIE, Bellingham, Washington), 4271:111–121. Kowalik, W.S., J.R.P. Lyon, and P. Switzer, 1983. The effects of additive radiance terms on ratios of Landsat data, Photogrammetric Engineering & Remote Sensing, 49:659–669. Leachtenauer, J.C., K. Daniel, and T.P. Vogl, 1997. Digitizing Corona imagery: Quality vs. cost, Corona: Between the Sun & the Earth, The First NRO Rreconnaissance Eye in Space (R.A. McDonald, editor), American Society for Photogrammetry and Remote Sensing, Bethesda, Maryland, pp. 189–203. Norman, B.N., and H.B. Brian, 1992. Objective image quality measure derived from digital image power spectra, Optical Engineering, 31(4):813–825. Rocadenbosch, F., A. Comeron, G. Vazquez, A. Rodriguez, C. Soriano, and J.M. Baldasano, 1998a. Inversion of atmospheric optical parameters from elastic-backscatter lidar returns using a Kalman filter, Proceedings, Atmospheric Propagation, Adaptive Systems, and Lidar Techniques for Remote Sensing II, 19–24 July, San Diego, California (SPIE, Bellingham, Washington), 3494:227–238. Rocadenbosch, F., G. Vazquez, and A. Comeron, 1998b. Adaptive filter solution for processing lidar returns: Optical parameter estimation, Applied Optics, 37:7019–7034. Rosenfeld, A., and A.C. Kak, 1982. Digital Picture Processing, Second Edition, Academic Press, New York, N.Y., 435 p. Sadot, D., and N.S. Kopeika, 1994. Effects of absorption on images quality through a particulate medium, Applied Optics, 33: 7107–7111. Sadot, D., R. Rosenfeld, G. Shuker, and N.S. Kopeika, 1995. High resolution restoration of images distorted by the atmosphere, based upon average atmospheric MTF, Optical Engineering, 34:1799–1807. Schowengerdt, A.R., 1983. Techniques for Image Processing and Classification in Remote Sensing, Academic Press, New York, N.Y., 249 p. Switzer, P., W.S. Kowalik, and J.R.P. Lyon, 1983. Estimation of atmospheric path radiance by the covariance matrix method, Photogrammetric Engineering & Remote Sensing, 47:1469–1476. Vermote, E.F., and D. Tanre, 1997. Second simulation of the satellite signal in the solar spectrum 6S: An overview, IEEE Transactions on Geoscience & Remote Sensing, 35(3):675–686. Woods, J.W., and C.H. Radewan, 1977. Kalman filtering in two dimensions, IEEE Transactions on Information Theory, 23(4):473–482. Woods, J.W., and V.K. Ingle, 1981. Kalman filtering in two dimensions: Further results, IEEE Transactions on Acustics, Speech, Signal Processing, 29(2):188. Zhang, J.Y., and W. Steenaart, 1989. High speed Kalman filtering for image restoration, Proceedings, Visual Communications and Image Processing IV, July, San Diego, California (SPIE, Bellingham, Washington), 1199:125–135. Zilberman, A., N.S. Kopeika, and Y. Sorani, 2001. Laser beam widening as a function of elevation in the atmosphere for horizontal propagation, Proceedings, Laser Weapons Technology II, 16–20 April, Orlando, Florida (SPIE, Bellingham, Washington), 4376:177–188. (Received 06 May 2002; accepted 21 November 2002; revised 07 February 2003)
