IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. ..... tion Processing Society of Japan, Operations Research Society of Japan, and.
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 34, NO. 5, SEPTEMBER 1996
1151
Enhancement of Low Spat'ial Resolution Image Based on High Resolution-Bands Ryuei Nishii, Member, ZEEE, Saeko Kusanobu, and Shojiq Tanaka, Member, ZEEE
Abstract-Thermal infrared measurements of Band 6 acquired by Landsat TM sensor have lower spatial resolution than those of the other six bands. In this paper, we propose a statistical approach to enhance the resolution of low spatial resolution image by using remaining bands. We employ a multivariate normal distribution for the seven band values. The values of Band 6 are predicted by the conditional expectations. Validity of our procedure is examined by mean squared errnrs based on actual images.
30m
5
6
7
8
9
10
11
12
13
14
15
16
I. INTRODUCTION
T
HE BAND 6 of the Landsat Thematic Mapper (TM) sensor is physically important because it is a measurement on heat radiation. Unfortunately, its resolution is 120 m x 120 m, while that of the other six bands is 30 m x 30 m. This difference of resolutions causes many difficulties in analyzing TM data. For example, discriminant analysis on TM data is performed by omitting values of Band 6 in many cases. Such a strategy may lose much information. If one can enhance the resolution of Band 6, 7-dimensional data of high resolution would be helpful in many applications. In [l], a deterministic approach to estimate proportions of categories in pixels is taken to this problem. However, the method discussed there is not flexible because it is assumed that the number of land-cover categories is nine (fixed). See [2] for reference. Meanwhile, the approaches developed by [3] and [4] are more flexible. First,-the images of high resolution are used for clustering spectrum characteristics, see, e.g., [5]. Second, the value of Band 6 at each pixel is renewed by the estimated mean of Band 6 of the corresponding cluster. This method may be powerful, but needs much computation. Another method is to correct Band 6 by adding a linear combination of the remaining band values. There is no theoretical justification how to determine the coefficients of the linear combination. In this paper, we will establish the method based on the statistical treatment by which the coefficients are reasonably estimated. In Section 11, we consider a local window consisting of 16(= 4 x 4) pixels of size 120 m x 120 m at which Band 6 is observed, In Section 111, we employ a multivariate normal distribution for the joint distribution of 7 band values. Manuscript received November 6, 1995; revised April 17, 1996. R. Nishii is with the Faculty of Integrated Arts and Sciences, Hiroshima University, Kagamiyama, Higashi-Hiroshima 739, Japan. S. Kusanobu is with the Graduate School of Engineering, Hiroshima University, Kagamiyama, Higashi-Hiroshima 739, Japan. S. Tanaka is with the Faculty of Engineering, Yamanashi University, Takeda, Kofu 400, Japan. Publisher Item Identifier S 0196-2892(96)06822-2.
120m Fig. 1. Pixel numbers in the local window at which Band 6 is observed.
Section IV proposes a predictor for Band 6 based on the conditional distribution. Band 6 is corrected by adding a linear combination of high resolution data. The proposed procedure is examined by mean squared errors (MSE's) based on Landsat 5 TM image of Hiroshima City in Section V. Further, Band 6 is enhanced and it is shown that our method reduces the conditional variance into half of the variance of raw data, which means our method provides an accurate predictor for Band 6. Final conclusion is given at Section VII. In the Appendix, we generalize our consideration into a general case. 11.
OBSERVATIONS IN THE
LOCALWINDOW
Consider a local window consisting of 16 pixels of size 120 m x 120 m at which Band 6 is observed. Each pixel is numbered from z = 1 to 16, see Fig. 1. Let xbt be a random variable of Band b at the ith pixel of P the local window. Put
xz = (xlz,x22,
' ' ' 3
x 5 z , x72)'
and
= x6z
where the symbol ' denotes the transposition of the vectodmatrix. In our setup, observations on X, of high resx z z ,. . . ,x5,,m)' are available for olution, say 2, = (xl,, z = 1 , 2 , . . . ,16. On the other hand, each Y, of low resolution is not observed, but the averaged value 5 / 1 6 = y is observed, say 3. Under these conditions, our aim is to predict the values Y, of Band 6 of all pixels based on the observations q ,. . . ,q 6 and in the local window. Fig. 2 gives an example of observed band values in the local window of actual data. Basically, Band 6 is observed once in each local window. Our data is bulk corrected, so y,, corresponding to observed values of Band 6 at zth pixel, are smoothed (resampled) and have small fluctuations.
0196-2892/96$05.00 0 1996 IEEE
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 34, NO. 5, SEPTEMBER 1996
1152
expressed by a linear mapping A 2 with a matrix A defined by 38
37
29
28
31
30
29
21
30
26
27
24
35
30
25
27i:
Values of Band 7
0
.. .
A=
0 16
. . .. .. 0 0 1 0
0 0 ,0
24
zli: Values of Band 1
0 0
'I6
0
0
0 0 ..
0 0 ..
I,
0
0 0
1 1
'.. '..
0 0
. .. 0
. * '
1
'
e
'
...
0
: 98 x 112.
(2)
The formulas (1) and (2) show that the joint distribution of A 2 is given by the multivariate normal distribution with mean vector A p and covariance matrix ACA', i.e.,
yi: Values of Band 6
Fig. 2. Band values in the local window. Bands 1 , 2 , 7 are of high resolution, and Band 6 is of low resolution.
111. DISTRIBUTIONS RELATING TO BAND 6
x)',
z = 1, 2, We assume that the random vectors ( X i , ..., 16 are independently distributed as a 7-variate normal distribution with unknown mean vector ( p : ,ut)' and unknown covariance matrix
where 16V = u3. Using the joint distribution (3), we can derive the conditional distribution of YIS as follows. The covariance matrix ACA' appearing in (3) is partitioned as
This distribution is denoted by
where pa:6 x 1. Here, we suppose that C x z : 6 x 6, myy and oxy:6 x 1 are common in the local window. Consequently, the joint distribution of the vector of all band values 2 ( X i ,Y1,.. ,X i 6 , Y 1 6 ) ' : 112 x 1 in the local window is given by the multivariate normal distribution with mean vector p and covariance matrix C , where
oyy- &EF;1J2 l
(5)
= 150yy.,/16
where myu.x is a conditional variance defined by gyy 2
= g y y - o;yc,-,loxy.
(6)
In behalf of the joint distribution (3) and the relations (4) and (5), the conditional distribution of Y 1 6 given X 1 = XI,.. . ,XI6 = X I 6 and = j j is deduced as
N(y + @;,C,-,l(J;16
-
+ 61, x , 15ayy x
/
q
whereI = x,/16: 6 x 1. Here, 516 denotes a difference between two conditional expectations, and is defined for general a by
6%x See, e.g., [6] for the definition and properties of multivariate normal distributions. For simplicity, at first we shall derive the joint distribution of Y16 (unobservable Band 6 at 16th pixel) and of all random vectors whose observations are available. The random vector . . ,xi6, 16u, &)': 98 x 1with 1 6 Y = E:!?l y3 is
(xi, xb,.
= (uz- o;y,x;p,)
-
(v - o;,C,-,1TL)
(7)
where ji = p,/16: 6 x 1. For general i, the conditional distribution of Y, is analogously derived as -
KI(x1 N
=
N(y
+
. ,XI6 = 2 1 6 , Y = ?J} o;yCi2(~c?-I) 5, x , 1 5 c ~ ~ ~ . ~ / (8) 16). '
+
NISHII et al.: ENHANCEMENT OF LOW SPATIAL RESOLUTION
-LOCAL
1153
i.e.,
WINDOW
16
FYY
OXY
where 3 and j j are sample means of high-resolution band values and of band 6 values, respectively. 2) The block estimate +block: Consider a block cdnsisting of 7 x 7 local windows, see Fig. 3. We have essentially 49 observations in the block. First, derive 49 mean vectors of observations on 7 band values obtained in local windows in the block. Next, calculate a sample covariance matrix based on 49 mean vectors which would give an estimate for (1/16) Fig. 3. Block consisting 49 local windows. The sample covariance matrix, calculated by 49 mean vectors of local windows, is used for the enhancement of Band 6 values of the shaded local window.
By virtue of the joint distribution ( 3 ) ,we obtain the conditional distribution of Y , given = j j as
yZl{Y = 7j)
N
N(7j+ V, - V , 15aYy/16).
IV. PREDICTION FOR BAND6 In this section, we will propose a predictor of Band 6 based on the conditional distribution (8). We make the following assumption on the mean vector in each local window. A l ) The difference &.= defined by (7) is sufficiently small. We propose a predictor for Y , by
3 + ~'(z, -I) with 7 = Ei2uZy:6 x 1
Then find an estimate for 7,say +block. 3 ) The global block estimate +g-block : This estimate is similar to.+blockexcept the sample covariance matrix is estimated by mean vectors of all local windows in the whole image. 4) The global estimate +global: The global estimate is also derived by the whole image. Take only one observed 7-variate vector in each local window and calculate the sample covariance matrix. Using the local estimate of 7,the local predictor of Y; are defined by y!ocal
(9)
which is .the conditional expectation in (8). Thus, Band 6 is corrected by the linear combination of high resolution data. Obviously, the predictor in the local window satisfies C,'z,{jj ~'(5, - Z)} = 167j, which is a canonical relation required by the sensor system. The simple predictor 7 j for Y , is derived under the following assumption: A2) The difference v, - V is sufficiently small. Then, its variance is given by 15ayy/16. On the other hand, the conditional variance of ?j 7'(za- I)is given by 15cYyx/16. Therefore, the ratio between the conditional and the unconditional variances is equal to
+
+
Throughout this paper, we assume the assumption Al). The assumption A2) is tight and does not fit for enhancement. A2) is assumed only for the case to consider (10). The prediction formula (9) contains the unknown regression coefficient vector 7 = E ; ~ o , ~ . The following estimates for 7 are obtained when C,,,(T,~ and oYyare supposed to be common locally or globally. 1) The local estimate ?local: The local estimate for the regression vector is given by +local = ?;2bzy, where Ex, and bxy are calculated by the sample covariance based on the 16 observed vectors in each local window,
.
h\
--Y
+ +;oca1 (2,
-
I).
711)
Similarly, according to the estimation methods for 7,weqefine , and pflobal. The local estimate has an advantage that it can be obtained by the data within each local window. However, it is composed by only 16 observations and y; is essentially observed once, so images based on this estimator expected to be unstable. Concerning the block estimate, we use only one mean vector in each local window for estimation. A block size with 3 x 3 local windows would be too small. We choose blocks of size 7 x 7 for the sake'of acquiring a stable estimate for 7.We will return to this point in Section VII. ?:lock,
v.
COMPARISON OF PREDICTORS
The proposed method is applied to Landsat 5 TM image of Hiroshima City, Japan, taken at Oct. 23, 1990. (428 lines x 560 pixels). Figs. 4 and 5 are gray scale images based on Bands 7 and 6, respectively. Again, it is noted that the values of Band 6 are bulk corrected. The performance of predictors (9) with estimates ?block,?g-block, and ?global are examined by using Only is not available high resolution data. (The local estimate in this case.) 1) Discard Band 6 to yield only six full-resolution bands. 2) 4 x 4 average one out of these bands in order to simulate the resolution of Band 6. 3) Predict the low resolution band values by ?:lock, I;;-block, and ?%global.
1154
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL 34, NO 5 , SEPTEMBER 1996
Fig 4. High resolution TM image based on Band 7 of size 428 x 560 Hiroshima City, Japan at Oct. 23, 1990.
Fig. 5
Low resolution TM image based on Band 6 of size 428 x 56Qt Hiroshima City, Japan at Oct. 23, 1990.
4) Compare MSE’s between predicted and observed values. The MSE’s between predicted and observed values of Band j for j = 1,. . . , 5 and 7 are tabulated in Table I. Columns of Table I: Mean, Block, G-block, and Global denote MSE’s corresponding to in each local window, Y ; l o c k , ?f-block, and Pfloba1, respectively. Table I proves that our predictors work well. Especially, the values of Band 2 are very accurately predicted by other five full-resolution band values. By this experiment, the global predictor shows the best performance and the block predictor comes next. We anticipate that this is true for Band 6.
VI. ENHANCEMENT OF TM BAND6 The image based on the bulk corrected data yz is a little superior to that of 8. Hence, we propose procedures for improving the image by adding linear combinations of fullresolution bands. The following four predictors for Band 6 are corresponding to the estimation procedures for 7,and are applied to the TM data of Hiroshima. The local predictor Yfcal and the block predictor Yzbiock: Define two predictors for Band 6
NISHII et al.: ENHANCEMENT OF LOW SPATIAL RESOLUTION
1155
TABLE I11 SAMPLE COVARIANCE/CORRELATION OF yt AND Xgloba1,SAMPLE SIZE=
Band
1 2 3 4 5 7
Mean
I1
II II
28.50 13.65 36.80 59.57 I 141.69 50.04
I
I
Block
G-block
Global
4.616* 0.748 2.292** 27.767** 17.474* 7.306*
5.935 0.737* 2.636 42.968 17.517 7.441
4.289** 0.702** 2.375* 31.323* 15.776** 6.442**
I
I
I
I
YG 31.2534 0.9632
TABLE I1
Band 3 Band 4 Band 5 Band 7 Band 6 163.095 102.977 250.166 173.884 44.587 110.279 88.289 186.267 121.620 26.504 185.939 152.654 320.829 207.789 43.828 0.567 389.750 487.523 225.855 9.576 0.814 0.855 834.758 456.866 58.812 0.908 0.681 0.942 281.921 48.305 0.576 0.087 0.365 0.516 31.132
Fig. 6. Enlarged low resolution image based on the rectangular region of Fig. 5.
where +local and are the local and the block estimates defined in the previous section. The global block predictor Y:-block: The global block estimate for 7 is given by -&,lock (0.7274, -1.0999,0.1186, -0.1617,0.1228,0.0638). Thus, we propose the global block predictor %g-block - yz
+ 0.7274(~1,- Z1) - 1.0999(~2,
-2
+ 0.1186(23, - 23) 0.1617(~4, + 0.1228(~5, 5 5 ) + 0.0638(%7, -
-
2)
- Z4) -
3 1.5942 34.4279
The global predictor Y;loba1: Sample covariances and correlations calculated by 107 x 140 = 14980 observations selected in each local window are listed in Table 11. If the covariance matrix of (X:, E)’ is common in the whole image, Table I1 yields the global estimate of 7 as (0.3871, -0.3126, -0.0032, -0.0809, -0.0088,0.1489)’. Thus, the global predictor in this case is given by
SAMPLE COVARIANCES/CORRELATIONS OF HIROSHIMA TM IMAGE, SAMPLE SIZE= 14980
Band 1 Band 2 157.020 99.260 0.969 66.831 0.955 0.989 0.416 0.547 0.691 0.789 0.827 0.886 0.638 0.581
239680
ezglob a1
E7). (14)
i/zglobal - yz -
+ O.3871(~1,
- -1)
O.O032(~3,- Z3) 0.0088(~5,- 2 5 )
-
- 0.3126(~2,- Z,)
0.0809(~4,- Z 4 ) - Z7).
+ 0.1489(~7,
(15) To observe in detail, we enlarge the sea and commercial area of Hiroshima. This area of size 100 x 116 is marked in Fig. 5. Fig. 6 is the enlarged image based on yz. This image is enhanced by four formulas (12)-(15) in Fig. 7. The image based on (15) is superior to the other images because the breakwater and roads can be easily detected. Now, the sample covariance and correlation between yz and in the whole image are listed in Table 111. The mean coincides with that of yz. 132.733 of Table I11 shows that the variances of enhanced values are greater than of yz because enhanced values have bigger fluctuations. However, using Table I1 we can estimate the conditional variance (6) as 16.40. This implies that the ratio (10) is 0.526. Thus, our procedure reduces the conditional variance into half of the variance of raw data by using information based on high resolution band data under the assumption A2). Figs. 8 and 9 show the difference betweenpriginal values yz in Fig. 2 and enhanced values Rblock and Kglobal,respectively. It is seen that corrects the raw data yz largely than Rblockdoes. is given Finally, the enhanced whole image based on by Fig. 10.
VII.
CONCLUDING REMARKS
All predictors (12)-(15) for Band 6 are of the form that the raw band 6 value is corrected by the linear combination of high resolution values. Such a correction is theoretically justified. We will summarize their features. 1) The predictors (12)-( 15) satisfy the desirable condition such that the average of enhanced values in the local window is equal to the observation on Band 6. 2) The predictor of the form (9) is simple and unbiased under the assumption Al). Its conditional variance is smaller than the variance of jj under A2).
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL 34, NO 5, SEPTEMBER 1996
1156
Fig. 7
Enhanced images by four predictors. (a)
Fig. 8. The difference ?:lock
- yz
%local
in (12). (b) %'lock
in the local window used in Fig. 2.
3) The global predictor (15) shows the best performance among four predictors, and the block predictor (13) comes next. The most difficult problem is in estimation of the regression vector 7 because the fluctuations in yt are much smaller than in Y,. The goodness of the global predictor may come from that the predictor is based on the stable estimate +g-block, but +g-block may still estimate 7 downward. The block estimate is fair in this sense because 7 is estimated by the data such that all band values are averaged in each local window. Actually,
in (13). (c)
%g-'lock
in (14). (d)
Fig. 9. The difference ?:lobagl
%gglobal
in (15).
- yz in the local window used in Fig. 2.
f l u c t u a e i n the elements of +g-block given in (14) are bigger than those of (15). The better estimate for mZy would improve predictors, and this problem remains open. We also examined an iterative use of our enhancement procedures. By the numerical example, this procedure behaves like an edging process. However, it is not seen the improvement of MSE's. Thus among four predictors, we recommend the global predictor (15) for Band 6. If the fluctuation in band values in the whole image is big, we expect that the block predictor
NISHII ef al.: ENHANCEMENT OF LOW SPATIAL RESOLUTION
Fig. 10. Enhanced image based on
Rglobal
1157
in (15). Hiroshima City, Japan at Oct. 23, 1990.
works better than the global predictor does. The other problem to find the optimal block size is left for future investigation. Finally, we retum to the normality assumption on 7 band values. Of course, this assumption is too tight in actual data. However, our predictors proposed here still remain useful in the sense that they are linear approximations to predictors based on conditional expectations under nonnormal case.
random vector is derived in the same way by 0 0 0
4 0
0 IP
0 0 0
...
0 0
...
0 0
. . . IP ... 0 ... 0
0 14
0
APPENDIX MORE GENERALSETTING Our treatment in Sections I11 and IV is immediately extended in the general setting. Let random vectors (X:,Y:)’: ( p q ) x 1,i = 1 , 2 , . . . , m , be independently normally distributed as
+
The joint density in the above yields the following conditional distribution of Y , -
Y,I{X1 = 2 1 , . . . , X m = z m , Y=g} N
.. E,,
E,,
E,,
E,, .
0 0
0 0
..’ ... .
.
0 0
.
E,, . . . E,, ” *
0 0
E,, E,,
The special case with p = 6, q = 1, and m = 16 has
Nq(3 + CyzE$(xm
-
z) +
6 2 2 ,
(1 - ~ / m ) C y yz )
wherem:= E,”=,x,:dxl,E,,, = Xyy-X,,X;~EZy: q x q and Sa ,= (v,- EyzE;2pa) - (v - E,,E;;p): q x 1 with mp = C,”=yj: p x 1. If the length of the vector S,, is sufficiently small, we propose the predictor for Y , by ya - y
+
~ y z ~ ~ : (-x 2). a
The unknown matrix CyxE;L: q x p will be estimated locally or globally.
been already considered in this paper. Suppose that observed ACKNOWLEDGMENT vectors x; on X ; are available, whereas the average of Y;’s The authors wish to thank two anonvmous referees for their is only observed as 3: q x 1. Our problem is to predict each careful readings and valuable comments. Their comments led Y , :q x 1by using 5 1 , ’ . ‘ ,xm and 3. Put m y = Y j and mD = CyEl uI.Then, the joint distribution of the following the draft into the drastic revision. Specifically, the referees
*
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 34, NO. 5, SEPTEMBER 1996
1158
proposed the block estimate and the comparison of procedures based on the MSE’s. Also, they suggested the iterative use of the enhancement procedures. These points, discussed in Sections IV, V, and VII, respectively, are greatly useful to improve the draft.
Saeko Kusanobu received the B S degree in mathematics from Shimane University in 1994 and the M S degrees in integrated arts and sciences from Hiroshima University in 1996 She is a student of the Graduate School of Engineering, Hlroshima University Her interest includes remote sensing data and ,image analysis based on stahstical techniques
REFERENCES M Inamura, “Improvement of spatial resolution for low spatial resolution thermal infrared image using high spatial resolution visible and near-infrared images,” Trans Inst Electron, Inform, Commun Eng , Part A, vol J 71-A, pp 497-504, 1988 (in Japanese) M Inamura, H Toyota, and S Fujimura, “Exterior algebraic processing for remotely sensed multispectral and multitemporal images,” ZEEE Trans Geosci Remote Sensing, vol 1, pp 112-118, 1982 B Zhukov, D Oertel, and F Lanzl, “A multiresolution multisensor technique for satellite remote sensing,” in Proc 1995 Int Geoscz Remote Sensing Symp , vol I, pp 51-53, 1995 B Zhukov, M Berger, F Lanzl, and H Kaufmann, “A new technique for merging multispectral and panchromatic images revealing sub-pixel spectral variation,” in Proc 1995 Int Geoscz Remote Sensing Symp , VOI 111, pp 2154-2156, 1995 R 0 Duda and P.E Hart, Pattern Classzjication and Scene Analysis. New York Wiley, 1973. C R Rao, Linear Statistical Inference and Its Applications, 2nd ed New York Wiley, 1976, ch 8, Section A
Ryuei Nishii (M’96) received the B.S. degree in mathematics from Nagoya University in 1976 and the M.S. and D.Sc. degrees in mathematics from Hiroshima University in 1978 and 1981, respectively. He was a Research Associate with the Department of Mathematics, Hiroshima University from 1980 to 1985, and moved to the Faculty of Integrated Arts and Sciences of the same university. Presently, he is an Associate Professor in the same faculty He was a Research Scientist with the University of Pittsburgh from 1986 to 1987, and was a Visiting Associate Professor with the Institute of Statistical Mathematics from 1992 to 1993 His current research interests are in the applications of statishcal techniques to remote sensing data and image analysis. Professor Nishii is a member of the Japan Statistical Society, Japan Mathematical Society, and Remote Sensing Society of Japan.
Shojiro Tanaka (M’89) received the M Sc. degree in environmental science in 1986 and the P h D degree (thesis) in environmental planning science in 1992 from Hiroshima University. He was a Research Associate with the Informahon Processing Center, Hiroshima University from 1987 to 1992, and he is currently an Associate Professor with the Department of Computer Science, Faculty of Engineering, Yamanashi University. His interests are in mathematical and statistical modeling of environment, remote sensing, and geometric extension of data models in DBMS. His broad topics converge on one point: actual solution or tool-building for environmental problems. Professor Tanaka is a member of ACM, Japan Statistical Society, Information Processing Society of Japan, Operations Research Society of Japan, and Remote Sensing Society of Japan