Illinois Institute of Technology, Chicago, IL, 60616. ABSTRACT. In this paper, we develop a multichannel formulation of a single channel filtering problem in the ...
RESTORATION OF SINGLE CHANNEL IMAGES WITH MULTICHANNEL FILTERING IN THE WAVELET DOMAIN
Mark R. Banham', Ntkolas P. Galatsanos2, Aggelos K . Katsaggelos' and Hector Gontale2 'Department of Electrical Engineering and Computer Science, Northwestern University, Evanston, Illinois 60208-31 18 2Department of Electrical and Computer Engineering, Illinois Institute of Technology, Chicago, IL, 60616
ABSTRACT In this paper, we develop a multichannel formulation of a single channel filtering problem in the wavelet domain. This technique is based on an orthogonal matrix representation of the 2-D separable wavelet t r a n s form. We show that semi-block circulant (SBC) structures resulting from a similarity transformation with this matrix operator permit a multichannel treatment of a noisy blurred image, yielding an adaptive restoration technique in the wavelet frequency domain. This work presents a new formulation of the 2-D separable wavelet decomposition that clearly produces matrices which may be block-diagonalized with an array Fourier transform. The method described here is a new tool, which builds on multichannel image restoration techniques developed in the past, but uses these ideas in the context of a single channel image, and offers much potential for improved adaptive image restoration in the wavelet domain.
1.
INTRODUCTION
There has recently been a substantial interest in utilizing wavelet transforms for a variety of signal processing tasks. In particular, wavelet transforms have been extensively used for analyzing and processing speech signals, and compactly encoding still images and image sequences. In this paper, we utilize wavelets for yet another signal processing application, image restoration. The wavelet transform offers the advantage of localizing spatial information by representing a signal as a series of low resolution approximations along with details lost when making these approximations. In the context of images, the details generally correspond to edge and texture information and the low resolution components represent the slowly changing flat regions of the image. The separable 2-D wavelet transform can be used to decompose an image into 4 channels, referred to as LL,LH,HL and HH. Here the two letters, L or H , correspond t o the type of filtering performed along the
rows and columns of the image. L represents a lowpass wavelet filtering operation, H represents the highpass counterpart. The transform may be applied to the LL band to generate 4 new bands at the next lowest resolution. The important advantage of this representation is that the scale changes with resolution. Thus, only a coarse scale is needed t o represent the low resolution information, but the high resolution details require a finer scale. The transformation is orthogonal, so the number of coefficients in the transform is equal to the number of pixels in the original image. Given that the wavelet transform can be used to decompose an image into 4 channels, we can develop a method based on multichannel image restoration techniques to restore a single channel image. Each of the four channels contains different spatial information, but each also has some remaining correlation with neighboring channels along one orientation. In multichannel restoration, the available channels of an image are restored simultaneously, thus taking advantage of the between-channel correlations [l, 21. We will discuss a method for performing image restoration with a spatially adaptive Wiener filter in the two dimensional wavelet domain. A technique is shown for writing a block-circulant (BC) operator as a semi-block circulant (SBC) operator in the wavelet domain, and as a blockdiagonal operator in the wavelet-frequency domain. A semi-block circulant matrix, in this case, is circulant at the block level, but has sub-blocks which may have an arbitrary structure [3, 41. It is interesting to find that the standard single channel Wiener filter may be written as an SBC problem in the wavelet domain, and thus easily solved in the discrete wavelet frequency domain. The benefit of entering the wavelet domain is the ability to place multichannel constraints on what would otherwise be a single channel problem, thus producing a restoration technique which is spatially adaptive in the detail bands of the wavelet decomposition. In particular, we show that we can adapt the restoration by weighting the restored detail bands according to the local variance activity in
0-7803-0720-8/92 $3.00 01992 IEEE
problem. A beneficial aspect of this is that we can arrive at all of the multichannel statistics by simply passing the filter though a similarity transformation which brings the problem into the wavelet domain. We can then take the wavelet transform of our observation, and, with discrete Fourier transforms, filter in the wavelet-frequency domain, before taking appropriate inverse transforms to return to the spatial domain. In expressing the wavelet transform, we can write the convolution operations with circulant operators because we will assume a circulant treatment of the borders in the this decomposition. This will provide the means to block-diagonalize the problem later. We can write a matrix which combines the lowpass filtering of one row followed by decimation as,
the observed subbands, and regularize the restoration by weighting the multichannel frequency components according to the behavior of each individual sub-matrix in the filter. The development of the multichannel technique involves a useful mathematical formulation which offers a new tool for processing two dimensional signals in the wavelet-frequency domain. 2.
MULTICHANNEL FORMULATION
We begin by discussing the wavelet-based multichannel decomposition. This multiresolution decomposition may be obtained by passing an image through a treestructured quadrature mirror filter bank (with wavelet filters as the operators) [5]. For the development of this technique, we will use orthogonal wavelets. The results are obtained with the Daubechies wavelets of support 8 [7]. Since the row and column operations are separable, we can write each band of the single level multiresolution decomposition in terms of the product of two matrices with the image, f , lexicographically ordered by rows. For example, to obtain the LL band, we can take the product of 8, and 8, with f , where 8, rep:+ sents lowpass column filtering and decimation, and H, represents lowpass row filtering and decimation. Here we will assume the image t o be of size N x N, making f a row-ordered N 2 x 1 vector. Our goal is to formulate an orthonormal matrix, W,which will provide a 2-D wavelet decomposition of f , and preserve, in some special way, the block circulant structures of standard linear operators. Such operators include D, the blur matrix, and Rjf the correlation matrix found in the two dimensional Wiener filter. This work relies on the knowledge that we can reformulate a multichannel restoration problem by ordering the image data into vectors composed of samples from each channel [3, 41. Since the multichannel Wiener filter may be implemented efficiently in the discrete frequency domain, it seems advantageous to treat image restoration in the wavelet domain as a multichannel
I I
ho hz
0 0
0 0
0 0
... h ~ - 1 ... ...... 0
ho
hi hL-1
...
hi 0
0
0
ho
0 0
1
qxN where h, represents an orthogonal lowpass wavelet filter of length L [SI. If the operator H, represents the process of filtering and decimating one row, we can process the lowpass part of all rows with a matrix of the form 8, = diag [HrH,.. *HrHr]qxNz. The highpass operations are simply an extension of the lowpass formulation with the highpass wavelet filter, g,, replacing h, in the equation above to give us the matrices G, and The column filtering and decimation operations applied to the same rowlexicographically ordered f must be written in a different form. We can represent the lowpass filtering and decimation of one column with
e,.
. . . . . . . . . . . . . . . hL-1 0 . . . . . . . . . . . . . . . hz 0 ... ... hi 0 . . . . . . . . . . . . . . . . . . hL-1 0 . . . . . . . . . . . .
...... h ~ - 1 0 . . . . . . . . . . . . . . . . . . . . . hL-1
0
hl
...
ho
... . . . . . .
1559
0
hi
......
0
......
hi hS
0
O . . . h o O . . . O
0 0
... ...
...... ......
1
The product H,H,f then results in a vector of one column of the resulting LL subband. The subscripts indicating the size of each matrix should help to make these matrix multiplications clear. We can now write the operation on all columns of the row filtered and decimated image in the form
,
-q
operator,
LWf, where the - notation will indicate the wavelet domain. In order to obtain the wavelet operator, W, we must observe the form of the rows of the matrix products in Eqs.(l)-(4). In particular, we will stick with the convention of examining the lowpass (LL) part of the problem and indicate that the other bands have identical properties as far as the structure of the matrices is concerned. The product of the matrices H,Hr is a matrix with the following form
1:
x q
where H: indicates H, with each row circu1ar.j shiftelby n. In this notation, the multiplication of HcHr by f would result in an LL subband which is ordered lexicographicaily by columns. To maintain the rowlexicographic ordering, and t o obtain a beneficial semicirculant structure, we reorder the rows of H: so that this structure is preserved. It is here that the interesting circulant structures of the 2-D operator that we are building begin to emerge. Each row of the reordered H:, which we call H,, is a circular shift by one of the previous row, for the first N / 2 rows. This comes from the fact that the reordering takes one row from each of the N / 2 H:'s and places them in succession. Since Ha is a circular shift of every row of H: by one, this results in the circulant property of the first N / 2 rows. However, after the first N / 2 rows of H,, we acquire the next N / 2 by taking the second row of each H: and grouping them together. It is clear that row two of H: is a circular shift by N of row one of H:. Therefore, every N / 2 rows in H, is simply equivalent to the previous set of N / 2 rows, shifted by N . This pattern is useful in formulating a complete 2-D decomposition with one matrix operator, which we will investigate shortly. Recognizing that the highpass filtering counterparts of each of the matrices defined here is simply acquired by replacing the lowpass filter coefficients with those of the highpass wavelet filter, we can make the following statement. The conventional subbands in a single level wavelet decomposition may be represented in lexicographically row-ordered form in the following manner with the matrices defined thus far,
This is a useful notation, however, we would also like to express the decompsition in terms of a single matrix
... C
D
...
9
where each A, B,C ... is an x N matrix with a shift-2 circulant structure like that found in H,. We now define the ( N 2 x N 2 ) matrix, W, to represent a 2-D wavelet operator which is comprised of the matrix products in Eqs.(l)-(4) and _has reordered rows such that the wavelet transform, f , is made up of lexicographically row-ordered vectors. Each vector contains 4 coefficients from the same spatial position from each of the four channels: LL,LH,HL,and HH. Thus, W may be partitioned into submatrices of support 2N x 2 N . While this is circulant at the block level (i.e. each 2N x 2N block is shifted by 2N to the right in each block-row) the submatrices are of a unique structure. Each 2N x 2N sub-block can be partitioned into 2 matrices of size 2N x N both of which are sub-block shift-2 circulant with sub-blocks of size 4 x 4. In other words, these sub-blocks possess a structure which allows W to be completely defined by the components of its first 4 rows. We have formulated W in this way because we wish to obtain an SBC matrix from a BC matrix under a similarity transformation with W. In particular, we will be interested in matrices which may be blockdiagonalized with the 2-D array Fourier transform. These types of matrices will be denoted as S B C ~ Din this paper. Such a matrix for our purposes will be circulant at the large block level (2N x 2 N ) , as is W, but each 2N x 2N sub-block must be SBC with sub-blocks , of size 4 x 4. Note that W is not S B C ~ D however, WAWT,with A a BC matrix, is SBCZD. Before examining this more closely, we can point out the properties of W which are beneficial for our purposes. First, W is an orthogonal matrix, that is,
1560
WTW = I. This means that the wavelet transform, as we know, is invertible, and the inverse is simply obtained by premultiplying f by W'. This is a result of the fact that the wavelet filters and their even translates form an orthonormal basis for one dimension [8]. The two dimensions in our problem are orthogonal, so the rows of W form a two dimensional orthonormal basis for the space of signals under consideration. In essence, the multiplication of W with f results in the projection of f onto four orthogonal subspaces. The rows of H,, G r , H, and c, are orthonormal, and in combination, form the basis functions for projecting f onto each orthogonal subband. Another important property of W , as eluded to above, is that when A is a BC matrix, WAWT is an S B C ~ Dmatrix. Thus, the circulant property is partially preserved under the similarity transformation used here. This fact allows us to rewrite a single channel linear filtering problem in the wavelet domain as a multichannel formulation. To obtain the transformation WAWT, we observe that AWT has 2 N x N sub-blocks. Each of these, in turn, has N x 4 sub-blocks which are vertical shifts by two of the neighboring the N x 4 sub-block. AWT is completely defined by four of its rows, namely rows 1,2,N+1 and N + 2 . These rows can be obtained with 4 wavelet transforms. Now, due to the special structure of AWT, we find that WAWT is S B C ~ D .This is a result of the alignment of terms in the sub-blocks in the multiplication. Because of this fact, WAWT it is completely defined by its first 4 rows and we require only 4 additional wavelet transforms to obtain it. WAVELET-BASED FILTERING
3.
The development of the multichannel Wiener filter for single channel images is obtained in the following way. (see also [3, 4]), Let y be the the blurred and noisy observation, and f be the estimate of the original image obtained with a single channel Wiener filter,
f
= RffDH[DR/fDH+ RUu]-'y,
where Rff is the covariance matrix of the original image which we can approximate as block circulant as in [l]. bW is the covariance matrix of additive white Gaussian noise, .,"I, and D is the blur operator which we take to be block circulant also. The superscript indicates the Hermitian transpose operator. We see from the formulation of W in Sec. 2. that we can write
Wf
= WRfjWTWDHWTW
+ WbuWT]
9
i.
= Rff D H [ D R f , D H
+ R,,]-'y
(5)
which we can block diagonalize with the 2-D Array [3, 41, to obtain Fourier Transform operator,; ;Cl
which is a block diagonal problem requiring the inversion of only 4x4 matrices to invert the N 2 x N 2 matrix \1Ir = @ ~ j Q , - ~ @ z@%. The @ terms represent the block diagonal counterparts in the frequency domain of each SBC matrix in the wavelet domain in Eq.(5). Equation (6) is a primary result of multichannel restoration which we now apply to a single channel case. The interesting characteristic about this filtering approach is that the beneficial SBC structure leads to a method for frequency domain restoration of an image, with the spatial localization of the wavelet transform preserved. As expressed in Eq.(6), this waveletfrequency domain filter is equivalent to the single channel Wiener filter. In the next section, we address the problem of exploiting the nonstationary characteristics of the detail bands to improve the restoration while maintaining the powerful multichannel formulation developed here.
+
4.
ADAPTIVE FILTERING
The multichannel problem investigated here, is directly related to much of the previous multichannel restoration work, in particular [I, 21. However, here we place appropriate constraints on the restoration to take advantage of the spatial information inherent in the wavelet transform. One of the most beneficial ways to take advantage of the wavelet domain representation, is to replace the auto- and cross-correlation terms in the wavelet domain representation , WRjj W T ,with the actual sample correlations between subbands in the decomposition. In other words, we can experimentally measure or estimate the multichannel statistics of the original image bands, and gain by taking advantage of the individual properties of each band. This is a departure from the single channel implementation where the restoration is limited by the stationary estimate of the original image spectrum. We will refer to this technique as "Approach 1". Here the only constraint applied to the problem is the use of the actual correlations between subbands, and not the multichannel statistics obtained by decomposing the single channel correlation matrix. In all of the cases reported here, the cross-correlation between signals a and b is estimated by
l M Rab(j, k.) = ~2
M
n = l m=l
1561
+
w ( n ,m ) a ( j n , k.
+ m)b(n,m)
where M is the lag of window function w(rn,n) and R,r,(j,k) is the correlation at spatial location (j,k). To find the original image autecorrelation we simply replace a and b with f , and to find the degraded image autocorrelation, we replace a and b with y. We use a 2-D Bartlett window for w ( m , n ) . To estimate the aut* and cross-correlations in the wavelet domain, we replace a and b with the appropriate subband channels. Next, we consider an adaptive approach that can be utilized in the wavelet domain in order to preserve some of the nonstationary characteristics of the image. After restoring the image, we can weight the restored detail bands by a local variance measure obtained from the original observation subbands. This allows for the reduction of some of the filtered noise effects found in the detail bands, but still preserves the edges that are sharp in the restored bands. A similar treatment of the upper bands was also implemented for the independent subband restoration (non-multichannel) case in [9]. Additionally, we can regularize the problem in the frequency domain by measuring the determinant of each 4x4 submatrix which we invert in the blockdiagonalized filter, Eq.(6), and adding a small value along the diagonal of each submatrix with a determinant smaller than a given threshold. This has the benefit of limiting the amplification of any noise present at those frequencies, but in a manner which considers the relationship between bands. Thus, we are regularizing multiple frequencies separated by octave bands at one time. The combination of adaptive methods described in this paragraph will be referred to as “Approach 2”. For this approach, we utilize WRjj W Tobtained from the measured Rjj of the degraded image, and do not replace the correlation terms in the wavelet domain as in Approach 1. 5.
EXPERIMENTAL RESULTS
In this section, we present some results based on the implementation of the wavelet-based multichannel filtering methods discussed above. The purpose is to demonstrate some of the ideas developed here, and to give some sense of the possible advantages of such a filtering approach. We have examined several filtering cases here, using the standard 256x256 “Cameraman” image with different degradations. To evaluate the quality of the restored images, we use the improvement in signal-to-noise ratio, or ISNR, as the figure of merit. This is defined in decibels as,
where f ( r n , n ) and y(m,n) are the original and de-
I
13x1 Uniform Blur Single Channel Approach 1
30 dB 6.652 6.457
9x1 Uniform Blur
30 dB 6.226 6.685 30 dB 5.577 6.405
40 dB 10.231 11.051
50 dB 12.238 15.979
ISNR Single Channel Approach 2 -_ 13x1 Uniform Blur Single Channel Approach 2
I 40 dB I 50 dB I
9.903 I 10.189 I 40 dB 9.254 9.960
11.682 13.381 50 dB 11.322 13.751
Table 2: Spectra computed from degraded image graded intensity co-mponents, respectively, at spatial location rn, n, and f(m, n) is the corresponding filtered estimate. First we implemented a full band (single channel) Wiener filtering using the original spectrum, and here show the results in comparison to the wavelet-based multichannel method using Approach 1. Table 1 shows these results for a 9x1 and 13x1 uniform horizontal blur with additive noise at 30,40, and 50 dB BSNR, or blurred signal to noise ratio. It can be seen here that at high BSNR, the wavelet domain offers a substantial improvement over the single channel restoration approach due to the nonstationary treatment of the bands. As the degradation due to noise becomes more severe, however, Approach 1 suffers from to the fact that the detail bands in the decomposition are largely composed of noise. Table 2 shows results obtained by applying the adaptive technique of Approach 2 (weighting the detail bands and regularizing the filter in the frequency domain), to the same degradation cases. Here we treat the problem as if we did not have access to the original image, and estimate the spectra from the degraded data with windows of the same size in the wavelet domain, and in the spatial domain. This approach is seen to produce the minimization of filtered noise effects which manifest themselves in the detail bands, by weighting the influence of those bands accordingly. LFrom these results, we see that given the true correlations between subbands, we achieve the best improve-
1562
ment in signal to noise ratio. One possible extension of this technique, as mentioned above is then to estimate the correlations between subbands from the noisy data before using that as part of the restoration procedure defined here. It is also apparent that gains can be made by treating the detail bands as nonstationary signals, and weighting them appropriately. The results presented here indicate that the multichannel approach to single channel image restoration provides some new motivations for employing nonstationary, or spatially adaptive restoration. The wavelet-based multichannel approach also offers potential for use with variety of linear filtering techniques which were previously limited to a purely stationary treatment of the signals under consideration.
REFERENCES [l] N . P. Galatsanos and R. T. Chin, “Digital restoration of multichannel images,” IEEE Trans. Acoust., Speech, Signal Proc., vol. 37, pp. 415-421, March 1989. [2] N. P. Galatsanos, A. K. Katsaggelos, R. T . Chin, and A. D. Hillery “Least squares restoration of multichannel images,” IEEE Duns. Acoust., Speech, Signal Proc., vol. 39, pp. 2222-2236, March 1991. [3] K. T. Lay, “Maximum likelihood iterative image identification and restoration ,” Ph. D. dissertation, Northwestern University, Dec. 1991. [4] A. K. Katsaggelos, N . P. Galatsanos, K. T . Lay and W. Zhu, “Multi-channel image identification and restoration based on the EM algorithm and crossvalidation,” Proc. SPIE Intl. Symp. Optical Applied Science and Engg., July 1992. [5] S . G. Mallat, “Multifrequency channel decomposition of images and wavelet models,” IEEE Trans. Acoust., Speech, Signal Proc., vol. 37, pp. 20912110, Dec. 1989. [6] W. H. Press, “Wavelet transforms: a primer,” Harvard-Smi t hsonian Center for Astrophysics, Preprint No. 3184, 1991. [7] I. Daubechies, “Orthonormal bases of compactly supported wavelets,” Commun. Pure Appl. Math., vol. 41, pp. 909-996, NOV. 1988. 181 M. Vetterli and C. Herley, “Wavelets and filter banks: theory and design,” IEEE Trans. Signal Proc. Sept. 1992. [9] J . Woods and J . Kim, “Image identification and restoration in the subband domain,” IEEE Proc. ICASSP, Vol. 111, pp. 297-300, March 1992.
1563
