Reduction of Boundary Artifacts in Image Restoration

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 5, NO. 4, APRIL 1996

611

Reduction of Boundary Artifacts in Image Restoration Farzin Aghdasi and Rabab K. Ward

Abstract-The abrupt boundary truncation of an image introduces artifacts in the restored image. The traditional solution is to smooth the image data wing special window functions such Bs Hamming or trapezoidal windows. This is followed by zeropadding and linear convolution with the restoration filter. This hethod improves the results but still distorts the image, especially fit the margins. Instead of the above method, we propose a differfat procedure. This proceduire is simple and exploits the natural property of “circular” or periodic convolution of the discrete Fourier transform (DFT). Instead of padding the image by zeros, it is padded by a reflected version of it. This is followed by “circular” convolution with the restoration filter. This procedure is shown to lead to better res#torationresults than the windowing and linear convolution techniques. The computational effort is also improved since our method requires half the number of computations required by the conventional linear deconvolution method.

I. INTRODUCTION

F

OR THE removal of certain well-characterized degradations from images, image restoration techniques are employed. These, however, generally introduce various artifacts of their own in the image. Consider an image of a scene obtained, for example, by a photographic camera. The observed image gl(z,y) is cut from the background due to the finite aperture of the irnage acquisition device. Since the reconstruction filter is usually a convolutiqnal filter and not a point-wise operation, the restored image f (z,y) will contain artifacts due to the abrupt truncated boundaries in g1(x,y). Several authors have studied these artifacts. Tekalp and Sezan have identified this problem as one of four types of image restoration artifacts [ 11. In particular, the artifacts associated with the image boundary truncation can dominate the restored image under certain conditions. Woods was the first to discuss the boundary truncation artifact [2]. Tan et al. have discussed the boundary artifacts in image restoration, but they have not considered extending the ilmage to reduce these artifacts [3], r41. The image degradation model is usually assumed to be that of a linear convolution and additive noise. In this paper, we do not consider the noise ,and concentrate on the deblurring. Even though the omission of the noise may not hold in actual Manuscript received July 29, 1094; revised December 21, 1994. This work was supported in part by a Canadian Commonwealth Scholarship. F. Aghdasi was with University of British Columbia, Vancouver, V6T 124 Cadada. He is now with the Department of Electrical Engineering, University of Witwatersrand, Johannesburg, South Africa. R. K. Ward is with the Department of Electrical Engineering, University of British Columbia, Vancouver, V6T 124 Canada. Publisher Item Identifier S 1057-7149(96)02758-3.

situations, our analysis serves as an appropriate starting point due to its mathematical tractability. This problem is also of relevance in the restoration of large images, where a large image is subdivided into subimages, and each subimage is restored separately. Additionally, the computational load of restoration is reduced by image subdivision. The traditional solution to the image truncation problem is to employ special windows such as trapezoidal, Hamming, or Hanning to smooth the effect of truncation and the literature has many examples of such windows [ 5 ] , [6]. This is followed by zero padding the observed image to the length of the restoration filter; DFT operations are then used for deconvolution. The zero padding is necessary to achieve linear deconvolution (convolution) since such DFT processes are by nature circular or periodic ones. Fig. l(a) shows a degraded image obtained by imaging a step edge with a CCD camera using a square aperture. Fig. l(b) shows the restored image using the linear deconvolution only. The reconistruction filter is a modified smooth Wiener filter that was designed from the knowledge of the camera modulation transfer function. We used the following filter

where a = 0.01 gave the visually best restored image. The ringing artifacts produced by frequency domain filtering are clearly visible in the restored image. In Fig. l(c), we illustriite the use of a trapezoidal window. The same Wiener filter as in Fig. l(b) is employed to restore the observed noisy step edge. The original image was multiplied by a trapezoidal window such that a band of 16 pixels wide around the image was attenuated linearly while the central region of 32 x 32 pixels were unaltered. Clearly, the step edge itself becomes sharper, reversing the blurring effect of the camera; however, the margins of the image are severely distorted. In this paper, we examine some simple measures to minimize the effects of these boundary truncation artifacts. We show that the natural circular deconvolution (convolution) process of DFT can be used to advantage in lessening the image truncation effects. We find that when the image is globally stationary, then padding with the image itself (instead of zeros) leads to better expected results. We then relax the condition of global stationarity to that of local stationarity and propose a simple padding method by which the results are

1057-7149/96$05.00 0 1996 IEEE

JEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 5 , NO. 4, APRIL 1996

612

(a)

(b)

(C)

(4

Fig. 1. Restoration of a step edge acquired by a digitizing photographic camera: (a) Observed image of the step edge; (b) restored image using linear deconvolution. Notice the boundary truncation artifact; (c) restored image of the step edge; a trapezoidal was applied to the data prior to its deconvolution; (d) restored image of the step edge using our proposed approach.

further improved. Whenever convenient, and without loss of generality, we will illustrate our approach using 1-D images.

We show in Appendix A that L-1

11. PROBLEM DESCRIPTION AND LINEAR DECONVOLUTON

j=1

RESTORATION BY

J

The problem could be modeled in more than one way. We choose a model by which the results of the different deconvolution methods may be easily compared. Consider a long 1-D signal f(n)whose length >> M and its blurred versiong(n).Letgl(n) = {go,g1,... , g ~ , - ~betheobserved } truncated section of g ( n ) . g 1 ( n ) is limited to a width of M pixels. Let us select the corresponding M points

x

C(f(-n) - (-l)”(M

- n))WNnk

(5)

n=l

for 0

5k 5N

-

I, where W N = e -27r-

In the space domain, the error sequence e l ( n ) is

{fo,fl,.‘.,fM-l)

and assume that the observed g l ( n ) (of size M ) is due to the convolution of h(n) with { fo, . . . , fhl-1) plus modeling errors. Let us pad the M selected points of f ( n ) with zeros to obtain the set f l ( n ) fl(.)

={~‘~,~,~,~,fO,f1,~~~,f~--l,~

Thus f ( n )-fi(.)

Please note that in the above, we assumed h(n) to be in the form {ho,h l , . . . ,h ~ - l } . If, however, h(n)is symmetric and is represented in the form { h - ~. .,. , ho,. . . , h a } , then (6) becomes (6a) which appears a:the expression at t i e bottom , , ~ ~ ~ , ~ } . of~ the page. Applying the linear deconvolution restoration filter H , ( k ) = to G1( k ) in (4) results in

&

o,o,. ’ . > 0, f M , f M + l , .

= {. . . , f - 2 , f-1,0,

. .>.

(2) After padding g1(n)with zeros, the resulting model is

+

g l ( n ) = h(n)* f l ( n ) e l ( n ) for 0 _< n _< N - 1

(3) In addition. the error in the restoration is

where h(n),which is the finite impulse response, is of length

L(< M ) and where N = 2M. The error sequence e l ( n ) is the result of the truncation of g ( n ) and assuming the only observed segment of g(n) is g1(n). Taking the N-point DFT of (3), we get

Gi(k)= H ( k ) . F i ( k )

+El(k).

(4)

i.e., the error in restoration using the conventional zero padding and linear deconvolution method is given by (10) and (5).

613

AGHDASI AND WARD: REDUCTION OF BOUNDARY ARTIFACTS IN IMAGE RESTORATION

111. RESTORATION BY CIRCULAR DECONVOLUTION

for the circular deconvolution case, will be less than that of the linear deconvolution case E;””(k).Appendix B shows that

Consider again the degraded image g(n). Truncating g(n) to L-3 length M to obtain the observed gl(n) and padding the latter by zeros creates severe edges within the signal gl(n), which &(k) = h(j)W$ causes the errors in (10) or (5). Thus, instead of assuming j=1 the observed g1 ( n ) is a truncated signal, let us assume g1 ( n ) j as a period in a periodic signal, and let us investigate the x -- f ( M - n ) ] [ l + (-l)k]w;nk (22) n=l effects of the periodic deconvolution of this period with the reconstruction filter h,(n). Let us denote two periods of g1(n) for 0 5 k 5 N - 1, andl in the space domain, we can show as g 2 ( n ) . g2(n) is thus the N-sample sequence that e2(n) is

C[f(-n)

92(n) = {go,..~,gM-l,90,...,9111--1j

(1 1)

and assume g 2 ( n ) to be one period of a periodic signal. The assumption that the period of g2(n) contains two periods of g1(n) will have no effect on the result of the computations, it will facilitate the comparisons with the results of the linear deconvolution because now, all sequences are of length N ( = 2M). Let us consider a periodic signal f 2 ( n ) whose period is

{ f o , f l , . . . , f A 4 -1, f o , f l , . . .

1

(12)

fM-11.

) f ( n ) is Thus the deviation of f ~ ( n from f ( n )- f2(72) = {. . . , f - - ’ 2 - f M - 2 , f-1 - fM-l,O,. fiM-.fo,fiM+1-fi,...j.

* *

, 0,

4.)

c

=

h(j)[f(n - j ) -- f 2 ( n- j ) ] , for 0 5 n 5 M - 1 for M 5 n 5 N - 1 e2 (n - M ), (23)

Comparing (22) and (5), we note that the error & ( k ) is a function of [ f ( - n ) - f ( M - n ) ] [ l (-1)”], whereas E l ( k ) (5) is the same function as & ( k ) but of the quantity [ f ( - n ) - (-l)”f(M - ,.)I. This is restated as

+

for k even El ( 5 ) is a function of [ f ( - n )- f ( M - n ) , ( [ f ( - n ) + f ( M - n ) , for k odd (24) and

(13)

& ( k ) is the same function but of 2 [ f ( - n ) - f ( M - n ) ] , for k even {0> for k odd.

We assume that g2(n) is the result of the circular convolution of one period of f 2 ( n ) with h(n) plus modeling errors ga(n) = h(n) 0 f 2 ( n )

+

0

e2(n)

In IN

-

1

where @ is the circular convolution operator. Applying the restoration filter h,(n) whose length L, we get & ( n ) = h,(n) @ g2(n)

0

5n5N

- 1.

(14)

5M (15)

Since g(n) was originally formed by linear convolution of f and h, and j$ in (15) is produced by circular deconvolution f2(n)

# f(rr) 0 5 n 5 M

-1

(16)

in general due to the “wraparound” effect of the periodic convolution. Taking the N-point DFT of (14), we get G2(k) = H ( k ) * & ( k )

+ E2(k)

0

5k5N

-1

(17)

and applying the circular deconvolution inverse filter H , ( k ) = to (17), we obtain

&

@ k ) = HT(JG) . G2(k) = F2(IC) H,(IC) . E z ( k ) 0

+

(18) I k I N - 1 (19)

and thus, the error in the restoration is

EY(IC) := :=

F2(k)

(20)

HT(IC). E2(k)

(21)

P2(IC) -

i.e., it is exactly the same as, the linear deconvolution case (lo), except that here, the term El ( k ) is replaced by E2 ( I C ) . Thus, if E2 ( k ) < El ( k ) ,then E;,””( IC), which is the error in restoration

(25)

We note that exp{Ez(k)} I exp{El(k)} ‘dk, for the following cases: 1) The object f ( n ) is globally stationary. In this case, exp{f(n)) = constant ‘dn, and thus, from (25), exp{Ez(k)} = 0 ‘dk, but exp{El(k)) # 0 for IC odd (24). The same result also holds in the special case when the ob-ject has a constant grey level i.e., f ( n ) = k Vn. 2) L < M and the areas around the boundaries of the object (of width L - 1) are globally stationary. Since L < M and f ( n ) - f 2 ( n ) := 0 for n = 0 , . . . ,M - 1, then (23) becomes e2(n) =

E,”=;’ h ( n ) [ f ( n--)

(II

e2(n - M )

for 0 5 n 5 L - 2 for L - 1 5 n 5 M - 1 forMSnIN-1.

Now, if all the ternns { f ~ - ~ + l , . . . , f i ~ - 2 , f i ~ -and ~ j { f - ~ + 1 , .. . ,f - z , j ’ - l } have the same expected value, then exp{f(n) - f 2 ( n ) }= 0 for -L 1 5 n 5 -1, and thus, exp{ea(n)} = 0 for 0 5 n I N 1. The above assumes that the impulse response is of the shape { h o , . . . ,-l}. h ~ However, if h(n) = {h-y,...,ho,...,h,-r} 2 , thenexp{e2(n)}=0,05 n 5 N - 1 if all the terms in { f-=, . . . ,f o , . . . ,f E } and in { f M - y , . . . , f ~ -. .~ . ,’f,+y> , have equal expected values. In this case, however, we can show from (6) and (2) that exp{el(n)} # 0.

+

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 5, NO. 4, APRIL 1996

614

where

A'

= H,(k) . E 3 ( k ) .

4

Note that (31) is the same as (10) and (21) except for the E3 ( k ) term. We have shown in the previous section that when the areas around the boundaries of the image are globally stationary, the expected error E p ( k ) is less than that of E p ( k ) . For the present case, we show below that when the impulse response h(n) is symmetric, then the expected error E p ( k ) is less than that of E p ( k ) when the areas around the boundaries of the object are only locally stationary. Local stationarity is a more relaxed condition than global stationarity. We show in Appendices A and C that for symmetric h(n), the error term E l ( k ) is given by

___, B'

B

(31)

(L-l)/2

El(k)=

A

h(j)W$ j=1

Fig. 2. Extension of the image by mirroring in the

2

and y directions.

3) The object is truley periodic with period M . In this case, & ( k ) = 0 'dk, i.e., the error vanishes when circular convolution is used. This fact suggests how structural features within the image can be exploited to reduce boundary truncation artifacts. BY I v . RESTORATION

IMAGE

EXTENSION

n=M

AND PERIODIC CONVOLUTION

In this section, we use the wraparound errors of the circular convolution to our advantage so that they lessen the effects of the boundary truncation artifacts. Our proposed solution is to form a new image g 3 ( z , y ) as in Fig. 2. The new image is N x N ( N = 2 M ) pixels. The top right quadrant is the mirror image of the observed g1 (x,y) about the AA' axis, and the bottom half is the mirror image of the top half about the BE' axis. As in Section 11, we shall also use circular convolution. The newly formed image in the 1-D case is

, ,..

g3 (n)= ( 9 0 9 1

' i

j=1

g M -1 i g M - 1 i '

for 0 5 k 5

N - 1, whereas the error term & ( k ) is given by

+

E 3 ( k ) = El(k) W,-'"E+k) and hence (L-1)/2

E3(k) =

h(j)W$ j=1

Ln=-j

(26)

. . i go}.

We assume that the point spread function is symmetric. Let us consider a periodic signal f 3 ( 7 2 ) whose period is {fO,

fli

".

fM-1, fM-1,

" '

(27)

,fO>.

j=1

Thus

ri-i

f(n)-f3(n)=

{...,S-Z

-fl,f-l-

f M - fM-1, fM+l

fO,O,OiO,~~~, - fM-2,.

'

.). (28)

In the frequency domain, using N-point DFT and circular convolution, we have

G3(k) = H ( k ) . F3(k) The circular convolution of h,(n) is

g3(n)

+E3(k).

(29)

with the reconstruction filter

# 3 ( k ) = H T ( k ). G 3 ( k ) 0 5 k 5 N = F3(k) +E?@)

-

1 (30)

(33)

AGHDASI AND WARD: REDUCTION OF BOUNDARY ARTIFACTS IN IMAGE RESTORATION

615

Fig. 3. Image of Bayan: (a) Original; (b) 64 x 64 section cut from the original larger imagel; (c) 64 x 64 truncated section of the blurred image of Fig. 8(a) blurred by the out-of-focus model of the digitizing camera; (d) result of restoring by linear deconvolution; (e) result of restoring by linear deconvolution after windowing the data by a trapezoidal window; (0result of restoring using image extension and circullar deconvolution; (g) difference image of (b) and (c); (h) difference image of (b) and (d); (i) difference image of (b) and (0.

Comparing the expressions for & ( k ) in (32) and E3(k) in (34), we note that they have the same form except that in & ( k ) , every pixel value f ( n ) has been replaced by the difference of two pixel values, where unlike E2(k), the two pixels lie in the same laical neighborhood. Thus, when the object is locally stationary in the region of the truncation boundaries, then exp(&(lc)) = 0 V k , and thus exp(E3)

< exp(E1).

(36)

We therefore conclude that our proposed method of extension of the observed image by reflection followed by circular deconvolution leads to smaller expected errors in the restored image when the truncateld image is locally stationary at its boundaries.

V. COMPUTATIONAL LOAD

For our proposed method (Section 111), we use the N ( = ZM)-point DFT. In the 2.-D case, we can show that the computational load compared with the circular convolution (i.e., without any zero padding) will increase from O ( M 2log, M ) to 0(4M2log, 2M). However, since the resulting extended image g3(n) is real and symmetric, considerable gains can be made. Taking the N-point DFT of (26), after a spatial shift of one half a pixel, we get G3(k) = 2 w Gl(k))

l \

(37)

M-l

= 2Re