Noise reduction filters for dynamic image sequences - Semantic Scholar

Noise Reduction Filters for Dynamic lmage Sequences: A Review JAMES C. BRAILEAN, MEMBER, IEEE, RICHARD P. KLEIHORST, MEMBER, IEEE, SERAFIM EFSTRATIADIS, MEMBER, IEEE, AGGELOS K. KATSAGGELOS, SENIOR MEMBER, IEEE, AND REGINALD L. LAGENDIJK, MEMBER, IEEE In this paper, a thorough review is presented of noise reduction j l t e r s for digital image sequences. Detailed descriptions of several spatiotemporal and temporal noise reduction algorithms are provided. To aid in comparing between these different algorithms, we classrfy them based on their support (i.e., 3 - 0 or I - D j l t e r ) and whether or not motion compensation is employed. Several algorithms from each of the four categories are implemented and tested on real sequences degraded to various signal-to-noise ratios. These experimental results are discussed and analyzed to determine the overall advantages and disadvantages of the four general classijications, as well as, the individual jlters.

I. INTRODUCTION In the past several years, applications utilizing digital video have increased dramatically. For instance, the pervasive use of video conferencing seen today is a direct result of the advances in video processing and the cost advantages provided by digital video. By utilizing digital video, compression algorithms are able to achieve higher compression ratios than their analog counterparts, which reduce the required bandwidth of the transmission channel. Also, through the use of application specific integrated circuits, the cost of hardware required to implement these algorithms is greatly reduced. Other applications that have benefited by the use of digital video include, but are not limited to, multimedia services, autonomous navigation, Manuscript received 1994; revised May 1995. This work was supported in part by NATO under Grant Number 0103/88. J. C. Brailean is with Motorola Corporate Systems Research Laboratory, Schaumburg, IL 60196 USA. R. P. Kleihorst was with the Information Theory Group, Delft University of Technology, Delft, The Netherlands; he is now with Philips Research Laboratories, Eindhoven, The Netherlands. S. Efstratiadis is with the Department of Electrical Engineering, Aristotelian University, Thessaloniki, Greece. A. K. Katsaggelos is with the Department of Electrical Engineering and Computer Science, McCormick School of Engineering and Applied Science, Northwestem University, Evanston, IL 60208-31 18 USA. R. L. Lagendijk is with the Information Theory Group, Department of Electrical Engineering, Delft University of Technology, Delft, The Netherlands. IEEE Log Number 9413089.

motion analysis, object tracking, surveillance, astronomical, and medical imaging. It is mentioned here that this paper addresses dynamic image sequences in general, of which video sequences is a special case. However, both terms are used interchangeably. In conjunction with this increased use of digital video has been the increased demand for noise filtering algorithms. These algorithms are critical, since the presence of noise in a video sequence degrades both its visual quality, as well as, the effectiveness of subsequent processing tasks. For example, the coding efficiency obtained for a particular image sequence is decreased by the presence of noise. This is due to the apparent increase in the sequence’s entropy, caused by the noise. Therefore, filtering methods for reducing this noise are desired not only to improve the visual quality, but also to increase the performance of subsequent processing tasks such as coding, analysis or interpretation. In this paper, we provide a comprehensive review of the important existing noise reduction techniques for dynamic image sequences. We include both a qualitative and quantitative comparison of these techniques. The results from several experiments using real sequences degraded to various signal-to-noise ratios are presented for quantitative evaluation. The degradation model we are using is given by

where f ( i , j , k ) and g ( z , j , k ) denote the original and observed image intensity, respectively, of the kth frame at spatial location ( i , j ) , and n(z,j,k ) is an independent additive Gaussian noise term. It should be noted that (1) implies a 3-D sampling or progressive scanning of the video sequence. This does not however, imply that the filters discussed in this paper are not suitable for use with interlaced video sequences. For instance, the even and odd fields can be processed by these filters as two separate progressively scanned image sequences; or the fields can be merged into a single frame prior to filtering.

0018-9219/95$04.00 0 1995 IEEE PROCEEDINGS OF THE IEEE, VOL. 83, NO. 9, SEFEMBER 1995

The image sequence filtering problem is to find an estimate f ( i , j , k) of the true image sequence f ( i , j ,IC) based on the noisy observation g ( i , j , k ) . The sources of noise which can corrupt an image sequence are numerous. Examples of the more prevalent ones include the noise introduced by the camera, shot noise which originates in the electronic hardware, and thermal or channel noise. The noise introduced by most sources is generally characterized as additive Gaussian, therefore (1) applies. Poisson noise is present in sequences acquired under quantum-limited conditions. Readers interested in the filtering of quantumlimited sequences can consult, i.e., [17], [lX], [%], 1721 and the references therein. The top level classification of the existing image sequence filtering algorithms is based on the dimensionality of the region of support of the filter. More specifically, we consider 1-D temporal filters, and 3-D spatiotemporal filters. The existing filters are further classified into nonmotion compensated and motion compensated. Motion compensation is one of the ways for removing the temporal nonstationarity of the sequence which is due to its dynamic nature. The motion compensated sequence can now be modeled as a temporally stationary field. Alternatively, a filter can be designed based on a nonstationary model of a sequence in the temporal (implicit motion compensation), as well as, in the spatial direction. The switching of one filter to another can for example take place at the location of spatial and/or temporal edges. Additionally, the filters are grouped into finite impulse response (FIR) and infinite impulse response (IIR) filters. We have chosen this classification, which although clearly not unique, has been found to be very natural and effective in the presentation of this material. In the last two decades an enormous amount of research in the area of image processing has focused on the problem of restoring 2-D images. The result of these efforts is a large and diverse collection of restoration techniques for a variety of problems (for a recent review and classification of image restoration algorithms, see [47]). In filtering each frame individually it is implied that an image sequence is temporally independent. However, by ignoring the temporal correlations that exist in an image sequence, suboptimal estimates are obtained and the spatial filters tend to introduce temporal artifacts into the filtered sequence. These artifacts, which generally appear as abrupt changes in the intensity values, result from different luminance estimates of the same pixel in different image frames. The oversmoothing of an image sequence is a second type of artifact that can occur from spatial filtering. Therefore, due to the limited usefulness of 2-D filtering techniques in image sequence filtering, and the numerous reviews already existing on the filtering of still images, 2-D filtering techniques are not further discussed in this paper. Spatiotemporal filters take advantage of the correlations that exist in both the temporal and spatial directions. Therefore, they tend to be less sensitive to the nonstationarities in both the spatial and temporal directions. However, a strictly stationary assumption can result in both spatial as well as

m1.y lnpvt s-.Dee

ri1ter.d Output 6e.qu.w. IllteIUitY ri1t.r

c

--1

m

mtim z.tiutor

-

I

+ -1

temporal artifacts. The two main approaches in avoiding these artifacts are adaptivity and motion compensation. Clearly, 2-D spatial and 1-D temporal filters may be thought of in most cases as special cases of 3-D spatiotemporal filters. However, in keeping in agreement with the way the various algorithms appeared in the literature, 1-D temporal filters will be presented separately from the 3-D filters. Temporal filters avoid the artifacts introduced by spatial filters by modeling the image sequence as a series of 1D pixel trajectories. That is, the intensity at each spatial position is considered as a 1-D signal that transverses along the temporal axis. Therefore, temporal variations in the same pixel are avoided. The oversmoothing of spatial boundaries is also avoided by assuming that the pixels of an image frame are spatially independent. However, due to motion the characteristics of the temporal correlation vary. Therefore, the temporal signal is a nonstationary signal. If stationarity is assumed in developing the temporal filter, then smoothing artifacts, similar to those introduced by spatial filters, will be introduced around the moving objects in the sequence. In general, there are two approaches in addressing the nonstationarity of the temporal signal. The first, is to make the filter adaptive, and the second is to remove the nonstationarity by motion compensating the image sequence. Both approaches are successful in reducing the motion artifacts; yet there are difficulties associated with each of them. These difficulties are discussed further in the remainder of this paper. As described above, the second level in the classification of the filtering algorithms is determined by the use (or not) of explicit motion information and compensation (motion information is implicitly utilized by a number of algorithms). The use of a motion estimator is therefore required. The problem of estimating the 2-D motion components in an image sequence is a very challenging one, due to its ill-posedness [7]. Furthermore, the problem is aggravated due to the presence of noise. Ideally, as shown in Fig. 1 , an estimator which simultaneously estimates the motion and intensity values is desired [ 111. However, most motion compensated (MC) filters in the literature treat the motion estimation and filtering, as shown in Fig. 2, as two separate problems which are solved sequentially. The performance of the MC noise smoothing filter clearly depends on the accuracy of the motion estimator. This dependence, however, is not the same for all filters. In other words, the

BRALEAN et al.: NOISE REDUCTION FILTERS FOR DYNAMIC IMAGE SEQUENCES: A REVIEW

1213

is a stationary process. This can result in a filter that severely degrades a sequence in regions where this assumption fails (spatial as well as temporal edges). Therefore, motion compensation and/or adaptivity are two extremely important components of a spatiotemporal filter. We review several nonadaptive and adaptive, nonmotion compensated and motion compensated spatiotemporal filters. Fig. 2. Standard approach to motion and intensity estimation.

use of a more accurate motion estimator might improve the performance of one filter considerably more than the performance of another filter. Furthermore, this relationship (between the accuracy of the motion field and performance of the filter) cannot be determined analytically, but instead needs to be determined experimentally. Due to this reason, we were faced with the problem of choosing the appropriate motion estimator for the algorithms to be tested. For purposes of consistency we chose to use only two motion estimators, even though a different motion estimator might have been used when the algorithm was initially proposed. Furthermore, we do not consider filters where a preprocessing smoothing step is performed for improving the performance of the motion estimator. This is beyond the scope of this paper. The goal of this paper is to provide a comprehensive review of the currently available noise reduction algorithms. In trying to meet this goal, we include in Sections I1 and I11 a detailed description of several spatiotemporal and temporal noise reduction algorithms. These particular algorithms were chosen for this review due to either their popularity and/or the uniqueness of their approach. To aid in distinguishing between particular algorithms, we further subdivide both the spatiotemporal and temporal algorithms into motion compensated and nonmotion compensated. As was mentioned above, this classification approach is not unique. However, we have found it to be very effective in the organization of this review, especially when attempting to analyze the performance of each algorithm type. A final analysis based upon an algorithm’s performance as measured by the experimental results found in Section V is provided in the concluding section. It should be noted that due to practical considerations not all algorithms described in Sections I1 and I11 are implemented and tested experimentally. Instead a representative group of algorithms was selected and tested from each category.

n.

SPATIOTEMPORAL FILTERING OF IMAGE SEQUENCES

As mentioned in the introduction, spatiotemporal filters represent the most general class of filters for removing noise from an image sequence. That is, both spatial and temporal filters can be considered as a special case of a spatiotemporal filter. By taking advantage of the correlations that exist in all directions of a video sequence, spatiotemporal filters potentially can provide higher noise suppression than their 1-D and 2-D counterparts. To utilize this correlation, many spatiotemporal filters assume that a video sequence 1274

A. Nonmotion Compensated Spatiotemporal Filtering Many of the nonmotion compensated spatiotemporal approaches currently available were developed by generalizing well known 2-D filtering techniques. For example, a simple approach to spatiotemporal filtering is to extend the support of a 2-D finite impulse response (FIR)filter in the temporal direction. The resulting 3-D weighted averaging filter is defined as

where S denotes the 3-D support of the filter and w ( p , ql 1) the filtering coefficients or weights indicating the relative importance of each element contained in the spatial and temporal extents of the filter. Various filters can be derived depending on the extent of S and the way the coefficients w ( p , q, 1) are defined and adapted. An averaging filter is the simplest form of (2), according to which the input elements are assumed to be of equal importance (i.e., w ( p , q , l ) = C , a normalizing constant). For this specification of the weights, the amount of the noise reduction is dependent on the length of the filter. That is, the larger the averaging window, the greater the reduction in noise power. However, in areas where motion has occurred or spatial edges are present, the averaging of these pixels can cause severe artifacts. Due to these artifacts, objects appear blurred in the filtered sequence. In these areas, averaging should be kept to a minimum. Therefore, a tradeoff exists between the noise suppression capabilities of the filter and the severity of artifacts introduced into the estimated sequence. To avoid these artifacts while maintaining good noise suppression, the weights w ( p lq, 1) can be made adaptive. For instance, in areas where motion is detected the weighting given to the frames farthest from the frame of interested should be lessened. This will reduce the amount of temporal smoothing applied to these areas. However, for the areas where the motion is small, all frames can be considered of equal importance. A similar procedure could be used to adapt the spatial weights when the filter is operating near a spatial edge. A stationary choice of the filter coefficients w ( p ,q , 1) is represented by the 3-D Wiener filter. It was used in [68] to remove noise and a deterministic degradation from an image sequence, using computational techniques reported in [33], [48]. Equation (2) can also be written and implemented in the 3-D discrete frequency domain. There are two major disadvantages with this approach; the first is the requirement that the 3-D autocorrelation function for the PROCEEDINGS OF THE IEEE, VOL. 83, NO. 9, SEPTEMBER 1995

:IL ............

..............

”

‘1..........................................................................................................

...........-................. ^..

”

I

frame

frame

a ,

I

I

I

I

i

30

40

50

60

70

-1 -1 m , 21 30 40 50

60

70

21

- PE

MSOS

--

OS

SNRi (db) for 2QdB Trevor

1

0

0’

frame

- PE

..I.. MSOS

--

OS

SNRi (db) for 30dB Trevor

.----

-ll

...................

-3

-5-

.............................. ...... .“..............................................................

.........

-5

I

-7

-

frame I

I

I

frame

-7

1

21

- PE

30

40

...... MSOS - - 0s

50

60

70

Fig. 3. Experimental results for the nonmotion compensated spatiotemporal filters.

original sequence is known a priori. The second is the 3-D stationarity assumption. This assumption, especially in the temporal direction, is detrimental to the performance of the 3-D Wiener filter. An adaptive choice of the coefficients w(p, q , I) is based on the contour plot method [20]. The filter coefficients are adapted by assigning large weights if g(2 - p , j - q , IC - I) belongs to the same object as g ( i , j , IC). The contribution of this method is how it checks whether pixels belong to the same object. This is done by attaching an altitude function a ( p , q, Z)V(p,q , 1) E S to the current point. The value of the altitude measures the minimal effort of climbing from g(Z,j, IC) to g(2 - p , j - q , IC - I ) :

(3)

The conversion from altitude a ( p , q, I) to the filter weights w ( p ,q , r ) is done by means of the function

where the parameter c serves as a normalization constant and a and /3 are tuning parameters. The results gained by the contour plot method are good for sequences that are corrupted with light to moderate level of noise. Too severe a noise level will cause a degradation in the performance of the edge detector, which in turn will effect the quality of the filtered sequence. A quadratic FIR Volterra filter was used by Chan et al. [ 161 in an attempt to model the nonstationarities of an image sequence. The filter is described by

which involves the minimum over a certain set of “no return”paths P leading from (2,j,IC)to ( z - p , j - q , I C - I ) . The function “edge” denotes the edge value at g ( r ) in the direction of O ( T 1).The set of paths is limited to avoid laborious calculation efforts.

+

. g(2 - p , j

- q , k - Z)g(2 - p / , j - ql, IC

(5)

BRAILEAN et al.: NOISE REDUCTION FILTERS FOR DYNAMIC IMAGE SEQUENCES: A REVIEW

_ _ - ~

~~

- Z/)

~

1275

~-

~

SNRI (dB) for lOdB Trevor

SNlU (db) for lOdB mobile

’1

7 1

5 1

frame 4 ,

I

I

1

20 MC-SPE

10

- MC-PE

11

1

I

30

- - MC-ASPE

40

- MC-PE ’1

-1 ,.a .

.........-...~“*.,......p”.v..’ .......-...../”..... .....

, . e . . .

frame I

I

10

20

- MC-PE

..... MC-SPE

. ......

. . ..e e

.e..***

1’

i

I

04*1-

0.6

o-a

frame I

-

10 MC-PE

*.I*

I

I

20

30

MC-SPE

710

”.,

.......-..-....

8.

8.:;

Y

’

*

I --I I

I*

frame I

I

I

i

I

..

1

1

.

.....i..AJ ‘

1 1

30 40 MC-ASPE

SNRL (db) for 3OdB mobile OS6

frame

60

..... MC-SPE ..MC-ASPE

2

1

’

5I0

SNRI (db) for 2OdB Trevor

SNlU (db) for 2odB mobile

3 1

40 I

30 I

- - MC-ASPE

i 40

SNRi (db) for 30dB Trevor 1

......... ,4*..*”‘.& .. .... $ ..

\ f.., p’

..

,*..-*-

.. $ k - l ( i , j and ) J E i k - ' ( i , j ) are the horizontal and vertical components of an estimate of the motion between frames k and k - I . By motion compensating the sequence prior to averaging, the temporal stationary assumption holds over a much larger region. This allows for the support of the filter to be increased in the temporal direction, improving the filter's ability to suppress noise without incurring additional artifacts due to motion. Another version of (13) was proposed by Kalivas and Sawchuk [44]. This method is based on a segmentation of the image sequence into objects and background. That is, each pixel in the frame of interest is classified as either belonging to an object or to the background. Only pixels classified as objects are motion compensated. The weights w(p, q , 1) are adjusted by determining the number of pixels within the filter's support that have the same classification as the pixel under consideration. Only pixels with the same classification are used in determining the average. A disadvantage of this filter is its dependence on the quality of the segmentation, which degrades with the seventy of the noise. Another adaptive version of (13) was proposed by Ozkan et al. [68]. For this adaptive weighted averaging (AWA) filter the weights are defined by (see (14) at bottom of page) with K a normalizing constant, and a and E tuning coefficients. If the difference between the current pixel and the pixel at location ( i - p , j - q, k - I ) in the filtering mask is less than E , then g ( i - p , j - q , k - 1) is used in the averaging filter. To protect against the effects of noise, t2 is set to twice the value of the noise variance. The parameter a is a penalty parameter controlling how rapidly a weight should reduce as a function of the mismatch between image values. If a = 0 the filter mimics the spatiotemporal average; if a is large, then 1 ae2 !z a c 2 and the AWA filter performs averaging only over the matching image values. Motion compensated recursive spatiotemporal filters are natural extentions of the 3-D nonmotion compensated counterpart of (6) and (7). Equation (6) remains the same in this case. However, the prediction is performed according to

where [ul, U ' , us]= [z,j,k],and f ^ ( u qis ) the 1-D estimate of f(z,j,k) in the direction of uq. Each of the gain terms aq(uq)is obtained recursively, with steady state being reached in only a few recursions. The value of the gain at steady state is dependent upon the correlation coefficients pq and the signal-to-noise ratio (SNR). A closed form expression for the gain is given by [25], [46]

+

q = 1,2,3.

The edge adaptive spatiotemporal version of the filter in (16) was proposed in [26], [46]. In this approach, the input sequence was modified according to

Such a filter was implemented by Katsaggelos et al. [46] and is referred to as the MC 3-D point estimator. The effect of the motion compensation is that the stationary assumption of the image sequence in the temporal direction becomes more valid. In other words, if the correlation model of (8) is used, the temporal correlation coefficient pt approaches the value of one. A variation of the MC 3-D point estimator was considered in [46].

where wq(uq) is a weight which depends on the output of an edge detector and cq is a scaling parameter. Replacing y q ( u q ) in (16) with G(uq)results in the edge adaptive filter. This filter can actually be characterized as edge enhancing, since it enhances the edges based on the output of an edge detector, once in steady state. Therefore, the performance

n W ( P , q1I)

=

^k,k-l

1 + a ( M A X ( E ' , ( g ( i , j , k )- g ( i - p - d,

(i - p , j - q ) , j - d;'k-l(i- p , j - q ) , k - I ) ) ' )

BRAILEAN er al.: NOISE REDUCTION FILTERS FOR DYNAMIC IMAGE SEQUENCES: A REVIEW

(14)

1279

of the filter is dependent upon the performance of the corresponding edge detector. Recently, Markov random field (MRF) models have gained in popularity as a method of modeling image sequences. This increase in popularity is due to the flexibility offered by MRF’s in expressing properties of the original image sequence with the use of probability distributions. It is known [34] that a Markov random field f’ is completely characterized by its Gibbs distribution, which is given by

spatiotemporal noise filter. The terms required in (20) for determining the MAP estimate were defined in [38] as

where K is a normalizing constant and p is a parameter that reflects the activity of the sample field. The “energy function” U(f’(z,j, k)) is defined as

where p ( g ( z , j , k ) ] f ’ ( i , j k)) , results from a Gaussian noise assumption combined with the observation model given in (1) and 0%is the variance of the term inside the bracket in (21). The energy function U ( f ’ ( i , j ,k)lfp(z7j,k)) required for P(f’(z,j,k)l f p ( i , j 7 k ) is ) used to model the piecewise continuous characteristics of an image sequence. That is through this term the amount of smoothing applied in a particular region is varied depending upon the amount of spatial and temporal ^activity. Specifically, the potential functions V(f’(z,j, k)lfp(i7j, k)) are defined as

dg(z,j7

k)lf’(i>j> k))

and

r,s,t€C

where the clique system C is a local set of pixels connected to the pixel of interest. It is through the “potential functions” V (f’) that relationships between local pixels of an MRF are characterized. The obvious advantage for using MRF models to model an image sequence is that the Gibbs equivalence provides an expression for the a priori distribution. Such an expression in general is difficult to obtain, yet crucial in solving for the maximum a posteriori (MAP) estimate. The problem of finding the MAP estimate f(z,j,k) from a noisy image sequence g ( i , j , k) can be described as

f(i7j,

+

f$z)p(f’(i7j7k)Ig(z,j>I C ) ,

fp(z,j,

IC))

(19) where f p ( i , j ,k) denotes all previous estimates up to the current estimate f(Z,j,k). Rewriting (19) using Bayes’ rule results in

where the conditional probability density function p ( g ( i , j , k)lf’(z,j, k),f p ( i 7 j k)) , is obtained from the observation model (1) and the conditional discrete probability P ( f ’ ( i , j ,k)lfp(i7j,k)) from the Gibbs equivalence. Such an approach was taken by Hong and Brzakovic [38] in the development of their motion compensated

W ( i , j , P , n) = 1280

with (see (24) at top of page) where the coordinates ( p , q ) are used to pick out neighboring pixels contained within the corresponding clique system C. The effect of these terms are to “encourage” or “discourage” the use of information from neighboring pixels contained in the clique system C. That is, if the difference between the motion compensated estimates f ( z - dkik-’(z,j),j - d : l k - ’ ( i 7 j ) , k - 1) and f(p - d$”’(p, q ) , q - d;>”’(p, q ) , k - 1) is small, it is assumed that the strong correlation exists between pixels ( i , j ) and ( p , q ) , which indicates that the filter should provide smoothing in this direction. However, if the opposite is true indicating the presence of an edge, the filter should not smooth in this particular direction. The spatial adaptivity required to control the amount of smoothing applied in any particular region is provided by w ( i , j , p ,q ) . The prediction error between f ’ ( i , j , k) and f ( z - d ~ ~ ~ - z ( z 7 j ) - d ~ ~ ~ - zk-1) ( z 7 jis) 7also assumed to be a Gaussian random variable, with variance a&(i,j , k). The parameter CY controls the influence of the previous frame. Hong and Brzakovic proposed solving (20) using iterative condition modes. A disadvantage with the above approach is that the energy function U ( f ’ ( i , j ,k)lfp(i,j,k)) is highly dependent upon

255 - If(i - dilk-’(Z7j),j- d : l k - ’ ( z 7 j ) , k - 1) - f ( p - d $ i ” - l ( p , q ) , q - d ~ ~ ” - l ( p , q ) ,-k 1)1 255

(24)

PROCEEDINGS OF THE IEEE, VOL. 83, NO. 9, SEPTEMBER 1995

~

_____

the accuracy of the estimated motion or displacement vector field (DVF). However, since the DVF is estimated in a separate step there is no mechanism to locally verify the accuracy of these displacement vectors prior to their use in the filter. Brailean and Katsaggelos [lo], [ll] have proposed a recursive, model-based MAP estimator that simultaneously estimates both the displacement and the intensity fields from the observed image sequence. The advantages of estimating these two fields simultaneously, is that information is made available to the estimator about the reliability of estimates that it is dependent upon. This is critical, since to estimate either of these two fields reliably requires an estimate of the other. To model both the displacement and intensity fields the authors developed MFW models. Specifically, to model the DVF the vector coupled Gauss-Markov (VCGM) model was used [12], while the intensity field was modeled using the spatiotemporal coupled Gauss-Markov (STCGM) model [lo], [ll]. The VCGM and STCGM models are defined in terms of the reduced-order state vectors, Sd(i,j , IC) and S f ( i ,j , I C ) [47], respectively. The models are given by

and C ' f ( e > j i l i )are matrices of the corwhere A'd(iyj>k) responding prediction coefficients, d(i - 1 , j 2,IC) = [d,(i - l , j + 2, k ) , d,(i - 1 , j 2, k ) y , W y > j q i , jIC), and W'f ('73,') ( i lj , k ) are Gaussian noise components, and

+

+

p d , ,L?f and U f contain the elements required to complete the equality. It should be noted that the temporal support of the STCGM model is contained in Uf. Both models are dependent upon the value of the line processes, Zd(z,jl k ) and Z f ( i , j lk ) . The components of Z d ( i , j , IC) and Zf(i,j,IC) are binary random variables called line elements. Each line element takes on a value that indicates the presence or absence of a discontinuity or edge in the field. With the knowledge of the line process, a set of prediction coefficients is chosen that excludes the points that are separated by this discontinuity. The estimates of the displacement and the intensity fields are obtained by maximizing the joint a posteriori probability density function with respect to the intensity field S f ( i l j I,C ) , the displacement field S d ( i l j ,IC), and the corresponding line processes Z f ( i , j lk ) ,Z d ( i , j , I C ) . Therefore, the joint a posteriori pdf is given by P ( S d ( i , j , IC), W I . i , k ) , Z d ( i > . A IC), Zf(i,j,k)lg(4.7., I C ) ) . This simultaneous approach can be formulated as

BRAUEAN

et

Utilizing Bayes rule, the conditional joint a priori distribution can be isolated, as was done in (20). From the Markovian assumption used in developing the VCGM and STCGM models, it is known that the a priori distribution is also Gibbsian. The exact energy functions defined by Brailean and Katsaggelos are given in [l 11, [12]. Knowledge that both the additive noise in the observation model as well as the uncertainty in the VCGM and STCGM models are Gaussian was used to solve (27). This resulting set of filtering equations are similar to that of the Kalman and extended Kalman filters for Sf and S i , respectively. By formulating a simultaneous estimator for the DVF and the intensity field, the simultaneous displacement and intensity estimation (SDIE) algorithm couples the two estimation processes. This coupling allows information regarding the accuracy of the estimates {o be passed between the estimators. The estimates Zd and 1, are determined through the use of a decision rule that chooses the most probable state for each, based on the neighboring line elements. The algorithm also allows for the accurate estimation and segmentation of the DVF from noisy-blurred sequences, which in turn can be used for other purposes, such as compression [5], analysis, or interpretation [12]. OF IMAGE SEQUENCES 111. TEMPORAL FILTERING To fully exploit the noise reduction capabilities of a spatiotemporal filter without introducing into the sequence visually degrading artifacts, requires that the filter be adaptive both spatially and temporally. Such a filter, however, can be very costly in terms of its computational requirements. Previously, these costs have prevented many adaptive spatiotemporal filters from being used in certain applications, such as real-time video processing. This was one of the reasons that most of the early filters were 1-D temporal filters. These filters offer reasonable noise suppression, especially in temporally static areas of the sequence, where the human viewer has a high sensitivity to noise. In this section, we discuss several nonmotion compensated and motion compensated temporal filters. As a general comment, all 1-D version of the filters presented in Section I1 represent valid temporal filters. The ones that have already appeared in the literature, are discussed next.

A. Nonmotion Compensated Spatiotemporal Filtering As previously mentioned, the problem of estimating the motion in an image sequence is further compounded by additive noise. Due to the lack of robust motion estimators, early attempts at temporally filtering image sequences were restricted to simple frame averaging techniques [41]. Such a filter results by restricting the region of support S of the 3-D fi!ter in ( 2 ) to the temporal direction. That is, an estimate f ( i , j , k ) is obtained by

al.: NOISE REDUCTION FXLTERS FOR DYNAMIC IMAGE SEQUENCES: A REVIEW

1281

SNRi (db) for lOdB mobile

5 -

SNRi (dB) for lOdB Trevor

.-- - .--. .'- - ., ................. ~. .... cc

_--__-_----.--____-a

......... .... ./...-'...*

- _ _ - I - - - - _

..............................

................

...........

a..\

"

4

,.'

t

1-

1-

frame

frame 0 1

I

I

I

10

20

30

1 40

0

I

1

i

I

I

SNRi (db) for 20dB Trevor

.--------. ._____-------__ _ _ - - I

0

-1

4 ....... .........................................

- TM

-5

I

I

I

10

20

30

..... TA ..TLS

I 40

1 YGiiie

.-.TA

h..

i' ."...." frame

30

- TM .....

I

I

1

I

50 TA ..TLS

60

70

s'o

$0

7'0

40

1

-" 21 I

-TM

.........

I""'

%

I

21

-5

................... ..........................................................

.

'., ..............

-2 -4

............ frame

1

-1

..."*

io

io

--TLS

Fig. 6. Experimental results for the nonmotion compensated spatiotemporal filters.

where M is the causal and N the noncausal extents of the averaging window and w(1) are the weights or the filter coefficients. Generally, the elements of an averaging filter were assumed to be of equal importance (i.e., w(1) = l / ( M + N + 1)).For this choice of w(Z) the amount of the noise reduction is limited by the length of the filter. The larger the averaging window, the greater the reduction in noise power, at the same time however, the more visually noticeable the artifacts resulting from motion within the sequence (as discussed in Section 11, this same tradeoff exists for the spatiotemporal averaging filter as well). These motion artifacts can be avoided by adapting the filtering weights ( ~ ( 1 ) according ) to the amount of motion detected in a particular area of the image sequence. Huang and Hsu [41] used (28) with implicit motion compensation. The idea behind implicit motion compensation is to always apply the temporal filter in the direction of highest correlation. This direction is chosen by searching over a finite number of directions and selecting the one with the smallest variance estimate. The results provided in [41] indicate that the implicit motion compensation was more robust than the explicit motion estimator used in the study, moderate noise cases. 1282

A disadvantage of the temporal averaging filter is its need to buffer several frames of an image sequence. For many applications this large memory requirement is prohibitive. A recursive temporal filter has the advantage that, in general, fewer frames are required to be buffered. Furthermore, the adaptability of these filters can be easily achieved, since there are fewer parameters to control. The general form of a recursive 1-D temporal filter is the same as (6). It is repeated here for ease of presentation,

f(4.L IC)

= (1 - 42,i k ) ) f b ( L k IC)

+ 44.Ak M 4 i IC).

(29) The prediction f b ( z , j l IC), is formed using only past estimates in the temporal direction. For example, using pv = p h = 0 a 1-D temporal filter described by (29), (7)-( 10) results. Using (29), McMann er al. [61], Dennis [21], and later Crawford [ 191, developed adaptive temporal filters for reducing the camera noise present in video and television sequences. In each case, the predicntion is set equal to the previously filtered frame, that is, fb(i,j, IC) = f(z,j,k l), while the gain is used to control whether the filter PROCEEDINGS OF THE IEEE, VOL. 83, NO. 9, SEPTEMBER 1995

..

.................. -.-.-, c,m.II* .............-.-:. . e . . , . '

.

-.-.-*""-"r.

\,,-.

---%

,

e----

\-e\-*-,,.#-.

-#,.

-,.* frame

-1 1

1

I

1

i

10

20

30

40

frame -1 21

I

I

I

I

1

30

40

50

60

70

SNRi (db) for 30dB Trevor

SNRi (db) for 30dB mobile

........................................................ .... .......".....I .................. ................................ . I . . . . . . .

I

Fig. 7. Experiemental results for the motion compensated temporal filters.

removes noise or forwards the observation. For instance, if a(i,j , I C ) = 1 the observation g ( i , j , IC) is forwarded; however, if a(ilj,IC)= 0 then the previous frame is forwarded. An advantage of this approach is that the response of the filter can be changed by varying a single parameter. If a(z,j,IC) is chosen to provide a low enough cutoff frequency, then the ability of this filter to suppress noise can surpass that of the nonrecursive averaging filter, while requiring only a single frame storage. Therefore, the choice of a(i,jlIC)is critical in determining the performance of this type of filter. Dennis [21] proposed determining a(z,j,IC) through the use of a global optimality criterion. The disadvantage of such an approach is the underlying assumption that the image sequence is stationary in the temporal direction. Due to this assumption, artifacts similar to those associated with an averaging filter are introduced into the filtered sequence. To alleviate this problem, McMann er al. [61] and Crawford [ 191 proposed using motion detection to selectively apply the filter to areas considered to be temporally stationary. This adaptive approach for determining a can be described

where the threshold T is used to suppress the effects of the noise in the detection process. For this configuration, when the error, (g(z,j,IC) - f ( i , j , k - 1)). is large, indicating motion, the filter is disabled to avoid introducing artifacts in these regions. For the areas where the motion is small or nonexistent, a(z,j,IC) is set equal to ag.In the work by McMann et al. [61], ag was set to zero, thus removing any averaging from the filter. Therefore, no real noise reduction was achieved for this choice of asunless the previous frame was not degraded. The approach taken by Crawford was to adapt ag based on the difference between g ( z , j , IC) and f ( i , j , k - 1). That is,

BRAILEAN et al.: NOISE REDUCTION FILTERS FOR DYNAMIC IMAGE SEQUENCES A REVIEW

~~~~~

~

~~

~

1283

where /3 is chosen to suppress the effects of noise and y is used to adjust the amount of smoothing applied by the filter. Clearly, when the difference between two adjacent frames is small, indicating little motion, ag is set so that the response of the filter is altered to provide more smoothing. However as the error in the frame difference increases, the response of the filter begins moving to that of an all-pass filter. In general, the disadvantage of these types of recursive adaptive filters, is that the noise cannot be reduced in the moving areas. Although the noise in these areas is masked by the human visual system [67], it will still present a problem in slowly moving objects, or in objects that start and stop within an image sequence. An FIR filter which has the form of (29) has been proposed by Martinez and Lim [60]. In this method, the prediction is represented by the local mean, that is, f b ( i , j , k ) = ,Gf(z,j,k ) , while the gain is given by (1 1). As was mentioned in Section 11-A, a recursive form of this filter is derived by using an AR model with equal coefficients. In [60] the local statistics were computed using a noncausal 1D window. To avoid motion related artifacts, Martinez and Lim chose to implement their filter using implicit motion estimation. Therefore, when the filter is able to track the direction of greatest correlation (i.e., c ; ( i , j , k ) is small), the estimate reduces to ,Gf(i,j , k ) . For the case of additive white Gaussian noise, this is the minimum mean square error estimate. If the filter is unable to track the motion, then a j ( i ,j , k ) is large and the filter is disabled. Although implicit motion compensation was employed for its noise reduction capabilities, it is clear that this characteristic is limited to image sequences containing only small amounts of very simple motion. An alternative class of nonmotion compensated temporal filters consists of the ordered statistics (OS) based filters. OS filters operate on the ordered values within the observation window, ignoring correlation and time information in favor of magnitude information. An early attempt at applying OS to the problem of image sequence filtering was the straightforward temporal median filter proposed by Huang and Hsu [41] and later by Naqvi et al. [64]. This approach can be described as

For data corrupted by Gaussian noise the sample average approximates the maximum likelihood estimate. However, because of the nonstationarity of the temporal signals, local distributions within a window are generally not Gaussian. Therefore, a sample median filter provides a better estimate of the original signal. Temporal median filters are very suitable for removing “impulsive” noise. This is because most of the temporal signals not containing noise are root signals of the median filter [64]. However, temporal median filters are not suited for sequences with a high amount of noise. First, this is because the temporal signals behave Gaussian for high noise levels, and therefore taking the temporal average will be superior to the median filter. 1284

Second, temporal median filters introduce artifacts at noisy spatiotemporal edges. The estimated “position” of the edge can, because of noise, differ for various points along the line. This manifests itself as artifacts on the edge in the resulting sequence. Kleihorst et al. [46], [49] have used adaptive OS based estimators along the temporal direction. The function of these estimators was to estimate the nonstationary part of the temporal signal. Then, the signal was decomposed into its stationary and nonstationary part. Since it was assumed that the noise was a stationary signal and therefore mapped to the stationary part, only this part was filtered by a recursive least squares noise filter. Finally, the filtered stationary part and the nonstationary part were combined again to produce the overall estimate.

B. Motion Compensated Temporal Filtering To avoid the problems associated with nonmotion compensated temporal filtering, several approaches have been developed that motion compensate the image sequence prior to temporal filtering. For instance, the motion compensated version of (28) is given by N

f(Z,j,k ) =

~ ( ~ ) g ( i - ~ ~ l ~ - l ( i , ~ ) , j - ~ k-l). ~l~-~(z,j), l=-N

(33) Huang and Hsu [41] used (33) with equal weights. A differential technique was used for estimating the motion. These displacement vectors are then used to motion compensate the corresponding frames. As already mentioned implicit motion compensation proved to be more robust than explicit motion estimation in this case [41]. The 1-D temporal version of the AWA filter in (14) was also used in [69]. A problem with the above approach is that in the temporally stationary areas, motion compensation tends to color the noise. This is a result of the motion estimator’s attempt to find the best match between corresponding noisy areas. To prevent this, Boyce [14] proposed to switch adaptively between motion compensated and nonmotion compensated temporal averaging. This algorithm is the 1-D version of the 3-D spatiotemporal algorithm discussed in Section 11-B. Using a block matching algorithm, Boyce proposed switching between these two options based on the mean absolute difference (within each block) before and after motion compensation. Specifically, motion compensation is not used if

M A D , < ,OMADmc

or if

MAD, < T

(34)

where MAD,, and M A D , are the mean absolute differences obtained with and without motion compensation, respectively. The parameters T and /3 are determined based upon a particular probability of false alarm and noise variance. In an approach similar to Boyce’s, Kalivas and Sawchuk [44] proposed an algorithm in which the image sequence is segmented and motion compensated prior to temporal averaging. From this segmentation of the image sequence, PROCEEDINGS OF THE IEEE, VOL. 83, NO. 9, SEPTEMBER 1995

Table 1 Summary of the Filters Implemented in This Study With 1 = Nonmotion Compensated Spatiotemporal Filter, 2 = Motion Compensated Spatiotemporal Filter, 3 = Nonmotion Compensated Temporal Filter, 4 = Motion Compensated Temporal Filter Filter Description Least Squares Point Estimator Multistage Ordered Statistics Ordered Statistics Motion Compensated Adaptive Weighted Averaging MC Point Estimator MC Separable Point Estimator MC Adaptive Separable Point Estimator Simultaneous Displacement and Intensity Estimator MC Ordered Statistics Temporal Average Temporal Median Temporal Least Squares Adaptive Weighted Averaging MC Temporal Point Estimator MC Adaptive Nonlinear MC Temporal Least Squares

MC-AWA

Classification 1 1 1 2

Motion Estimation NIA NIA NIA BM

Equation Number (7) (32) (11) (14)

Reference 1451 [41 1541 [691

MC-PE MC-SPE MC-ASPE

2 2 2

PR PR PR

(7) (16) (16)

1451 [461 [461

SDIE

2

PR-sim

(26)

[111

MC-OS TA

2 3 3 3 4 4 4 4

BM NIA NIA implied BM PR PR BM

(11)

1541 [411

Filter Acronym PE MSOS

os

TM TLS AWA MC-TPE MC-AN MC-TLS

each pixel is labeled as either belonging to an object or as background. Motion estimation and compensation are performed only for the pixels that belong to an object. Pixels classified as background are temporally averaged along the same spatial coordinates (no motion compensation), while pixels belonging to an object are filtered along the estimated motion trajectory. In all cases, the above proposed motion compensated temporal averaging filters provided filtered sequences with fewer artifacts, as compared to their nonmotion compensated counterpart (see (28)). However, these proposed filters still suffer from similar disadvantages as the nonmotion compensated temporal averaging. First, the noise reduction capability of these filters is dependent on their length. Secondly, these approaches can introduce artifacts in the areas where motion is not tracked, especially in the newly exposed or occluded areas. It should be noted that these two problems are inversely related. That is, as the length of the filter is increased, so is the degradation to the estimated image sequence caused from motion related artifacts. Katsaggelos et al. [45] proposed a multiple frame input temporal motion compensated filter. It is the 1-D temporal version of (15), that is described by (29) with the prediction given by

(28) (32) (11) (14) (11) (36) (1 1)

[MI 1601 [681 [451 ~ 3 1 [601, [731

specifically, the gain is defined according to

f

ab,

if IA(i,j,&I

5 7b

if lA(Zljld)l> 7,

where d = [ ~ ~ 1 ~ - ' ( Z 1 j ) , ~ ~and ~ ~lA(Z,jld)l - ' ( i , ~ )=] ~ Jg(i,j,IC) - j ( i - @ J ( i , j ) , j - @ y i , j ) , I C - 1)l. Clearly, when the motion is tracked, the filter has a cutoff frequency determined by ab. As the prediction error increases, the filter smoothly transitions to an all pass frequency response and no filtering is performed, since a, obtains a value close to unity. The motion estimator that they employed in their filter was the one developed by Netravali and Robbins [65], [66]. As was the case with the differential motion estimator mentioned above, the performance of the Netravali and Robbins motion estimator degrades significantly in the presence of noise. Therefore, the Dubois and Sabri filter performs well when the noise power is small; however, for more severe noise cases a more robust estimator is desirable. There have been several modifications to the Dubois and Sabri recursive filter since its introduction. For instance, Reinen [70] proposed a second order variation of this filter for use in L removing Gaussian noise from an NMR image sequence. f&j, IC) = ~ y ( z ) f ( z - ~ ~ ~ ~ - ~ ( z , j ) , j - ~ IC-l) ~ ~ ~ - ~Similarly, ( 2 1 j ) Martinez's , filter [60] has also been modified in an k 1 attempt to improve its performance. For instance, in [73] (35) a robust explicit motion estimator [31] was used instead where y(Z) are AR model prediction coefficients. As with of the implicit motion estimation technique developed by the temporal averaging filters, the size of the temporal Martinez and Lim. The use of this explicit motion estimator support is limited by the accuracy of the motion estimates. did improve the filter's performance. A nonlinear motion compensated version of the general temporal recursive filter, given in (29), was proposed by Dubois and Sabri [23]. In their filter the prediction is obtained by using L = 1 and y(l) = 1 in ( 3 9 , while the gain a(z,j , IC) is adaptively controlled according to the prediction error along the estimated motion trajectory. More

IV. EXPERIMENTAL RESULTS In this section, we compare the performance of several of the filters described in this paper. Specifically, the filters implemented are given in Table 1. These particular filters were

BRAILEAN et al.: NOISE REDUCTION FILTERS FOR DYNAMIC IMAGE SEQUENCES: A REVIEW

~ _ _ _ ~ ~

~~

--

1285

-

chosen to provide a comprehensive comparison between the four filtering categories identified, namely nonmotion compensated spatiotemporal (class 1) and motion compensated spatiotemporal (class 2), nonmotion compensated temporal (class 3), motion compensated temporal (class 4). Each filter listed in Table 1 was applied to degraded versions of the real 256 x 256 image sequences “Mobile” (frames 1-40) and “Trevor” (frames 21-70). The “Trevor” and “Mobile” sequences were chosen because of their popularity and availability, as well as the scene content contained in each sequence. For instance, the “Trevor” sequence contains a single object which moves across a stationary (but very textured) background. On the other hand, the “Mobile” sequence contains several moving objects imposed on a moving background. These sequences were degraded using simulated additive white Gaussian noise to signal-to-noise ratios (SNR’s) of 30, 20, and 10 dB. The expression used for calculating the SNR is given by

(37) where the scalar K denotes the number of frames in the sequence. A figure of merit used to evaluate the performance of a filter is the improvement in SNR at each frame, expressed in decibels (dB), according to f 256 256

\

where f(z,j,k ) denotes the filtered result. If the entire filtered sequence is considered, then the total or average improvement in SNR can be expressed as

(39) Although it has been argued that these types of metrics may not always reflect very accurately the image quality, we have decided to use both of them for two reasons. The first is the lack of any other universally accepted metric, while the second is the need to insure compatibility so that 1286

these results can be used as a bench mark to compare future filters against. To aid in analyzing these results, we present both the performance at each frame over the sequence as well as the average improvements. In the description of the motion compensated filters provided in Sections 11-B and 111-B, it was assumed that an estimate of the DVF was available for motion compensating the sequence. The problem of estimating the motion within an image sequence is formidable [ l ] and has attracted the attention of many researchers over the past several years. This has led to the development of numerous approaches to solving the DVF estimation problem, which include but are not limited to, feature model-based [42], transform-based [30], stochastic-model based [12], [27], [28], [57] region matching [9], [36], [37], and spatiotemporal gradient-based [8], [65], [66] approaches. For an application such as image sequence filtering, it is obvious that the algorithm chosen must not only provide accurate estimates of the DVF, but it must also be robust to the effects of noise. Recently, several noise robust approaches have been developed for estimating the DVF. Since the purpose of this paper is to review and evaluate image sequence filtering techniques, we have decided to use only two motion algorithms. This was done for removing any variation in the performance that may result due to the implementation of each filter with a different motion estimation algorithm. This decision was also possible because all but one of the filtering algorithms used either a block matching or a pel recursive algorithm to estimate the motion. Therefore, it was decided to keep the type of motion estimator originally used in a filter, for consistency. That is, filters originally implemented using a block matching algorithm were implemented here using the block matching algorithm described in [36], [37]. Filters implemented using a pel recursive algorithm were implemented using the pel recursive algorithm described in [27]. These two particular algorithms were chosen, due to proven ability to provide quality estimates of the DVF in the presence of noise. The actual motion estimation scheme used by each filter is also provided in Table 1. Based on the plots and images shown in Figs. 3-14 and Tables 1-3 we have reached the following general conclusions:

1) SNR and SNRi increases by increasing the support of the filter (1-D to 3-D) and by introducing motion compensation. As shown Figs. 8-14, this general trend is also consistent with the visual quality of the filtered sequences. Although not ideal for discussing video filtering results, due to the elimination of temporal effects, these frames do illustrate this general trend. For instance, the lack of motion compensation (Fig. 9) results in severe temporal smoothing. This is especially evident near the hands in Fig. 9. Also, as the support increases, the amount of smoothing or noise reduction increases. However, the ability of the filter to adapt is very important for preserving edges and reducing the impact of motion compensation errors. PROCEEDINGS OF THE IEEE, VOL. 83, NO. 9, SEPTEMBER 1995

Average Improvement in SNR (SNR) for the “Mobile” Sequence

Takle 2

Average Improvement in S N R (SER) SNR=lO dB SNR=20 dB SNR=30 dB 4.7 1.o 0.1 1.1 3.5 -5.1 3.9 1.4 0.5 4.1 7.3 0.1 1.5 5.5 0.2 2.2 0.4 5.9 0.2 1.5 6.2 4.4 0.7 7.6 4.1 7.6 0.5 -12.9 -3.5 2.8 -11.4 -2.8 2.0 -1.2 2.2 4.1 3.1 0.4 2.5 0.8 0.0 3.6 0.9 4.2 2.6 0.5

Filter PE MSOS

os

MC-AWA MC-PE MC-SPE MC-ASPE SDIE MC-OS TA TM TLS AWA MC-TPE MC-AN MC-TLS

I :::

Fig. 8. Frame 5 of the noisy “Trevor” sequence ( S N R = 20 dB).

Table 3 Average Improvement in SNR (SNR) for the “Trevor” Sequence

Average Improvement in SNR (SRR) SNR = 10 dB S N R = 20 dB SNR= 30 dB 5.0 1.I 0.3 2.1 -3.1 3.9 2.9 1.2 0.2 2.6 -1.0 7.0 2.1 5.6 0.3 2.4 0.4 6.3 2.2 6.2 0.2 1.7 0.7 4.0 -0.1 7.3 2.4 - 10.9 -11.5

Filter PE MSOS

os

MC-AWA MC-PE MC-SPE MC-ASPE SDIE MC-OS

15

iA MC-TPE MC-AN MC-TLS

; 2.6

0.3 0.3

SNRi increases as the SNR decreases (for the three SNR levels we have tested). This is due to the increase in the value of the numerator in the SNRi formula (the numerator decreases as the SNR decreases but not as fast as the numerator increases). The relative performance of the filters in terms of SNRi is consistent in most cases for both sequences and all three SNR’s we experimented with (the exceptions to this conclusion will be addressed below). Some of the inconsistencies are due to the particular characteristics of the sequences used. The “Trevor” consists of a moving object in a stationary background. This results in a large area where the assumption of temporal stationarity holds, benefiting those filters which rely on such an assumption. The sequence “Mobile” consists of moving objects in a moving background. This results in large areas where the assumption of temporal stationarity does not hold, therefore penalizing filters which rely on this assumption. This is evident in Fig. 6 (curves in the middle and bottom plots), where the TA and TM algorithms

Fig. 9. Frame 5 of the noisy ‘Trevor” sequence ( S N R = 20 dB) filtered by the TA algorithm (SNR = -3.7 dB).

are compared. TM outperforms the TA algorithm for the “Mobile” sequence at 20 dB and 30 dB SNR, since the TM algorithm preserves the temporal edges. On the other hand, the TA algorithm outperforms the TM algorithm for the “Trevor” sequence at the same SNR levels, due to the reduction in the variance of the noise at the large temporally stationary areas. 5) Some filters are more sensitive than others to the tuning of the parameters involved. For the experimental results presented we have attempted to optimize such parameters. 6) The sensitivity of the MC-AN filter at an SNR = 10 dB (“Trevor” sequence) results from the tuning of the parameters to provide an overall best possible improvement in SNRi. However, in obtaining this result the parameter a, is set to a value which is less sensitive to the motion and therefore SNRi decreases for frames 45-55. where there is considerable motion. The following particular conclusions have been reached for the 1-D and 3-D filters. More specifically, regarding the 1-D filters:

7) The use of motion compensation provides a considerable improvement over nonmotion compensated

BRAILEAN et al.: NOISE REDUCTION FILTERS FOR DYNAMIC IMAGE SEQUENCES: A REVIEW

_

_

~

~

1287

-

filters at high SNR’s (20 and 3 0 dB). This can be observed, for example, when viewing Figs. 9 and 10 or comparing the performance of the TLS and MCTLS algorithms report in Figs. 6 and 7. These results are expected, since on one hand the estimation of the motion (and therefore the compensation of the motion) is quite accurate at high SNR’s, while on the other hand the averaging of nonmotion compensated frames provides a small benefit (since the noise level is low) while deteriorates the quality of the estimated image by smoothing out the temporal edges. 8) The use of motion compensation provides smaller improvement over nonmotion compensated filters at low SNR’s (10 dB). This is explained by the fact that the estimation of the motion is quite erroneous at low SNR’s, and therefore the compensation introduces errors as well. The overall improvement provided by the filter is dominated by the averaging of the noise samples, regardless of the accuracy (or lack of it) of the motion compensation. Overall, MC helps even with the basic motion estimators we used which do not take the presence of noise into account, since the averaging is performed in the direction of maximum correlation. It is expected that SNRi will further increase by the use of MC at low SNR’s if more sophisticated motion estimators are used. 9) The dependency of SNRi on the amount of motion in the sequence can be observed in the filters which do not use motion compensation (primarily in the plots on the right column of Fig. 6). As expected, during the frames of considerable motion (frames 25-35 and 45-55 in the Trevor sequence), nonmotion compensated filtering introduces considerable blurring of the temporal edges, which decreases the SNRi. This decrease in SNRi is not as severe as the SNR decreases (the SNRi curves become more flat as the SNR decreases) for the reasons provided in point (8) above. This is clearly not as noticeable in sequences with moderate motion, such as the “Mobile” sequence (left column plots in Fig. 6). It must be taken into account that although motion compensation is very effective at improving the stationarity assumption of a temporal signal, not all the nonstationarities which occur in this signal are the result of motion. That is, scene and illumination changes as well as occlusions lead to nonstationarities in the temporal signal. Since motion compensation fails to remove these nonstationarities in the temporal signal, the ability of a motion compensated temporal filter to adapt locally is extremely valuable. Regarding the 3-D filters, the following conclusions can be drawn: 10) The performance of the filters which do not use motion compensation does not depend on the amount of motion in the sequence, as was the case with the 1-D filters (curves of Fig. 3). This is apparent when 1288

-~

Fig. 10. Frame 51 of the noisy “Trevor” sequence (SNR = 20 dB) filtered by the MC-TPE algorithm (SNR = -1.4 dB).

comparing Figs. 9 and 11 and is due to the fact that the use of nonmotion compensated spatiotemporal information has a blurring effect on the estimated sequence which is not made more severe during the frames of considerable motion. Also as was mentioned in ( 5 ) , we attempted to optimize the filter’s performance through the available tuning parameters. For the nonmotion compensated spatiotemporal filters this required careful adjustment of the parameters controlling the influence of the elements in the temporal direction on the filtered solution. In other words, these parameters were set to values which minimized the distortions caused by motion. 11) In agreement with the above comment (10) the amount of motion in the sequence effects the performance more in the MC filters (Figs. 4 and 5 , right column, middle, and bottom). Also in agreement with comment (9), the ability of a filter to locally adapt is very important for addressing motion compensation failures. This is evident when comparing the results provide in Figs. 4, 5, 13, and 14. V. CONCLUSION In this paper, we have presented a review of noise removal techniques for digital image sequences. Four major categories of filters were identified: 1) nonmotion compensated spatiotemporal, 2 ) motion compensated spatiotemporal, 3) nonmotion compensated temporal, and 4) motion compensated temporal filtering. For each category we compared several algorithms, pointing out the similarities and differences in each approach. To further distinguish between these algorithms, experimental results for several filters from each category were also presented and their implications were discussed. In general it was found that for both the spatiotemporal and temporal filters, motion compensation improved the performance of these filters. By motion compensating, the assumption of stationarity in the temporal direction holds over a much larger portion of the sequence. Therefore, PROCEEDINGS OF THE IEEE, VOL. 83, NO. 9, SEPTEMBER 1995

Fig. 11. Frame 51 of the noisy “Trevor” sequence (SNR = 20

Fig. 13. Frame 51 of the noisy “Trevor” sequence (SNR = 20

dB) filtered by the PE algorithm (SNR = -1.6 dB).

dB) filtered by the SDIE algorithm (SNR = -3.9 dB).

Fig. 12. Frame 51 of the noisy “Trevor” sequence (SNR = 20 dB) filtered by the MC-PE algorithm (SNR = -1.9 dB).

Fig. 14. Frame 51 of the noisy “Trevor” sequence (SNR = 20 dB) filtered by the MC-OS algorithm (SNR = -2.2 dB).

the motion artifacts characteristic of the nonmotion compensated filters are greatly reduced in an image sequence restored using the motion compensated versions of these filters. A disadvantage of motion compensation is the additional required computational cost. Therefore, for applications where computational complexity of an algorithm is a concern and the SNR of the input image sequence is still relatively high, a nonmotion compensated filter maybe more appropriate. For instance, at 30 dB the performance of the PE filter is comparable to the performance of the motion-compensated filters. Another interesting trend observed in these results is the importance of adaptability both spatially as well as temporally. The performance of the adaptable filters is significantly better at low SNR’s. However, as the SNR increases the performance of these filters decrease. The reason for this decline can be explained as follows. A filter’s ability to adapt, especially in the temporal direction, is extremely important at low SNR’s. This is due to the overall degradation in temporal correlation caused by the noise. For the motion compensated filters, this degradation in the temporal correlation component results from inaccuracies in the motion estimates which are also caused by the noise.

However, as the SNR increases, so does the reliability of the estimated motion field. Thus the importance of temporal adaptability is diminished for both motion compensated and nonmotion compensated filters at high SNR’s. It is clear from this review that although many image sequence filtering approaches exist, there is no one filter that can be singled out as the solution for all applications. That is, for applications constrained by computational complexity and memory availability, such as in real time high SNR applications, a nonmotion compensated temporal filter would be the most attractive. Depending on the amount of memory available to the filter, a nonmotion compensated spatiotemporal filter may also be appropriate. However, for a low SNR application that does not have these constraints, a motion compensated adaptable filter would provide the highest quality (in an MSE sense) filtered sequence. REFERENCES [I] J. K. Aggarwal and N. Nandhakumar, “On the computation of motion from sequences of images-a review,” Proc. IEEE, vol. 76, pp. 917-935, Aug. 1988. [21 M. B. Alp and Y. Nuevo, “3-dimensional median filters for image sequence processing,” in Proc. IEEE Int. Con$ Acoust., Speech, Signal Process., May 1991, Toronto, Canada, vol. 4, pp. 2917-2920.

BRAILEAN er al.: NOISE REDUCTION FILTERS FOR DYNAMIC IMAGE SEQUENCES: A REVIEW

._

~~

__

-_

1289

~

[3] J. M. M. Anderson and G. B. Giannakis, “Image motion estimation algorithms using cumulants,” IEEE Trans. Image. Process., vol. 4, pp. 346-352, Mar. 1995. [4] G . R. Arce, “Multistage order statistic filters for image sequence processing,” IEEE Trans. Acoust., Speech, Signal Process., vol. 39, pp. 1147-1163, May 1991. [5] M. R. Banham, J. C. Bralean, C. L. Chan, and A. K. Katsaggelos, “Low bit rate coding using robust motion vector regeneration in the decoder,” IEEE Trans. Image Process., vil. 3, pp. 652-665, Sept. 1994. [6] H. C. Bergmann, “Displacement estimation based on the correlation of image sequences,” in IEEE Proc. Int. Con$ on Electron. Image Process., July 1982, pp. 215-219. [7] M. Bertero, T. A. Poggio, and V. Torre, “Ill-posed problems in early vision,” Proc. IEEE, vol. 76, pp. 869-889, Aug. 1988. [8] J. Biemond, L. Looijenga, D. E. Boekee, and R. H. J. M. Plompen, “A pel-recursive Wiener-based displacement estimation algorithm,” Signal Process., vol. 13, no. 4, pp. 399412, Dec. 1987. [9] M. Bierling, “Displacement estimation by hierarchical block matching,” in SPIE Proc. Vis. Com. and Image Process., Nov. 1988, Cambridge, MA, vol. 1001, pp. 942-951. [lo] J. C. Brailean and A. K. Katsaggelos, “Simultaneous displacement estimation and restoration of noisy-blurred image sequences,” IEEE Trans. Image. Process., vol. 4, Sept. 1995. [ 111 -, “Recursive displacement estimation and restoration of noisy-blurred image sequences,” in Proc. IEEE Int. Con$ Acousr., Speech, Signal Process., Apr. 1993, Minneapolis, MN, vol. 5, pp. V-273-V-276. [ 121 __ , “A recursive nonstationary MAP displacement vector field estimation algorithm,” IEEE Trans. Image Process., vol. 4, pp. 416429, Apr. 1995. [13] C. D. Bowling and R. A. Jones, “Motion compensated image coding with a combined maximum a posteriori and regression algorithm,” IEEE Trans. Commun., vol. COM-33, pp. 844-857, Aug. 1985. [14] J. M. Boyce, “Noise reduction of image sequences using adaptive motion compensated frame averaging,” in Proc. IEEE Int. Con$ Acoust., Speech, Signal Process., Mar. 1992, San Fransisco, CA, vol. 3, pp. 461-464. [15] C. Cafforio and F. Rocca, “The differential method for image motion estimation,” in Image Sequence Processing and Dynamic Scene Analysis, T. S. Huang, Ed. Berlin: SpringerVerlag, 1983, pp. 104-124. [16] C. L. Chan, A. K. Katsaggelos, and A. V. Sahakian, “Linearquadratic noise smoothing filters for quantum-limited images and image sequences,” IEEE Trans. Image Process., vol. 4, Sept. 1995. [17] -, “Image sequence filtering in quantum-limited noise with applications to low dose fluoroscopy,” IEEE Trans. Med. Imaging, vol. 12, pp. 6 1 M 2 1 , Sept. 1993. [ 181 -, “Recursive locally linear MMSE motion-compensated image sequence filtering under quantum-limited conditions,” J. Vis. Comm. and Image Repr., vol. 4, no. 4, pp. 349-363, Dec. 1993. [19] D. I. Crawford, “Spatio/temporal prefiltering for a videoconference coder,” in Proc. Int. ZEE Conf on Electron. Image Process., July 1982, York, UK, pp. 236-242. [20] C. B. Dekker, A. J. E. M. Janssen, and P. J. van Otterloo, “The contour plot method for noise reduction in digital video,” Acta Electronica, vol. 27, no. 1-2, pp. 119-131, 1985. [21] T. J. Dennis, “Nonlinear temporal filter for television picture noise reduction,” IEE Proc., vol. 127, Pt G, no. 2, Apr. 1980. [22] J. N. Driessen and J. Biemond, “Motion field estimation for 2-D Kalman filtering,” in SPIE Proc. Vis. Comm. and Image Process., Nov. 1991, Boston, MA, vol. 1605, pp. 511-521. [23] E. Dubois and S. Sabri, “Noise reduction in image sequences using motion compensated temporal filtering,” IEEE Trans. Commun., vol. COM-32, pp. 826-831, July 1984. [24] E. Dubois, “Motion compensated filtering of time-varying images,” Multidimensional Syst. and Signal Process., vol. y, pp. 211-239, Mav 1992. [25] S. N. Efstratiidis and A. K. Katsaggelos, “Formula for a steady state gain of a recursive estimator,” Electron. Lett., vol. 26, no. 20, pp. 1682-1684, Sept. 1990. [26] S. N. Efstratiadis, “Recursive motion compensated image sequence estimation,” Ph.D. dissertation, Dept. EECS, Northwestem Univ., Evanston, IL, 1991. 1290

[27] S. N. Efstratiadis and A. K. Katsaggelos, “Nonstationary AR modeling and constrained recursive estimation of the displacement field,” ZEEE Trans. Circ. Syst. for video Tech., vol. 2, pp. 334346, Dec. 1992. [28] -, “An adaptive regularized recursive displacement estimation algorithm,” IEEE Trans. Image Process., vol. 2, pp. 341-352, Sept. 1993. [29] 0. E. Erdogan, “A multichannel regularized iterative algorithm for the restoration of noisy and blurred image sequences,” M.S. thesis, Northwestern Univ., Evanston, IL, June 1993. [30] D. Fleet, Measurement of Image Velocity. Norwell, MA: Kluwer, 1992. [31] S. V. Fogel, “Estimation of the velocity vector fields from time varying image sequences,” CVGIP: Image Understanding, 1991. [32] N. P. Galatsanos and R. T. Chin, “Digital restoration of multichannel images,” IEEE Trans. Acoust., Speech, Signal Proc., vol. 37, pp. 415421, Mar. 1989. [33] N. P. Galatsanos, A. K. Katsaggelos, R. T. Chin, and A. D. Hillery, “Least squares restoration of multichannel images,” IEEE Trans. Signal Process., vol 39, pp. 2222-2236, Oct. 1991. [34] S. Geman and D. Geman, “Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images,” IEEE Trans. Putt. Anal. and Much. Intell., vol. PAMI-6, pp. 721-741, Nov. 1984. [35] H. Gharavi and M. Mills, “Block matching motion estimation algorithms-New results,” IEEE Trans. Circ. and Syst., pp. 649451, May 1989. [36] G . de Haan and H. Huijgen, “Motion estimator for TV-picture enhancement,” in Proc. 4th Int. Workshop on HDTVand Beyond, Sept. 1991, Torino, Italy. [37] G . de Haan, “Motion estimation and compensation,” Ph.D. dissertation, Dept. Elect. Eng., Delft Univ. Tech., The Netherlands, Sept. 1992. [38] L. Hong and D. Brzakovic, “Bayesian restoration of image sequences using 3-D Markov random fields,” in Proc. IEEE Int. Conf Acoust., Speech, Signal Process., May 1989, Glasgow, UK, vol. M1.5, pp. 1413-1416. [39] B. K. P. Hom and B. G . Shunck, “Determining optical flow,” Art$ Intell., vol. 17, pp. 185-203, 1981. Robot Vision. Cambridge, MA: MIT Press, 1986. [40] -, [41] T. S. Huang, Ed., Image Sequence Analysis. Berlin: SpringerVerlag, 1981. [42] T. S. Huang and A. N. Netravali, “Motion and structure from feature correspondences: A review,”Proc. IEEE, vol. 82, pp. 252-267, Feb. 1994. [43] J. R. Jain and A. K. Jain, “Displacement measurement and its applications in interframe image coding,” IEEE Trans. Commun., vol. COM-29, Dec. 1981. [44] D. S. Kalivas and A. A. Sawchuk, “Motion compensated enhancement of noisy image sequences,” in Proc. IEEE Int. Con$ Acoust., Speech, Signal Process., Apr. 1990, Albuquerque, NM, pp. 2121-2124. [45] A. K. Katsaggelos, J. N. Driessen, S. N. Efstratiadis, and R. L. Lagendijk, “Temporal motion compensated noise filtering of image sequences,” in SPIE Proc. Vis. Comm. and Image Process., Nov. 1989, Boston, MA, pp. 61-70. [46] A. K. Katsaggelos, R. P. Kleihorst, S. N. Efstratiadis, and R. L. Lagendijk, “Adaptive image sequence noise filtering methods,” in SPIE Proc. Vis. Comm. and Image Process., Nov. 1991, Boston, MA, vol. 1606, pp. 716-727. [47] A. K. Katsaggelos, Ed., Digital Image Restoration, vol. 23. New York: Springer-Verlag, 1991, chapt. 1. [48] A. K. Katsaggelos, K. T. Lay, and N. P. Galatsanos, “A general framework for frequency domain multichannel signal processing,” IEEE Trans. Image Process., vol. 2, pp. 417-420, July 1993. [49] R. P. Kleihorst, R. L. Lagendijk, and J. Biemond, “Nonlinear filtering of image sequences using order statistics,” in Proc. 12th Symp. on Inform. Theory in the Benelux, May 1991, Veldhoven, The Netherlands, pp. 49-55. [50] R. P. Kleihorst, “Nonlinear filtering of image sequences using order statistics,” Chartered Designer (TWAIO) Report, Delft Univ. Tech., Delft, The Netherlands, Jan. 1992. [51] R. P. Kleihorst, G . de Haan, R. L. Lagendijk, and J. Biemond, “Noise filtering of image sequences with double compensation for motion,” in Proc. 13th Symp. on Inform. Theory in the Benelux, June 1992, Enschede, The Netherlands, pp. 197-204. PROCEEDINGS OF THE IEEE, VOL. 83, NO. 9, SEPTEMBER 1995

“An order statistics supported non-linear filter with application to image sequence filtering,” in Proc. IEEE Winter Workshop on Nonlinear Digital Signal Process., Jan. 1993, Tampere, Finland, pp. 1.2-2.1-1.2-2.5. -, “Motion compensated noise filtering of image sequences,’’ in Proc. Eusipco ’92, Aug. 1992, Brussels, Belgium, pp. 1385-1388. R. P. Kleihorst, R. L. Lagendijk, and J. Biemond, “Noise reduction of severely corrupted image sequences,’’ in Proc. IEEE Int. Con$ Acoust., Speech, Signal Process., Apr. 1993, Minneapolis, MN, vol. 5, pp. 293-296. T. Koga et al.“Motion-compensated interframe coding for video conferencing,” in Proc. 1981 NTC, Dec. 1981, pp. G5.3.1G5.3.5. A. C. Kokaram and P. J. W. Rayner, “A system for the removal of impulsive Noise in image sequences,” in SPIE Proc. vis. Comm. Image Process., Nov. 1992, Boston, MA, vol. 1818, pp. 322-33 1. J. Konrad and E. Dubois, “Bayesian estimation of motion vector fields,” IEEE Trans. Putt. Anal. and Mach. Intell., vol. 14, pp. 910-927, Sept. 1992. D. T. Kuan, A. A. Sawchuk, T. C. Strand, and P. Chavel, “Adaptive noise smoothing filter for images with signal dependent noise,” IEEE Trans. Putt. Anal. and Mach. Intell., vol. PAM-7, pp. 653465, Mar. 1985. C. T. Lee et al.,“Postprocessing of video sequence using motion dependent median filters,” in SPIE Proc. vis. Comm. Image Process., Nov. 1991, Boston, MA, vol. 1606, pp. 728-734. D. Martinez and J. S. Lim, “Implicit motion compensated noise reduction of motion video signals,” in Proc. 1EEE lnt. Conf: Acoust., Speech, Signal Process., 1985, Tampa, FL, pp. 375-378. R. H. McMann, et al., “Digital noise reducer for encoded NTSC signals,” SMPTE J., vol. 87, pp. 129-133, Mar. 1978. H. Musmann, P. Pirsch, and H. Grallert, “Advances in picture coding,” Proc. IEEE, vol. 70, pp. 523-542, Apr. 1982. N. M. Namazi and C. H. Lee, “Nonuniform image motion estimation from noisy data,” IEEE Trans. Acoust., Speech, Signal Process., vol. 38, pp. 364-366, Feb. 1990. S. S. H. Naqvi, N. C. Gallagher, and E. J. Coyle, “An application of median filters to digital television,” in Proc. IEEE Int. Con6 Acoust., Speech, Signal Process., Tokyo, Japan, 1986, pp. 2451-2454. A. N. Netravali and J. D. Robbins, “Motion-compensated television coding: Part I,” The Bell Syst. Techn. J., vol. 58, no. 3, pp. 629468, 1979. -, “Motion-compensated television coding: Some new results,” The Bell Syst. Techn. J., vol. 59, no. 9, pp. 1735-1745, 1980. A. Netravali and B. Haskell, Digital Pictures, Representation and Compression. New York: Plenum, 1988. M. K. Ozkan, A. T. Erdem, M. I. Sezan, and A. M. Tekalp, “Efficient multiframe Wiener restoration of blurred and noisy image sequences,’’ IEEE Trans. Image Process., vol. 1, pp. 4 5 7 4 7..-’ 6 Oct. 1992 [69] M-K. Ozkan, M. I. Sezan, and A. M. Tekalp, “Adaptive motioncompensated filtering of noisy image sequences,’’ IEEE Trans. Circ. and Syst. for video Tech., vol. 3, pp. 277-290, Aug. 1993. [70] T. A. Reinen, “Noise reduction in heart movies by motion compensated filtering,” in SPIE Proc. Vis. Comm. Image Process., Nov. 1991, Boston, MA, vol. 1606, pp. 755-763. [71] A. Rosenfeld and A. C. Kak, Digital Picture Processing. New York: Academic, 1982. [72] K. Sauer and B. Liu, “Nonstationary filtering of transmission tomograms in high photon counting noise,” IEEE Trans. Med. Imaging, vol. 10, pp. 445-452, Sept. 1991. [73] M. I. Sezan, M. K. Ozkan, and S. V. Fogel, “Temporally adaptive filtering of noisy image sequences using a robust motion estimation algorithm,” in Proc. IEEE Int. Con$ Acoust., Speech, Signal Process., May 1991, Toronto, Canada, vol. 4, pp. 2429-2432. [74] R. C. Triplicane, “Recursive restoration of noisy image sequences,” M.S. thesis, Northwestern Univ., Evanston, IL, Oct. 1989. W. Woods and J. Kim, “Motion compensated spatiotempo[75] .I. ral Kalman filter,” in Motion Estimation and Image Sequence Processing, M. I. Sezan and R. L. Lagendijk, Fds. Boston: Kluwer, 1993. -,

[76] J. W. Woods, S. Dravida, and R. Mediavilla, ‘‘Image estimation using doubly stochastic Gaussian random field models,” IEEE Trans. Putt. Anal. and Mach. Intell., vol. PAMI-9, pp. 245-253, Mar. 1987.

James C. Brailean (Member, IEEE) received

the B.S.E.E. degree from the University of Michigan in 1985. He received the M.S.E.E. degree from the University of Southern California in 1988, and the Ph.D. degree in electrical engineering from Northwestern University in 1993. From 1985 to 1989, he was a Member of Technical Staff at Hughes Aircraft Co., Space and Communications Group, Los Angeles, CA, working on advanced communication systems. He is currently a Staff Engineer at Motorola, Corporate Research, Schaumburg, IL, where he is working on video compression. His research interests are in video compression, motion estimation, and image sequence restoration. Dr. Brailean is a Hughes Aircraft Murphy and Cabell Fellow. He is a member of Tau Beta Pi, Eta Kappa Nu, and SPIE.

Richard P. Kleihorst (Member, IEEE) was born in Schiedam, The Netherlands. He received the MSc. degree in electrical engineering, the chartered designer’s degree, and the Ph.D. degree from Delft University of Technology, The Netherlands, in 1989, 1992, and 1994, respectively. At present, he is with Philips Research Laboratories, Eindhoven, The Netherlands, in the Digital VLSI Group, where he is investigating the limitations and possibilities of integration of digital video processing algorithms. From 1990 to 1994 he was a Research Assistant in the Laboratory for Information Theory of Delft University of Technology. The research during this period was focused on the application of order-statistic operators to noise filtering of image sequences. In 1989, he worked at the Philips Research Laboratories in Eindhoven, The Netherlands, on fuzzy classification techniques for character descriptions. His current research interests include digital-signal processing, with emphasis on image and image-sequence processing and restoration of signals and various other objects.

Serafim Efstratiadis (Member, IEEE) was born

in Thessaloniki, Greece, in 1964. He received the Diploma degree in electrical engineering from Aristotle University of Thessaloniki, Greece, in 1986, and the M.S. and Ph.D. degrees in electrical engineering from Northwestern University, Evanston, IL, in 1988 and 1991, respectively. From 1987 to 1991 he was a Teaching Assistant and a Research Assistant at the Department of Electrical Engineering and Computer Science of Northwestem University. During 1991-1992 he was First Assistant and Head of the Digital TV Group at the Signal Processing Laboratory of the Swiss Federal Institute of Technology, Lausanne, Switzerland. Since 1993 he has been assistant director of the Information Processing Laboratory, Aristotle University of Thessaloniki, Greece. His research interests include multidimensional signal processing, motion estimation and compensation, mono/stereoscopic imagehide0 coding and transmission, medical image processing and analysis, and multimedia applications. Dr. Efstratiadis is a member of the Technical Chamber of Commerce of Greece.

BRAILEAN et al.: NOISE REDUCTION FILTERS FOR DYNAMIC IMAGE SEQUENCES: A REVIEW

1291

Aggelos K. Katsaggelos (Senior Member, IEEE) received the Diploma degree in electrical and mechanical engineering from the Aristotelian University of Thessaloniki, Thessalonila, Greece, in 1979, and the M.S. and Ph.D. degrees, both in electrical engineering, from the Georgia Institute of Technology, Atlanta, GA, in 1981 and 1985, respectively. In 1985 he joined the Department of Electrical Engineering and Computer Science at Northwestern University, Evanston, IL,where he is currently an Associate Professor. During the 1986-1987 academic year he was an Assistant Professor at Polytechnic University, Department of Electrical Engineering and Computer Science, Brooklyn, NY. His current research interests include image recovery, processing of moving images (motion estimation, enhancement, very low bit rate compression) and computational vision. He has served as an Associate Editor for the IEEE TRANSACTIONS ON SIGNAL PROCESSING (199&1992), and is currently an area editor for the journal GRAPHICAL MODELS AND IMAGE PROCESSING. RESTORATION(Springer-Verlag, 1991). He is the editor of DIGITAL IMAGE Dr. Katsaggelos is an Ameritech Fellow, a member of SPIE, the ON MEDICAL IMAGING Steering Committees of the IEEE TRANSACTTONS and the IEEE TRANSACTIONS ON IMAGE PROCESSING, the IEEE TECHNICAL COMMI~TEES ON VISUAL SIGNALPROCESSINGAND COMMUNICATIONS AND ON IMAGE AND MULTI-DIMENSIONAL SIGNAL PROCESSING, the Technical Chamber of Commerce of Greece, and Sigma Xi. He was the General Chairman of the 1994 Visual Communications and Image Processing Conference (Chicago).

1292

~~

Reginald L. Lagendijk (Member, IEEE) was born in Leiden, The Netherlands, in 1962. He received the M.Sc. and Ph.D. degrees in electrical engineering from the Delft University of Technology in 1985 and 1990, respectively. In 1987 he became an Assistant Professor in the Laboratory for Information Theory of the Delft University of Technology. He was a Visiting Scientist in the Electronic Image Processing Laboratories of Eastman Kodak Research in Rochester, NY, in 1991. Since 1993 he has been an Associate Professor in the Laboratory for Information Theory of the Delft University of Technology. Dr. Lagendijk is author of the book Iterative Identification and Restoration of Images (Kluwer, 1991). and coauthor (with M. I. Sezan) of the book Motion Analysis and Image Sequence Processing (Kluwer, 1993). At present his research interests include theory and implementations of multidimensional signal processing methods and communication theory, with emphasis on filtering, compression, and analysis of image sequences. ON IMAGE Dr. Lagendijk is Associate Editor of the IEEE TRANSACTIONS PROCESSING, and a member of the IMDSP Technical Committee of the SP Society.

PROCEEDINGS OF THE IEEE, VOL. 83, NO. 9, SEPTEMBER 1995