Membership function adaptive fuzzy filter for image ... - IEEE Xplore

M.K. Güllü and S. Ertürk: Membership Function Adaptive Fuzzy Filter for Image Sequence Stabilization

1

Membership Function Adaptive Fuzzy Filter for Image Sequence Stabilization M.K. Güllü and S. Ertürk, Member, IEEE Abstract — In this paper a novel adaptive fuzzy image sequence stabilization system is proposed for the motion correction part of digital image stabilizers. The proposed stabilizer is based on smoothing of absolute frame positions. Initially a short mean filter is applied to raw absolute frame displacements as pre-process, to reduce the dynamic range of the fuzzy system input. This enables the fuzzy system to give appropriate responses with a fewer number of membership functions, hence provides improved performance at reduced computational load. Fuzzy stabilization is then achieved through fuzzy correction mapping. Output membership functions of the fuzzy system are continuously adapted so as to constitute a membership function adaptive fuzzy filtering process. The proposed membership function adaptive fuzzy filter based image sequence stabilization system is shown to provide excellent stabilization as well as intentional camera movement preservation performance, superior to previously reported systems.1 Index Terms — Fuzzy logic, Image sequence analysis, Nonlinear estimation, nonlinear filters. I.

INTRODUCTION

An important problem in image sequences captured by hand held digital cameras or cameras mounted on a moving framework, is originated by the fluctuations caused due to unwanted camera motions. Image stabilization is performed to eliminate undesired fluctuations to obtain a compensated sequence that reflects smooth camera movements only [1]. Digital image stabilization has two typical applications in the field of consumer electronics: consumer video cameras and prospective mobile visual communication equipment. Compact consumer video cameras with powerful zooms are likely to result in the fluctuation of images due to camera shake, and various digital image stabilizing systems have already been proposed to improve visual quality through stabilization [2]. Furthermore, with recent advances in wireless technology, image stabilization systems are being explored for integration into wireless video communication equipment for the stabilization of acquired sequences before transmission [3], to improve visual quality and enhance compression performance by eliminating fluctuations caused by device instabilities. Digital image stabilization systems consist of two parts: the motion estimation system that evaluates global interframe motion vectors and the motion correction system that 1 This work was supported by the Turkish Scientific and Technical Research Council, under grant EEEAG/101E006. M. K. Güllü and S. Ertürk are with the Department of Electronics & Telecommunications Engineering, University of Kocaeli, Veziroglu Campus, Izmit 41040, Turkey ([email protected], [email protected]).

Contributed Paper Manuscript received October 20, 2003

compensates for fluctuations. While undesired global motion effects might be translation, rotation or zoom based, translational jitter is the most commonly encountered case resulting in visual quality degradation. Therefore most stabilization systems aim translational fluctuation compensation only and such systems are referred to as two-dimensional (2D) stabilization systems. Stabilization systems that aim to stabilize rotation or scale variations in addition to translational jitter are referred to as three-dimensional (3D) stabilization systems, which have a significantly higher computational complexity as it is required to estimate 3D motion parameters. Motions encountered in image sequences can typically be classified into two types: global frame motion and local object motion. The global frame motion is of main interest for the image stabilization system as it reflects the movements of the imaging system or framework. The global motion estimation accuracy is vital for the stabilization system as incorrect motion vectors will directly result in unsuccessful motion correction. Full-search (FS) frame matching with mean absolute difference (MAD) or mean square error (MSE) criteria can be considered as the optimal solution for detecting translational motion vectors, however the computational load is prohibitively high. Furthermore the image frame might be dominated by foreground objects affecting the global motion estimation performance. Various techniques have been proposed to reduce the computation load and improve accuracy of the motion estimation part of digital image stabilizers. Global motion estimation using local motion vectors of sub images has initially been presented in [4], where four sub-images located in the corners of the image frame are defined and block-matching is employed to obtain local subimage motion vectors. Using a median filter for the local motion vectors, and also including the global motion vectors detected for the previous image frame, the global motion vector of the current frame is assigned. The approach of median filtering of sub-image motion vectors has also been adopted in many subsequent systems, which concentrated on reducing the computational cost through different matching criteria or search techniques. Global motion estimation based on edge pattern matching has been proposed in [5]. An iterative multiresolution scheme that estimates affine motion parameters between levels of the Laplacian pyramid has been described in [6]. A multiresolution feature based motion estimation system has been presented in [7]. Edge pattern matching and multiresolution techniques have shown to reduce the computational load at the expense of accuracy or complexity, and are therefore rather unsuited for a real-time

0098 3063/04/$20.00 © 2004 IEEE

2

image stabilization system. Fast motion estimation based on bit-plane and gray-coded bit-plane matching has been proposed in [8] and [2] respectively. It has been shown that gray-coded bit-plane matching is superior to straight bit-plane matching in terms of global motion vector estimation accuracy, yet the accuracy is degraded compared to MAD or MSE matching [9]. Digital image stabilization with sub-image phase correlation based global motion estimation has been proposed to provide improved motion estimation accuracy as well as intelligent combining of sub-image motion vectors [10]. For the stabilization of fluctuations, image frames are repositioned inversely to jitter by an appropriate amount referred to as the correction vector. The motion correction system is usually not desired to remove all of the detected global motion, as this might result in the loss of image content if the sequence contains substantial camera movement. Motion vector integration (MVI) [2,3,4,8] and frame position smoothing (FPS) [11,12,13,14,15] have been proposed to obtain appropriate correction vectors for 2D stabilization. MVI constitutes a first-order low-pass infinite impulse response (IIR) filter that integrates differential motion vectors to smoothen the global movement trajectory by simple realtime operation. However, the smoothened movement trajectory is delayed with respect to the actual camera gross displacements due to integrator operation, imposing larger frame shifts than actually required for stabilization. Moreover, in sequences that contain deliberate movements such as pan, the MVI system is likely to reach the limit of permitted frame shifts rather rapidly, in which case the stabilization performance is degraded to keep the image content within the visible frame [16]. FPS is based on low pass filtering of absolute frame positions, and achieves successful stabilization performance with retained smooth camera movements. FPS has initially been utilized in off-line DFT filtering based stabilization [11]. Real-time operation has been accomplished with Kalman filtering based stabilization [12]. The stabilization performance has then been improved by Fuzzy adaptive Kalman filtering, introducing a stabilization system that adjusts to changes in camera motion characteristics [13]. The fact that stabilization performance of Kalman filtering can be improved by fuzzy adaptive Kalman filtered frame position smoothing, has motivated entirely fuzzy stabilization systems. Fuzzy stabilization systems have shown to provide excellent stabilization performance if membership functions are optimized to motion dynamics [14]. Membership selective fuzzy stabilization was proposed in this course, having the stabilization system select between a pre-determined set of membership functions according to instantaneous motion characteristics, so as to improve stabilization and intentional motion preservation performance of fuzzy stabilization [15]. This paper proposes a superior image stabilization system that utilizes adaptive membership functions for the fuzzy stabilizer, with membership functions being repositioned according to instantaneous global motion dynamics. The membership function adaptive fuzzy stabilization system is

IEEE Transactions on Consumer Electronics, Vol. 50, No. 1, FEBRUARY 2004

shown to provide excellent stabilization as well as intentional camera movement preservation performance, superior to previously reported systems. II. FUZZY IMAGE SEQUENCE STABILIZATION Fuzzy logic based systems have extensively been used in image and signal processing applications in recent years [17, 18]. Fuzzy systems provide an effective and smart approach to nonlinear and uncertain systems. A typical application of fuzzy logic in image and signal processing has been noise filtering [19]. In this paper, fuzzy logic is utilized in an estimator structure referred to as a fuzzy filter, which is used to stabilize fluctuations encountered in the image frame position signal. A. Fuzzy Estimation A standard discrete time-invariant system can be defined by the process and observation equations

xk +1 = f (xk ) + wk

(1)

z k = h(x k ) + v k

(2)

where k is the time index, x k is the state vector, z k is the measurement vector, wk is the process noise and vk is the measurement noise. To find an estimate xˆ k to the discrete time signal, the commonly used estimator architecture known as the recursive predictor-corrector is given by

xˆ k = fˆ (xˆ k −1 ) + g (z k , xˆ k −1 )

(3)

where fˆ (.) shows an estimate of f (.) which maps the state

from one time step to the next, and g (.) is the correction function of the system. Fuzzy logic can be used as correction function of the estimator in a similar approach to [20]. The fuzzy estimator structure, which estimates the stabilized frame position by a constant velocity camera model [12], can be implemented as

xˆ k− = xˆ k −1 + Tvˆk −1 xˆ k =

xˆ k−

+g

(

z k , xˆ k−

(4)

)

(5)

where k is the time index, xˆ k− is the a priori estimate of x at time k , xˆ k is the a posteriori estimate of x at time k , T is the update period of the estimator, z k refers to raw frame positions and Vˆ is an estimate of the rate of change of frame motion speed, which is typically obtained from

(

)

Vˆk −1 = xˆ k −1 − xˆ k −2 / T

(6)

The correction function g (.) typically has an uncertain mathematical

model

as

instantaneous

fluctuations

are

M.K. Güllü and S. Ertürk: Membership Function Adaptive Fuzzy Filter for Image Sequence Stabilization

3

unknown. In this paper it is proposed to implement the correction function g (.) by a fuzzy system with adaptive membership functions. B. Fuzzy Filter Parameters The structure used for the fuzzy correction system is displayed in Fig. 1. The implemented correction system has two inputs and one output. The input variables are given by

I1k = z k − xˆ k−

(7)

I 2 k = I1k − I1k −1

(8)

(I1) depends on the difference between the k -th index of unfiltered, i.e. raw image sequence frame positions, and the a priori estimate xˆ k− . Input 2 (I 2) is where input 1

calculated from the difference between current and previous indexes of input 1. knowledge base

Input 1 Input 2

fuzzifier

defuzzifier

output

fuzzy inference engine

Fig. 1. Fuzzy correction system block diagram.

In the proposed stabilization system five Gaussian MFs are used for each input and the output. The number of membership functions has been selected so as to obtain decent performance with as few membership functions as possible to maintain low system complexity. Gaussian membership functions are preferred for their smooth transition and simple adaptability. A Gaussian MF is entirely specified by the two parameters {c,σ } , where c represents the MFs mean (center location) and the standard deviation σ determines the width. A Gaussian membership function can be expressed as

y = gaussian( x; c, σ ) =

1  x −c  −   e 2 σ 

2

(9)

In the design of the fuzzy system, membership functions and the rule base play key roles for optimal system performance. The input membership functions and the initial output membership functions are prearranged experimentally for optimal performance, and output membership functions are varied adaptively during operation again by a fuzzy system to have the stabilization system react to changes in global motion dynamics. The utilized input and initial output membership functions are displayed in Fig. 2. The constructed rule base containing 25 rules is shown in Table I.

Fig. 2. Membership functions for fuzzy filter (a) input 1 (b) input 2 (c) output and (d) surface. (NB = negative big, N = negative, Z = zero, P = positive, PB = positive big)

C. Preliminary Mean Filtering Large fluctuations in the frame position signal have been observed to obstruct optimum working conditions for the fuzzy

4

IEEE Transactions on Consumer Electronics, Vol. 50, No. 1, FEBRUARY 2004

filter. Hence, these fluctuations are preliminary reduced by a mean operation. This operation calculates mean values using present and past three values of the frame position vector, as shown in equation (10).

z k −1 = (a k + ak −1 + ak −2 + ak −3 ) / 4

(10)

Input1

TABLE I RULE BASE FOR THE FUZZY FILTER*

NB N Z P PB

NB NB N N N Z

Input2 N Z N N N Z Z Z Z Z Z P

P Z Z Z P P

PB Z P P P PB

performance is achieved. In order to have the stabilization system compensate for intensive fluctuations as well as follow the camera movement trajectory, it is proposed in this paper to adapt two of the output MF functions according to the rate of change of undesired fluctuations by a fuzzy logic adaptation system. It has been observed that the position of two output membership functions, namely the negative (N) and positive (P) MFs, directly affects the operation of the fuzzy filter. This feature has been used to introduce adaptation into the fuzzy stabilization system by varying the positions of these two membership functions as shown in Fig. 4.

*NB = negative big, N = negative, Z = zero, P = positive, PB = positive big

Raw frame positions before and after preprocessing are denoted as ak and z k respectively. The result of mean filtering of the absolute frame position signal for the “car” test sequence is given in Fig. 3 for the vertical direction. It is clearly seen that large amplitude fluctuations are reduced after the simple mean operation, yet fluctuations are preserved even if of lower amplitude. The fuzzy stabilization system uses the outcome of the mean operation as input to accomplish stabilization.

Fig. 3. Mean operation results for the vertical direction of test sequence “car”.

D. Membership Function Adaptive Fuzzy Stabilization In order to have the fuzzy filter operate successfully with a limited number of membership functions, so as to keep the computational complexity at a reasonable level, every membership function has to be defined to comprise a limited range. However the stabilization system is clearly required to adjust to changes in global motion dynamics so as to ensure that optimal stabilization and long-term motion preservation

Fig. 4. Changing the distance between N - P output MFs center locations.

The position of the negative and positive membership function center locations with respect to each other, directly determines the operation of the fuzzy filter. If the distance between membership function centers is increased, i.e. the two membership functions are shifted further apart, the fuzzy filter will track the global frame position more closely, but the stabilization intensity will decrease. This is shown in Fig. 5(a), where it is possible to see that if the two membership functions are distant, the stabilization system output closely follows the mean operation result that is fed into the stabilization system, preserving some fluctuations. In terms of classical filter theory the cut-off frequency of the fuzzy filter is increased as the two membership functions are shifted further apart. On the contrary, if the distance between membership function center locations is reduced, the fuzzy filter can diverge from the global camera movement trajectory, but the stabilization performance will increase. This is shown in Fig. 5(b), where it is possible to see that if the two membership functions are close, the stabilization system output is very smooth, but can diverge from the camera movement trajectory. In terms of classical filter theory the cut-off frequency of the fuzzy filter is decreased as the two membership functions are shifted further close to each other.

M.K. Güllü and S. Ertürk: Membership Function Adaptive Fuzzy Filter for Image Sequence Stabilization

Fig. 5. Fuzzy filter results for “car” sequence with interval between N–P MFs center locations changed: (a) maximum interval (b) minimum interval.

The adaptation of the fuzzy filter output MFs is constituted according to the rate of change of undesired fluctuations by one input and one output Mamdani type fuzzy inference system with fixed input and output membership functions. Hence a fuzzy adaptation system is utilized. The fuzzy adaptation system uses the second input of the stabilization system ( I 2 ) for adaptation. The fuzzy filter N – P output MFs interval is updated from frame to frame, depending on the fuzzy adaptation system output. The rule base for the adaptation system is given in Table II, input and output membership functions and the surface of the fuzzy adaptation system is shown in Fig. 6. TABLE II RULE BASE FOR THE FUZZY FILTER

Input

NB

N

S

P

PB

NB

N

S

P

PB

Fig. 6. Fuzzy adaptation system: (a) input MFs (b) output MFs surface.

5

(c)

III. EXPERIMENTAL RESULTS Stabilization results of the proposed membership function adaptive fuzzy stabilization system are presented for two test sequences, referred to as the “bike” sequence and the “car” sequence, with different global motion characteristics. The “bike” sequence was captured from a domestic VHS camcorder mounted rigidly to one side on the rear carrier of a moving motorcycle, aiming forward past the rider. The sequence contains extensive fluctuation due to camera instability and the frame position signal trajectory reflects the movement of the motorcycle. The “car” sequence has been captured by a hand held camcorder with deliberately introduced high amplitude fluctuations in the vertical and extensive pan with changing direction in the horizontal motion component. Stabilization results for the proposed MF adaptive fuzzy system are presented for the “bike” and “car” sequences in Fig. 7 and Fig. 8 respectively, together with the previously proposed fuzzy stabilization system with fixed membership functions [14] and also membership function selective fuzzy stabilization [15]. The membership functions of the fuzzy stabilization system with fixed membership functions are

6

IEEE Transactions on Consumer Electronics, Vol. 50, No. 1, FEBRUARY 2004

adjusted for optimal stabilization performance for the “bike” sequence. Hence it is seen that the fuzzy stabilization, membership selective fuzzy stabilization and the proposed membership function adaptive fuzzy stabilization systems perform about the same for the “bike” sequence. However if the same membership functions are used for the “car” sequence which displays different motion dynamics, the fuzzy stabilization system with fixed membership functions is unsuccessful in tracking the global motion trajectory, resulting in very large correction vectors. The membership function selective fuzzy stabilization system performs better and usually tracks the motion trajectory closely, however fails if there is an abrupt change in the motion dynamics as is seen in Fig. 8 (b). Membership function adaptive fuzzy stabilization on the other hand performs successful stabilization tracking the global movement trajectory closely even if there are abrupt changes in the motion dynamics.

Fig. 8. Stabilization results for the “car” sequence (a) vertical frame position (b) horizontal frame position.

IV. CONCLUSION

Fig. 7. Stabilization results for the “bike” sequence (a) vertical frame position (b) horizontal frame position.

This paper has presented an original membership adaptive fuzzy image sequence stabilization system. Initially a short mean filter is applied to raw absolute frame displacements as pre-process to reduce the dynamic range of the fuzzy system input. Fuzzy stabilization is achieved through fuzzy correction mapping for the stabilization process that is defined in the form of a standard discrete time-invariant system. Output membership functions of the fuzzy system are continuously adapted in order to improve stabilization performance. The presented membership function adaptive fuzzy stabilization system has shown to closely follow the smoothened camera movement trajectory even if there are abrupt changes in camera motion dynamics, in which cases previously reported systems have shown to fail. The proposed system constitutes a membership function adaptive fuzzy filter based image sequence stabilization system which is shown to provide excellent stabilization and intentional camera movement preservation performance, superior to previously reported systems.

M.K. Güllü and S. Ertürk: Membership Function Adaptive Fuzzy Filter for Image Sequence Stabilization

REFERENCES [1] [2] [3] [4] [5] [6] [7] [8] [9]

[10] [11] [12] [13] [14] [15]

[16]

[17] [18] [19] [20]

A. Rosenfeld, “Guest Editorial”, Real-Time Imaging, vol. 2, pp. 269, 1996. S.J. Ko, S.H. Lee, S.W. Jeon and E.S. Kang, “Fast digital image stabilizer based on gray-coded bit-plane matching”, IEEE Trans. Consum. Electron., vol. 45, pp. 598-603, 1999. A. Engelsberg and G. Schmidt, “A comparative review of digital image stabilizing algorithms for mobile video communications”, IEEE Trans. Consum. Electron., vol. 45, pp. 591-597,1999. K. Uomori, A. Morimura and H. Ishii, “Electronic image stabilization system for video cameras and VCRs”, J. Soc. Motion Pict. Telev. Eng., vol. 101, pp. 66-75, 1992. J.K. Pail, Y.C. Park and D.W. Kim, “An adaptive motion decision system for digital image stabilizer based on edge pattern matching”, IEEE Trans. Consum. Electron., vol. 38, pp. 607-615, 1992. A. Burt and P. Anandan, “Image stabilization by registration to a reference mosaic”, in Proc. of ARPA Image Understanding Workshop, pp. 425-434, 1994. C. Morimoto and R. Chellappa, “Fast electronic digital image stabilization for off-road navigation”, Real-Time Imaging, vol. 2, pp. 285-296, 1996. S.J. Ko, S.H. Lee and K.H. Lee, “Digital image stabilizing algorithms based on bit-plane matching”, IEEE Trans. Consum. Electron., vol. 44, pp. 617-622, 1998. S. Ertürk, “Locally refined Gray-coded bit-plane matching for block motion estimation”, ISPA'03, Proc. 3rd IEEE R8-EURASIP Symposium on Image and Signal Processing and Analysis, Rome, pp. 128-133, 2003. S. Ertürk, “Digital image stabilization with sub-image phase correlation based global motion estimation” IEEE Trans. Consum. Electron., accepted for publication, Nov. 2003. S. Ertürk and T.J. Dennis, “Image sequence stabilization based on DFT filtering”, IEE Proc. on Image Vision and Signal Processing, vol. 127, pp. 95-102, 2000. S. Ertürk, “Real-time digital image stabilization using Kalman filters,” Real-Time Imaging, vol. 8, pp. 317-328, 2002. M.K. Güllü, E. Yaman and S. Ertürk, “Image sequence stabilization using fuzzy adaptive Kalman filtering,” Electronics Letters, vol. 39, no. 5, pp. 429-431, 2003. M.K. Güllü and S. Ertürk, “Fuzzy Image Sequence Stabilisation,” Electronics Letters, vol. 39, no. 16, pp. 1170-1172, August 2003. M.K. Güllü and S. Ertürk, “Image sequence stabilization using membership selective fuzzy filtering,” ISCIS XVIII Eighteenth International Symposium on Computer and Information Sciences, accepted for presentation. S. Ertürk, “Image sequence stabilisation: Motion vector integration (MVI) versus frame position smoothing (FPS),” Proc. 2nd IEEE R8EURASIP Symposium on Image and Signal Processing and Analysis, ISPA'01, Croatia, pp. 266-271, 2001. D.V. Ville, M. Nachtegael, D.V. Weken, E.E. Kerre, W. Philips and I. Lemahieu, “Noise reduction by fuzzy image filtering,” IEEE Trans. Fuzzy Systems, vol. 11, no. 4, August 2003. C.F. Juang and C.T. Lin. “Noisy speech processing by recurrently adaptive fuzzy filters,” IEEE Trans. Fuzzy Systems, vol. 9, no. 1, pp. 139-152, February 2001. F. Russo and G. Ramponi, “A fuzzy filter for images corrupted by impulse noise,” IEEE Signal Processing Letters, vol. 3, no. 6, pp. 168170, June 1996. D. Simon, “Fuzzy logic for digital phase-locked loop filter design,” IEEE Trans. Fuzzy Systems, vol. 3, no. 2, pp. 211-218, May 1995.

7

M. Kemal Güllü was born in Denizli, Turkey, in 1980. He received B.Sc and M.Sc degrees in electronics and telecommunications engineering from the University of Kocaeli, Kocaeli, Turkey, in 2001 and 2003 respectively. Currently, he is working as a Research Assistant in the Department of Electronics and Telecommunications Engineering, University of Kocaeli. His main research interests are in signal, image and video processing, in particular image and video restoration and image sequence stabilization. He has several published papers on the subject of image sequence stabilization.

Sarp Ertürk (M’99) Born in Ankara, Turkey, on June 4, 1974. He received his B.Sc. in Electrical and Electronics Engineering from Middle East Technical University, Ankara in 1995. He received his M.Sc. in Telecommunication and Information Systems and Ph.D. in Electronic Systems Engineering in 1996 and 1999 respectively from the University of Essex, U.K. From 1999 to 2001 he carried out his compulsory service at the Army Academy, Ankara. Since 2001 he has been with the University of Kocaeli, Turkey, where he is currently appointed as Associate Professor. His current research interests are in the area of digital signal and image processing.