A real-time gray projection algorithm for electronic ...

A real-time gray projection algorithm for electronic image stabilization YANG Weiping, ZHANG Zhilong,

ZHANG Yan,

LU Xinping,

LI Jicheng,

SHI Zhiguang

（ATR Lab, School of Electronic Science and Engineering, National University of Defence Technology, Changsha, Hunan, 410073, P.R.China）

ABSTRACT Electronic digital image stabilization technique plays important roles in video surveillance or object acquisition. Researchers have presented many useful algorithms, which can be classified to three kinds: gray based methods, transformation based methods and feature based methods. When scenario is simple or flat, feature based methods sometimes have imperfect results. Transformation based methods usually accompany large computation cost and high computation complexity. Here we presented an algorithm based on gray projection which divided the whole image into four sub-regions: the upper one, the bottom one, the left one and the right one. For making the translation estimation easier, a central region is also chosen. Then the gray projections of the five sub-regions were counted. From the five pairs of gray projections five group offsets including rotation and translation were obtained via cross correlation between current frame and reference frame gray projections. Then according to the above offsets, the required parameters can be estimated. The expected translation parameters(x axis offset and y axis offset) can be estimated via the offsets from the central region image pair, the rotation angle can be calculated from the left four groups offsets. Finally, Kalman filter was adopted to compute the compensation. Test results show that the algorithm has good estimation performance with less than one pixel translation error and 10 percent rotation error. Based on this kind of gray projection algorithm, a real-time electronic digital image stabilization system has been designed and implemented. System tests demonstrate the system performance reaches the expected aim. Keywords: image dithering, electronic image stabilization, gray projection, cross correlation Biography: YANG Weiping(1968~), male, Shaanxi, China, received the B. S. degree in the College of Air Force Telecommunication Engineering, the M. S. degree and the Ph. D degree in School of Electronic Science and Engineering, National University of Defense Technology. His research interests are mainly about target detection, target tracking, signal processing and image information processing.

1. INTRODUCTION With the development of video acquisition technology and decrease of the cost of such acquisition devices, it becomes possible to store and process large data volumes. Therefore video cameras become extremely popular in consumer market and their use in both industry and military is quickly growing. Because of the motion of carrying platform, unwanted, parasitic vibrations in video sequences harm the performance and value of such devices significantly. Both

hardware and image processing approaches to video stabilization have been developed. The first approach, called optical stabilization, consists in implementing an optical system that compensates for unwanted camera motion using motion sensors and active optical system. This approach is the most powerful, but it makes the system significantly more expensive and, therefore, usually it is not chosen for many application situations. The second approach, electronic image stabilization (EIS), consists in performing post-processing of the video sequence to eliminate unwanted motion in the video caused by carrying platform vibration or image dithering. It consists of two parts: motion estimation and motion compensation. Due to its low cost and high precision, EIS approach was paid more attentions and has been applied into broad fields. The task of EIS is to remove the irregular global motion effects (jitter) so as to obtain a compensated image with the smooth global movements only. Generally, the vibration of the camera does not modify the individual frame content, but only shifts and rotates the image. Therefore, the image stabilization must be carried out in the domain of global motion description with respect to an anchor frame. In order to compensate for the unwanted irregular motion, intentional component of the transformation between frames has to be estimated. The intentional motion usually is slow and smooth compared with the irregular jitter/image dithering. For this reason, to estimate the intentional motion parameters from the image sequence is a key step. After the intentional motion parameters are estimated, by “subtracting” them from the observed motion parameters, the irregular motion can be obtained and used to transform (warp) the frames. With regard to EIS, researchers have presented many useful algorithms, which can be classified to three kinds: gray based methods[4,6], transformation based methods and feature based methods[1-3,5,7,8]. As one of gray based methods, gray projection method has many advantages, such as less computation cost, scenario flexibility and robusticity. Because there may be little effective features when scenario is simple or flat, feature based methods sometimes have imperfect results. Like feature based methods, transformation based methods usually accompany large computation cost, high computation complexity and low background adaptability. Here we presented an algorithm which divided the whole image into four sub-regions, that is to say, the upper one, the bottom one, the left one and the right one(The concrete flow chart can be see in Figure 1). In order to make the translation estimation easier, a central region is also chosen. Then the gray projections of the five regions were counted. The following step was to calculate the correlation coefficients between the reference frame projections and the current frame gray projections. From the five pairs of gray projections five group offsets including rotation and translation were obtained via cross correlation of gray projections between current frame and reference frame. Then according to the above offsets, the required parameters can be estimated. The expected translation parameters(x axis offset and y axis offset) can be estimated via the offsets from the central region image pair, the rotation angle can be calculated from the left four groups offsets. Finally, the Kalman filter was adopted to compute the compensation. Experimental results show that the algorithm has good estimation performance, which it can nearly get rid of the impact of the unsteady platform and make the image sequence clear and stable with less than one pixel translation error and 10 percent rotation error. The size of image region is also discussed. Test results show that the region size is a compromise of precision and reliability. Based on this kind of gray projection algorithm, a real-time electronic digital image stabilization has been designed and implemented. System tests demonstrate the system performance reaches the expected aim. The paper is organized as follows: In Section 2 the algorithm is described and its flow diagram is given. The experiment results are presented in Section 3, then followed by the discussion and conclusion.

2. GRAY PROJECTION BASED ELECTRONIC IMAGE STABILIZATION The flow chart of the real-time electronic image stabilization system based on region gray projection algorithm can be shown in Figure 1. Unlike other gray projection algorithms, our presented algorithm divides the input image into four sub-images. These four sub-images are chosen as Figure 2. For the sake of movement estimation, a central image block is chosen as well. The red rectangle stands for the central image region. Considering computation cost and estimation precision, the size of these five sub-images can be set according to the requirements. Therefore, the region selection strategy has two schemes. In Figure 2(a), the selection of the sub-images is simple, and in Figure 2(b), the region selection can bring some advantages such as less computation, high estimation precision, and so on.

Odd frame Motion Est

Region gray projection

Motion Comp Video

Input

A/D

Video

Synthesis Motion Est

Region gray projection Even frame

D/A

Output Video

Motion Comp

EIS

Fig 1 The flow chart of the presented real-time electronic image stabilization processing system

x 2 1

5

3

4

y (a) scheme 1

(b) scheme 2

Fig 2 The region selection strategy of region gray projection algorithm

Suppose f(x,y) (M×N pixels) stands for the input image, (P XU , P YU ), (P XB , P YB ), (P XL , P YL ), (P XR , P YR ), and (P X , P Y ) stand for the gray projection of the upper sub-image(sub-image 2), bottom sub-image( sub-image 4), left sub-image(sub-image 1), right sub-image(sub-image 3), and central sub-image(sub-image 5), respectively, thus these gray projections can be expressed as

PXU ( y ) 

M 1

PXB ( y ) 



f (i, y ) ,

i 0

M 1

PXL ( y ) 

PXR ( y ) 



f (i, y ) ,

i 0

M / 2 1

 f (i, y) , i 0

M 1

 f (i, y) ,

iM / 2

PYU ( x) 

PYB ( x) 

N / 2 1

 f ( x, j ) ,

x=0,1,…,M-1, y=0,1,…,N/2-1

(1)

j 0

N 1

 f ( x, j ) ,

x  [0,M-1],

y  [N/2,N-1]

(2)

jN / 2

N 1

PYL ( x)   f ( x, j ) , x  [0,M/2-1], y  [0,N-1]

(3)

j 0

N 1

PYR ( x)   f ( x, j ) , x  [M/2,M-1], y  [0,N-1] j 0

(4)

PX ( y ) 

3 M / 4 1

 f (i, y) ,

PY ( x) 

iM / 4

3 N / 4 1

 f ( x, j ) ,

x  [M/4,3M/4-1], y  [N/4,3N/4-1]

(5)

jN / 4

Once the gray projection values are obtained, the motion between the reference frame and the current frame can be estimated if the former frame is regarded as the reference frame. Suppose (P XUc , P YUc ), (P XBc , P YBc ), (P XLc , P YLc ), (P XRc , P YRc ), and (P Xc , P Yc ) stand for current image frame’s gray projection of the upper sub-image(sub-image 2), bottom sub-image( sub-image 4), left sub-image(sub-image 1), right sub-image (sub-image 3), and central sub-image(sub-image 5), respectively, then the correlation coefficients(considering the amount of calculation, here the correlation coefficients are the simplified ones) for the five sub-image pairs can be expressed as N / 21 1 C XU (t )   | PXUc ( j  t )  PXU ( j ) | , t [-nDisX, nDisX] N / 2  | t | j 0

(6)

0 j  t  N / 2

1 CYU (t )  M|t |

M 1

| P

i 0 0 i  t  M

YUc

(i  t )  PYU (i ) | ,

t  [-nDisY, nDisY]

(7)

Where nDisX and nDisY stand for the possible maximum frame movement in x direction and y direction, respectively. Similarly, we can get other correlation coefficients C XB (t ) , CYB (t ) , C XL (t ) , CYL (t ) , C XR (t ) , CYR (t ) , C X (t ) , and

CY (t ) . Then the offsets where have the maximum coefficients can be found from the correlation coefficients sequences. Suppose xU , yU , xB , y B , xL , yL , xR , y R , x , and y stand for the estimated offsets, then x , and

y can be regarded as the translation parameters because they are calculated from the central region. And then the rotation parameters can be estimated from the other four pairs of offsets(can be seen in Figure 3),

tg1 

xU  xB , yB  yU

tg 2 

y  1  y tg1   U  B  , 2  xU xB 

yR  y L xR  xL

(8)

1  x x  tg 2    R  L  2  yR yL 

(9)

Where ( xU , yU ), ( xB , y B ),( xL , y L ), and ( xR , y R ) is the coordinate of region 2,4, 1 and 3 in reference image, respectively; ( xUc , yUc ),( xBc , y Bc ),( xLc , y Lc ), and ( xRc , y Rc ) is the coordinate of region 2,4, 1 and 3 in current image, respectively. 1 and  2 are the estimated rotation parameters. x

O

(x U ,y U ) (x Lc ,y Lc ) (x L ,y L )

(x Uc ,y Uc ) θ1

(x R ,y R )

θ2 (x Rc ,y Rc ) (x Bc ,y Bc )

(x B ,y B )

y Fig 3 Rotation parameter calculation via four pairs of offsets(without translation movement)

In formula (9), the estimation of the rotation parameters may have large errors because of lesser offsets or big divide overflow risk when the offsets are zero. For avoiding the above mistakes, only formula (8) is used to estimate the rotation parameters. Moreover, because the rotation angle is small in general, formula (8) can be simplified as

1 

xU  xB y B  yU ,

2 

y R  y L xR  xL

(10)

Finally, the rotation angle θ is estimated by



1   2 2

(11) After the movement parameters are worked out, the next step is to do the movement compensation via Kalman filter. That is to say, the translation parameters( x and y ) and the rotation parameter(  ) need to be filtered and obtained the movement compensation parameters. To x , the state model of Kalman filter can be described as follow,

X (k )  ( k , k  1) X (k  1)  (k  1)W ( k  1)

(12)

The observation model is

Z (k )  C (k ) X (k )  V (k ) Where X ( k )  X ( k )

(13)

T vX  , W(k) and V(k) are white noise with zero mean, and

1 0 , CovV ( k )  R (k ) , Var X (0)  P (0) CovW (k )    0 1 





E X (0)W T ( j )  0 , EX (0)V T ( j )  0 , CovW (k )V T ( j )  0 1 T   (k , k  1)   , 0 1 

1 0 , (k , k  1)    C ( k )  1 0 0 1 

Through filtered by the above filter, the real movement compensation value x* can be obtained. Similarly, y * and

 * can be obtained. Using these three parameters, the irregular global motion effects can be removed, and the image with the smooth global movements only can be obtained.

 x  cos *  y    *    sin 

 sin  *   x  x*       cos *   y  y * 

(14)

If the rotation angle  * is small, then formula (14) can be simplified as

 x   1  y   *   

  *   x   x*       1   y  y * 

Where (x,y) is the coordinate of the input image, ( x, y) is the coordinate of the compensated image. The process flowchart is given in Table 1.

(15)

Table 1 The steps of the EIS algorithm based on sub-image gray projections Step 1: Initializing the variables; Step 2: Input the first image frame, which is regarded as reference image, calculate regional gray projections (P XU , P YU ), (P XB , P YB ), (P XL , P YL ), (P XR , P YR ), and (P X , P Y ); Step 3: Input the second image frame, which is regarded as current image, calculate regional gray projections (P XUc , P YUc ), (P XBc , P YBc ), (P XLc , P YLc ), (P XRc , P YRc ), and (P Xc , P Yc ); Step 4: Calculate correlation coefficients C XU (t ) , CYU (t ) , C XB (t ) , CYB (t ) , C XL (t ) , CYL (t ) , C XR (t ) , CYR (t ) , C X (t ) , and CY (t ) Step 5: Calculate the five pairs offsets; Step 6: Estimate translation and rotation parameters; Step 7: Filter the movement parameters using Kalman filter; Step 8: Compensate the current image using formula (15); Step 9: Input a new image frame, assign the old current image as the reference image, and update the current image frame by the new image; Step 10 Terminate if interrupted or finished, otherwise, go to step 2 and continue the process.

3. EXPERIMENTAL RESULTS The EIS algorithm based sub-image gray projections was tested using both simulated and practical image sequences. Experimental results show that the algorithm has perfect stabilization performance. Then the EIS algorithm was implanted into the DSP based processing system. Performance test and evaluation show that the system can run in real time and have good expected effects. Some raw images extracted from an image sequence and their stabilized images(The image size is 768*576 pixels, and the frame rate is 25 Hz) are given in Figure 4. The numbers in the brackets are the adjustment values relative to reference image frame: offset in x direction, offset in y direction and rotation angle. In fact, the estimated motion between current frame and reference frame is often larger than the filtered compensation. The detailed unwanted or irregular motion can be seen in Figure 5. The solid line stands for the irregular movement and the dashed line stands for the filtered movement. Because of the tiny rotation angle, the rotation parameter can be ignored while compensating the images. In Figure 6, one simulation result for simulated image sequences is given, where the red line stands for the filtered position, the black one stands for the real pixel position and the blue one stands for the simulated pixel position that the random movement was added. For testing the effect of the rotation estimation, a special test is designed to do so. A special image sequence is made by rotating a practical image at a set of predetermined angle. Then the rotation estimation error can be obtained by comparing the estimated angles with the scheduled ones. Table 2 shows that the rotation angle estimation error is superior to 10 percent.

(a) the 86th frame raw image

(b) compensated image of the 86th frame(4,16,0)

(c) the 98th frame raw image

(d) compensated image of the 98th frame(2,2,0)

(e) the 123th frame raw image

(f) compensated image of the 123th frame(4,0,0)

X Coordinate

Y Coordinate

Fig 4 Some images extracted from a real image sequence and its stabilized images

Frame (a) The position change in X Coordinate

Frame (b) The position change in Y Coordinate

Fig 5 Image stabilization process results for the sequence showed in Fig 4 (The solid line stands for the irregular movement, the dashed line stands for the filtered movement)

4. DISCUSSION The experiment results show that the presented algorithm is effective. But its performance relies on the quality of image sequence and the distribution of image background. From the description of the algorithm, we can easily find that the algorithm will lose effectiveness when the image background is flat or is distributed only in few parts. Considering this case, some special process was adopted in the algorithm, for example, the offsets were kept and not to be updated temporarily.

Moreover, the block sizes of the sub-regions are sensible to the precision of the algorithm. When the image background is simple, the larger block size may induce lower image stabilization precision. So on this condition, the block size can be reduced to a smaller one. These five sub-regions can have some overlaps. Generally, a smaller block size may bring higher precision, but the unsteady risk may increase as well. Therefore, a compromising block size should be chosen. Here, the size of the central sub-region can be chosen as the half size of the image, and the sizes of other four sub-regions can be chosen as less than the half image size, typically, these four sub-regions can be located in corresponding image parts with about 10 pixels indented.

(a) The position change in X Coordinate

(b) The position change in Y Coordinate

Fig 6 Image stabilization results for a simulated image sequence Table 2 Results of rotation angle estimation test Image number

1

2

3

4

5

Practical angle(degree)

5.00

-5.00

3.50

2.80

-3.00

Estimated angle(degree)

4.71

-4.62

3.73

2.95

-3.26

Estimation error(%)

5.8

7.6

6.6

5.4

8.7

5. CONCLUSION In this paper we have presented an algorithm for image stabilization based on sub-region gray projection. An effort has been made to keep the computational load low. Then a real-time processing system is designed and realized. The next step is to study how to remove the motion between the odd frame and even frame so that the image can be enhanced in some extent.

6. ACKNOWLEDGMENTS

The authors would like to thank Dr Lu Xiaowei and Mr Yuan Aibo for their enthusiastic discussion, and other people related to video acquisition, hardware and software debugging.

REFERENCES [1] Gong Weiguo, Wang Xiaoli, Li Zhenghao, “High accuracy electronic image stabilization method using feature matching”, Application Research of Computers, 27(7), 2751-2753 (2010). [2] Wang Chuntao, Kim Jin-Hyung, Byun Keun-Yung, Ni Jiangqun, and Ko Sung-Jea, “Robust digital image stabilization using the Kalman filter”, IEEE Transactions on Consumer Electronics, 55(1), 6-14(2009). [3]Zhu Juanjuan, Guo Baolong, “Electronic image stabilization system based on global feature tracking”, Journal of Systems Engineering and Electronics, 19(2), 228–233(2008). [4] Ertürk, S., “Digital image stabilization with sub-image phase correlation based global motion estimation”, IEEE Transactions on Consumer Electronics, 49(4), 1320-1325(2003). [5] Zhou Yu-bin, Zhao Yue-jin, “Real-time digital image stabilization based on contour”, Journal of Beijing Institute of Technology, 12(Suppl.), 62-65(2003). [6]Ko Sung-Jea, Lee Sung-Hee, Jeon Seung-Won, Kang Eui-Sung, “Fast digital image stabilizer based on gray-coded bit-plane matching”, IEEE Trans on Consumer Electronics, 45(3), 598-603(1999). [7] Marcenaro, L., Vernazza, G., Regazzoni, C.S., “Image stabilization algorithms for video-surveillance applications”, Proceedings of 2001 International Conference on Image Processing, 1, 349 – 352(2001).

[8] Hsu Sheng-Che, Liang Sheng-Fu, Lin Chin-Teng, “A robust digital image stabilization technique based on inverse triangle method and background detection”, IEEE Transactions on Consumer Electronics, 51(2), 335-345(2005). [9] Andrew Litvin, Janusz Konrad, Karl, W.C., “Probabilistic video stabilization using Kalman filtering and mosaicking”, IS&T/SPIE Symposium on Electronic Imaging, Image and Video Communications and Proc., 20-24(2003).