Motion Detection Based on Optical Flow and Self ... - ScienceDirect.com

Available online at www.sciencedirect.com Available online at www.sciencedirect.com

Procedia Engineering

ProcediaProcedia Engineering 00 (2011) 000–000 Engineering 15 (2011) 3471 – 3476 www.elsevier.com/locate/procedia

Advanced in Control Engineering and Information Science

Motion Detection Based on Optical Flow and Self-adaptive Threshold Segmentation Shui-gen Wei a, Lei Yang a, Zhen Chen a ,Zhen-feng Liu a , a* a

Lab of Computer Vision, School of Information Engineering, Nanchang Hangkong University, Nanchang city, Jiangxi province, 330063, China

Abstract

A motion detection method based on optical flow and self-adaptive threshold segmentation is proposed in this paper. Optical flow is calculated from image sequences firstly, and then is normalized to a gray image. A binary image is got by Otsu method from normalized gray image. An adaptive window is obtained from binary image. Otsu method is used again to obtain a binary image from window image. Noise is removed by close operation from the window binary image and moving object area is found out. Experimental results show that this method is suitable and effective. © 2011 Published by Elsevier Ltd. Selection and/or peer-review under responsibility of [CEIS 2011] Open access under CC BY-NC-ND license.

Keywords: Image processing, Motion detection, Normalization, Self-adaptive window, Otsu method ;

1. Introduction Motion detection is a hot study field of computer vision. Its purpose is to extract moving object area in image sequences. Extracting moving object effectively and exactly is the foundation of tracking and sorting of moving target in computer vision. Up to the present, there are several proposals. Three typical approaches are mostly used in motion detection: temporal difference[1-4], background subtraction[5-7] and optical flow analysis[8-10]. Optical flow provides all motion information. But optical flow computation methods usually are too complex to use in real-time applications if without special hardware. Background subtraction can extract the most precise foreground by modeling the background, but it is sensitive to scene changes caused by light and weather etc. Temporal difference computes the difference between two or three consecutive frames. It is good at adapting to the dynamic environments, but generally poor at extracting enough relevant feature pixels, resulting holes being generated in the moving object. Every

* Corresponding author. Tel.: +086-13979171345. E-mail address: [email protected]

1877-7058 © 2011 Published by Elsevier Ltd. Open access under CC BY-NC-ND license. doi:10.1016/j.proeng.2011.08.650

3472 2

Shui-gen et al. / Procedia Engineering 15 (2011) 3471 – 3476 Shui-gen WeiWei et al/ Procedia Engineering 00 (2011) 000–000

method has advantages and disadvantages. To avoid the disadvantages of single method, a motion detection method is given based on optical flow and self-adaptive threshold segmentation in this paper. Window image is got by optical flow. Then, moving object area is obtained in the window image by Otsu method and close operation. 2. Optical Flow Calculation In 1981, Horn and Schunck[11-12] deduced a basic equation of optical flow estimation when the interval of consecutive frames was short, and the gray change in the image was also small. If at time t, coordinates of a pixel on the image with its gray value is I(x, y, t). At time (t + Δt), the pixel has moved to new position. Its location on the image becomes (x + Δx, y + Δy), and the gray value becomes I(x + Δx, y + Δy, t + Δt). Based on an assumption that intensity is conserved, and an equation is got as bellow. dI ( x, y, t ) / dt = 0 （1） From (1), the follow equation is obtained: I ( x, y , t ) = I ( x + Δx, y + Δy , t + Δt ) （2） Supposing u and v are two components of optical flow along x coordinate and y coordinate. They are defined as follows. u = dx / dt , v = dy / dt （3） From a Tayor expansion of (2), the gradient constraint equation is easily derived: ∂I dx ∂I dy ∂I + + =0 （4） ∂x dt ∂y dt ∂t i.e.

I xu + I yv + It = 0

（5）

This is the basic optical flow equation. Where I x denotes the partial x coordinate derivative of I ( x, y , t ) , I y denotes the partial y coordinate derivative of I ( x, y , t ) , and I t denotes the partial time derivative of I ( x, y , t ) . Equation (5) is also expressed as bellow.

（6）

∇I • U + I t = 0

Where ∇I = ( I x , I y ) denotes gradient direction，and U = (u, v ) denotes optical flow. There are two unknown components of U in (6), and basic optical equation is only one linear equation. There for, it is an ill-state problem. Further constraints are necessary to solve for both components of U. Horn and Schunck combined the gradient constraint with a global smoothness term to constrain the estimated velocity by minimizing  ⎧⎨α 2 ⎡( ∂u ) 2 + ( ∂u ) 2 + ( ∂v ) 2 + ( ∂v ) 2 ⎤ + ( I u + I v + I ) 2 ⎫⎬dxdy = min  （7） x y t ∫∫ ⎩ ⎢⎣ ∂x ∂y ⎥⎦ ∂x ∂y ⎭ Iterative equations are used to minimize (7) and obtain velocity: ⎧ I u (k ) + I y v (k ) + I t ij ⎪u ( k +1) = u ( k ) − x ij ⋅ Ix ij 2 2 ⎪ ij I I y2 + + α x ⎪ （8） ⎨ ⎪ I x u (k ) + I y v (k ) + I t ij ij ⎪v ij( k +1) = vij( k ) − ⋅Iy 2 2 ⎪⎩ α + I x + I y2 T

Shui-gen Wei et al. / Procedia Engineering 1500 (2011) 3471 – 3476 Shui-gen Wei et al / Procedia Engineering (2011) 000–000

Where k denotes the iteration number, u 0 and v 0 denote initial velocity estimates which are set to zero. u k and v k denote neighbourhood average of u k and v k . 40 iterations is used in all testing bellow. Because of global smoothness, velocity of background pixels near moving object boundary is not zero, while in real they are stationary. 3. Optical Flow Normalization

Optical flow value of pixel (x, y) is calculated by the follow. V ( x , y ) = u ( x, y ) 2 + v ( x, y ) 2

（9）

The maximum Vmax and the minimum Vmin of optical flow value are got as follow. Vmax = max ( u( x, y ) 2 + v ( x, y ) 2 )

( x, y ) ∈ C

Vmin = min( u( x, y ) 2 + v ( x, y ) 2 )

( x, y ) ∈ C

Let

V ( x, y ) （10） × 255] Vmax − Vmin Where f ( x, y ) is the normalized data of optical flow value. C is the set of all pixels coordinates in the image. The value of f (x, y) is from 0 to 255 and can be displayed as a gray image. f ( x, y ) = int[

4. Translating Gray Image into Binary Image

Otsu[13] method is a calculation method to find the best threshold by the rule of the maximum variance difference between classes based on least square method. It is supposed that the initial image gray level is defined from 0 to (L -1), and ni is the pixel number that gray level is i [0≤ i ≤(L-1)]. The pixels of image is N. The probability of gray level i is defined as （11） pi =ni／N Two classes C0 and C1 are defined by threshold T as below. ⎧ ⎪C 0 I ( x, y ) ≤ T （12） ⎨ ⎪⎩C1 I ( x, y ) > T Parameters are defined as follow. T T ⎧ ⎧ w0 = P ( C 0 ) = pi ipi / w0 ⎪ ⎪ μ0 = ⎪ ⎪ i =0 i =0 （13） ⎨ ⎨ L −1 L −1 ⎪ w = P (C ) = ⎪μ = = − p 1 w ip / w 1 i 0 i 1 ⎪ 1 ⎪ 1 i =T +1 i =T +1 ⎩ ⎩ Where w0 stands for the probability of class C0, and w1 denotes the probability of class C1. μ 0 denotes the mean gray value of class C0, and μ1 is the mean gray value of class C1. Mean gray value of the whole image μ is defined follow.

∑

∑

μ=

L −1

∑ ip i =0

i

∑

∑

3473 3

3474 4

Shui-gen et Procedia al. / Procedia Engineering 15 (2011) 3471 – 3476 Shui-gen WeiWei et al/ Engineering 00 (2011) 000–000 T ⎧ 2 p (i −μ 0 ) 2 i ⎪σ0 = w0 ⎪ i =0 ⎨ L −1 p ⎪σ 2 = (i − μ1 ) 2 i ⎪ 1 w 1 i =T +1 ⎩

∑

（14）

∑

Where σ 02 denotes the variance of C0 class, and σ 12 is variance of C1 class. Variance within classes is defined as follow. σ w2 = w0σ 02 + w1σ 12 （15） The variance between classes can also be defined following. σ b2 = w0 ( μ 0 − μ ) 2 + w1 ( μ1 − μ ) 2 （16） Gray value T* can be searched out from zero to (L -1) to got maximum variance between classes. T* is the best threshold T to be found. Let ⎧ ⎪0 f ( x, y ) ≤ T * （17） I 2 ( x, y ) = ⎨ ⎪⎩1 f ( x, y ) > T * A binary image including moving object area is got by T*. The white area in binary image denotes the big optical flow value. Because of globe smoothness constrain in Horn’s algorithm, the white area is bigger than moving object area. The black area is the background. 5. Obtaining Self-adaptive Window

A rectangle window can be got by searching the white area as follow. Rmin = min( x ∈ I 2 ( x, y ) = 1) Rmax = max ( x ∈ I 2 ( x, y ) = 1) C min = min( y ∈ I 2 ( x, y ) = 1)

（18）

C max = max ( y ∈ I 2 ( x, y ) = 1) Where Rmin and Rmax are the minimum and maximum row coordinates of the white area, C min and C max are the minimum and maximum column coordinates of the white area. i.e., the window image is got as following. I window = I ( Rmin : Rmax , C min : C max )

（19）

Where I ( x, y ) is gray value of the original image. 6. Moving object area detection

Otsu method is used again to get the best threshold T to image Iwindow, and binary image I3 can be obtained by following formula.

⎧ ⎪0 I 3 ( x, y ) = ⎨ ⎪⎩1

I window ( x, y ) ≤ T I window ( x, y ) > T

（20）

3475 5

Shui-gen Wei et al. / Procedia Engineering 1500 (2011) 3471 – 3476 Shui-gen Wei et al / Procedia Engineering (2011) 000–000

There may be some noses in the binary image. Close operation is used for removing these noses. A structuring element g is constructed as 7×7 or 5×5 ones square array. Moving object area (MOA) is found out by Equation (21) + g)○ - g IMOA=(I4○ （21） 7. Experimental Result and Analysis

7.1. The Experimental Result and Analysis of Cube Image Sequences The result of synthesized cube motion image sequences computed by the method here is in figure 1. It shows that the moving object area by the algorithm is satisfied.

（a）

(b)

(e)

(c)

(f)

(g)

(d)

(h)

(i)

Fig. 1. (a) The first image of cube image sequences. (b) The second image of cube image sequences. (c) Optical flow of cube image sequences. (d) Normalization of optical flow value. (e) Binary image after normalization. (f) Self-adaptive window image. (g) Binary image of window image. (h) Result of close operation. (i) The comparison between the result (detected by the method here) and real moving object area.

7.2. The Experimental Result and Analysis of Box Image Sequences Box image sequences are regarded as another example here. It is a realistic image sequences we take. The motion detection result is shown in figure 2. It shows that the difference between the result (by algorithm in this paper) and the real moving object area is small.

(a)

(f)

(b)

(g)

(c)

(d)

(h)

(i)

(e)

(j)

Fig. 2. (a) The first image of cube image sequences. (b) The second image of cube image sequences. (c) Optical flow of cube image sequences. (d) Normalization of optical flow. (e) Binary image after normalization. (f) Self-adaptive window image. (g) Binary image of window image. (h) Inverse image of (g). (i) Result of close operation. (j) The comparison between the result (detected by the method here) and real moving object area.

3476 6

Shui-gen et Procedia al. / Procedia Engineering 15 (2011) 3471 – 3476 Shui-gen WeiWei et al/ Engineering 00 (2011) 000–000

8. Conclusion

This proposal is an algorithm based on optical flow field and self-adaptive threshold segmentation. It could detect the moving object area precisely. The method is suitable to image sequences in the condition that camera is stationary while objects are moving. The experimental results show that this method in this paper is suitable and effective. Acknowledgements

This research is supported by National Natural Science Foundation of China (No.60963003), the Key Project of Chinese Ministry of Education (No.206080), Aeronautical Science Foundation (No.2008CZ56002), Natural Science Foundation of Jiangxi Province (No. 2007GZS1900, 2009GZS1109), International cooperation Project of Jiangxi Province (No. 2060402108) and Education Department Technology Project of Jiangxi( (No.GJJ09018). References [1] Lipton A, Fujiyoshi H and Patil R. Moving target classification and tracking from real-time video. In: Proc IEEE Workshop on Applications of Computer Vision, Princeton, NJ; 1998; 8-14. [2] Anderson C, Bert P and Vander Wal G. Change detection and tracking using pyramids transformation techniques. In: Pro SPIE Conference on Intelligent Robots and Computer Vision, Cambridge, MA; 1995; 579: 72-78. [3] Foresti G, Murino V, Regazz oni C. Vehicle recognition and tracking from road image sequences. IEEE Transaction on Vehiclar Technology, 1999; 48(1): 301-317. [4] WEI Shui-gen, CHEN Zhen, DONG Hua. Motion Detection Based on Temporal Difference Method and Optical Flow field. 2009 Second International Symposium on Electronic Commerce and Security, Nanchang; 2009; p.85-88. [5] WANG Liang, HU Wei-Ming, TAN Tie-Niu. A Survey of Visual Analysis of Human Motion. Chinese J. Computers, 2005; 25(3): 225-237(In Chinese). [6] Haritaohlu I, Harwood D and Davis L. W. real-time surveillance of people and their activities. IEEE Trans Pattern Analysis and Intelligence, 2002; 22(8): 809-830. [7]WANG Lanxun, LI Keyi, ZHU Di. An Adaptive Background Model Method. Journal of Hebei University. 2010; 30(4):428433. [8]LI Jin-zong, YUAN Lei, LI Dong-dong. Tracking method of moving target based on detection of feature optical flow. Systems Engineering and Electronics, 2007; 27(3): 422-426. [9]Meyer D, Denzler J and Niemann H. Model based on extraction of articulated objects in image sequences for gait analysis. In: Pro IEEE International Conference on Image Processing, Santa, California; 1997; p:78-81. [10]Memin E, Dence P. Estimation and object-based segmentation of the optical flow with robust techniques. IEEE Transaction on Image Procession, 1998; 7(5): 703-719. [11] B. Horn B. Schunck. Determining optical flow. Artificial Intelligence, 1981; 17: 185~203. [12] J. L. Barron, D. Fleet, S. S. Beauchemin. Performance of optical flow techniques. International Journal of Computer Vision, 1994; 12(1):43~77. [13] Otsu N. A threshold selection method from gray-level histograms. IEEE Transactions on System Man and Cybernetic, 1979; 9(1):62-66.