A Convergence Proof for the Horn-Schunck Optical ... - Springer Link

A Convergence Proof for the Horn-Schunck Optical-Flow Computation Scheme Using Neighborhood Decomposition Yusuke Kameda1 , Atsushi Imiya2 , and Naoya Ohnishi1 2

1 School of Science and Technology, Chiba University, Japan Institute of Media and Information Technology, Chiba University, Japan

Abstract. In this paper, we prove the convergence property of the HornSchunck optical-flow computation scheme. Horn and Schunck derived a Jacobi-method-based scheme for the computation of optical-flow vectors of each point of an image from a pair of successive digitised images. The basic idea of the Horn-Schunck scheme is to separate the numerical operation into two steps: the computation of the average flow vector in the neighborhood of each point and the refinement of the optical flow vector by the residual of the average flow vectors in the neighborhood. Mitiche and Mansouri proved the convergence property of the Gauss-Seidel- and Jacobi-method-based schemes for the Horn-Schunck-type minimization using algebraic properties of the matrix expression of the scheme and some mathematical assumptions on the system matrix of the problem. In this paper, we derive an alternative proof for the original Horn-Schunck scheme. To prove the convergence property, we develop a method of expressing shift-invariant local operations for digital planar images in the matrix forms. These matrix expressions introduce the norm of the neighborhood operations. The norms of the neighborhood operations allow us to prove the convergence properties of iterative image processing procedures.

1

Introduction

In this paper, we prove the convergence property for the Horn-Schunck opticalflow computation scheme. First, we derive a proof for the original Horn-Schunck scheme. Second, we evaluate the convergence rate. Finally, we introduce a method of selecting the regularization parameter for accurate computation. The main idea of the Horn-Schunck method for optical-flow computation is the decomposition of the Laplacian to the neighborhood average and the subtraction of the value at each point from the neighborhood average. Then, they derived the Jacobi method for optical-flow computation. Therefore, in this paper, we clarify the mathematical properties and evaluate the operator norm of the neighborhood operations in digital image processing. In signal processing and analysis, it is well known that a shift-invariant linear operation is expressed as a convolution kernel. Furthermore, a linear transform V.E. Brimkov, R.P. Barneva, H.A. Hauptman (Eds.): IWCIA 2008, LNCS 4958, pp. 262–273, 2008. c Springer-Verlag Berlin Heidelberg 2008

Convergence Proof for the Horn-Schunck Optical-Flow Computation Scheme

263

in a finite dimensional space is expressed as a matrix [2,17,6]. It is also possible to express a shift-invariant operation as a band-diagonal matrix [3,2,16]. However, this expression is not usually used in signal processing and analysis. In numerical computation of the partial differential equations, approximations of the partial differentiations in discrete operations are one of the central issues [4,7,17]. The discrete approximations of the partial differentiations are called the neighborhood operations in digital signal and image processing. To analyze and express digital image transformations from the viewpoint of functional analysis, we introduce a method of describing the neighborhood operations in the matrix forms. Optical flow is an established method of motion analysis in computer vision [8,11,1] and has been introduced to many application areas such as cardiac motion analysis [14,18], robotics [12,15,5], and visualization in chemical sciences [13]. However, there still exist mathematical problems concerning to accurate and stable computation of optical-flow. There are two types of evaluation methods on the schemes for optical-flow computation. The first one is a numerical-based analysis of the accuracy of the solution using normalized phantoms, that is, an evaluation of the differences between the numerical results and the ground truth for the synthetic data images with a predesigned motion field. The second one is a mathematical-theory-based evaluation, that is, clarification of the convergence and stability of the algorithm employing numerical analysis. From the viewpoint of mathematical-theory-based evaluation, we derive the convergence property on a variational optical-flow computation method proposed by Horn and Schunck [8]. Horn and Schunck derived the Jacobi-based-method for the computation of the optical-flow vector of each point [8] as the motion of each point on the image1 . The basic idea of the Horn-Schunck scheme is to separate the numerical operation into two steps: the computation of the average flow vector in the neighborhood of each point and the refinement of the optical flow vector at each point by the residual of the average flow vectors in the neighborhood. In their paper [8], the mathematical proof for the convergence property of the algorithm was not dealt with. The convergence of the scheme was later examined numerically [1]. Therefore, it might be understood that the convergence of the scheme depends on the input images. The first numerical scheme for computing optical-flow is later 1

In their original paper[8], they said We now have a pair of equations for each point in the image. It would be very costly to solve these equations simultaneously by one of the standard methods, such as Gauss-Jordan elimination [11, 13]. The corresponding matrix is sparse and very large since the number of rows and columns equals twice the number of picture cells in the image. Iterative methods, such as the GaussSeidel method [11, 13], suggest themselves. (n+1)

n = 12 (fi+1 + However, the method has the same structure as the iteration form fi n 1 fi−1 ) − gi for solving the equation gi = 2 (fi+1 − 2fi + fi−1 ), which is derived as the 2 numerical approximation of g = dd2 x f .

264

Y. Kameda, A. Imiya, and N. Ohnishi

extended to the three-dimensional problem for the computation of cardiac optical flow. Mitiche and Mansouri [10] proved the convergence property of the GaussSeidel-method-based scheme for the Horn-Schunck-type minimization using the algebraic property that the large system matrix of the problem is symmetry. Furthermore, they proved the convergence property for the Jacobi-type scheme of the Horn-Schunck-type minimization. In this paper, we derive an alternative proof for the original Horn-Schunck scheme and evaluate the convergence rate. Furthermore, we introduce a method of selecting the regularization parameter which guarantees accurate computation.

2

Optical Flow Computation

2.1

Optical Flow and Regularization

For functions in two-dimensional Euclidean space R2 , setting f (x − u, t + 1) and f (x, t) to be the images at times t + 1 and t, the small displacement u of each point x is called the optical flow of the image f . For a spatio-temporal image f (x, t), x = (x, y) , the total derivative is given as d ∂f dt f = ∇f u + , dt ∂t dt

(1)

where u = x˙ is the motion of each point x. Optical flow constraint [8,11,1] d dt f = 0 implies that the motion u of the point x is the solution of the singular equation, ∇f u + ft = 0. To solve this equation, the regularization method is employed to minimize the criterion (∇f u + ft )2 dx + αtr∇u∇u dx, (2) J(u) = R2

where u is the vector gradient of vector u, which is given as ∇u = (∇u, ∇v) for u = (u, v) . We call, in this paper, optical-flow computation by the minimization of eq. (2) the Horn-Schunck method. Furthermore, the numerical algorithm to solve eq. (2) is called the Horn-Schunck scheme for optical flow computation. The Euler-Lagrange equation of the energy function of eq. (2) is Δu =

1 1 (∇f u + ft )∇f = (Su + ft ∇f ), α α

(3)

where S = ∇f ∇f is called the structure tensor of f at point x. We adopt the natural boundary condition ∂∂n f = 0, where n is the unit outward normal vector on the boundary of the domain. 2.2

The Horn-Schunck Scheme

We assume that the sampled image f (i, j) exists in the M × M grid region, that is, we express fij as the value of f (i, j) at the point (i, j) ∈ Z2 . The natural boundary condition, that is, the Neumann condition, for discrete flow vectors uij = (u(i, j), v(i, j)) = (uij , vij ) , i, j = 1, 2, · · · , M

(4)


is

u1 1 − u2 2 = 0, u1 j − u2 j = 0, u1 M − u2 M−1 = 0, ui 1 − ui 2 = 0, uM j − uM−1 j = 0, uM 1 − uM−1 2 = 0, uM j − uM−1 j = 0 uM M − uM−1 M−1 = 0.

265

(5)

The discrete version of eq. (3) becomes, Luij =

1 (S ij uij + sij ), S ij = ∇fij ∇fij . sij = (∂t f )ij ∇fij , α

(6)

where fij = f (i, j, t) is the sampled function of f (x, y, t) at time t. Setting N4 (fij ) to be the operation to compute N4 (fij ) = av4 fij =

1 (fi+1 j + fi−1 j + fi j+1 + fi j−1 ) , 4

(7)

the Laplacian operation L with the four-neighborhood is expressed as Lfij = avfij − fij .

(8)

Using N4 , eq. (6) is rewritten as α(N4 (uij ) − uij ) = (S ij uij + sij ), 2 ≤ i, j ≤ M − 1.

(9)

Setting N4 (uij ) = uij , we have the equation (αI 2 + S)uij = αuij − sij .

(10)

For the matrix T ij = trS × I − S ij using the relation, (αI + S ij )(αI + T ij ) = α(α + trS ij )I,

(11)

we have uij = uij −

1 (S ij uij + sij ), α + trS ij

(12)

and the iteration form (m+ 12 )

uij

(m+1)

uij

(m)

= N4 uij

(m+ 12 )

= uij

−

1 (m+ 1 ) (S ij uij 2 + sij ). α + trS ij

(13)

In the original Horn-Schunck scheme, the first equation of the iteration is the weighted summation in the eight-neighborhood of the point (i, j) . Then, setting N to be an appropriate operation to compute the weighted summation in an appropriate neighborhood, we replace the first equation to (m+ 12 )

uij

(m)

= N uij .

(14)

266


Therefore, we have the iteration form (m+1)

uij

(m+ 12 )

uij

(m+ 12 )

= uij

−

1 (m+ 1 ) (S ij uij 2 + sij ), α + trS ij

(m)

= N uij , if 2 ≤ i, j ≤ M − 1,

(m+1)

(m+1)

(m+1)

(m+1)

(m+1)

(m+1)

= u2 2 , u1 j = u2 j , u1 M = u2 M−1 , u1 1 (m+1) (m+1) (m+1) otherwise.(15) ui 1 = ui 2 , uM j = uM−1 j , (m+1) (m+1) (m+1) (m+1) (m+1) (m+1) uM 1 = uM−1 2 , uM j = uM−1 j , uM M = uM−1 M−1 , The second term of the right-hand side of eq. (13) is 1 1 1 1 (S ij u(m+ 2 ) + sij ) = (∇fij u(m+ 2 ) + ∂t fij )∇fij ). α + trS ij α + trS ij

(16)

Equation (16) implies the next property. (m+ 12 )

Proposition 1. If uij

is the solution of the equation ∇fij u + (∂t f )ij = 0, (m+ 12 )

then we have the relation um+1 = uij ij the flow vector of the point.

3 3.1

(17)

, that is, the iteration does not update

Matrix Expression of Problem Matrix Expressions of Neighborhood Operations

Since the second-order discrete differentiation is ∂2 u =

u(i + 1) − 2u(i) + u(i − 1) , 2

(18)

the M × M second-derivative matrix is tridiagonal [4,7]. For Dirichlet mann boundary conditions, the derivative matrices are ⎞ ⎛ ⎛ −2 1 0 0 · · · 0 0 −1 1 0 0 · · · 0 ⎜ 1 −2 1 0 · · · 0 0 ⎟ ⎜ 1 −2 1 0 · · · 0 ⎟ 1⎜ 1⎜ ⎟ ⎜ ⎜ D1 = ⎜ 0 1 −2 1 · · · 0 0 ⎟ D2 = ⎜ 0 1 −2 1 · · · 0 2 ⎜ .. .. .. .. . . .. .. ⎟ 2 ⎜ .. .. .. .. . . .. ⎝ . . . . ⎝ . . . . . . . ⎠ . . 0

0

0 · · · 0 1 −2

0

0

and Neu0 0 0 .. .

⎞ ⎟ ⎟ ⎟ ⎟, ⎟ ⎠

0 · · · 0 1 −1

(19) respectively. Using D1 and D 2 , the discrete Laplacian operations for twodimensional discrete functions with the Dirichlet and Neumann boundary conditions, are expressed as L1 = I M ⊗ D 1 + D 1 ⊗ I M , L2 = I M ⊗ D 2 + D2 ⊗ I M ,

(20)


267

for 1 ≤ i, j ≤ M , respectively, where I n is the n × n identity matrix and A ⊗ B is the Kronecker product of matrices A and B [4]. Setting ⎞ ⎞ ⎛ ⎛ 0 1 0 0 ··· 0 0 1 1 0 0 ··· 0 0 ⎜1 0 1 0 ··· 0 0⎟ ⎜1 0 1 0 ··· 0 0⎟ ⎟ ⎟ 1⎜ 1⎜ ⎟ ⎜0 1 0 1 ··· 0 0⎟ ⎜ (21) B1 = ⎜ ⎟ , B2 = ⎜ 0 1 0 1 · · · 0 0 ⎟ , 2 ⎜ .. .. .. .. . . .. .. ⎟ 2 ⎜ .. .. .. .. . . .. .. ⎟ ⎝. . . . ⎝. . . . . . .⎠ . . .⎠ 0 0 0 ··· 0 1 0

0 0 0 ··· 0 1 1

the matrix N ε , ε ∈ {1, 2}, N ε = (B ε ⊗ I M + I M ⊗ B ε ),

(22)

is the averaging operation in the four-neighborhood of each point with Dirichlet and Neumann boundary conditions, respectively. Let ρ(A) be the spectrum of the matrix A. Since B ε = Dε + I M ,

(23)

N ε satisfies the property ρ(N ε ) < 1. The discrete Laplacian Lε is expressed as Lε u = N ε u − u. 3.2

(24)

Discrete Model

For the sampled optical flow vector uij = (uij , vij ) , we define two vectorizations of the sampled function as ⎞ ⎛ u11 ⎜ v11 ⎟ ⎛ ⎞ ⎟ ⎜ u11 ⎜ u12 ⎟ ⎟ ⎜ u12 ⎟ ⎜ ⎜ ⎟ ⎜ ⎟ (25) v = ⎜ . ⎟ = vex u11 , u12 , . . . , uMM = ⎜ v12 ⎟ ⎜ .. ⎟ ⎝ .. ⎠ ⎜ . ⎟ ⎟ ⎜ uMM ⎝ uMM ⎠ vMM and

⎛

⎜ ⎜ u = vec(u11 , u12 , · · · , uMM ) = vec ⎜ ⎝

u 11 u 12 .. .

⎞

⎛

⎟ ⎜ ⎟ ⎜ ⎟ = vec ⎜ ⎠ ⎝

u MM

u11 u12 .. .

v11 v12 .. .

⎞ ⎟ ⎟ ⎟. ⎠

(26)

uMM vMM

For these vectorizations, we define the permutation P as P v = u.

(27)

268


For the vector function uij = (uij , vij ) on the discrete plane Z2 , we have the matrix equation for the optical flow computation as Lε u =

1 1 Su + s, ε ∈ {1, 2} α α

(28)

for 1 ≤ i, j ≤ M and ε ∈ {1, 2}, where L := I 2 ⊗ Lε

(29)

S = P Diag(S 11 , S 12 , · · · , S MM )P ⎛ ⎞ s11 ⎜ s ⎟ ⎜ 12 ⎟ s = vec ⎜ . ⎟ = P t ⎝ .. ⎠ s MM t = vec s11 s12 · · · sMM .

4

(30)

(31)

(32)

The Horn-Schunck Scheme with Four-Neighborhood

Using N ε , the matrix form of the Horn-Schunck scheme is expressed as u(m+1) = N 4 u(m) − P F −1 P (SN 4 u(m) + s),

(33)

where F = αI + Diag (trS 11 I 2 , trS 12 I 2 , · · · , trS MM I 2 ) = αI + Diag(S ij ) = Diag(αI 2 + S ij )

1 1 1 −1 F = Diag I 2, I 2, · · · , I2 . α + trS 11 α + trS 12 α + trS MM

(34) (35)

Horn and Schunck [8] derived eq. (33) for the pointwise expression. From these expression, we have the relations u(m+1) − u(m) = N 4 (u(m) − u(m−1) ) − P F −1 P S(N 4 (u(m−1) ) − N 4 (u(m) )) = (I − P F −1 P S)N 4 (u(m−1) − u(m) ) = (I − P F −1 Diag(S ij )P )N 4 (u(m−1) − u(m) ) = P I − F −1 Diag(S ij ) P N 4 (u(m−1) − u(m) )

(36)

and |u(m+1) − u(m) | ≤ ρ(I − F −1 Diag(S ij ))ρ(N 4 )|u(m) − u(m−1) |

(37)

Here, ρ(N 4 ) < 1 and ρ(I − F

−1

Diag(S ij )) = max 1 − ij

trS ij . α + trS ij

(38)


269

Since trS ≥ 0 and α > 0, we have 0
b > 0.

(46)

ρ(N a8 b ) ≤ aρ(N 1 ) + bρ(M 2 ) < 1,

(47)

Since

the original Horn-Schunck scheme converges for the Dirichlet condition. In the original Horn-Schunck numerical scheme, the parameters a and b were selected to be 23 and 13 . Therefore, we have the local operation on the discrete Laplacian as Lf (i, j) = av8 f (i, j) − f (i, j),

(48)

for ⎛ ⎞ f (i − 1, j + 1) +2f (i, j + 1) +f (i + 1, j + 1) 1 ⎝ +2f (i − 1, j) +2f (i + 1, j) ⎠ . (49) av8 f (i, j) = 12 +f (i + 1, j + 1) +2f (i + 1, j + 1) +f (i + 1, j + 1) If i, j < 1 and M < i, j, we set f (i, j) = 0. Theorem 2. The classical Horn-Schunck scheme for the two-dimensional problem generates sequences of solutions which converge to the solutions of the discrete equation of the Euler-Lagrange equation for α > 0.


6

271

Conclusions

In this paper, we directly proved the convergence property of the optical-flow computation without any assumptions on the system matrices. Furthermore, we introduced an iteration form which does not depend on the images. Moreover, we showed that the selection method of the regularization parameter which guarantees accurate and stable computation. This research was performed using the support by Grants-in-Aid for Scientific Research from JSPS Japan.

References 1. Barron, J.L., Fleet, D.J., Beauchemin, S.S.: Performance of optical flow techniques. International Journal of Computer Vision 12, 43–77 (1994) 2. Bellman, R.: Introduction to Matrix Analysis, 2nd edn. SIAM, Philadelphia (1987) 3. Bracewell, R.N.: Two Dimensional Imaging. Prentice-Hall, Englewood Cliffs (1995) 4. Demmel, J.W.: Applied Numerical Linear Algebra. SIAM, Philadelphia (1997) 5. Enkelmann, W.: Obstacle detection by evaluation of optical flow fields from image sequences. Image and Vision Computing 9, 160–168 (1991) 6. Grossmann, Ch., Terno, J.: Numerik der Optimierung. Teubner (1997) 7. Grossmann, Ch., Roos, H.-G.: Numerik partieller Differentialgleichungen. Trubner (1994) 8. Horn, B.K.P., Schunck, B.G.: Determining optical flow. Artificial Intelligence 17, 185–204 (1981) 9. Hwang, S.-H., Lee, U.K.: A hierarchical optical flow estimation algorithm based on the interlevel motion smoothness constraint. Pattern Recognition 26, 939–952 (1993) 10. Mitiche, A., Mansouri, A.-R.: On convergence of the Horn and Schunck optical-flow computation. IEEE, Image Processing 13, 848–852 (2004) 11. Nagel, H.-H.: On the estimation of optical flow: Relations between different approaches and some new results. Artificial Intelligence 33, 299–324 (1987) 12. Ohnishi, N., Imiya, A.: Featureless robot navigation using optical flow. Connection Science 17, 23–46 (2005) 13. Ruhnau, P., Knohlberger, T., Schnoerr, C., Nobach, H.: Variational optical flow estimation for particle image velocimetry. Experiments in Fluids 38, 21–32 (2005) 14. Sachse, B.F.: Computational Cardiology, Modeling of Anatomy, Electrophysiology, and Mechanics. In: Sachse, F.B. (ed.) Computational Cardiology. LNCS, vol. 2966, Springer, Heidelberg (2004) 15. Santos-Victor, J., Sandini, G.: Uncalibrated obstacle detection using normal flow. Machine Vision and Applications 9, 130–137 (1996) 16. Strang, G., Nguyen, T.: Wavelets and Filter Banks. Wellesley-Cambridge (1997) 17. Varga, R.S.: Matrix Iteration Analysis, 2nd edn. Springer, Heidelberg (2000) 18. Zhou, Z., Synolakis, C.E., Leahy, R.M., Song, S.M.: Calculation of 3D internal displacement fields from 3D X-ray computer tomographic images. In: Proceedings of Royal Society: Mathematical and Physical Sciences, vol. 449, pp. 537–554 (1995)

272


Appendix Original Formulation of Horn and Schunck In the original paper [8], Horn and Schunck adopted the relation Δuij ∼ = 3(av8 uij − uij ) for

⎛ ⎞ f (i − 1, j + 1) +2f (i, j + 1) +f (i + 1, j + 1) 1 ⎝ +2f (i − 1, j) +2f (i + 1, j) ⎠ . av8 f (i, j) = 12 +f (i + 1, j + 1) +2f (i + 1, j + 1) +f (i + 1, j + 1)

If i, j < 1 nd M < i, j, we set f (i, j) = 0. Then, we have the approximate numerical Euler-Lagrange equation 3α(av8 u − u) = Su + s. Therefore, replacing 3α with β we have the equation β(av8 u − u) = Su + s. Inverse of a Matrix For S = ss and T = trS × I − S, we have the relation S 2 = trS × S, T S = 0, and T s = 0. Furthermore, for μ ≥ 0, (I + μS)(I + μT ) = (1 + μtrS)I. and

1 (I + μT ). 1 + trS Since for S ij , we have the orthogonal decomposition

⊥ trS ij 0 ∇fij ∇fij⊥ ∇fij ∇fij S ij = |∇fij | |∇f ⊥ | ⊥| |∇fij | |∇fij 0 0 ij (I + μS)−1 =

and 1 ∇fij I 2 = |∇f ij | α + trS ij

⊥ ∇fij ⊥| |∇fij

1 α+tr S ij

0

we have the relation F −1 ij S ij

1 ∇fij = I 2 S ij = |∇f ij | α + trS ij

Therefore, ρ(I 2 − F −1 ij S ij )

⊥ ∇fij ⊥| |∇fij

0

1 α+tr S ij

tr S ij α+tr S ij

0

⊥ ∇fij ∇fij ⊥| |∇fij | |∇fij

0 0

⊥ ∇fij ∇fij ⊥| |∇fij | |∇fij

.

trS ij trS ij −1 , ρ(I − P F P S) = max 1 − . = 1 − ij α + trS ij α + trS ij


273

Spectrums of Matrices For the matrices D1 and D 2 , setting D 1 U = Λ1 U , D 2 V = Λ2 V where U and V are orthogonal matrices, and Λ1 = Diag λ1M , λ1M−1 , · · · , λ11 , Λ2 = Diag λ2M , λ2M−1 , · · · , λ21 ,

(50)

the eigenvalues are

λ1k = − 1 − cos

π k, M +1

π , λ2k = − 1 − cos k, . M

Since B ε = Dε + I, we have the eigenvalues of B 1 and B 2 μ1k = cos

π π k, μ2k = cos k. M +1 M

Therefore, ρ(B 1 ) < 1 and ρ(B 2 ) < 1. Furthermore, the eigenvalues of N 2 and M 1 are μij = μ2i +μ2j and κij = μ1i μ1j . Therefore, we have the relation ρ(N 2 ) < 1 and ρ(M 1 ) < 1.