Integrating Stereo Disparity and Optical Flow by Closely-coupled Method

J OURNAL OF PATTERN R ECOGNITION R ESEARCH 1 (2012) 175-187 Received Nov 15, 2009. Revised Nov 15, 2011. Accepted Nov 30, 2011.

Integrating Stereo Disparity and Optical Flow by Closely-coupled Method Sang Hyun Han [email protected] Department of Electrical Engineering, Pohang University of Science and Technology, Pohang, Gyungbuk 790-784, Republic of Korea Yan Sheng

[email protected] Department of Electrical Engineering, Pohang University of Science and Technology, Pohang, Gyungbuk 790-784, Republic of Korea

Hong Jeong

[email protected]

Department of Electrical Engineering, Pohang University of Science and Technology, Pohang, Gyungbuk 790-784, Republic of Korea

WWW. JPRR . ORG

Abstract As a convergence method for stereo matching and motion estimation, this paper presents an equation, called Disparity-Optical flow Equation, that relates disparity with optical flow in rectified images. Considering this equation as a constraint, this paper suggests an algorithm, called Simultaneous Disparity and Optical Flow under Epipolar Constraints, that efficiently determines the disparity and optical flow. This algorithm is completely different from the previous approaches that try to compromise the outputs of the two modules. The experiments show that the algorithm can resolve the ambiguous situations in which either stereo matching or motion estimation fail to yield satisfactory results. Keywords: Disparity, Optical flow, Stereo Matching, Motion Estimation, Rectification

1. Introduction One of the major goals of computer vision is the reconstruction of shapes from 2D images; this process is known as shape from modules including shading, stereo, motion and texture. The stereo module is one important member of this group. The fundamental goal of this module is to estimate the depth of image by computing the disparity between two images by stereo matching. The performance is bounded by the projection model used, and by optical obscurity and occlusion. Stereo matching can be classified into area-based methods [24, 23] and featurebased methods [16]. The other taxonomy is based on the concept of stochastic method, such as fast belief propagation [18, 21], maximum likelihood estimation by dynamic programming [28] and maximum a posteriori estimation using Bayesian estimation theory [9, 1] or using belief propagation [26, 7]. In addition, the soft computational methods are applied, such as fuzzy theory [27] or neural network [20]. An exhaustive taxonomy and evaluation of stereo matching algorithms is available in [22]. Another important module is the motion estimation. Motion is an important feature of image sequences; it reveals the dynamics of scenes by relating spatial image features to temporal changes. Optical flow is the recovery of motion information from an image sequence; this technique is widely used to compute depth [29], surface orientation and 3D structure [12, 17]. The computation of optical flow can date back to Horn and Schunck [11] who proposed an assumption that the image brightness at a given point should only change due to motion. Since then, various algorithms have been proposed using this brightness constancy assumption [13]. An excellent survey on performance of different algorithms is given by Barron et al. [2]. c 2012 JPRR. All rights reserved. Permissions to make digital or hard copies of all or part of this work for personal or

classroom use may be granted by JPRR provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or to republish, requires a fee and/or special permission from JPRR.

S ANG H YUN H AN , YAN S HENG , H ONG J EONG

The approach in the next level is the integration of the two modules to reduce ambiguities in solutions. One approach is to combine the information obtained separately from motion and stereo modules. Depths are recovered separately from both modules and then combined to make the final decision as scene structure [4, 10] or 3D scene flow [15]. The ultimate approach is to combine information from the stereo and motion modules even before the depth estimation. Coupled Markov random fields defining interprocess interactions do this by exploiting some natural constraints between the discontinuity of optical flow and the discontinuity, or occlusion in disparity [19, 3, 25]. Other methods calculate the transition of a stereo rig in a time sequence by tracking feature point and using the transition information to achieve stereo matching [5, 14]. In summary, merging motion and stereo might be realized on the system level (Fig. 1(a)). We

(a) Loosely-coupled manner

(b) Closely-coupled manner Fig. 1: Integrating stereo and motion.

call this loosely-coupled method. In this method, the stereo matching and optical flow module each estimates the depth, then the two estimates are combined to provide a better one. However, this approach is limited in that the uncertainties in optical flow and disparity cannot be corrected before depth estimation, so this might cause serious errors in final decisions. Differently from the previous methods, this paper finds the explicit relationships between optical flow and disparity in the image without requiring estimates of the distances to surfaces (Fig. 1(b)). We call this closely-coupled method. The discovered equation, disparity and optical flow equation (DOE), relates disparity to optical flow in the image planes. Compared to the loosely-coupled methods, this closely-coupled method allows less ambiguity. By DOE, this paper suggests an algorithm that computes the disparity and optical flow efficiently. Previously, we derived a recursive algorithm to acquire the disparity [30], whereas in this paper, we determine the disparity and optical flow simultaneously. This paper is organized as follows. § 2 defines the optical setting for both stereo matching and motion estimation. § 3 derives the DOE, which relates stereo disparity with optical flow in the image sequence. Using this relationship, § 4 determines the disparity and optical flow in a tight manner using Simultaneous Disparity and Optical Flow under Epipolar Constraints (SDOE). Experimental result are given in § 5 followed by conclusion in § 6.

2. Problem Statements Consider the case (Fig. 2), in which the origins Ol and Or are, respectively, the projective centers for the left and right images, Itl and Itr ; the image planes are coplanar and the optical axis are parallel, separated by a baseline B. In this setting, a 3D point P = (X, Y, Z)T is projected onto the two 176

I NTEGRATING S TEREO D ISPARITY AND O PTICAL F LOW BY C LOSELY- COUPLED M ETHOD

z

V P(X,Y,Z)

pl

pr vr

vl

Ol

y

Or

x

B

Fig. 2: The parallel optical alignment

image planes as pl = (xl , yl )T and pr = (xr , yr )T . This alignment is called parallel optics and the epipolar lines, on which corresponding points are collinear. The purpose of this simple alignment is to use the rectified images, for which image analysis is much more convenient. The rectified environment can be related with the convergent optics with projective mapping if needed. Two images of the same size M × N are captured at constant intervals of time t. In 3D-space, the point P is generally moving with a velocity V = (U, V, W )T , where (U, V, W ) describes the translation components in X,Y and Z direction. Accordingly, the two projected points also move on the image planes, with vl = (ul , vl ) and vr = (ur , vr ); this is called optical flow. Also, the two projected points are separated with disparity d on both image planes. Incidentally, the rotational movement is omitted to make the problem simple. If the sampling rate is fast enough or, equivalently, the rotation is slow enough, this assumption is correct for general case. For a given point P in 3D-space, the image points pl and pr in homogeneous form have relationships with the projective matrix M [8]: ( pl = Ml P (1) pr = Mr P . The matrix M consists of the internal matrix Mi and the external matrix Me : M = Mi Me . Meanwhile, the stereo pair has the following relationship with the fundamental matrix F [6]: pTr F pl = 0,

(2)

where −T F = Mir EMil−1 .

The matrix E is called the essential matrix; it establishes a natural link between the epipolar constraint and the extrinsic parameters of the stereo system. (1) and (2) are the bases of all the derivations in this paper and lead to the derivation of the relationships between disparity and optical flow. 177


3. Relationships between Optical flow and Stereo Disparity Optical flow and disparity are simply two different 2D-space measures on the image plane of the same phenomena in the 3D-space scene. Therefore, strong relationships must exist between the two measures. Also, all of these relationships must be geometrical and can be derived from (1) and (2), which cover the original position and the two projected positions. This section derives the explicit forms first for the special case with parallel optics in which the epipolar lines are collinear. Then, the general case in which the optical axes are oriented arbitrarily is derived. First, consider the optical flow. When a scene point P = (X, Y, Z)T is perspectively projected to the point p = (x, y, f )T , their relationship is p=f

P . Z

(3)

When the point p is projected on a charge-coupled device, measurements must be converted from meters to a number of pixels. Assuming that radial distortions of the lens can be neglected and that the image center is the origin of the image reference frame, (3) can be rewritten in the image reference frame as ( xim = sfx X Z, (4) f Y yim = sy Z , where sx and sy denote the horizontal and vertical effective pixel size in metric respectively in units of m/pixel. For simplicity, we omit the subscript im indicating image (pixel) coordinates, and write (x, y) for (xim , yim ). Let Ix = f /sx and Iy = f /sy , because sx = sy in most cases, for simplicity, we omit the subscript x or y of f . Then, we convert (4) to the form ( x=fX Z, (5) Y y = fZ. In all the subsequent equations, lower-case letters have units of pixels and upper-case letters have units of meters. Taking the time derivative of both sides of (3) yields v=f

ZV − W P , Z2

(6)

where v = (u, v, 0)T is the optical flow of the image point in unit of pixel and V = (U, V, W )T is the velocity of the corresponding scene point. 3.1 Parallel optical axes Consider the special case with parallel optics in which the epipolar lines are collinear. In this case the relationship between disparity d and depth Z along with baseline B is d=

fB . Z

(7)

Taking the time derivative of both sides, one obtains d′ = −

f BW . Z2

(8)

To distinguish the denotation of disparity at time t from the time derivative of disparity, we write them as dt and d′ , respectively. 178


The three vectors, V = (U, V, W )T , vl = (ul , vl )T , and vr = (ur , vr )T of P must be related. Writing (6) in components, according to original point Ol and Or , becomes ( (ul , vl ) = Zf (U, V ) − f ZW2 (X, Y ), (9) (ur , vr ) = Zf (U, V ) − f ZW2 (X − B, Y ). Substituting (8) to (9) yields the equation: (

ul − ur = d′ , vl − vr = 0.

(10)

These relationships are very important because they directly relate disparity with optical flow. We call this the DOE. The first equation specifies that the difference between the two optical flows is identical to the difference between the two disparities. The second equation represents the epipolar assumption: the corresponding points are on the same epipolar line whenever the point moves. There are two equations and five unknowns, so three more equations must be obtained from the optical flow matching and the stereo matching. 3.2 Arbitrarily-oriented optical axes Now we consider general case in which the optical axes are arbitrarily aligned, as described by (1) and (2). In this case, the disparity can be defined in a general setting as: d = pl − pr . Taking the time derivative of (11) and (2) yields ( vrT F pl + pTr F vl = 0, d′ = vl − vr .

(11)

(12)

We call these constraints the Generalized Disparity-Optical flow Equation (GDOE). The relations in (12) involve only with the image quantities and not with depth or velocities themselves. First equation in (12) relates optical flows in stereo pair images with positions and fundamental matrix. The second equation relates disparity and optical flow, specifying that the time differential of the disparity must be equivalent to the difference between the optical flows in stereo pair images. In real applications, using DOE is much more convenient than GDOE. The general case can be covered by following the three steps. First, the two image pairs are rectified according to the fundamental matrix. Next, the stereo matching and optical flow computation are performed in subsequent steps. Finally, the depth and velocity are recovered by inverse projection. Therefore, we confine ourselves in the rectified images.

4. Using DOE in Stereo and Optical Flow Matching So far, we have derived the relationships, DOE and GDOE. To obtain the five unknowns, three more equations are needed. This problem can be solved in the following way (Fig. 3). For two consecutive time frames, two l , I r ) and (I l , I r ). It is assumed that the variable d are already pairs of images are defined, (It−1 0 t t t−1 l r determined for (It−1 , It−1 ). Then, the problem is to determine (ul , vl , ur , vr , d) at t for (Itl , Itr ). According to this figure, one can consider that the problem is to find the two other points at t, given one point and its corresponding point at t − 1. Due to DOE, the degree of freedom becomes 179


dt 1

t1 Vl

Vr

t dt Fig. 3: The four correspondence problem.

three. For example, the unknowns can be defined as either (ul , ur , vl ) at t or (ul , d′ , vl ) at t. To determine the three variables, we need at least three more equations. These equations come from the matching both for optical flow and stereo as will be explained below. First, consider the optical flow matching. Under the standard brightness constancy assumption [11], we have I(x, y, t) = I(x + uδt, y + vδt, t + δt).

(13)

Taking Taylor expansion of (13) [11] to its first order yields the form of data conservation constraint: Ix u + Iy v + It = 0,

(14)

where Iz = ∂I ∂z . The constraint on the optical flow expressed by this equation is illustrated in (Fig. 4). It is worth noting that there are two unknowns, u and v, in one equation. So, (14) is rank deficient

u It/!Ix

v It/!Iy Fig. 4: The dashed line is the constraint line of optical flow. Any vector which ends on this line is a solution of (14). So, recovering the optical flow is an ill-posed problem known as aperture problem

to solve for u and v. This is often called the aperture problem. As a consequent, the optical flow cannot be computed without introducing some additional constraints such as smoothness constraints in monocular situation. However, in binocular case, this problem can be made fully ranked by introducing the disparity d (or d′ ). 180


4.1 Pixel-wise solution Consider the parallel optical axes alignment for two cameras and put DOE as well as data conservation constraint (14) together, we have   ul − ur = d′ ,    v − v = 0, r l Irx ur + Iry vr + Irt = 0,    I u + I v + I = 0, lx l ly l lt

(15)

a where Iaz = ∂I ∂z . The interpretation of four equations is drawn in Fig. 5a. The vertical separation between the

u

u l

l t

l

x

r r t x

Vl

l t

x

t

r x

r

Vl

Vr

Vr

v

v l

l t y

r r t y

r r t y

(a) Normal case

l

l t

y

(b) Degenerate case

Fig. 5: Relationship between d′ and (xl , xr ).

two lines defined by the optical flow is d′ . Once d′ is known, xl and xr are determined uniquely. However, if the two lines are in parallel, there is no unique solution. This case is the singular case in the following matrix inversion. Rewrite (15) in matrix form:    ′  d ul 1 −1 0 0     0 0 1 −1 ur   0    Ilx 0 Ily 0   vl  =  −Ilt  . −Irt vr 0 Irx 0 Iry 

(16)

This can be written as Ax = b, where x is the unknown. If −Irx Ily + Ilx Iry 6= 0, (16) becomes   ul ur  1  =  vl  Ilx Iry − Ily Irx vr

 −Irx Ily d′ − Iry Ilt + Ily Irt −Ilx Iry d′ − Iry Ilt + Ily Irt     Ilx Irx d′ + Irx Ilt − Ilx Irt  Ilx Irx d′ + Irx Ilt − Ilx Irt 

(17)

It is worth noting that when −Irx Ily + Ilx Iry = 0, there is no unique solution for (16) (Fig. 5b). That is, if accidentally Iry /Irx = Ily /Ilx holds, the flow vectors cannot be determined. One way to deal with this is to set the optical value in the direction of image intensity gradient. 181


The optical flow is a function of d′ : u vl (d ) = l = vl  " #  I I  lx lt 1  , if Ilx Iry − Ily Irx = 0,  2 +I 2 − Ilx ly Ily"Ilt # ′−I I +I I  −I I d  rx ry rt ly lt ly 1  , otherwise.   Ilx Iry −Ily Irx I I d′ + I I − I I rx lt lx rx lx rt u ′ vr (d ) = r = vr  " #  I I  rx rt 1  , if Ilx Iry − Ily Irx = 0  2 +I 2 − Irx ry Iry " Irt # ′−I I +I I  −I I d  ry ry rt lx lt ly 1  , otherwise.   Ilx Iry −Ily Irx I I d′ + I I − I I ′

lx rx

rx lt

(18)

lx rt

So far, it is assumed that d′ is given and (xl , vr ) are unknowns. To obtain d, we need one more equation, which is the stereo matching equation. In real application, d′ is regarded as the difference of disparity between consecutive frames: d′ = d − d0 .

(19)

As discussed, the problem needs three equations and two were found. The remaining one is the stereo matching:

Ir (x, y) − Il (x + d, y) = 0.

(20)

Here d = d0 + d′ with d0 being the disparity in the previous frame. As a result, the four variables can be computed by solving (18) and (20). 4.2 Patch-wise solution The equations, (18) and (20), must be extended to wider neighbors to deal with noise and other image deficiencies. This can be realized either by using the pixel equation on the pyramid structure or the equation extended with neighborhood, that is to be derived here. The patch-wise solution can be obtained by the aid of Lagrangian. Define the vector I˜x at Q:   w1 I(x1 )  w2 I(x2 )    ˜ I= (21) . ..   . wn2 I(xn2 )

The weighting term wi gives more weight to the pixels that are close of center of Q than those at the periphery. Then, (15) can be rewritten as minimizing the cost function Φ[v] in terms with v: Φ[v] =

X

[||I˜rx ur + I˜ry vr + I˜rt ||2 + ||I˜lx ul + I˜ly vl + I˜lt )||2 ]

xi ∈Q

182


subject to (

ul − ur = d′ , vl − vr = 0.

With the Lagrange multipliers λ1 and λ2 , construct the Lagrange function Λ[v, λ1 , λ2 ]: Λ[v, λ1 , λ2 ] = Φ[v] − λ1 (ul − ur − d′ ) − λ2 (vl − vr ).

(22)

∇v,λ1 ,λ2 Λ[v, λ1 , λ2 ] = 0,

(23)

Solving

one obtains 

−1     ˜T ˜ T I˜ T I˜ 0 1 0 2Irx Irx 0 2I˜ry −2I˜rt ur rx rx   T I˜ T I˜  ul   0 2I˜lx 0 2I˜ly 0 lx lx −1   −2I˜ltT I˜lx         T T T  vr  2I˜ I˜ry 0 2I˜ry I˜ry 0 0 1  −2I˜rt I˜ry    =  rx   .  vl   0 T I˜ T I˜ 2I˜lx 0 2I˜ly 0 −1  −2I˜ltT I˜ly  ly ly       λ1   −1  1 0 0 0 0   d′ λ2 0 0 0 −1 1 0 0

(24)

4.3 Energy function To solve either (18) (or (24) ) and (20) in five variables, some kind of tracking or searching is needed. Depending upon the search method, different algorithms may result. However, one of the efficient methods is to compute the stereo matching along the epipolar line (Fig. 6). dt 1

t1 Vl

Vr

t a dt Fig. 6: The possible solutions. For every possible disparity of a given point at time t in right image, there is only one corresponding optical flow in left and right image in non-degenerate case.

Along the epipolar line, the matching errors are accumulated and must be minimized. First, all the parent nodes must be found. The nodes are found using the optical flow previously computed. In this figure, for a pair (alt , art ), there can be many parent nodes (alt−1 , art−1 ). Only the parent nodes that minimize the accumulated cost must be chosen. The principle is that for a good pair of corresponding points, their previous points must also be a good pair. For the given d at (x, y, t), (vl , vr ) can be calculated with (16). Then, the previous node at t − 1 can be calculated by (x, y) − v. This method is described as follows. X min (Itr (x, y) − Itl (x + d, y))2 d

x

r l + λ(It−1 ((x, y) − vr ) − It−1 ((x + d, y) − vl )2 + µ|∇d|2 .

183


Here, (d, vr , vl ) satisfies (16). In the ordinary stereo matching, the second term is missing. The optimization is done line by line basis. This can be realized by the optimization methods such as dynamic programming, belief propagation (BP) and graph cuts, though BP is chosen in this paper. 4.4 BP realization From (18), we regard that the optical flow in the left and right images as a function of d′ and can be written as vl (d′ ) = [ul , vl ]T , vr (d′ ) = [ur , vr ]T ,

(25)

respectively. The framework for the problem can be regarded as searching disparity at each pixel. We assume that the disparity should vary smoothly almost everywhere except at some region, such as object boundary, where intensity changes drastically. Therefore, we can define a cost function aimed at finding a optical disparity as X X V (dp , dq ) + Dp (dp ) (26) E(d) = (p,q)∈N

where subscripts p and q denote the neighbor position and V (dp , dq ) is the smoothness cost function, which is defined as V (dp , dq ) = min(ρ|dp − dq |, dc ),

(27)

where dc is a truncation term that prevents smoothing around edges. The data term, DP (d), is the cost of assigning d as the disparity at each pixel and is normally referred to as the data cost which is defined as Dp (d) = |Irt (x, y) − Ilt (x + d, y)|

+ α|Irt−1 ((x, y) − vr ) − Ilt−1 ((x + d, y) − vl )|. (28)

5. Experimental Results The experiments need 3D video images, such as 3D movies. However, synthetic images are also needed since various effects including noise, illumination and geometrical distortion can be controlled as desired. The two types of images are used here. 5.1 Synthetic images The images are synthesized so that periodic patterns are included. Estimating accurate disparity in such images is a very difficult task, because of ambiguity in matching. Fig. 7 shows a periodic pattern at time t − 1 and t, where for a given point in right image there exist many matching points in the left image. If the nearest matching is regarded as right matching, the result is not satisfactory as in Fig. 8(a). However, if the information of motion between t − 1 and t is available, the proposed algorithm can recover the disparity correctly in the entire region (Fig. 8(b)). This is due to the small block that breaks the periodic pattern at t − 1. It is easy to infer the right disparity by the position of the block. This information also propagates to the next frame with object movement. See Fig. 8(b), where the disparity is correctly recovered in the region around the moving block. Besides disparity, the optical flow has been recovered as shown in Fig. 5.1. 5.2 3D Video streams Next, two stereo video streams are experimented (Fig. 9). The images are obtained by 3D cameras and thus appropriately rectified for experimentation. 184


(a) Left image at t

(c) Left image at t − 1

(b) Right image at t

(d) Right image at t − 1

Fig. 7: Two frames of test images 0 10 20 30 40 50 60 70 80

0

(a) Disparity map without motion information BP

(b) Disparity map by SDOE

10

20

30

40

50

60

70

80

(c) Optical flow by SDOE

Fig. 8: The estimated disparities and optical flows

Fig. 9(a) is the original left image of video that two trains are running in opposite direction; one train on near side is running towards right and the other on far side is running towards left. Besides the trains, the background is stationary. Observe the wire pole standing upright near the center. Similarly, Fig. 9(d) is the original left image and the two trains are running in opposite direction. The ordinary stereo matching, by BP, yields the result in Fig. 9(b). One can see the erroneous disparity region on the right side of the wire pole, which must be due to the homogenous texture. This can be drastically improved by the proposed algorithm, as can be seen in Fig. 9(c). Fig. 9(e) is also resulted from the ordinary stereo matching applied on Fig. 9(d). When compared with result from SDOE (Fig. 9(f)), our method improves the disparity map on the center part where BP method generates erroneous bumps region. In addition, the shape of panel area is more reasonable in SDOE than BP. The information from motion estimation helps stereo matching around the homogeneous region. Also, the other regions are also improved as can be observed.

6. Conclusion In this paper, we suggested a geometrical relationship called a disparity-optical flow equation and an algorithm SDOE for calculating stereo and optical flow together. Our algorithm is proposed to improve stereo matching and motion estimation. Experiment results demonstrate that our proposed algorithm can boost the performance of disparity considerably and given reasonable optical flow calculation at the same time. Further work should be done in two directions. First, a suitable public test stereo time sequence images are still not available. So it is important to generate good test images with reasonable ground truth of disparity and optical flow. Second, this algorithm may be improved by using more than one previous frame to estimate disparity and optical flow. 185


(a) left image of 3D movie

(b) Disparity map by BP

(c) Disparity map by SDOE

(d) left image of 3D movie

(e) Disparity map by BP

(f) Disparity map by SDOE

Fig. 9: A frame of 3D movie and the disparity result

Acknowledgments This work has been supported by the following funds: the Brain Korea 21 Project, the Ministry of Knowledge Economy, Korea, under the Core Technology Development for Breakthrough of Robot Vision Research support program supervised by the National IT Industry Promotion Agency.

References [1] Edgar Arce and J.L. Marroquin. High-precision stereo disparity estimation using hmmf models. Image and Vision Computing, 25(5):623 – 636, 2007. [2] J.L. Barron, D.J. Fleet, and S.S. Beauchemin. Performance of optical flow techniques. International Journal of Computer Vision, 12:43–77, 1994. [3] S.P. Clifford and N.M. Nasrabadi. Integration of stereo vision and optical flow using markov randomfields. Neural Networks, 1:577–584, 1988. [4] T. Dang, C. Hoffmann, and C. Stiller. Fusing optical flow and stereo disparity for object tracking. In IEEE International Conference on Intelligent Transportation Systems, pages 112–117, 2002. [5] Fadi Dornaika and Ronald Chung. Stereo correspondence from motion correspondence. Computer Vision and Pattern Recognition, 1:70–75, 1999. [6] O. Faugeras. Three-Dimensional Computer Vision: A geometric viewpoint. The MIT Press, 1993. [7] Pedro F. Felzenszwalb and Daniel P. Huttenlocher. Efficient belief propagation for early vision. volume 1, pages 261–268, 2004. [8] David A. Forsyth and Jean Ponce. Computer vision : a modern approach. London : Prentice Hall, 2003. [9] Salvador Gutierez and Jose Luis Marroquin. Robust approach for disparity estimation in stereo vision. Image and Vision Computing, 22(3):183 – 195, 2004. [10] A.Y.K. Ho and T.C. Pong. Cooperative fusion of stereo and motion. Pattern recognition, 29(1):121–130, 1996. [11] Berthold K.P. Horn and Brian G. Rhunck. Determining optical flow. Artificial Intelligence, 17:185–203, 1981. [12] Tony Jebara, Ali Azarbayejani, and Alex Pentland. 3d structure from 2d motion. IEEE Signal Processing Magazine, 16(3), May 1999.

186


[13] Pierre-Marc Jodoin and Max Mignotte. Optical-flow based on an edge-avoidance procedure. Computer Vision and Image Understanding, 113(4):511 – 531, 2009. [14] S. Kagami, Y. Takaoka, Y. Kida, K. Nishiwaki, and T. Kanade. Stereo correspondence from motion correspondence. pages 3858–3863, 2005. [15] R. Li and S. Sclaroff. Multi-scale 3D scene flow from binocular stereo sequences. Computer Vision and Image Understanding, 110(1):75–90, 2008. [16] M.Lew, T.Huang, and K.Wong. Learning and features selection in stereo matching. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(9):869–882, 1994. [17] T. Morita and T. Kanade. A sequential factorization method for recovering shape and motion from image streams. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(8):858–867, 1997. [18] Intae Na, Junghun Choi, and Hong Jeong. Robust fast belief propagation for real-time stereo matching. volume 2, pages 1175–1179, 2009. [19] Nasser M. Nasrabadi, Sandra P. Clifford, and Yi Liu. Integration of stereo vision and optical flow by using an energy-minimization approach. Journal of the Optical Society of America, 6:900–907, 1989. [20] NM Nasrabadi and CY Choo. Hopfield network for stereo vision correspondence. IEEE transactions on neural networks, 3(1):5–13, 1992. [21] Sungchan Park and Hong Jeong. Memory efficient iterative process on a two-dimensional first-oder regular graph. OPTICS LETTERS, 33:74–76, 2008. [22] D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two frame stereo correspondence algorithms. Interational Journal of Computer Vision, 47:7–42, 2002. [23] Mikhail Sizintsev and Richard P. Wildes. Coarse-to-fine stereo vision with accurate 3d boundaries. Image and Vision Computing, 28(3):352 – 366, 2010. [24] L. Di Stefano, M. Marchionni, S. Mattoccia, and G. Neri. A fast area-based stereo matching algorithm. Image and Vision Computing, 22:983–1005, 2002. [25] G. Sudhir, Subhashis Banerjee, K.K. Biswas, and R. Bahl. Cooperative integration of stereopsis and optic flow computation. J. Opt. Soc. Am. A, 12:2564–2572, 1995. [26] Jian Sun, Heung yeung Shum, and Nan ning Zheng. Stereo matching using belief propagation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25:787–800, 2003. [27] G. Tolt and I. Kalaykov. Measures based on fuzzy similarity for stereo matching of color images. Soft Computing-A Fusion of Foundations, Methodologies and Applications, 10(12):1117–1126, 2006. [28] P. H. S. Torr and A. Criminisi. Dense stereo using pivoted dynamic programming. Image and Vision Computing, 22(10):795 – 806, 2004. [29] Jens Vogelgesang, Alex Cozzil, and Florentin W¨org¨otterl. A parallel algorithm for depth perception from radial optical flow fields. Artificial Neural Networks, 1112(4):721–725, 1996. [30] Sheng Yan and Hong Jeong. Stereo and optical flow calculation using recursive filter. In International Conference on Information Engineering and Computer Science, volume 1, pages 126–129, 2009.

187