Dense Multi-Frame Optic Flow for Non-Rigid Objects ... - CiteSeerX

Dense Multi-Frame Optic Flow for Non-Rigid Objects using Subspace Constraints

?

Ravi Garg1 , Luis Pizarro2 , Daniel Rueckert2 and Lourdes Agapito1 1 2

Queen Mary University of London. Mile End Road, London E1 4NS, UK Imperial College London, South Kensington Campus, London SW7 2AZ, UK

Abstract. In this paper we describe a variational approach to computing dense optic flow in the case of non-rigid motion. We optimise a global energy to compute the optic flow between each image in a sequence and a reference frame simultaneously. Our approach is based on subspace constraints which allow to express the optic flow at each pixel in a compact way as a linear combination of a 2D motion basis that can be pre-estimated from a set of reliable 2D tracks. We reformulate the multi-frame optic flow problem as the estimation of the coefficients that multiplied with the known basis will give the displacement vectors for each pixel. We adopt a variational framework in which we optimise a nonlinearised global brightness constancy to cope with large displacements and impose homogeneous regularization on the multi-frame motion basis coefficients. Our approach has two strengths. First, the dramatic reduction in the number of variables to be computed (typically one order of magnitude) which has obvious computational advantages and second, the ability to deal with large displacements due to strong deformations. We conduct experiments on various sequences of non-rigid objects which show that our approach provides results comparable to state of the art variational multi-frame optic flow methods.

1

Introduction

Dense registration of deforming surfaces from a monocular image sequence continues to be one of the unsolved fundamental problems in computer vision, despite the attention it has received from the community for decades. Its applications are numerous from video augmentation to non-rigid structure from motion or medical imaging. For instance, non-rigid structure from motion [1–3] and scene-flow techniques for deformable surfaces [?] rely on the efficient estimation of image correspondences. However, most state of the art algorithms only use sparse features obtained with local feature matching techniques such as Lucas and Kanade’s popular tracker [5]. While this algorithm provides good matches in areas with rich texture, it fails to provide reliable solutions in texture-less areas with vanishing ?

This work is supported by the European Research Council under ERC Starting Grant agreement 204871-HUMANIS. We are grateful to Visesh Chari for sharing his optical flow code and to Adrien Bartoli to share T-shirt sequence.

2

R. Garg et al.

gradients due to the well known aperture problem. Therefore, most current nonrigid structure from motion algorithms are limited to sparse 3D reconstructions. Irani’s was the first work to exploit rank constraints in the case of rigid objects to obtain optic flow in areas with one dimensional or no texture [6]. She proved that the optic flow vectors lie on a lower dimensional subspace and therefore the flow at each point can be expressed as a linear combination of a low-rank motion basis. This constraint was then used for estimating dense correspondences, by requiring that all corresponding points across all video frames reside in the appropriate low-dimensional linear subspace. However, although this algorithm indeed allows to solve the aperture problem, it performs poorly in areas of uniform intensity. Moreover, this approach cannot cope with large displacements since they rely on the linearised brightness constancy assumption which assumes small displacements. Besides, since they do not impose smoothness constraints, the resulting optic flow is not regular. The rank constraint was later extended to the non-rigid case by Torresani et al. [7] and Brand [8]. These methods also minimise the linearised brightness constancy and therefore they suffer when there are image displacements larger than a few pixels or local appearance changes due to large deformations. Besides, although in theory these non-rigid approaches are dense, in practice they have only been used to extend the tracking to features which display the aperture problem (such as edges or degenerate features) instead of computing optic flow values for every pixel in the image. In contrast, variational methods allow to formulate the optic flow problem in its continuous form. Pioneered by Horn and Schunck [9], the optic flow problem is formulated as the optimization of an energy functional with a regulariser that allows to fill in textural information into non-textured regions from their neighbourhoods, making dense flow field estimation possible. Moreover, recent developments in variational optical flow [10, 11] have proposed numerical strategies to solve the non-linearised brightness constancy constraint in order to cope with large displacements. Although geometric constraints have been incorporated before in the computation of optic flow, this has been in the case of rigid scenes. For instance, fundamental matrix priors have been proposed within vatriational approaches [14–16] to improve the accuracy of optic flow. To the best of our knowledge this is the first approach to apply subspace constraints to the case of non-rigid motion in a variational framework. In this paper we propose to marry the ideas of using subspace constraints to constrain the optic flow and solving the non-linearised brightness constancy constraint within a variational approach. This allows us on the one hand to reduce the dimensionality of the multi-frame optic flow problem drastically and on the other to be able to cope with large displacements while imposing smoothness on the optic flow taking advantage of the variational formulation. We focus on the more challenging problem of non-rigid motion and adopt a multi-frame optical flow approach by optimising a global energy.

Dense Multi-Frame Optic Flow for Non-Rigid Objects

2

3

Low-rank non-rigid subspace constraints

Our approach is based on the assumption that the motion of a non-rigid object can be represented as a linear combination of K basis shapes which encode the mean shape and its main modes of deformation. This low-rank constraint, first proposed by Bregler et al. [17], has allowed the simultaneous estimation of 3D non-rigid shape and motion. More importantly for the scope of this paper, it also induces a subspace rank constraint on the 2D motion of image points. In the next two sections we describe the linear basis shape model that underpins our formulation and the linear subspace constraint satisfied by the multi-frame correspondences. 2.1

Linear basis shape model

The 3D reconstruction of a rigid body from a monocular sequence under the affine projection assumption was pioneered by Tomasi and Kanade in [18] using a factorization approach. This assumption was then relaxed to extend structure from motion algorithms to the case of deformable objects. Most non-rigid structure from motion algorithms are based on the low rank basis shape model defined by Bregler et al. [17] in which the deformable 3D shape is represented as a linear combination of a set of basis shapes with time varying coefficients. In the case of deformable objects the observed 3D points change as a function of time. In this paper we use the low-rank shape model defined by Bregler et al. [17] in which the P observed 3D points deform as a linear combination of a fixed set of K rigid shape bases according to time varying coefficients. In K P this way, Sf = lf k Bk where the matrix Sf = [Sf 1 , · · · Sf P ] is the 3D shape k=1

of the object at frame f , the 3 × P matrices Bk are the shape bases and lf k are the coefficient weights. If we assume an orthographic projection model, the coordinates (xf j , yf j ) of the 2D image points j observed at frame f are then related to the coordinates of the 3D points according to the following equation: !   K X x x · · · x f 1 f 2 f P ^ Wf = = Rf lf k Bk + Tf (1) yf 1 yf 2 · · · yf P k=1

where Rf is a 2 × 3 truncated rotation matrix and the 2 × p matrix Tf aligns the image coordinates to the image centroid. When the image coordinates are registered to the centroid of the object and we consider all the frames in the sequence, we may write the measurement matrix as:   x11 x12 · · · x1P     y11 y12 · · · y1P  l11 R1 . . . l1K R1 B1     .. .. ..  ..   ..  = M ^ =  ... . . . Wf =  ... 2F ×3K B3K×P . . .  .  .     xF 1 xF 2 · · · xF P  lF 1 RF . . . lF K RF BK yF 1 yF 2 · · · yF P 2F ×P (2)

4

R. Garg et al.

Since M is a 2F × 3K matrix and B is a 3K × P matrix, in the case of deformable structure the rank of ^ Wf is constrained to be at most 3K. This rank constraint forms the basis of the factorization method for the estimation of 3D deformable structure and motion [17]. Interestingly, the motion and shape matrices can exchange their roles as basis and coefficients and we can either interpret the 2D tracks as the projection of a linear combination of 3D basis shapes(Bk ) or as the linear combination of a 2D motion basis encoded in matrix M. This concept of 2D trajectory basis was introduced by Torresani et al. [19] as an extension to the non-rigid case of the subspace constraints proposed Irani [6]. 2.2

2D low-rank trajectory basis

Let us denote the 2D motion of point j with respect to its position in the reference image x0j = (x0j , y0j ) by the vector wj = [u1j u2j · · · uF j |v1j v2j · · · vF j ]T

(3)

where uij = xij − x0j and vij = yij − y0j . We now consider P pixels or image features observed in the image and stack their corresponding multi-frame 2D motion vectors horizontally to form a 2F × P measurement matrix W W = [w1 w2 · · · wP ]

(4)

It is easy to observe that W is also rank constrained, therefore the multi-frame motion vector wj of any point j can be expressed as linear combination of R basis motion vectors Qr R X wj = Lrj Qr (5) r=1

· · · QvF r ]T and Lrj with r = 1 · · · R are the where Qr = coefficients that multiply basis vectors Qr to obtain wj . We may now rewrite this equation for every point in matrix form as   u Q11 · · · Qu1r · · · Qu1R  .. .. ..   L1 · · · L1 · · · L1   . . .   L21 · · · L2j · · · L2P   u   u  QF 1 · · · Qu1r · · · QuF R   j P  Q   .1 (6) W2F ×P =  .. ..  = Qv L  Qv11 · · · Qu1r · · · Qv   .   1R . . .    . .. ..  LR · · · LR · · · LR  .. 1 j P . .  | {z } QvF 1 · · · Qu1r · · · QvF R LR×P | {z } [Qu1r

Qu2r

· · · QuF r |Qv1r

Qv2r

Q2F ×R

It is easy to see that the motion basis matrix Q is independent of the number of points. Therefore it is possible to pre-compute Q from a small subset of “reliable” point tracks by truncating the SV D decomposition of the corresponding measurement matrix Wrel to rank R. svd (Wrel ) = |{z} U |{z} ΛVT Q

Lrel

(7)

Dense Multi-Frame Optic Flow for Non-Rigid Objects

5

The “reliable” tracks are those where the texture of the image is strong in both spatial directions which can be selected using Shi et al. [20]. Now the motion of any other point, or set of points, in the image can be encoded in terms of the known basis Q and only the new coefficients L need to be computed. This will form the basis of our re - parametrisation of the multi-frame optic flow problem. In the next section we will introduce our variational approach to solve the multi-frame dense optic flow problem for non-rigid motion using rank constraints.

3

Dense optic flow with subspace constraints

As we mentioned above, current non-rigid structure from motion approaches are limited to sparse 3D reconstructions as they rely on sparse feature tracking techniques such as the method of Lucas and Kanade [5]. This local method copes with the well known aperture problem in areas rich of texture, but fails to estimate motion in (flat) regions with vanishing image gradients. This limitation was overcome by the pioneered work of Horn and Schunck [9] who formulated a global energy functional with a regulariser that allows to fill in textural information into flat regions from their neighbourhoods, making dense flow field estimation possible. More advanced methods for dense motion recovery have been proposed in the literature. Amongst them, the combined local and global (CLG) approach of Bruhn et al. [21] casts the methods of Lucas-Kanade and Horn-Schunck into a unifying variational formulation; and the large displacement optical flow (LDOF) method of Brox and Malik [22] integrates rich feature descriptors into a variational optic flow approach. In this section we aim at combining dense motion estimation with the subspace rank constraints described in the previous section following variational principles. As this is the first attempt of that kind -to the best of our knowledge-, we embed the rank constraints in the original approach of Horn and Schunck as a proof of concept that subspace constraints can be used to determine multi frame optic flow. Let I0 , If : Ω ∈ R2 → R be the reference and the target images to be registered (or matched). To compute the optic flow (u(x), v(x))> for all x := (x, y)> ∈ Ω, the Horn-Schunck [9] method minimises an energy functional of the form E = Edata + α Ereg : Z 
2
 E(u, v) = If (x + u, y + v) − I0 (x, y) + α |∇u|2 + |∇v|2 dx , (8) Ω

where Edata and Ereg penalise deviations from the model assumptions and α > 0 acts as a regularisation parameter. The assumption in Edata is that the grey value of a “moving” pixel remains constant in both images, while Ereg assumes that the optic flow varies smoothly in space. It is important mentioning that since we are dealing with a sequence, we adopt a non-linearised data constraint in order to cope with large displacements following [10, 11]. This choice has the disadvantage that the energy functional may be non-convex, and hence with multiple local minima. We discuss how to alleviate this problem later in this section.

6

3.1

R. Garg et al.

Horn-Schunk approach with subspace constraints

We now extend the variational model (8) for tracking a non-rigid object along an image sequence I1 , . . . , IF using subspace constraints. Following the derivations from Section 2.2, we can express the optic flow between a reference image I0 and a target image If as a linear combination of 2D motion basis uf (x) = Quf L(x) ,

(9)

vf (x) = Qvf L(x) ,

(10)

which holds for all frames f = 1, . . . , F . The (1 × R)-vectors Quf and Qvf correspond to the f -th row of the motion matrix basis Qu and Qv respectively. Assuming that the motion basis can be pre-estimated using a set of reliable 2D tracks, we reformulate the optic flow computation as the estimation of the (R × 1)-vector of flow coefficients L(x), for all x ∈ Ω. The advantage of this parameterisation is that the functions L(x) are shared by both components of the flow along the whole image sequence, which drastically reduces the number of unknowns. Therefore, using the above parameterisation we proposed the following extension of the Horn-Schunck approach for the dense multi-frame estimation of the functions L: Z X F
2 If (x + Quf L(x), y + Qvf L(x)) − I0 (x, y) dx E(L) = Ω f =1

+α

Z X F


|∇(Quf L(x))|2 + |∇(Qvf L(x))|2 dx .

(11)

Ω f =1

By noticing that the motion basis matrix Q does not vary spatially (cf. (6)) and that its column-vectors are constraint in P orthonormal (cf. (7)), the smoothness P (11) can be simplified to r,f (Quf,r 2 + Qvf,r 2 )|∇Lr (x)|2 = r |∇Lr (x)|2 , where Lr (x) is the r-th element of L(x). This means that we are actually imposing homogeneous regularisation on the multi-frame motion basis coefficients. With that simplification, the proposed energy functional reads   Z F R X X
2  E(L) = If (x + Quf L(x), y + Quf L(x) − I0 (x, y) + α |∇Lr (x)|2  dx . Ω

f =1

r=1

(12) Before discussing the minimisation strategy for this energy, it is important to state that once the functions L have been estimated we can densely compute the optic flow between all target images I1 , . . . , IF and the reference image I0 via the equations (9)-(10). Thanks to the non-linearised grey value constancy assumption we can cope with large displacements, which we will demonstrate in the experimental section. This is one of the characteristics that distinguishes our approach from other methods such as [6, 7] that assume small deformations as they rely on linearised data constraints. Moreover, by exploiting a global variational formulation we can truly compute dense and smooth flow fields without

Dense Multi-Frame Optic Flow for Non-Rigid Objects

7

worrying about the aperture problem in areas of partial (one-dimensional) or null textural information. One could argue that in (12) a pre-computed motion basis needs to be available before estimating L. This is though a common and very useful practice in numerous applications where the tracked objects undergo particular motions that could be represented by suitable bases. Such bases are frequently computed from sparse data points, so the need for estimating the motion coefficients and then the optic flow densely is compelling. Our approach is going in that direction, and in the future we expect to jointly estimate the motion basis and the motion coefficients for specific applications. In terms of dimensionality reduction, the number of unknowns in the functional (12) is R × P , where R is the number of 2D motion bases and P the number of pixels, compared to the 2 × F × P unknowns in a multi-frame optical flow estimation. This way we reduce the number of variables by a factor of 2 F/R with F  R. In a typical experiment we would consider sequences with around F = 50 frames and use an average of R = 5 basis. 3.2

Minimisation

The proposed energy functional E(L) from (12) can be minimised by solving the Euler-Lagrange equations [23] for all r = 1, . . . , R 0=

F  X

 If z (Quf,r If x + Qvf,r If y ) − α ∆(Lr ) ,

(13)

f =1

with reflecting boundary conditions, where (If x , If y )> := ∇If , If z := If − I0 and ∆ := ∂xx + ∂yy is the Laplacian operator. As we mentioned above the energy (12) may be non convex due to the nonlinearised data constraint. For the same reason, we obtain a highly non linear system of equations (13). Following [11] we solve the system of equations by the following multiresolution strategy with warping to avoid local minima and handle large displacements, discretising semi-implicitly the data term and fully implicit the regularisation term 0=

F  X

u k v k r Ifk+1 z (Qf,r If x + Qf,r If y ) − α ∆(L

k+1

 ) ,

(14)

f =1

with k the warping index for a pyramid level k iteration. We remove the nonlinearities cause by If z via the Taylor expansion k Ifk+1 z = If z +

R X  k u k If x Qf,η + Ifky Qvf,η dLη ,

(15)

η=1

where dL = (dL1 , . . . , dLR )> . We discretise the partial derivatives with standard finite differences and solve the resulting linear system of equations with the SOR solver [24].

8

4

R. Garg et al.

Experimental results

We have evaluated our approach on three different video sequences which display different types of deformations. In all three sequences we used the technique of Lucas and Kanade [5] to track a set of highly textured reliable points. The motion basis was in each case estimated by computing the singular value decomposition of the measurement matrix containing the reliable tracks and truncating to the chosen rank. Since these R basis tracks must encode the motion of any other pixel in the image, we are implicitly assuming that the reliable tracks cover the object and they represent a good sample of the deformations present in the sequence. Despite having significant texture, our sequences are challenging mainly because of the presence of complex deformations and large displacements. It is important to stress that we were not able to test our algorithm on the Middlebury database simply because there were no deformable motion sequences. Therefore we have not been able to test our algorithm on ground truth data. However, we provide various qualitative tests and a comparison with a state of the art variational method for large displacements [22]. Paper sequence: The first sequence shows a sheet of paper being bent backwards and has been used in non-rigid structure from motion methods [25] for 3D reconstruction. We used a 40 frame long subsequence and tracked corner features to estimate the basis. In this sequence we chose the rank to be 2 and considered only the first 2 basis tracks. Since the sequence had 40 frames, the number of variables to be estimated per pixel in the variational framework was reduced from 80 to just 2. We show results in Figure(1). The reference frame and two other frames of the sequence are shown on the top row. These show the extent of the deformation and the large displacements (a maximum of 58 pixels). The left column shows the reference image on top and the results of reverse warping (W −1 (If )) the other two frames to the reference frame. These results show that the algorithm can cope with the challenging displacements present in this sequence. The two rightmost plots in the middle row of Figure(1) show Middlebury colour coded optic flow results for every pixel. The colour indicates direction and the intensity indicates magnitude of the flow. These results prove the smoothness of our results. Finally, we have sampled the dense optic flow values and show the arrow values on those sampled locations. These plots also confirm the smoothness and the accuracy of the resulting optical flow. Face Paint sequence: We use a face paint sequence provided by an artist3 which has strong and fast deformations. Besides there is significant appearance change in most of the local features, further challenging our system. This is obvious by looking at the top row of Figure(2) which shows three images of the sequence (including the first/reference and last). In this case we chose a 40 frame long subsequence and needed as few as 4 basis tracks to encode the non-rigid motion. Figure(2) shows the results. Despite being a challenging sequence with self-occlusions (for instance in the eye area), large displacements of up to 27 3

This sequence is courtesy of James Kuhn

Dense Multi-Frame Optic Flow for Non-Rigid Objects

I0

I10

I40

W −1 (I10 )

Dense flow I10

Dense flow I40

W −1 (I40 )

Flow arrows I10

Flow arrows I40

9

Fig. 1. Results for the Paper sequence. Top row shows the reference image and two more images in the sequence (rightmost is the last frame). The left-most column shows the reference image on top and the result of reverse warping images I10 and I40 to the reference frame. Colour coded images represent the dense optical flow between each image and the reference frame. Arrows represent sampled flow vectors between each image and the reference frame.

10

R. Garg et al.

pixels and large appearance changes, the reverse warped images appear to be accurate and the colour coded optic flow results show smoothness in the flow. T-shirt sequence: This particular sequence has no motion blur, self or external occlusions but still has highly non-rigid and large deformations. We only found 90 reliable features with Lucas and Kanade’s tracking algorithm on the 60 frames of the sequence where a T-shirt is coming back to its rest state from a deformed one. The maximum displacement in this sequence was 50 pixels. In this case, we only needed 3 motion basis to run our variational multi-frame optic flow. Our results are shown in Figure(3). The bottom left image shows the result of reverse warping the last image of the sequence back to the first/reference frame. There are some obvious errors in the corner of bottom corner of the t-shirt. However, this was expected since there was is no texture there and therefore no features were found in that area to encode in the basis. The rest of the results show smooth flow with good accuracy, except in the corners where no features were tracked. The results follow the same layout as Figure(1). 4.1

Comparison with state of art variational optic flow method

Comparing our approach with other multi-frame optic flow algorithms is not straightforward. Our problem definition is different from other approaches since we compute the optic flow between a reference frame and every other frame in the sequence with subpixel accuracy. Note that the flow between the first and last image will be very large. Other approaches tend to compute frame-toframe flow with subpixel accuracy but in their case it is not possible to register the last frame to the reference without the use of interpolation. Therefore it is difficult to carry out a fair comparison. It is also worth mentioning that, unlike other authors [13], despite using image sequences in our experiments we do not assume temporal smoothness in optic flow. We only enforce spatial smoothness of the flow, which leads to smoothing of the motion basis coefficients. We have chosen to compare the performance of our algorithm with Brox and Malik’s large displacement optical flow (LDOF) [22] which integrates rich feature descriptors into a variational optic flow approach to compute dense flow. This approach can be considered the state of the art in the case of very large displacements, since it outperforms previous methods. Although both the data term and the regulariser are more advanced than the ones we have used in our variational formulation we thought it fair to compare our approach with the best performing method for large displacements, particularly since it integrates the use of features. We compute the flow between the reference frame and frame 10 of the face sequence using LDOF [22] and compare the results with our multi-frame approach. Figure (4) shows the detailed comparison. Note that we only used 60 Lucas-Kanade features in this case. The left-most images show the target image I10 and the reference frame I0 . The two middle images show the results of reverse warping the target frame I10 back to the reference frame with the LDOF algorithm and our own algorithm. Notice that the LDOF algorithm produces some artifacts in the warped images around the left collar of the shirt, the corner of the

Dense Multi-Frame Optic Flow for Non-Rigid Objects

I0

I10

I40

W −1 (I10 )

Dense flow I10

Dense flow I40

W −1 (I40 )

Flow arrows I10

Flow arrows I40

11

Fig. 2. Results for the frame 10 and frame 40 of Face-paint sequence with same layout as figure 1

12

R. Garg et al.

I0

I30

I60

W −1 (I30 )

Dense flow I30

Dense flow I60

W −1 (I60 )

Flow arrows I30

Flow arrows I60

Fig. 3. Results for the frame 10 and frame 40 of T-shirt sequence with same layout as figure 1

I10

I0

W −1 LDOF

Our W −1

Flow LDOF

Our Flow

Fig. 4. Left-most two: Target and reference images. Middle two: results of reverse warping target image into reference frame with LDOF (left) and our approach (right). Right two: colour coded optical flow.

Dense Multi-Frame Optic Flow for Non-Rigid Objects

13

lip and the eye. These images show very similar performance for both algorithms which is encouraging since the LDOF approach uses much more sophisticated data and regularization terms in their variational approach.

5

Conclusions and future work

We have presented a variational approach to computing dense multi-frame optic flow for non-rigid motion based on a re-parametrisation of the optic flow in terms of a linear combination of a 2D motion basis. The proposed energy formulation reduce number of variables to be computed one order of magnitude to increase computational speed and accuracy of optimisation by applying most generic rank constraint. It is conclusively proven with experiments that we can reduce the problem size and work on global optimization of brightness constancy without actual loss of useful information. The comparison of our new approach with a state of art optic flow method supports the feasibility of applying statistical rank constraints to the non-rigid registration problem and encourages us to investigate several possible extensions. Future work will include the use of a robust data term which could deal with occlusion and appearance changes by replacing the squared data fidelity term with a more robust one such as the L1 norm. More sophisticated regularisation terms could also be considered such as an anisotropic regulariser.

References 1. Bartoli, A., Gay-Bellile, V., Castellani, U., Peyras, J., Olsen, S., Sayd, P.: Coarseto-Fine Low-Rank Structure-from-Motion. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, Alaska. (2008) 2. Torresani, L., Hertzmann, A., Bregler., C.: Non-rigid structure-from-motion: Estimating shape and motion with hierarchical priors. IEEE Transactions on Pattern Analysis and Machine Intelligence 30 (2008) 3. Paladini, M., Del Bue, A., Stosic, M., Dodig, M., Xavier, J., Agapito, L.: Factorization for non-rigid and articulated structure using metric projections. In: IEEE Conference in Computer Vision and Pattern Recognition (CVPR 2009). (2009) 4. Pons, J., andO. Faugeras, R.K.: Multi-view stereo reconstruction and scene flow estimation with a global image-based matching score. International Journal of Computer Vision 72 (2007) 179–193 5. Lucas, B., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: Proc. International Joint Conference on Artificial Intelligence. (1981) 6. Irani, M.: Multi-frame correspondence estimation using subspace constraints. Int. J. Comput. Vision 48 (2002) 7. Torresani, L., Yang, D., Alexander, E., Bregler, C.: Tracking and modeling nonrigid objects with rank constraints. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition, Kauai, Hawaii. (2001) 8. Brand, M.: Morphable models from video. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition, Kauai, Hawaii. Volume 2. (2001)

14

R. Garg et al.

9. Horn, B., Schunck, B.: Determining optical flow. Artificial Intelligence 17 (1981) 185–203 10. Alvarez, L., Weickert, J., S´ anchez, J.: Reliable estimation of dense optical flow fields with large displacements. International Journal of Computer Vision 39 (2000) 41– 56 11. Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In Pajdla, T., Matas, J., eds.: Computer Vision – ECCV 2004, Part IV. Volume 3024 of Lecture Notes in Computer Science. Springer, Berlin (2004) 25–36 12. Weickert, J., Schn¨ orr, C.: A theoretical framework for convex regularizers in PDEbased computation of image motion. International Journal of Computer Vision 45 (2001) 245–264 13. Weickert, J., Schn¨ orr, C.: Variational optic flow computation with a spatiotemporal smoothness constraint. Journal of Mathematical Imaging and Vision 14 (2001) 245–255 14. Wedel, A., Pock, T., Braun, J., Franke, U., Cremers, D.: Duality tv-l1 flow with fundamental matrix prior. In: Image and Vision Computing New Zealand. (2008) 15. L. Valgaerts, A. Bruhn, J.W.: A variational approach for the joint recovery of the optical flow and the fundamental matrix. In: Pattern Recogination. Volume 5096., springer (2008) 314–324 16. Wedel, A., Cremers, D., Pock, T., Bischof, H.: Structure- and motion-adaptive regularization for high accuracy optic flow. In: IEEE International Conference on Computer Vision (ICCV), Kyoto, Japan (2009) 17. Bregler, C., Hertzmann, A., Biermann, H.: Recovering non-rigid 3D shape from image streams. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition, Hilton Head, South Carolina. (2000) 18. Tomasi, C., Kanade, T.: Shape and motion from image streams under orthography: A factorization approach. International Journal of Computer Vision 9 (1992) 19. Torresani, L., Bregler, C.: Space-time tracking. In: ECCV ’02: Proceedings of the 7th European Conference on Computer Vision-Part I, London, UK, SpringerVerlag (2002) 801–812 20. Shi, J., Tomasi, C.: Good features to track. In: 1994 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’94). (1994) 21. Bruhn, A., Weickert, J., Schn¨ orr, C.: Lucas/Kanade meets Horn/Schunck: Combining local and global optic flow methods. International Journal of Computer Vision 61 (2005) 211–231 22. Brox, T., Malik, J.: Large displacement optical flow: Descriptor matching in variational motion estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence (2010) To appear. 23. Gelfand, I.M., Fomin, S.V.: Calculus of Variations. Dover, New York (2000) 24. Weickert, J., ter Haar Romeny, B.M., Viergever, M.A.: Efficient and reliable schemes for nonlinear diffusion filtering. IEEE Transactions on Image Processing 7 (1998) 398–410 25. Varol, A., Salzmann, M., Tola, E., Fua, P.: Template-free monocular reconstruction of deformable surfaces. In: ICCV 2009. (2009)