Efficient Suboptimal Solutions to the Optimal Triangulation

Int J Comput Vis (2011) 91: 77–106 DOI 10.1007/s11263-010-0378-y

Efficient Suboptimal Solutions to the Optimal Triangulation F.C. Wu · Q. Zhang · Z.Y. Hu

Received: 17 December 2009 / Accepted: 20 August 2010 / Published online: 8 September 2010 © Springer Science+Business Media, LLC 2010

Abstract Given two images, the optimal triangulation of a measured corresponding point pair is to basically find out the real roots of a 6-degree polynomial. Since for each point pair, this root finding process should be done, the optimal triangulation for the whole image is computationally intensive. In this work, via the 3D cone expression of fundamental matrix, called the fundamental cone, together with the Lagrange’s multiplier method, the optimal triangulation problem is reformulated. Under this new formulation, the optimal triangulation for a measured point pair is converted to finding out the closest point on the fundamental cone to the measured point in the joint image space, then 3 efficient suboptimal algorithms, each of them can satisfy strictly the epipolar constraint of the two images, are proposed. In our first suboptimal algorithm, the closest point on the generating cone to the measured point is used as the approximation of the optimal solution, which is to find out the real roots of a 4-degree polynomial; in our second suboptimal algorithm, the closest point on the generating line to the measured point is used as the approximation of the optimal solution, which is to find out the real roots of a 2-degree polynomial. Finally, in our third suboptimal algorithm, the converging point of the Sampson approximation sequence is used as the approximation of the optimal solution. Experiments with simulated data as well as real images show that our proposed 3 suboptiF.C. Wu () · Q. Zhang · Z.Y. Hu National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, P.O. Box 2728, Beijing 100190, China e-mail: [email protected] Q. Zhang e-mail: [email protected] Z.Y. Hu e-mail: [email protected]

mal algorithms can achieve comparable estimation accuracy compared with the original optimal triangulation, but with much less computational load. For example, our second and third suboptimal algorithms take only about a 1/5 runtime of the original optimal solution. Besides, under our new formulation, rather than recompute the two Euclidian transformation matrices for each measured point pair, a fixed Euclidian transformation matrix is used for all image point pairs, which, in addition to its mathematical elegance and computational efficiency, is able to remove the dependency of the resulting polynomial’s degree on the parameterization of the epipolar pencil in either the first image or in the second image, a drawback in the original optimal triangulation. Keywords Fundamental cone · Optimal triangulation · Suboptimal triangulation · Sampson sequence

1 Introduction In computer vision, triangulation refers to the process of determining a 3D point given its projections onto two or more images and the corresponding camera projection matrices. This problem is trivial in theory, because this 3D point is the common point of the projection rays generated by the image points. However, in practice since the coordinates of image points can only be measured with limited accuracy, these rays do not always intersect in 3D space. This leads to find a 3D point that optimally fits the measured points, usually called the optimal triangulation. In the literature, various methods for the optimal triangulation have been proposed based on different optimality criteria, such as to minimize the sum of squared algebraic errors, the Sampson error (Sampson 1982; Torr and Zisserman 1998),

78

the sum of squared geometric errors, or squared reprojection errors (Hartley and Sturm 1997; Stewenius et al. 2005; Lu and Hartley 2007; Kahl et al. 2008; Hartley and Seo 2008; Olsson et al. 2009), or the geometric error under LP (p = 1, or ∞) norm (Hartley and Schaffalitzky 2004; Ke and Kanade 2007; Kahl and Hartley 2008). Among these optimality criteria, the minimization of the sum of squared geometric errors is the most interesting since it leads to the maximum likelihood estimation if the image measurement error obeys independent Gaussian law. In this paper, the optimal triangulation is meant to minimize the sum of squared geometric errors. The main problem for the optimal triangulation is the existence of local minima which prevent any iterative methods from finding out the global optimal solution with certainty, due to the non-convexity nature of the cost function. Thus, to explore new methods to find the global optimal solution has aroused great attention in the literature. For more than two images, the optimal triangulation problem is very difficult. Stewenius et al. (2005) showed that finding the global optimal solution from three images involves the roots of a polynomial of 47 degrees, and much higher degree polynomials are involved with more images. Recently, Hartley and Seo (2008) proposed a method for verifying whether a local minimum solution is globally optimal. Olsson et al. (2009) gave a new verification method. Since a branch and bound algorithm is embedded, their method always gives an answer: Either (i) a certificate of optimality for the local solution or (ii) the global optimal solution. However, this method is still computationally intensive. In the case of two images, Hartley and Sturm (1997) showed that, the optimal triangulation is solvable in closed form and propose an algorithm to find the global optimal solution. Their key ideas are that by using a single parameterization of epipolar line correspondences and Euclidian transformations of images, the cost function is reduced to a 6-degree polynomial, and then the global optimal solution is obtained by determining the real roots of this polynomial. In Hartley and Sturm’s algorithm, in addition to finding the roots of a 6-degree polynomial, the construction of Euclidean transformations depends not only on the fundamental matrix but also on the measured point, i.e., the Euclidean transformations are different for different measured points. Thus, for a problem with a large number of measured points, the algorithm’s computational complexity is intensive. In this paper, the optimal triangulation of two images is considered again. Our main contributions include: A new method for the optimal triangulation and three suboptimal triangulations, which all satisfy strictly the epipolar constraint. Experiments show that, our three suboptimal triangulations can largely reduce the computational complexity of the optimal triangulation, but have almost the same accuracy with the optimal triangulation.

Int J Comput Vis (2011) 91: 77–106

In this work, the optimal triangulation of two images is reformulated, and it is converted to finding the closest-point on the fundamental cone, a 3D cone in the joint image space defined by the fundamental matrix (Anandan and Avidan 2000), to the measured point. Via the upper left 2 × 2 submatrix of fundamental matrix, F2×2 , and the two epipoles, in the joint image space we construct a Euclidian transformation which transforms the fundamental cone into its canonical form, and then the optimal triangulation is converted to minimizing the geometric error subject to the canonical form, which is easily solvable through the Lagrange’s multiplier method. More precisely, we obtain the following results: CASE I: If F2×2 has two non-zero singular values, the optimal triangulation is reduced to finding the real roots of a 6-degree polynomial. In particular, it is reduced to finding the real roots of a 2-degree polynomial if F2×2 has two equal singular values. CASE II: If F2×2 has only one non-zero singular value, the optimal triangulation is reduced to finding the real roots of a 5-degree polynomial. And, it is reduced to finding the real roots of a 4-degree polynomial if the two epipoles are both at infinity. CASE III: If F2×2 has two zero singular values, the optimal triangulation has a linear solution. Under our new formulation, a new algorithm for the optimal triangulation is proposed. Though in general, our new algorithm also needs to finding the real roots of a 6-degree polynomial as did in the algorithm of Hartley and Sturm, the construction of the Euclidian transformation in our algorithm depends only on the fundamental matrix, in other words, the same Euclidian transformation is used for all the measured points, hence it is more computationally efficient. Either Hartley and Sturm’s optimal triangulation or our new optimal triangulation has a high computational complexity in practice, because in general, both of them need finding out the real roots of a 6-degree polynomial for each measured point. To reduce the computational complexity, this paper introduces three suboptimal triangulations, each of which satisfies strictly the epipolar constraint. From the polar plane of the measured point about the fundamental cone, a generating cone and a generating line are introduced, and then two approximately closest-points on the fundamental cone to the measured point are defined. Based on these, we give two efficient suboptimal triangulation algorithms since they need only either to find out the real roots of a 4-degree polynomial or a 2-degree polynomial. By considering the Sampson approximation (Sampson 1982; Torr and Zisserman 1998; Hartley and Zisserman 2003), we define a Sampson sequence for the measured point pair and propose another suboptimal triangulation, called the Sampson suboptimal triangulation.

Int J Comput Vis (2011) 91: 77–106

79

The remainder of this paper is organized as follows. Section 2 is a short review on the optimal triangulation algorithm by Hartley and Sturm (1997). In Sect. 3, the optimal triangulation is reformulated and followed by a new algorithm. Section 4 introduces the generating cone and generating line for the fundamental cone. Sections 5, 6 and 7 are our 3 suboptimal triangulations. Section 8 reports experimental results. Section 9 concludes this paper. 2 A Brief Review of the Optimal Triangulation Given a measured point correspondence x ↔ x , and the fundamental matrix F of the two images, the goal of the optimal triangulation is to compute the corrected correspondence, x¯ ↔ x¯  , that minimizes the following geometric error subject to the epipolar constraint, x¯ T F x¯ = 0: C = d 2 (¯x, x) + d 2 (¯x , x ).

(1)

This problem is equivalent to minimizing the following point-to-line distances d 2 (x, l) + d 2 (x , l )

(2)

l

where l and stands for all possible pairs of corresponding epipolar lines. Assume ¯l and ¯l minimize (2), then point x¯ is the closest-point on line ¯l to point x, and similarly point x¯  is the closest-point on ¯l to x . The main steps of the original optimal triangulation are shown in Table 1 (Hartley and Sturm 1997).

Table 1 Hartley and Sturm’s algorithm 1. Define two transformation matrices ⎛ ⎞ ⎛ 1 0 −x 1 0 ⎜ ⎟ ⎜ T = ⎝ 0 1 −y ⎠ , T  = ⎝0 1 0

0

1

0 0

−x 

⎞

⎟ −y  ⎠ . 1

These are in fact the translations that take x = (x, y, 1)T and x = (x  , y  , 1)T to their respective origin. 2. Replace F by T −T F T −T . The new F corresponds to the translated coordinates. 3. Compute the epipoles e and e such that F e = 0 and e T F = 0. Normalize (multiplication by a scalar) e such that e12 + e22 = 1 and do the same to e . 4. Construct the rotation matrices ⎞ ⎞ ⎛  ⎛ e1 e2 0 e1 e2 0 ⎟ ⎟ ⎜ ⎜ R = ⎝ −e2 e1 0 ⎠ , R  = ⎝ −e2 e1 0 ⎠ , 0

0

1

0

0

1

then Re = (1, 0, f )T and R  e = (1, 0, f  )T . 5. Replace F by R  F R T . The resulting F must have the form: ⎛  ⎞ ff d −f  c −f  d ⎜ ⎟ F = ⎝ −f b a b ⎠. −f d

c

d

6. Parametrize the pencil of epipolar lines in the first image with a parameter t, l(t) = (f t, 1, −t)T and compute the corresponding epipolar line in the second image, l (t) = (−f  (ct + d), at + b, ct + d)T . Then express explicitly the cost (2) as a function of t: C(t) =

t2 (ct + d)2 + . 2 2 1+f t (at + b)2 + f 2 (ct + d)2

(3)

7. From the following polynomial equation, solve for t to get 6 roots:

Remark 1 (a) The two transformation matrices are dependent on the given corresponding point pair in the above algorithm. In Sect. 3, we will show that it is possible to use a fixed transformation for all the point pairs. (b) Paradoxically in the above algorithm, the degree of the resulting polynomial could depend on whether the first image or the second image is used to parameterize the epipolar pencil in Step 6. In other words, given a pair of image points, it is possible to yield a polynomial with different degree if either the epipolar pencil in the first image or the epipolar pencil in the second image is parameterized. For example, if the second epipole is at infinity but the first one is not (f  = 0, f = 0) for a given image pair, and if the first image is chosen for the parameterization, the degree of the resulting polynomial is 6. However, if the second image is chosen for the parameterization, the degree of the resulting polynomial is 5. From the algorithmatical point of view, such a result seems not logic. We thought the degree of the resulting polynomial should reflect the nature of this point pair, not the choice of image for the epipolar pencil parameterization. In our new algorithm, such a problem is completely removed, which will be discussed in Appendix 2.

g(t) = t ((at + b)2 + f 2 (ct + d)2 )2 − (ad − bc)(1 + f 2 t 2 )2 × (at + b)(ct + d) = 0. 8. Evaluate the cost function (3) at the real roots of g(t). Also, find the asymptotic value of (3) for t = ∞, i.e., 1/f 2 +c2 /(a 2 +f 2 c2 ). Select the value tmin of t that gives the smallest value of the cost function. 9. Evaluate the line correspondence l(t) ↔ l (t) at tmin , and find x¯ and x¯  as the closest points on these lines to the origin. 10. Transform back to the original coordinates by replacing x¯ with T −1 R T x¯ and x¯  with T −1 R T x¯  . 11. The 3D point of x¯ ↔ x¯  is obtained by the DLT method.

3 Reformulation of the Optimal Triangulation and a New Algorithm

In this section, we reformulate the optimal triangulation problem via the fundamental cone in the joint image space, and introduce a new optimal algorithm under this new formulation. By the Anandan and Avidan (2000)’s studies, the epipolar constraint is equivalent to the following cone constraint in

80

Int J Comput Vis (2011) 91: 77–106

Table 2 The classification of the epiploes’ location, the canonical form of the fundamental cone, as well as the degree of the resulting polynomial Singular values of F2×2

Locations of the two epiploes

The Euclidian canonical form of 

The polynomial’s degree

CASE I: k=2

σ1 > σ2 > 0 σ1 = σ2 > 0

Neither of them is at infinity

σ1 (x32 − x12 ) + σ2 (x42 − x22 ) = 0 (x32 − x12 ) + (x42 − x22 ) = 0

6 2

CASE II: k=1

σ1 > 0, σ2 = 0

One of them is at infinity Both of them are at infinity

x32 − x12 + 2s(x2 + x4 ) = 0 x32 − x12 + t = 0

5 4

CASE III: k=0

σ1 = σ2 = 0

Both of them are at infinity

ax1 + bx2 + cx3 + dx4 + e = 0

1

the joint image space: ⎛

0 ⎜ 0 ⎜  ⎜ XT X = (x, y, x  , y  , 1) ⎜ F11 ⎜ ⎝ F21 F31 ⎛ ⎞ x ⎜y⎟ ⎜ ⎟ ⎟ ×⎜ ⎜x ⎟ = 0 ⎝ y ⎠ 1

0 0 F12 F22 F32

F11 F12 0 0 F13

F21 F22 0 0 F23

⎞

F31 F32 ⎟ ⎟ ⎟ F13 ⎟ ⎟ F23 ⎠ 2F33

(4)

where X = (xT , 1)T , defined by the corresponding point pair (x, y, 1)T ↔ (x  , y  , 1)T , is a point in the joint image space, Fij is the (i, j )-element of the fundamental matrix F and the cone  is called the fundamental cone. Hereafter, the homogeneous coordinates of a point in the joint image space is denoted by an uppercase letter and its non-homogeneous coordinates by the corresponding lowercase letter, such as X and x. And the measured corresponding point pair is called the measured point in the joint image space. Clearly, the optimal triangulation (1) is equivalent to the following minimization problem:  ¯ X) minX¯ d 2 (X, (5) ¯ T X ¯ = 0. subject to X In the paper, we use the following strategy to solve this minimization: (a) By the SVD of F2×2 and the two epiploes, where F2×2 is the upper left 2 × 2 submatrix of the fundamental matrix, construct a Euclidian transformation E in the joint image space, which transforms the fundamental cone  ˜ = E −T E −1 . into its canonical form,  (b) By the above Euclidian transformation, the minimization (5) is reduced to minimizing the geometric error subject to the canonical form:  ¯  , X ) minX¯ d 2 (X (5a) where X = EX. ¯ T  ¯  = 0, ˜X subject to X

(c) By the Lagrange’s multiplier method, the minimization (5a) is reduced to finding the real roots of a polynomial. The exact degree of this polynomial is given in the following proposition. Proposition 1 Let k = rank F2×2 and σ1 and σ2 (σ1 ≥ σ2 ≥ 0) be the two singular values of F2×2 , then we have the following conclusions as shown in Table 2. Proof Here, we give a proof for CASE I and CASE III (at the same time, give also details of the corresponding minimizations). The proof of CASE II is placed in Appendix 1. We at first give a proof for CASE III since it does not involve the construction of the Euclidean transformation. Proof of CASE III: From rank F2×2 = 0, F11 = F12 = F21 = F22 = 0, thus the fundamental cone degenerates to a 3D hyperplane: F31 x + F32 y + F13 x  + F23 y  + F33 = 0. Hence, for a measured point x = (x, y, x  , y  )T , the minimization (5) is linear and by the Lagrange’s multiplier method, we have x¯ = x − λb,

λbT b = bT x + F33 ,

where b = (F31 , F32 , F13 , F23 )T . Therefore, for CASE III the minimization (5) has the linear solution as follows: x¯ = x −

bT x + F33 b. bT b

(6)

Proof of CASE I: In this case, the fundamental cone  is a 3D cone in the joint image space, and neither of the two epipoles is at infinity. Let the two epipoles be e1 = (e1 , e2 , 1)T and e2 = (e1 , e2 , 1)T , then it is not difficult to see that the epipole correspondence E = (eT , 1)T , where e = (e1 , e2 , e1 , e2 )T , is the vertex of . In this paper, the vertex of a cone is also called the singular point of this cone. In order to solve the minimization (5), we give at first the ’s canonical form under Euclidian transformation. Since the fundamental cone is a 3D cone and its vertex E =

Int J Comput Vis (2011) 91: 77–106

81

(eT , 1)T is not at infinity, it can be expressed as:



T 0 F2×2 B −Be , where B = . = −eT B eT Be F2×2 0

we have the Lagrange’s equations:

Then, by the transformation matrix

I −e T= 0 1

(7)

which translates the vertex E to the origin, the fundamental cone can be expressed as:

B 0 . (8) T −T T −1 = 0 0 Let the SVD of F2×2 be U diag(σ1 , σ2 )V T , then the transformation matrix,

1 V V (9) W=√ 2 −U U is orthonormal, and it is easy to verify that the matrix B has the eigen-decomposition: B = W diag(−σ1 , −σ2 , σ1 , σ2 ) W T ,  

or W T BW = .

∂x¯1 L = 2(x¯1 − x1 ) − 2λσ1 x¯1 = 0,

(14a)

∂x¯2 L = 2(x¯2 − x2 ) − 2λσ2 x¯2 = 0,

(14b)

∂x¯3 L = 2(x¯3 − x3 ) + 2λσ1 x¯3 = 0,

(14c)

∂x¯4 L = 2(x¯4 − x4 ) + 2λσ2 x¯4 = 0,     ∂λ L = σ1 x¯32 − x¯12 + σ2 x¯42 − x¯22 = 0.

(14d)

From (14a) to (14d), we have x1 , 1 − λσ1 x3 x¯3 = , 1 + λσ1 x¯1 =

x2 , 1 − λσ2 x4 x¯4 = . 1 + λσ2 x¯2 =

(15)

By substituting (15) into (14e), we obtain the following equation of λ:

x32 x12 − σ1 (1 + λσ1 )2 (1 − λσ1 )2

x22 x42 = 0, + σ2 − (1 + λσ2 )2 (1 − λσ2 )2 or equivalently



Thus, under the Euclidian transformation

T −W T e W E= 0 1



(10)

s(λ) = σ1 (1 + λσ2 )2 (1 − λσ2 )2   × (1 − λσ1 )2 x32 − (1 + λσ1 )2 x12 + σ2 (1 + λσ1 )2 (1 − λσ1 )2   × (1 − λσ2 )2 x42 − (1 + λσ2 )2 x22 = 0.

the fundamental cone has the following canonical form ˜ = E −T E −1 = diag(, 0), 

(16)

(11)

 = diag(−σ1 , −σ2 , σ1 , σ2 ). 

Let X be a measured point and X = EX = (x1 , x2 , x3 , x4 , 1)T , then the ’s canonical equation is σ1 (x32 − x12 ) + σ2 (x42 − x22 ) = 0.

(12)

Since Euclidian transformation preserves the distance between two points, the minimization (5) is reduced to the following minimization problem:   minx¯ 4i=1 (x¯i − xi )2 (13) subject to σ1 (x¯32 − x¯12 ) + σ2 (x¯42 − x¯22 ) = 0. This minimization can be easily solved by the Lagrange’s multiplier method. From the Lagrange’s function of (13) 4       L= (x¯i − xi )2 + λ σ1 x¯32 − x¯12 + σ2 x¯42 − x¯22 i=1

(14e)

(14)

This is a 6-degree polynomial equation in λ, its coefficients depend only on the two singular values of F2×2 and the coordinates of the measured point. By computing the values of the cost function in (13) at the real roots of this polynomial, we obtain the value λmin of λ that gives the smallest value of the cost function. Hence, substituting λmin into (15) leads ¯ that minimizes the cost to the corrected correspondence X function in (13). If F2×2 has two equal singular values, i.e., σ1 = σ2 , then (13) becomes   minx¯ 4i=1 (x¯i − xi )2 (17) subject to (x¯32 − x¯12 ) + (x¯42 − x¯22 ) = 0. By the Lagrange’s multiplier method, we obtain x1 , 1−λ x3 x¯3 = , 1+λ

x¯1 =

x2 , 1−λ x4 x¯4 = 1+λ x¯2 =

(18)

82

Int J Comput Vis (2011) 91: 77–106

Table 3 New triangulation algorithm (CASE I): Multiplier method for the optimal triangulation 1. By the SVD of F2×2 and the vertex of , define the Euclidian transformation matrix E. 2. Replace  by E −T E −1 and X by EX. The new  and X correspond to the transformed coordinates. ¯ by (20) and 3. If F2×2 has two equal singular values, then compute X go to the step 5. Otherwise, from (16), solve for λ to get 6 roots. 4. Evaluate the cost function in (13) at these real roots. Take the value λmin of λ that gives the smallest value of the cost function and sub¯ which minimizes the cost function. stitute it into (15) to obtain X ¯ ¯ by E −1 X. 5. Transfer back to the original coordinates by replacing X ¯ by the 6. Compute the 3D point of the corrected correspondence X DLT method.

and s(λ) = (1 − λ)2 (x32 + x42 ) − (1 + λ)2 (x12 + x22 ) = 0.

(19)

And solutions of (19) are: ⎧ √ ⎨ λ1 = (δ1 +δ2 )+2 δ1 δ2 , (δ2 −δ1 ) ⎩λ = 2

√ (δ1 +δ2 )−2 δ1 δ2 , (δ2 −δ1 )

where δ1 = x12 + x22 , δ2 = x32 + x42 . It is not difficult to verify that λ2 gives a smaller value of the cost function in (17) than λ1 . Thus, by (18) we obtain the optimal triangulation for the measured point: ⎧ x¯ = x1 = 1 (1 + √δδ2 δ )x1 , ⎪ ⎪ 1 2 ⎪ 1 1−λ2 2 ⎪ ⎪ x2 ⎪ 1 √δ2 )x2 , ⎨ x¯2 = 1−λ = (1 + 2 δ1 δ2 2 (20) x3 δ1 1 ⎪ √ = = (1 + )x , x ¯ ⎪ 3 3 1+λ2 2 ⎪ δ1 δ2 ⎪ ⎪ ⎪ ⎩ x¯ = x3 = 1 (1 + √δ1 )x . 4

1+λ2

2

δ1 δ2

4

In summary, if F2×2 has two equal singular values, the optimal triangulation has a quadratic solution.  Based on our above discussions, our new optimal triangulation algorithm for CASE I is outlined in Table 3. Remark 2 In general, both our new optimal triangulation and the optimal triangulation by Hartley and Sturm need to ⎧ ⎪ σ1 = ⎪ ⎪ ⎨



⎪ ⎪ ⎪ ⎩ σ2 =



√

(αβ  )2 +(α  β)2 (1+(αβ  δ)2 )+

√

(αβ  )2 +(α  β)2 (1+(αβ  δ)2 )−

find the real roots of a polynomial and do Euclidean transformation, but between them there exists some differences, which will be discussed in Appendix 2. 3.1 Translational configuration In this subsection, we consider the optimal triangulation for translational configuration. In general, the optimal triangulation for a translational configuration still needs to find the roots of a 6-degree polynomial. However, for the following two special configurations, the corresponding optimal triangulation has a linear solution and a quadratic solution, respectively: Configuration I: t3 = 0; Configuration II: t3 = 0, β  /α  = β/α and γ  /α  = γ /α, where, the translation between two views is t = (t1 , t2 , t3 )T , the intrinsic parameter matrices of the two cameras are: ⎞ ⎛  ⎛ ⎞ α γ  u α γ u K = ⎝0 β v⎠, K  = ⎝ 0 β  v ⎠ . 0 0 1 0 0 1 In the configuration I, the translation direction is parallel to the image plane, but there is no constraint on the camera intrinsic parameters. This is a widely used configuration in a stereo vision system, such as translation along the x-axis or the y-axis. In the configuration II, the translational motion is a general motion, but the camera intrinsic parameters should obey some constraints, for example the intrinsic parameters of the two cameras are the same. Here is a proof of our above results: Since the fundamental matrix can be expressed as F = K −T [t]× K −1 , by direct calculation we have ⎞ ⎛ 0 − αt3β ⎠. (21) F2×2 = ⎝ γ t3 t3 γ  ( − )    α αβ ββ α For the configuration I, F2×2 = 0 from (21), thus the optimal triangulation has a linear solution as shown in (6) by CASE III in Proposition 1. When t3 = 0, F2×2 has the following two non-zero singular values:

√ 2|t3 | (αβ  )2 +(α  β)2 (1+(αβ  δ)2 )2 −4(αβα  β  )2 √ 2|t3 | (αβ  )2 +(α  β)2 (1+(αβ  δ)2 )2 −4(αβα  β  )2

, ,

Int J Comput Vis (2011) 91: 77–106

where, δ =

1 γ ββ  ( α 

83

− γα ). Thus, σ1 = σ2 if and only if

(αβ  )2 + (α  β)2 (1 + (αβ  δ)2 )2 − 4(αβα  β  )2 = 0. The solutions of this equation in δ 2 are δ2 = −

α  β − αβ  αβα  β 

2

,

δ2 = −

α  β + αβ  αβα  β 

2 .

Then from δ 2 ≥ 0, we have δ = 0 and α  β − αβ  = 0. Thus, σ1 = σ2 if and only if γ  /α  = γ /α and α  /α = β  /β. Therefore, for the configuration II the optimal triangulation has a quadratic solution as shown in (20) by CASE I in Proposition 1.

4 Generating Cone and Generating Line In this section, we define two approximately closest-points for a measured point by introducing a generating cone and a generating line of the fundamental cone .1 For the convenience of presentation, we always assume rank F2×2 = 2, but the obtained results can be extended to the case of rank F2×2 = 1. Given a measure point X, we obtain a 3D hyperplane, = X, which is the polar plane of X about the fundamental cone . The vertex E of  is on this polar plane since T E = XT E = 0. Let X⊥ be the pedal of X on the plane , i.e. X⊥ is the closest-point on to X, and let L be the line passing through X and X⊥ . Then, the cone  and the line L have two intersections,2 of which the closest intersection point to the point X is denoted by Y, as shown in Fig. 1. Let  = Y and  =  ∩  , then  is the tangent plane of  at the point Y, a 3D hyperplane, and  is a 2D cone with the vertex E, called the generating cone of  at point Y. Let L be the line passing through the vertex E and the intersection Y, called the generating line of  at point Y. Clearly, L ⊂  ⊂ . Then, we can define two approximately closest-points for the measured point X as follows. Definition 1 The closest-point on the generating cone  ˜ is called the approximately to the measured point X, X, closest-point on the fundamental cone  to X. Similarly, the closest-point on the generating line L to the measured point ˜  , is also called the approximately closest-point on the X, X fundamental cone  to X. do not consider CASE III (rank F2×2 = 0) because in this case, the optimal triangulation has a linear solution.

1 We

2 In the case that the measured point is located in the cone’s interior, the two intersections must be real. When the measured point is located in the cone’s exterior, the two intersections are also real if the measured point is close to the cone. We always assume two real intersections exist since the measured point is close to the cone in practice.

Fig. 1 Approximations of the closest-point on the fundamental cone to a measured point and the Sampson approximation

In the next two sections, we will discuss how to find out these two approximately closest-points, which correspond to the minimization subject to the generating cone  or the generating line L respectively. Since the Euclidian transformation E in Sect. 3 preserves distance and orthogonality, we always assume here that the fundamental cone  and the measured point X are in the transformed coordinates.

5 Suboptimal Triangulation I: Minimization on Generating Cone ˜ on the generating cone  In this section, the closest point X to the measured point X will be used as the approximation of the closest point on the fundamental cone to the measured point. We consequently obtain our first suboptimal solution of the triangulation. ˜ is the solution of the following miniBy Definition 1, X mization: 

minx˜ (˜x − x)T (˜x − x) subject to  : x˜ T  x˜ = 0 ∧ x˜ T y = 0.

(22)

In order to solve this minimization, we need to determine the pedal X⊥ , the intersection point Y and the parameteriza˜ can be tion of  with 3 variables. And thus the solution X obtained by the following three steps:

84

Int J Comput Vis (2011) 91: 77–106

5.1 Determining the Pedal X⊥ Since x is the normal of , point X⊥ can be expressed as x⊥ = x + sx. Then, we obtain (x + sx)T x = 0 since x⊥ is on , and thus s=−

xT x xT x = − . (x)T x xT  2 x

So, x⊥ = x −

xT x x. xT  2 x

(23)



and let z = U z. Then, we obtain the canonical equation of the generating cone  :

0 ≤ t ≤ 1.

Then, from yT y = 0 and (23) we have ∗

∗

x (I + ts ) (I + ts )x = 0, T

z   z = 0. T

∗

where s = −

xT x xT  2 x

.

Or, equivalently xT x + 2s ∗ xT  2 x · t + s ∗2 xT  3 x · t 2 = 0.

(24)

The solutions of this quadratic equation are  −xT  2 x ± (xT  2 x)2 − xT x · xT  3 x . t± = s ∗ xT  3 x Thus,  y=

t+ x⊥ + (1 − t+ )x, t−

x⊥

+ (1 − t− )x,

if 0 ≤ t+ ≤ 1, else.

(˜x − x)T (˜x − x) = (QU T z − x)T (QU T z − x) = z z − 2z QT U x +   xT x .   T

T

c

subject to z T   z = 0. (25)

(26)

Q

i.e., x˜ T y = 0 ⇔ ∃z ∈ R 3 such that x˜ = Qz. The column orthonormal matrix Q can be determined as: Let e1 = (1, 0, 0, 0)T , e2 = (0, 1, 0, 0), e3 = (0, 0, 1, 0)T , e4 = y (0, 0, 0, 1)T and d = y . If d = e1 then Q = (e2 , e3 , e4 ). If d = e1 , then

(d − e1 )(d − e1 )T Q = I4 − 2 (e2 , e3 , e4 ). (27) (d − e1 )T (d − e1 )   H

From U ’s orthogonality and Q’s column orthogonality, we have

Therefore, the minimization (22) is reduced to  minz C(z ) = z T z − 2z T b + c

We first parameterize the generating cone  with 3 variables since  is a 2D cone. On the tangent plane  we select three orthonormal vectors, denoted by q1 , q2 and q3 , then the tangent plane  can be parameterized as: z = (z1 , z2 , z3 )T ∈ R 3

(29)

b

˜ on the Generating Cone 5.3 Finding the Closest-Point X

x˜ = (q1 , q2 , q3 ) z,  

(28)

QT Q = U T diag(σ1 , σ2 , σ3 ) U  

Clearly, the intersection point Y can be expressed as

T

zT QT Qz = 0. Let the eigen-decomposition of QT Q be

5.2 Finding the Intersection Point Y

y = tx⊥ + (1 − t)x,

This is because: Since the tangent plane  is a 3D-subspace of R 4 and d is its normal vector, {e2 , e3 , e4 } is an orthonormal basis of  if d = e1 , and thus Q = (e2 , e3 , e4 ). If d = e1 , then H is an orthonormal matrix (a 4-order Householder matrix) and H e1 = d, which leads to that {H e2 , H e3 , H e4 } is an orthonormal basis of  . And thus Q can be determined by (27). Now substituting (26) into (22), the generating cone  has the following expression with 3 variables:

By the Lagrange’s multiplier method, we have

T b1 b2 b3  , , z = 1 + λσ1 1 + λσ2 1 + λσ3

(30)

(31)

where the multiplier λ satisfies the following 4-degree polynomial equation: σ1 b1 (1 + λσ2 )2 (1 + λσ3 )2 + σ2 b2 (1 + λσ1 )2 (1 + λσ3 )2 + σ3 b3 (1 + λσ1 )2 (1 + λσ2 )2 = 0.

(32)

By evaluating the cost function in (30) at the real roots of (32), which has analytical solutions, we obtain the value λmin of λ that gives the smallest value of the cost function. Substituting λmin into (31), the resulting zmin will minimize the cost function in (30), and thus x˜ min = QT U T zmin will minimize the cost function in (22). Based on the above discussions, our first suboptimal the triangulation algorithm is outlined in Table 4. This algorithm is able to largely reduce the computational complexity of the original optimal triangulation since only a 4-degree, rather than a 6-degree, polynomial equation is solved.

Int J Comput Vis (2011) 91: 77–106

85

Table 4 Suboptimal triangulation I: Triangulation based on generating cone 1. By the SVD of F2×2 and the vertex of , compute the Euclidian transformation matrix E. 2. Replace  by E −T E −1 and X by EX. The new  and X correspond to the entities under the transformed coordinates. 3. Compute the pedal X⊥ by (23) and determine the intersection point Y by (25). 4. Construct a column orthogonal matrix Q such that x˜ = Qz is a parameterization of the tangent plane  and compute the eigendecomposition of QT Q, QT Q = U T   U . 5. Compute the canonical form of the generating cone  by the transformation z = U Qx. 6. Evaluate the cost function in (30) at the real roots of (32). Select the value λmin of λ that gives the smallest value of the cost function. 7. Compute z by substituting λmin into (31) and compute the approximately closest-point: x˜ = QT U T z . ˜ ˜ by E −1 X. 8. Transfer back to the original coordinates by replacing X ˜ by the 9. Compute the 3D point of the corrected correspondence X DLT method.

Table 5 Suboptimal triangulation II: Triangulation based on generating line 1. By the SVD of F2×2 and the vertex of , compute the Euclidian transformation matrix E. 2. Replace  by E −T E −1 and X by EX. The new  and X correspond to the entities under the transformed coordinates. 3. Compute the pedal X⊥ of X on its polar plane with (23). 4. Determine the intersection point Y by (25). Then compute the ap˜  with (34). proximately closest-point X ˜  by E −1 X ˜ . 5. Transfer back to the original coordinates by replacing X  ˜ 6. Compute the 3D point of the corrected correspondence X by the DLT method.

It is worth noting that our previous two suboptimal solutions will become the optimal solution if F2×2 has two equal singular values, which will be stated in Proposition 2. Proposition 2 If F2×2 has two equal singular values, then the suboptimal solutions based on the generating cone and that based on the generating line both become the optimal solution. Proof If the suboptimal solution based on the generating line is the optimal solution, then the suboptimal solution based on the generating cone must also be the optimal solution. So, here we need only to prove the case of the generating line. If both the two equal singular values are zero, then the fundamental cone  degenerates to a 3D hyperplane: bT x + F33 = 0,

where b = (F31 , F32 , F13 , F23 )T .

(35)

For a given measured point X = (xT , 1)T , its polar plane about  is   b = X = . bT x + 2F33 And thus, the pedal x⊥ of x on the plane and the midpoint x¯ of x⊥ and x are: x⊥ = x − x¯ =

2(bT x + F33 ) b, bT b

x⊥ + x bT x + F33 =x− b. 2 bT b

Then, we have 6 Suboptimal Triangulation II: Minimization on Generating Line

bT x¯ + F33 = bT x + F33 −

bT x + F33 T b b = 0, bT b

˜  on the generating line L In this section, the closet-point X to the measured point X is used as the approximation of the closest point on the fundamental cone, and then we obtain our second suboptimal triangulation. ˜  on the generating line L is To find the closest-point X very simple. Since the generating line L can be parameterized as x˜  = ty(−∞ < t < ∞), by determining the minimum of the following 2-degree polynomial of t

i.e., x¯ satisfies (35). By Definition 1 in Sect. 4, it is not difficult to see that x¯ is the suboptimal triangulations for the measured point based on the generating cone and the generating line. On the other hand, x¯ is the optimal triangulation for the measured point by (6) in Sect. 3. Therefore, the two suboptimal solutions are both the optimal solution. If the two equal singular values are non-zero, i.e., σ1 = σ2 = 0, by the discussions in Sect. 3 the fundamental cone  has the following canonical equation:

p(t) = (ty − x)T (ty − x) = yT yt 2 − 2yT xt + xT x

(x32 + x42 ) − (x12 + x22 ) = 0

we have T

y x x˜  = y. yT y

(33)

(34)

Hence, we have our second suboptimal triangulation algorithm outlined in Table 5, it only needs to solve the minimization of a 2-degree polynomial, and is very computationally efficient.

(36)

and the optimal triangulation for a measured point, x = (x1 , x2 , x3 , x4 )T ( corresponding to the transformed nonhomogeneous coordinate), can be expressed as: √ ⎞ ⎛ (1 + δ2 / δ1 δ2 )x1 √ ⎟ ⎜ (1 + δ2 / δ1 δ2 )x2 ⎟ 1⎜ ⎟ (37) x¯ = ⎜ √ ⎟ 2⎜ ⎝ (1 + δ1 / δ1 δ2 )x3 ⎠ √ (1 + δ1 / δ1 δ2 )x4

86

Int J Comput Vis (2011) 91: 77–106

where δ1 = x12 + x22 , δ2 = x32 + x42 . If we can prove that (37) is the suboptimal triangulation based on the generating line, then it must be the suboptimal triangulation based on the generating cone. And thus, both the two suboptimal triangulations are the optimal solution. Now we prove that (37) must be the suboptimal triangulation based on the generating line. Since the polar plane of the measured point x about the cone (36) is π = (−x1 , −x2 , x3 , x4 )T , we obtain the pedal of x on π, x⊥ = ⊥ 1 x − δδ22 −δ +δ1 π . Then, the line, z(s), passing through x and x can be expressed as: z(s) = x + s(x⊥ − x) = x + s

δ2 − δ1 π δ2 + δ 1

(−∞ < s < ∞).

Thus, by the two roots of the equation ((z3 (s))2 + (z4 (s))2 ) − ((z1 (s))2 + (z2 (s))2 ) = 0: √ √ 2 δ1 δ2 2 δ1 δ2 s1 = 1 + , s2 = 1 − , δ1 + δ 2 δ1 + δ2 we obtain the two intersections of z(s) and the cone (36): ⎞ ⎛ −δ1√ (1 + (δ +δδ2)+2 )x δ1 δ2 1 2 1 ⎟ ⎜ δ2 −δ1√ ⎟ ⎜ (1 + )x 2 ⎟ ⎜ (δ2 +δ1 )+2 δ1 δ2 ⎟, z1 = ⎜ ⎟ ⎜ δ1 −δ2√ ⎜ (1 + (δ +δ )+2 δ δ )x3 ⎟ 2 1 1 2 ⎠ ⎝ −δ2√ (1 + (δ +δδ1)+2 )x 4 δ1 δ2 2 1 ⎞ ⎛ −δ1√ (1 + (δ +δδ2)−2 )x δ1 δ2 1 2 1 ⎟ ⎜ δ2 −δ1√ ⎟ ⎜ (1 + )x 2 ⎟ ⎜ (δ2 +δ1 )−2 δ1 δ2 ⎟, z2 = ⎜ ⎟ ⎜ δ1 −δ2√ ⎜ (1 + (δ +δ )−2 δ δ )x3 ⎟ 2 1 1 2 ⎠ ⎝ −δ2√ (1 + (δ +δδ1)−2 )x 4 δ δ 2

1

1 2

where z1 is the closest point to the point x. Since the generating line, Lg , at z1 about the cone (36) can be expressed as z(s) = sz1 (−∞ < s < ∞), and it is not difficult to verify that

δ1 + δ2 1 smin = 1+ √ 2 2 δ1 δ2 such that z(smin ) is the closest-point on Lg to x, we obtain the suboptimal triangulation for the measured point x based on the generating line as:

δ1 + δ2 1 z1 1+ √ z(smin ) = 2 2 δ1 δ2 √ ⎞ ⎛ (1 + δ2 / δ1 δ2 )x1 ⎟ ⎜ √ ⎟ 1⎜ ⎜ (1 + δ2 / δ1 δ2 )x2 ⎟ (38) = ⎜ ⎟. √ 2 ⎜ (1 + δ1 / δ1 δ2 )x3 ⎟ ⎠ ⎝ √ (1 + δ1 / δ1 δ2 )x4

This is just (37), hence the suboptimal triangulation based on the generating line is the optimal solution.  7 Suboptimal Triangulation III: Triangulation via the Sampson Sequence In this section, we give our third suboptimal triangulation, called the Sampson suboptimal triangulation. The Sampson suboptimal triangulation is based on our introduced Sampson sequence, which will be elaborated in the following subsections. 7.1 Sampson Sequence for a Variety At first we introduce the Sampson sequence for a variety. The Sampson sequence is closely linked to the Sampson approximation, here is a brief review of the Sampson approximation. Let G be a variety defined by the equation G(x) = 0 (x ∈ R n ), or a set of algebraic equations. A measured point x is said to be from the variety G, is meant that its true value x¯ satisfies G(¯x) = 0. Since in practice it is unable to obtain the true value x¯ , the solution of the following minimization, which is called the closest-point problem, is usually considered as the estimation of x¯ : x¯ = arg min ¯x − x2 subject to G(¯x) = 0.

(39)

Let x = x¯ − x, then the minimization becomes x = arg min(x)T x subject to G(x + x) = 0.

(40)

This is a nonlinear minimization. To reduce the computational complexity, the first-order approximation of (40), called the Sampson approximation of the measured point x, is usually used. The Sampson approximation was first introduced by Sampson (1982) for conic fitting. In 3D reconstruction, the Sampson approximation was also used to iteratively fit the epipolar geometry and the multiple view geometry (Zhang 1998; Torr and Zisserman 1997, 1998), or to linearly solve the triangulation for two views, which does not satisfy strictly the epiploar constraint (Hartley and Zisserman 2003). The first order approximation of the constraint function in (40) can be expressed as: G(x + x) ≈ G(x) + J (x) · x

(41)

where J (x) is the Jacobian matrix of G at point x. The Sampson approximation is obtained by the following minimization with linear constraints x = arg min(x)T x subject to J (x) · x = −G(x).

(42)

Int J Comput Vis (2011) 91: 77–106

87

Using the Lagrange multiplier method, we can obtain the solution of (42) as: x = −J T (x)(J (x)J T (x))+ G(x).

(43)

Hence, the Sampson approximation of measured point x is xS = x − J T (x)(J (x)J T (x))+ G(x).

(44)

ellipsoid or cone can be expressed as



x B b  =0 G : G(x) = (xT , 1)T 1 bT c

(48)

where B is an invertible symmetric matrix. G is an ellipsoid if B is positive definite and c < bT B −1 b, and a cone if c = bT B −1 b. Clearly, the Jacobian matrix (a row vector) of G is

Clearly, if G is a linear variety, i.e., G(x) = Bx − b where B is a row-full-rank matrix, then the Sampson approximation xS must be on the variety. However, in general the Sampson approximation xS does not satisfy the equation G(x) = 0, i.e., it is not on the variety G. Next, we introduce the Sampson sequence for a measured point.

J (x) =

Definition 2 Given a measured point x from a variety G, we define a point sequence for it, called the Sampson sequence, as follows:

xS = x −

∂G = 2xT B + 2bT . ∂x

(49)

J (x) = 0 iff x is the centre of G, i.e. x = B −1 b. For a measured point x which is not the centre of G, its Sampson approximation is JT G(x). JJT

(50)

where Jn = J (xSn ), and xSn is called the n-th Sampson approximation of the measured point. If the Sampson sequence converges, then its limit point is called the Sampson point of the measured point.

Hereafter, we always assume the measured point is not the centre of G. Let B = U T U be the eigen-decomposition of B where U is an orthogonal matrix and  a diagonal matrix, then under the following Euclidian transformation



T −B −1 b x x U , = 1 1 0 1

Proposition 3 Suppose the Sampson sequence {xSn } converges to a point x∗ and J (x∗ ) is of row-full-rank, then x∗ is on the variety G.

G has the following canonical form:



 0 x  T ˜ : G(x ˜ ) = G =0 (x , 1)T 1 0 δ

Proof Since all the components of G are polynomials, both G and J are continuous. The continuity of J leads to that each sub-determinant of J is continuous. Thus, from the row-full-rank of J (x∗ ), there exists a natural number N such that for n ≥ N, J (xSn ) is of row-full-rank. So, for n ≥ N the (n + 1)-th Sampson approximation can be expressed as

˜ is a cone with the vertex at the where δ = c − bT B −1 b. G origin if δ = 0, an ellipsoid with the centre at the origin if δ < 0. Under the canonical form, the Sampson approximation of the measured point x is

xSn+1 = xSn − JnT (Jn JnT )−1 G(xSn ).

x = x −

xSn+1 = xSn − JnT (Jn JnT )+ G(xSn ),

n ≥ 0,

xS0 = x (45)

(46)

Then we obtain G(xSn ) = Jn (xSn

From − xSn+1 )

(47)

and, G(x

∗

S

) = lim (J (xnS ) · (xnS n→∞

S − xn+1 ) = J (x ∗ ) · 0 = 0.

Hence x∗ is on the variety G.



˜ ∂G ∂x



x T x + δ 2x T  2 x

=

∂G ∂x

·

∂x  ∂x

x .

=

∂G ∂x

(51)

(52)

· U T , we have

xS = U T x − B −1 b. S

(53)

Since xS and x S are related by (53), we can always assume that G has the canonical form (51) in our subsequent discussions without loss of generality. Now consider the relationship between the Sampson approximation of a measured point x and its pedal x⊥ on the 

7.2 Convergence of the Sampson Sequence for Cone and Ellipsoid

polar plane. Since the polar plane of x about G is (y) = T yT x + δ = 0 and J = ∂ ∂y |y=x = (x) ,

In this subsection, we consider the convergence of Sampson sequence for cone or ellipsoid. In n-D space, an (n − 1)-D

x⊥ = x − J T (J J T )−1 (x) = x −

xT x + δ x. xT  2 x

(54)

88

Int J Comput Vis (2011) 91: 77–106

By (52) and (54), xS = x −

1 xT x + δ x = (x + x⊥ ). 2 2xT  2 x

(55)

Fig. 2 The Sampson sequence {xSn } and the pedal sequence ⊥ S {x⊥ n } : ∠oxn xn = π/2 for a ⊥ cone, ∠oxn xSn > π/2 for an ellipsoid

Hence, we obtain an interesting geometric interpretation for the Sampson approximation, which is stated in Proposition 4 and will be used in the convergence proof of the Sampson sequence below. Proposition 4 As shown in Fig. 1, the Sampson approximation xS of a measured point x is the midpoint of x and its pedal x⊥ on the polar plane. If the measured point x is not the centre of G, i.e., x = 0, and is not on G for a cone, then its Sampson approximation xS is not the centre of G either. Here is a proof: If xS is the centre of G, from (52) we have x−

xT x + δ x = 0. 2xT  2 x

xT x + δ = 0. 2

And thus, = δ. If G is an ellipsoid, then < 0, this is impossible because  is a positive definite matrix. If G is a cone, then xT x = 0, this is in contradiction to the assumption that x is not on G. Hence, the Sampson approximation xS is not the centre of G. Based on the above discussions, in general the Sampson sequence of a measured point x can be expressed as xT x

(xSn )T xSn + δ S xn . 2(xSn )T  2 xSn

(56)

Proposition 5 If G is a cone or an ellipsoid, then the Sampson sequence {xSn } must converge to a point on G.

(xSn )T xSn + δ S xn , (xSn )T  2 xSn

− x⊥ n)=

4(xSn−1 )T  2 xSn−1

> 0,

T S ⊥ we have −(x⊥ n ) (xn − xn ) < 0, and thus T S ⊥ −(x⊥ n ) (xn − xn ) < 0. S ⊥ x⊥ n  · xn − xn 

S This leads to ∠ox⊥ n xn > π/2. Hence, for a cone or an ellipsoid we always have

n ≥ 0.

δ((xSn )T xSn + δ) 2(xSn )T  2 xSn

(58)

Let cn = xSn . By (54), there must be cn+1 < cn i.e., the sequence {cn } monotonically decreases. And thus {cn } must converge to a scalar α, xSn  = cn → α

(n → ∞).

(59)

S ⊥ Let an = x⊥ n − xn , bn = xn . Since cn+1 is a middle line S of the triangle ox⊥ n xn ,  1 cn+1 = 2(bn2 + cn2 ) − an2 . (60) 2

From (60), we have

bn2 ≤ cn2 − an2 . (57)

It is the pedal of xSn on the polar plane : xT xSn + δ = 0. By Proposition 4, xSn+1 is the midpoint of xSn and x⊥ n. If G is a cone, then the polar plane passes through the oriS gin o (the vertex of cone G), and ∠ox⊥ n xn = π/2, as shown in Fig. 2. S If G is an ellipsoid, then ∠ox⊥ n xn > π/2. This is because: From T S −(x⊥ n ) (xn

((xSn−1 )T xSn−1 + δ)2 (xSn−1 )T  3 xSn−1

S 2 2 cn2 = an2 + bn2 − 2an bn cos ∠ox⊥ n xn ≥ an + bn ,

Proof At first, we prove the convergence of {xSn }. Let S x⊥ n = xn −

= −δ +

S ∠ox⊥ n xn ≥ π/2.

xT x

xSn+1 = xSn −

(xSn )T xSn + δ

S cos ∠ox⊥ n xn =

Then, by left-multiplying its two sides with xT , we obtain xT x −

and

By (60) and (61), we obtain cn+1 ≤

1 2

or, (61)

 4cn2 − 3an2 , and thus

 2 2 an ≤ √ cn2 − cn+1 → 0 (n → ∞). 3

(62)

Then from (62) and   S T  1 ⊥  (xn ) xSn 1 S S xSn+1 − xSn  =  x n  = xn − xn  = an ,  2(xS )T  2 xS 2 2 n n we have xSn+1 − xSn  → 0 (n → ∞).

(63)

Int J Comput Vis (2011) 91: 77–106

89

By (59), {xSn } is a bounded sequence, and thus it has a converged sub-sequence, xSnk → x∗

(k → ∞).

(64)

Since xSnk +1 − x∗  ≤ xSnk +1 − xSnk  + xSnk − x∗ , by (63) and (64) we have xSnk +1 → x∗

(k → ∞).

And thus by induction, ∀m ≥ 1 there must be xSnk +m → x∗

Fig. 3 The measurement error, y⊥ , from the orthogonal dimension with G, should be ignored

(k → ∞).

So ∀ε > 0, ∃K and M such that for k ≥ K and m ≥ M, xSnk +m − x∗  < ε.

(65)

Let N = nK + M. Then for n > N , there is n = nk  + m (k  ≥ K, m ≥ M), and thus from (65) we have xSn − x∗  < ε. Therefore, the Sampson sequence {xSn } converges. Next, we prove that x∗ is on G. By left-multiplying the two sides (56) with (xSn )T , we have

Initiation: Given a termination threshold ε and let n = 0 and xS0 = x. 1. Sampson iteration: 1.1 xSn+1 = xSn − JnT (Jn JnT )−1 ϕ(xSn ). 1.2 If |ϕ(xSn+1 )| < ε, then output x¯ = xSn+1 and terminate. Otherwise, let n ← n + 1 and return to 1.1. 2. Reconstruction: Compute the 3D point of the corrected correspon¯ by the DLT method. dence X

Sampson point of the measured point from the fundamental cone satisfies strictly the epipolar constraint. In the Sampson suboptimal triangulation, we do not need taking the Euclidian transformation E to the fundamental cone as that in the previous two suboptimal triangulations. Because with the fundamental matrix F = (Fij ), the Sampson sequence can be directly computed:

(xSn )T xSn + δ = 2(xSn )T (xSn − xSn+1 ). And thus, (x∗ )T x∗ + δ = lim (xSn )T xSn + δ n→∞

= 2 lim (xSn )T (xSn − xSn+1 ) = 0. n→∞

Hence, x∗ is on G.

Table 6 Suboptimal Triangulation III: Triangulation based on Sampson sequence



Remark 3 Proposition 5 holds for any dimensional cone or ellipsoid in R n . For a k-dimensional cone or ellipsoid G in R n , R n has an orthogonal decomposition, R n = R k+1 ⊕ R n−(k+1) , such that G ⊂ R k+1 . Let x be a measured point from G, then it can be uniquely decomposed as x = y ⊕ y⊥ , y ∈ R k+1 , as shown in Fig. 3. Clearly, y⊥ is the measurement error from the orthogonal dimension with G, and it should be removed. And thus, the Sampson point of y in R k+1 can be considered as the Sampson point of x. For general varieties, does Proposition 5 still hold? Or, under what conditions does it keep true? These are still interesting open questions, and worthy of further investigating since they can be applied to the multi-view triangulation. 7.3 Sampson Suboptimal Triangulation Applying Proposition 5 to the fundamental cone, we can obtain our third suboptimal triangulation, the Sampson suboptimal triangulation, outlined in Table 6. By Proposition 5, the

xSn+1 = xSn − JnT (Jn JnT )−1 ϕ(xSn ), where, Jn = 2(xSn )T B +2bT , ϕ(xSn ) = (xSn )T BxSn +2bT xSn + 2F33 , ⎞ ⎛ F11 F21 ⎜ F12 F22 ⎟ ⎟, B =⎜ ⎠ ⎝ F11 F12 F21 F22 b = (F31 , F32 , F13 , F23 )T .

8 Experiments 8.1 Experiment with Simulative Data In our experiments, three typical cameras configurations are considered: Configuration I (Circular Configuration): As shown in Fig. 4 (left), the observed space points lie in a sphere of radius 4 units. The two cameras’ optical centers both lie on

90

Int J Comput Vis (2011) 91: 77–106

Fig. 4 Three typical configurations used in the experiments. Left: Circular configuration (r = 10); Middle: Radial Configuration (h = 10); Right: Lateral Configuration (h = 10)

the same circle centered at the sphere’s center with a fixed radius r = 10 units, and their optical axes pass through the sphere center. The base-height ratio, b/ h, is set to 1, 1/2, 1/20, and 1/40 3 respectively, where b is the baseline of the two camera, and h is the distance from the sphere’s center to the mid-point of the baseline. Configuration II (Radial Configuration): As shown in Fig. 4 (middle), camera 2 is just moved along the optical axis (z-axis) by b from camera 1. The observed points are in the sphere centered at the z-axis by a fixed h = 10 with a radius of 1 unit. The base-height ratio is set to 2/3, 3/5, 1/2, and 2/5 respectively. Configuration III (Lateral Configuration): As shown in Fig. 4 (right), this is the standard setup of a stereo system. Camera 2 is moved along the x-axis by b from camera 1. The observed points are in the sphere centered at the bisector of b and away from the x-axis by a fixed h = 10, whose radius is 4 units. The base-height ratio is set to 1, 1/2, 1/20, and 1/40 respectively. Under each of our three configurations, the internal matrices of the two cameras both are set as: ⎛ ⎞ 700 0 512 K =⎝ 700 512 ⎠ . 1 In each of our experiments, we choose randomly 50 3D points in the observed-point sphere, such that all their projected image points are in [0, 1024] × [0, 1024]. Then, Gaussian noise with zero mean and standard deviation σ is added to each image point. The noise level σ is varied from 0.5 to 3.0 with the steps of 0.5 pixels. For providing statistically meaningful results, at each noise level 500 trials are 3 When

the b/ h is small, it means the baseline is relatively short, the reconstruction performance degrades accordingly.

performed. The compared algorithms include DLT (Homogeneous linear triangulation), HSO (Hartley and Sturm’s optimal triangulation), ONO (Our new optimal triangulation), SOC (Suboptimal triangulation based on generating cone), SOL (Suboptimal triangulation based on generating line), and SSO (Sampson suboptimal triangulation). The evaluated performances include the following three aspects: (a) RMS error of reprojection. (b) RMS of the distance from the suboptimal solution to the global optimal solution (i.e., from the corrected image point of the suboptimal triangulation to the corrected image point of the optimal triangulation). (c) RMS error of 3D points (reconstructions) relative to their ground truths. 8.1.1 Experimental Results for the Configuration-I Experimental results for the configuration I are shown in Fig. 5, Fig. 6, Fig. 7 and Table 7. Figure 5 (left) shows the RMS error of reprojection. From this figure we can see that: The RMS error of the DLT is remarkably larger than that of all the other triangulations, and the RMS error curves of the three suboptimal triangulations are almost the same with that of the optimal triangulation. To further show the performance of the three suboptimal triangulations, Fig. 5 (right) shows the difference of their RMS errors to that of the optimal triangulation. It can be seen that, even if the SSO gives the largest difference among the three suboptimal triangulations, it is less than 10−5 pixels either with long or short baselines; and both the SOC and the SOL have the difference closing to zero. This indicates that the three suboptimal triangulations and the optimal triangulation give almost the same RMS error of reprojection. Figure 6 shows the RMS of the distance from the solution of the SOC (SOL, SSO) to the global optimal solution. From Fig. 6 we can see that: Both the SOC and the SOL have the

Int J Comput Vis (2011) 91: 77–106

91

Fig. 5 Experimental results for the configuration I. Left: RMS error of reprojection. Right: The difference of the RMS error of the three suboptimal triangulations to the RMS error of the optimal triangulation

RMS closing to zero, and their RMS almost do not change with the decrease of baseline length or the increase of data error. Though the RMS of the distances from the solution of the SSO to the global optimal solution increases with the increase of measurement error, it is no larger than 10−2 pixels at the measurement error of 3.0 pixels. These results indicate that, the SOC and the SOL have comparable estimation accuracy with the optimal triangulation, and the SSO gives also a good accuracy. From Fig. 7 which is about the RMS error of 3D points (reconstructions) relative to their ground truths, we can see that, in the case of long baselines, the DLT gives the largest the RMS error among all the triangulations. The three suboptimal triangulations and the optimal triangulation give almost the same RMS error curves. To further show the performance of the three suboptimal triangulations, Fig. 7 (right) shows the difference of their RMS errors to that of the opti-

mal triangulation. It can be seen that, the SSO gives the best accuracy for 3D reconstruction, and both the SOC and the SOL have the similar accuracy with the optimal triangulation. Table 7 is the average iteration number of the SSO for the measured points with Gaussian noise of zero mean and standard deviation 3.0. From this table, we can see that the Sampson sequence has almost a geometric convergence rate. Only with a small number of iterations, the SSO is sufficiently close to the Sampson point, which is on the fundamental cone. For example, the SSO requires only three iterations in order to close to the Sampson point in the accuracy of 10−9 . 8.1.2 Experimental Results for the Configuration II Experimental results are shown in Fig. 8, Fig. 9, Fig. 10 and Table 8. For the reprojection RMS errors, from Fig. 8 we can

92

Int J Comput Vis (2011) 91: 77–106

Fig. 5 (Continued) Table 7 Average iteration number of the SSO in the configuration I

ε = |ϕ(xSn )|