THE KERNEL OF A DIRECT CLOSED-FORM SOLUTION TO

THE KERNEL OF A DIRECT CLOSED-FORM SOLUTION TO GENERAL RELATIVE ORIENTATION Heping Pan Cooperative Research Centre for Sensor Signal and Information Processing SPRI Building, Technology Park, Adelaide, The Levels, SA 5095, Australia Email: [email protected]

KEY WORDS: Stereo geometry, stereo camera calibration, relative orientation, direct closed-form solution, photogrammetry. ABSTRACT

Recovering the relative geometry of two perspective stereo images is a classical problem in photogrammetry and also a basic problem in computer vision. General relative orientation refers to recovering ve relative orientation parameters and two principal distances from pure image measurements. This paper exposes the kernel of a new direct closed-form solution to this problem while the general frame of this solution was rst published in 1995 [Pan et al, 1995a, 1995b]. The approach recasts the explicit stereo coplanarity equation into an implicit form whose coecients, which can be put in a coplanarity matrix, are solvable via a direct solution from the image measurements. The two principal distances (focal lengths) are then fully determined by sixth-order algebraic equations whose coecients are fourth-order polynomials of coplanarity matrix elements. Fortunately, these sixth-order algebraic equations are solvable in closed-form. After the two principal distances have been solved, the relative orientation (5 parameters or, rotation matrix and baseline vector) are then solved successively also in closed-form. This direct closed-form solution is useful not only in real-time implementation of stereo vision, but also for providing the approximate values of unknown geometric parameters to high-precision photogrammetry, as well as for direct functional analysis of error propagation.

1 Introduction In 1995, the author with the help of his colleagues has fortunately discovered a direct closed-form solution to general relative orientation (to be explained below) and published two papers in a SPIE Photonics Conference on Videometrics VI [Pan et al, 1995a, 1995b]. Since then, interested readers have contacted the author for the kernel part of this solution, which was not included in the two 1995 papers. This paper intends to expose the kernel of this solution and also clarify some degenerate cases. We start with a historical review on the relative orientation problem in photogrammetry and its generalization in computer vision. We then clarify the problem in exact technical terms and formulate the basic stereo coplanarity equations in the explicit form and implicit form. After that, we derive the high order algebraic equations. We then show that these equations are solvable in a direct and closed form. Recovering the relative geometry of two stereo images is a classical problem in photogrammetry and also a basic problem in stereo vision. Technically, this problem includes interior orientation of each image and relative orientation of the two images, which are the prerequisite for object localization and surface reconstruction from the two images. Interior orientation of an image refers to the determination of 3 intrinsic parameters: the principal distance - also called focal length in an inexact sense - and the principal point position in a given image plane coordinate system. Relative orientation of two images refers to determination of the relative baseline (translation in motion vision) vector of two perspective centres and the relative rotation (matrix or angles) of one image relative to the other, which totally involves only ve parameters. Therefore, the relative geometry of two stereo images includes totally 11 parameters (two sets of three intrinsic parameters and one set of ve relative orientation parameters). However, if only homologous image point coordinates are measured, it is known that only seven parameters can be solved from the image measurements. Considering general application, the

principal point position can be determined by using ducial marks with metric cameras, or good cameras are manufactured such that the principal point can be very close to the image coordinate centre. Therefore, in the following discussions, we assume either the two principal points to be known a priori or suciently close to zero. Hence, we choose to study the problem of solving for two principal distance and ve relative orientation parameters, a problem we shall call general relative orientation. This problem is chosen because it contains a selection of the maximum number of parameters which is symmetric and practically relevant. The word 'general' means that we intend to generalize the standard relative orientation problem in photogrammetry by including also two principal distances. The importance of this problem is obvious: given two stereo images with unknown interior and relative orientations, it is impossible to reconstruct 3D object points or surfaces without solving the general relative orientation even a general image matching can be achieved [Pan, 1996]. This paper presents a direct closed-form solution to the general relative orientation problem. We shall concentrate on the existence and correctness of this solution. The error sensitivity analysis is beyond the scope of this paper. However, a numerical example and a real image application are provided. Camera calibration in photogrammetry terminology refers to determining the three intrinsic geometric parameters and all other camera distortions. This is routinely done by using control points. Self-calibration of cameras in photogrammetry refers to include the camera geometry and distortion parameters into the simultaneous adjustment of photogrammetric network. However, in computer vision literature, camera calibration refers to anything related to solving intrinsic and extrinsic geometry and distortion parameters of one, or two, or three cameras, thus may include interior and relative orientation. In a more general setup, we may have a network of overlapping images covering the object surfaces of interest completely. The recovery of the image overlapping topology and geometry of this image network may be better called im-

age resituation [Pan et al, 1995a, 1995b], meaning to recover the topological and geometric situation of all images of this network. Therefore, image resituation may be understood as a general umbrella covering interior and relative orientation, and camera calibration of any number of networked images. However, in this paper, we shall concentrate on the general relative orientation of two stereo views, though this background is clari ed. Most photogrammetrists consider the problems of interior and relative orientation problem to be already solved. These problems have been revisited in the recently years by computer vision specialists partly because the early photogrammetric literature is not easily available to computer vision community, and largely because the stereo vision system, e.g. robot head, touches somes aspects of the stereo geometry which may be rather dierent from the standard aerial photogrammetric applications. It is also the case in close-range photogrammetry. The formulation of the stereo geometry (interior and relative orientation, etc.) goes back to the period around the turn of the century to 30's [Heisse, 1863; Finsterwalder, 1899; Finsterwalder, 1932; Fourcade, 1903; Kruppa, 1913]. Analytical photogrammetry including interior orientation, camera calibration, relative orientation, absolute orientation, up to aerial triangulation network adjustment were intensively studied from the 50's to the 70's [Schut, 1955; Thompson, 1959; Schmid, 1954; Abdel, 1971; Stefanovic, 1973; Wang, 1979; Khlebnikova, 1983]. The relative orientation problem was studied by computer vision community rather from a setup of motion vision [Longuet et al, 1980; Longuet, 1981; Bruss et al, 1983; Horn (1990) bridged over the photogrammetry and computer vision communities, and gave a thorough description of relative orientation problem. In recent years, the problem of interior and relative orientation were revisited by many computer vision specialists as well as some photogrammetrists [Huang et al, 1989; Brandstatter, 1991; Faugeras et al, 1992; Hartley, 1992; Faugeras, 1993; Niini, 1993; Weng et al, 1993; Niini, 1994; Brandstatter, 1996].

perspective projection of each single image play the central role, so the relative orientation of multiple images may be totally avoided, but at the expense of having control points or other information. However, the coplanarity equations represent still the basic stereo geometry if only two images are available. It is especially useful for robot stereo vision systems and biological stereo vision study.

2 Explicit and Implicit Coplanarity Equations For the generality of stereo geometry, let us assume that there is a 3D global coordinate system O ? XY Z with origin O and orthogonal axes labelled XY Z , within which two stereo images, left and right, are situated. The 2D image coordinate system o ? xy and the 3D camera coordinate system C ? x~y~z~ will be local to the left image, where C is the perspective centre of the left camera. Symbols relating to the right image will be marked with a prime 0 , yielding the right image coordinate systems o0 ? x0 y0 and right camera coordinate system C 0 ? x~0 y~0 z~0 . Let c denote the principal point [i.e. the orthogonal projection of the perspective centre on the image plane] of the left image and f the left principal distance, i.e. f = Cc. The position C in O ? XY Z is denoted by (XC ; YC ; ZC ), the position c in o ? xy is (xc ; yc ). The orientation of C ? x~y~z~ in O ? XY Z is captured by an orthonormal rotation matrix R de ned by 3 successive rotation angles , , around C ? x~, C ? y~, and C ? z~ axes respectively,

R=

!

= R R R

1 0 0 ! 0 cos  sin  0 ? sin  cos  cos sin 0 ! ? sin cos 0 0 0 1

=

In the standard photogrammetric applications, relative orientation is solved through an iterative least-squares algorithm. A closed-form solution which is usually called relative linear transform was rst formulated by Thompson (1959) and further exploited by Stefanovic (1973). As Horn (1990) pointed out, Longuet-Higgin's essential matrix [Longuet, 1981] is similar to Thompson's transform. Of course, a number of properties of Longuet-Higgin's essential matrix and Faugeras' fundamental matrix [Faugeras et al, 1992] discovered in recent years are not explicitly known previously. On the other hand, Faugeras' fundamental matrix has other usefulness in image matching and motion vision. The work presented here may be considered as a follow-up of these earlier works. We present a new direct closed-form approach to the general relative orientation problem, i.e. solving for two principal distances and ve relative orientation parameters from pure image measurements. This problem was also tackled by Hartley (1992), but in a quite dierent way which is rather more complicated and less direct than our approach. In the next section, we rst clarify the relative geometry of two stereo images. Central to the stereo geometry is the epipolar constraint which is expressed by the coplanarity equation for each pair of homologous image points. It is also known that in photogrammetric network adjustment, such as bundle adjustment, the collinearity equations characterizing the

r11 r12 r13 r21 r22 r23 r31 r32 r33

cos 0 ? sin ! 0 1 0 sin 0 cos (1)

~ z ~ z’

~ y ~ x

C

~ y’

B (b ) C’

~ x’ f

p

p’

y (x ,yc ) c c o

f’ y’

x p(x,y)

o’ p’ (x’,y’) c’ (x’,y’) c c

x’

Z

P Y

O

X

Figure 1: Analytical Geometry of Two Stereo Images

2.1 Collinearity Equations

Suppose an object point P = (X; Y; Z ) in O ? XY Z is projected to an image point p(x; y) in o ? xy. Let us de ne three vectors for image point p(x; y)

p= p~ = p =

x y

!

1

x ? xc ! y ? yc = p; ?f x ! y = R p~ z

where

1 0 ?xc !

=

0 1 ?yc 0 0 ?f

(2) (3) (4)

(5)

Note that p~ denotes the image point p in C ? x~y~z~, p denotes the vector p~ in O ? XY Z . The perspective projection from P to p through the perspective centre is captured by a collinearity equation (P , C , p are collinear)

P = C + p or written in analytical form

X Y Z

!

=

XC YC ZC

!

+

r11 r12 r13 r21 r22 r23 r31 r32 r33

(6)

! x ? xc ! y ? yc ?f

(7)

Eliminating the scaling factor , we obtain two analytical equations for each pair of object-image points

X ? XC ) + r21 (Y ? YC ) + r31 (Z ? ZC ) x ? xc = ?f rr11 ((X ? XC ) + r23 (Y ? YC ) + r33 (Z ? ZC ) 13

(8) ? XC ) + r22 (Y ? YC ) + r32 (Z ? ZC ) y ? yc = ?f rr12 ((X 13 X ? XC ) + r23 (Y ? YC ) + r33 (Z ? ZC ) (9)

This is the most direct form of collinearity equations widely used in camera calibration, interior and exterior orientation, as well as bundle adjustment. When a sucient number of control points with known (X; Y; Z ) is given, it can be easily seen that all unknowns including geometric parameters (xc ; yc ; f ), (XC ; YC ; ZC ), (; ; ) as well as all other additional parameters characterizing camera distortions and other unknown object points can be solved through a simultaneous collection of collinearity equations. This is the essential idea of the bundle adjustment in photogrammetric network adjustment. The purpose for showing these collinearity equations is to clarify the relationship between these equations and the coplanarity equations which will be discussed below.

2.2 Explicit Coplanarity Equation When the only available information is a number of homologous image points, we can easily see that the collinearity equations given in (8) contain too many unknowns that cannot be solved from pure image measurements. Let B denote the baseline vector,

!

Bx By Bz

B=

XC ? XC 0 YC ? Y C 0 ZC ? ZC 0

=

!

(10)

As the magnitude of B cannot be determined without control information, let b denote a positively scaled version of B

b = (bx by bz )t = B;

>0

(11)

For any given object point P visible from the left and right images, there are two homologous image points p(x; y) and p0 (x0 ; y0 ). A simple fact is that the ve points P , p, p0 , C , and C 0 are coplanar, which can be captured by a cross product of three vectors being equal to zero [p b p 0 ] = p
(b  p 0 ) = 0 (12) This is the coplanarity equation so-called traditionally in photogrammetry. There are several ways of rewriting this coplanarity equation. One way is to write it in the simplest analytical form

bx by bz x y z = 0 x0 y0 z0

(13)

This is the most direct form of coplanarity equation used in iterative relative orientation in photogrammetry. We can also rewrite the coplanarity equation (12) in the form of matrix product, p
(b  p 0 ) = p t B p 0 = 0 (14) where B is a skew-symmetric matrix de ned by the elements of b: (bx ; by ; bz ),

0 ?bz by ! bz 0 ?bx ?by bx 0

B=

(15)

Using the notations and relations de ned by (2) - (4), we obtain pt t Rt B R0 0 p0 = 0 (16) We shall call this form the explicit coplanarity equation, because the role of every geometric parameters is explicit shown. This becomes more clear if we rewrite this equation in analytical form

  x y

t

1



0 11 r0 21 r0 31 r

1 0 0

0 12 r0 22 r0 32 r





0 1 0

?xc t ?yc ?f 0 ?bz

bz by

? 013 r0 23 r0 ) 33 r



0

bx

1 0 0

r11 r21 r31 by

?bx

0

0 1 0



r12 r22 r32

?x0c0 ?yc0 ?f

r13 r23 r33



t

 0  0 x y

1

= 0

(17)

2.4 Choices of the Reference System

2.3 Implicit Coplanarity Equation Let

A = R t B R0

(18)

equation (16) then becomes

p~ t A p~ 0 = 0

(19)

or, in expanded form,

x ? xc y ? yc ?f

!t a11 a12 a13 ! x0 ? x0c ! 0 0 a21 a22 a23 a31 a32 a33

y ? yc ?f 0

=0 (20)

where aij , i; j = 1; 2; 3, are the elements of A. We term equation (20) the rst form of the implicit coplanarity equation, and A the special coplanarity matrix. It is useful at this stage to consider folding (xc ; yc ; f; x0c; yc0 ; f 0 ) into A, thereby obtaining an equation which is made up explicit only of the purely measured coordinates of image points. Let D = t A 0 (21) equation (19) then can be rewritten as pt D p0 = 0

(22)

or, in expanded form,

! x0 ! ? x y 1
dd11 dd12 dd13 y0 = 0 (23) 21 22 23 d31 d32 d33

1

where dij , i; j = 1; 2; 3, are the elements of D. We term equation (23) the second form of the implicit coplanarity equation, and D the general coplanarity matrix. The equations (20) and (23) may be considered as a generalized reformulation of the relative linear transform rst introduced by Thompson (1959) and then followed by Stefanovic (1973). In the work of Longuet-Higgins (1981), the intrinsic parameters (xc; yc ; f ) and (x0c ; yc0 ; f 0 ) are assumed known, and a coordinate system equivalent to the second reference system described in section 2.4 is adopted. Longuet-Higgins' essential matrix is equivalent to the special coplanarity matrix, but in a less general setting. Faugeras et al (1992) considered the more general case in which the intrinsic parameters are unknown, but are xed across all images (with camera adjustment disabled, as would apply with a frozen mobile camera). A fundamental matrix was de ned which is essentially equivalent to the above general coplanarity matrix, D. Faugeras et al. actually considered a more general camera model with a greater range of linear distortions and associated free parameters than that considered here, but they assume these parameters have already been found by calibrating the camera: they do not attempt to solve for them along with rotation and baseline parameters for the case of two images. The reason we term matrix A in (20) and D in (23) the special and general coplanarity matrix respectively is to keep the consistency with the term coplanarity equation which is in existence for decades.

We now explain why a global coordinate system O ? XY Z is introduced at the beginning of this section. The reason is simple: because there are dierent ways of choosing the reference system for the general relative orientation, each with its trade-o. Reference System 1: The rst choice is to align the global coordinate system with baseline vector and the left (or right) image, such that O = C , O ? X = CC 0 , and the O ? XZ plane contains the left (or right) principal point c. The two-way ambiguity of the O ? Y axis's direction may be resolved by adoption of the right-hand rule. We now have that

b = (b; 0; 0)t ;

=0

(24)

where b is the magnitude of b. Because b cannot be determined, the degrees of freedom are now given by ( ; ; 0 ; 0 ; 0 ; xc ; yc ; f; x0c; yc0 ; f 0 ):

With this reference system, the explicit coplanarity equation (16) becomes pt t Rt R0 0 p0 = 0 (25)

where R = R R . The special coplanarity matrix A of (18) becomes A = R t R0 (26)

This reference system is inspired by the biological vision system. It is neutral and natural to take the baseline as the X -axis. With this reference system, the disparity for a pair of homologous points can be decomposed to horizontal and vertical components. Horizontal disparity can be de ned to the orientational dierence of the left and right viewing rays in the baseline direction, while the vertical disparity refers to the dierence in the direction perpendicular to the plane de ned by the baseline and the left principal axis. Reference System 2: The second choice is the coincidence of the global and image coordinate systems. Without loss of generality, we choose the left image system. When O ? XY Z coincides with C ? x~y~z~, the left view is de ned by the values of the intrinsic parameters (xc; yc ; f ) as well as the equalities

R = I;

(XC ; YC ; ZC )T = (0; 0; 0)T

(27)

where I is the identity matrix. The nature of the right view is xed by the speci cation of the intrinsic parameters (x0 0c; y0c0 ; f00 ), the matrix R0 (incorporating the rotation angles  , , ), and the direction of the baseline vector given by

B = (XC 0 ; YC 0 ; ZC 0 )T

(28)

With this reference system, the parameter set for characterising the stereo geometry becomes (0 ; 0 ; 0 ; bx ; by ; bz ; xc ; yc ; f;x0c ; yc0 ; f 0 ); where only two of the three components bx , by , bz need to be determined. Apparently, this reference system is more biased to the left (or right) image. Note that, for the remainder of this work, we shall assume this second reference system being used. Since we shall no longer deal with both left and right rotation matrices, we henceforth

drop the prime and refer to R0 as R = (rij ), i; j = 1; 2; 3, this being the rotation matrix mapping the right image system into the left image system. The special coplanarity matrix A (18) thus becomes

A = BR

Expanding this equation, we have

a211 + a212 + a213 = b2y + b2z a221 + a222 + a223 = b2x + b2z a231 + a232 + a233 = b2x + b2y a11 a21 + a12 a22 + a13 a23 = ?bx by a11 a31 + a12 a32 + a13 a33 = ?bx bz a21 a31 + a22 a32 + a23 a33 = ?by bz

(29)

which is identical to Longuet-Higgins' essential matrix.

2.5 Properties of Coplanarity Matrices

Similar to the properties of the essential matrix and the fundamental matrix found by Huang and Faugeras (1989) and Faugeras et al (1992), some obvious properties of the special and general coplanarity matrices A and D can be easily shown, 1. Because the determinant of matrix B is zero, from (18) and (21), we know

jAj = jDj = 0

(30)

2. From the implicit coplanarity equations (19) and (22), we know both A and D only determined to a scale factor. 3. From the above two properties, we know the degree of freedom (DoF) of D is 7, i.e.

DoF (D) = 7 (31) We also know that A is solely de ned by 5 relative orientation parameters, whatever these parameters are chosen, so

DoF (A) = 5

(32)

4. Because the rst property (30) is a third-order polynomial equation, we know the degree of linear freedom (number of linearly noncorrelated coecients) (DoLF) is 8, i.e.

DoLF (D) = 8

(33)

DoLF (A) = 8

(34)

Similarity

The third property (31) shows that at most only 7 parameters can be solved from the relative geometry of two images using only image measurements. It is another con rmation of seven degrees of freedom in relative stereo geometry known previously [Kruppa, 1913]. That is why we choose seven unknowns (two principal distances and ve relative orientation parameters) to de ne the problem of general relative orientation as the central topic of this work. It is the fourth property (33) and (34) that makes it possible to solve for D or A from pure image measurements via a simultaneous collection of implicit coplanarity equations. In addition to these simple properties, we will show more complicated properties by fully exploiting the orthonormality of the rotation matrix R. Using equation (29), we obtain a direct relation between A and B ,

A At = (B R)(B R)t = B R Rt B t = B B t

(35)

(36) (37) (38) (39) (40) (41)

Equations (36) { (38) are simple linear equations of b2x , b2y , and b2z , so b2 ! 0 1 1 !?1 a2 + a2 + a2 ! x

1 0 1 1 1 0

11

12

13

a221 + a222 + a223 (42) a231 + a232 + a233 From equations (39) { (41) we can also solve for b2x , b2y , and b2z as

=

b2y b2z

 b b b b b b T (bx ; by ; bz = xybyzxz ; xybxzyz ; xzbxyyz 2

2

2

)T

(43)

where

bxy = ?(a11 a21 + a12 a22 + a13 a23 ) bxz = ?(a11 a31 + a12 a32 + a13 a33 ) byz = ?(a21 a31 + a22 a32 + a23 a33 )

(44) (45) (46)

Substituting (43) into (42) gives 3 equations de ned purely in terms of the elements of A,

(?a213 + a223 + a231 + a232 + a233 ? a211 ? a212 + a221 + a222 ) (a21 a31 + a22 a32 + a23 a33 )+ 2(a11 a21 + a12 a22 + a13 a23 )(a11 a31 + a12 a32 + a13 a33 ) = 0 (47) (a213 ? a223 + a231 + a232 + a233 + a211 + a212 ? a221 ? a222 ) (a11 a31 + a12 a32 + a13 a33 )+ 2(a11 a21 + a12 a22 + a13 a23 )(a21 a31 + a22 a32 + a23 a33 ) = 0 (48) (a213 + a223 ? a231 ? a232 ? a233 + d211 + d212 + d221 + d222 ) (a11 a21 + a12 a22 + a13 a23 )+ 2(a11 a31 + a12 a32 + a13 a33 )(a21 a31 + a22 a32 + a23 a33 ) = 0 (49) These will later prove valuable in deriving a direct closedform solution for solving for two principal distances from the general coplanarity matrix D. Equations (47) - (49) are not known in the literature, which can only be derived with such elementary manipulations. Due to equation (32) and considering the properties in (30) and (34), we know that one of the three equations (47)-(49) is redundant. However, this redundancy is useful for robustness.

2.6 Outline of the Whole Procedure

In the following sections, we shall show how to recover the 2 principal distances and 5 relative orientation parameters through the implicit and explicit coplanarity equations from pure image measurements. The whole procedure consists of these steps: (1) Solving for the general coplanarity matrix D from eight or more pairs of homologous image points. (2) Solving for the 2 principal distances f and f 0 from the general

coplanarity matrix D via a novel direct closed-form solution. (3) Solving for the special coplanarity matrix A from D and f and f 0 also via a direct relation. (4) Solving for the baseline vector and rotation matrix from A via a novel direct solution. The problem of the rst step is equivalent to solving the relative linear transform of Thompson (1959), and the essential matrix of Longuet-Higgins (1981), as well as the fundamental matrix of Faugeras (1992). An obvious approach [Thompson, 1959; Stefanovic1973; Longuet, 1981; Khlebnikova, 1983] is to transform the implicit coplanarity equation (equivalent) to a normal linear equation by setting one of the implicit coecients dij 's to unit. The remaining eight unknowns can thus be solved from the linear equations, using eight or more pairs of homologous image points [Hartley, 1995]. This approach is known to be inecient and sensitive to noise in image measurements and to the near-singular cases [Brandstatter, 1996]. An alternative approach [Faugeras et al, 1992; Weng et al, 1993] makes use of normalization constraint 3 3 X X

i=1 j=1

dij = 1

(50)

and solves for the 9 unknowns of D (equivalent) via the singular value decomposition. This second approach is an improvement to the rst approach as we no longer need to arbitrarily set a non-zero dij to unit, which may actually be close to zero. Again, this approach is known to be still sensitive to image noise and distribution of image points. Another approach [Pan et al, 1995b] is to use an iterative least squares criterion that takes into account the image noise, distribution of image points and constraints. Importantly, the problem of enforcing the constraint (30) has been done by Zhang et al (1995). In addition, the problem of eliminating outliers has been done by Torr (1995). Considering that robust solutions for essential and fundamental matrix exist in the literature such as [Zhang et al, 1995; Torr, 1995], which are also applicable for coplanarity matrix D, this paper will concentrate on the direct closed-form solutions for the two principal distances after the general coplanarity matrix D is determined. The further part of the solution for the baseline vector and rotation matrix was described in detail in [Pan et al, 1995b]. Error analysis is beyond the aim of this paper.

3 The Algebraic Solution for the Two Principal Distances 3.1 Basic Relation and Constraint

In practical applications, we want to include two principal distances f and f 0 as unknowns, given only seven degrees of freedom, we must assume the principal point coordinates xc and yc to be known a priori (e.g. detected by using ducial marks for a metric camera), or to be suciently close to zero. In order to solve for two principal distances f and f 0 from the solved general coplanarity matrix D, we need to rst defactorize the principal points (xc; yc ) and (x0c; yc0 ) out from D. Using the equations (2) - (4), (5), and (21), we obtain D = t A 0 = tc tf A 0f 0c (51) where

c =

1 0 ?xc ! 0 1 ?yc ; 0 0 1

f =

1 0 0 ! 0 1 0 (52) 0 0 ?f

Let

D = tf A 0f (53) D is a special version of D, D = D; if (xc ; yc ) = (x0c ; yc0 ) = (0; 0) (54) In general when (xc ; yc ); (x0c ; yc0 ) are nonzero, but known a priori, D can be solved from D via (51) as 1 D = ?c t D 0? (55) c Note that D involves purely the 7 degrees of freedom in the

coplanarity constraint, which correspond to 7 unknowns (2 principal distances and 5 relative orientation parameters) in the general relative orientation, we may simply call D the coplanarity matrix. D has the same properties of D as listed in the subsection 2.5. In fact, we can also capture xc and yc into the image measurements. The implicit coplanarity equation (23) can be written as ! ? x ? x y ? y 1
D xy00 ?? xy00c = 0 (56) c c c

1

D de ned by (56) can still be solved using the general approaches for solving D, but we need to use x ? xc and y ? yc instead of x and y. In the remainder of this work, we shall assume D is solved either via (55) or directly via (56). The relation between D = (dij ) and A = (aij ) is 0 d11 d12 d 0 1 a11 a12 a13 ! f a21 a22 a23 = @ d21 d22 df 0 A (57) d d d a31 a32 a33 f f f f0 13

23

31

32

33

3.2 Algebraic Equations of Two Principal Distances

Applying expression (57) into (47) { (49), 3 equations in 2 unknowns f and f 0 are obtained hi1 + hi2 f 2 + hi3 f 02 + hi4 f 04 + hi5 f 2 f 02 + hi6 f 2 f 04 = 0

(i = 1; 2; 3)

(58)

where the hij 's are coecients directly computable from the elements dij 's, given by

h11 = d23 d333 h12 = (d213 + +d223 )d23 d33 h13 = (d231 + d232 )d23 d33 + (d21 d31 + d22 d32 )d233 h14 = (d231 + d232 )(d21 d31 + d22 d32 ) h15 = (d222 ? d211 ? d212 + d221 )d23 d33 +(?d213 + d223 )(d21 d31 + d22 d32 ) +2(d11 d21 + d12 d22 )d13 d33 +2(d11 d31 + d12 d32 )d13 d23 h16 = (d222 ? d211 ? d212 + d221 )(d21 d31 + d22 d32 ) +2(d11 d21 + d12 d22 )(d11 d31 + d12 d32 ) h21 = d13 d333 h22 = (d213 + d223 )d13 d33 h23 = (d231 + d232 )d13 d33 + (d11 d31 + d12 d32 )d233 h24 = (d231 + d232 )(d11 d31 + d12 d32 )

(59) (60) (61) (62) (63) (64) (65) (66) (67) (68)

h25 = (d211 + d212 ? d221 ? d222 )d13 d33 +(d213 ? d223 )(d11 d31 + d12 d32 ) +2(d11 d21 + d12 d22 )d23 d33 (69) +2(d21 d31 + d22 d32 )d13 d23 h26 = (d211 + d212 ? d221 ? d222 )(d11 d31 + d12 d32 ) +2(d11 d21 + d12 d22 )(d21 d31 + d22 d32 ) (70) h31 = d13 d23 d233 (71) 2 2     h32 = (d13 + d23 )d13 d23 (72) h33 = ?(d231 + d232 )d13 d23 ? (d11 d21 + d12 d22 )d233 (73) +2(d11 d31 + d12 d32 )d23 d33 (74) +2(d21 d31 + d22 d32 )d13 d33 2 2       h34 = ?(d31 + d32 )(d11 d21 + d12 d22 ) +2(d11 d31 + d12 d32 )(d21 d31 + d22 d32 ) (75) h35 = (d211 + d212 + d221 + d222 )d13 d23 +(d213 + d223 )(d11 d21 + d12 d22 ) (76) 2 2 2 2         h36 = (d11 + d12 + d21 + d22 )(d11 d21 + d12 d22 ) (77) From each of equations (58), f 2 can be expressed in terms of f 0 viz: + hi3 f 02 + hi4 f 04 f 2 = fi2  ? hhii21 + hi5 f 02 + hi6 f 04

(i = 1; 2; 3) (78) In theory, we have that f12 = f22 , f22 = f32 , and f12 = f32 .

Imposing each of these constraints in turn, then, after setting q = f 02 (79) we obtain three cubic algebraic equations in q given by

si1 + si2 q + si3 q2 + si4 q3 = 0; (i = 1; 2; 3) where the sij 's are coecients directly computable hij 's, given by s11 = ?h11 h25 + h12 h23 ? h13 h22 + h15 h21 s12 = ?h11 h26 + h12 h24 ? h13 h25 ? h14 h22 +h15 h23 + h16 h21 s13 = ?h13 h26 ? h14 h25 + h15 h24 + h16 h23 s14 = ?h14 h26 + h16 h24 s21 = ?h11 h35 + h12 h33 ? h13 h32 + h15 h31 s22 = ?h11 h36 + h12 h34 ? h13 h35 ? h14 h32 +h15 h33 + h16 h31 s23 = ?h13 h36 ? h14 h35 + h15 h34 + h16 h33 s24 = ?h14 h36 + h16 h34 s31 = ?h21 h35 + h22 h33 ? h23 h32 + h25 h31 s32 = ?h21 h36 + h22 h34 ? h23 h35 ? h24 h32 +h25 h33 + h26 h31 s33 = ?h23 h36 ? h24 h35 + h25 h34 + h26 h33 s34 = ?h24 h36 + h26 h34

3.3 The Case of Two Equal Principal Distances

In practical applications, there is an usual case where two principal distances are equal, i.e. f = f0 (93) This occurs in standard aerial photogrammetry and motion vision with a 'frozen' mobile camera. In this case, the algebraic equations of (58) becomes

hi1 + (hi2 + hi3 )f 2 + (hi4 + hi5 )f 4 + hi6 f 6 = 0 (i = 1; 2; 3) (94) These are cubic equations in an unknown f 2 , so f 2 can be solved in closed-form.

4 Cases of Degeneracy

As the degree of freedom of D is 7, it appears that if one of the elements of D constantly equals zero, we can not solve for all the 7 parameters from D . This becomes clear when we consider the relations between D and A, and A and B , R. Expanding equation (29) gives

A=

=

a11 a12 a13 ! a21 a22 a23 a31 a32 a33 by r31 ? bz r21 by r32 ? bz r22 by r33 ? bz r23 bz r11 ? bx r31 bz r12 ? bx r32 bz r13 ? bx r33 bx r21 ? by r11 bx r22 ? by r12 bx r23 ? by r13

! (95)

(80)

By xing a dij to zero, for a particular index of (i; j ), from relation (57), it implies

from

(96)

(81) (82) (83) (84) (85) (86) (87) (88) (89) (90) (91) (92)

Equations (58), (78) and (80) are the central result of this paper. Each of the equations (80) is cubic in q, and can therefore be solved in closed-form. Once q is solved, f 0 and f are then solved from (79) and (78). And the special coplanarity matrix A is obtained from (57).

dij  0 =) aij  0

That means there is a constant dependency between the baseline vector (bx ; by ; bz ) and the rotation angles (; ; ). Obviously, there are 9 such cases of degeneracy. Not every case is meaningful in practical setup of the stereo geometry. However, there is at least one practical case of degeneracy

d33  0 =) a33  0 =) bx r23 ? by r13  0

(97)

This degenerate case corresponds to the coplanarity of two principal axes which is the standard case of biological and robot stereo setup. In standard aerial photogrammetry, two stereo image planes tend to be parallel, which also corresponds to the coplanarity of two principal axes.

That two principal axes are coplanar naturally implies they are coplanar with the baseline vector

[b Cc

bx by bz 0 0 ? ?r13f 0 ?r23f 0 ?r33ff0

C 0 c0 ] =

 0

(98)

This can be reduced to

rb13x rb23y = bxr23 ? by r13  0

(99)

This shows that d33  0 corresponds to a particular geometry of stereo setup, which is the standard case of robot and biological vision. Further to show the constraint on the explicit parameters, replacing r13 and r23 by their trigonometric functions of rotation angles de ned by (1), equation (99) can be expanded as

With computed D in (106), coecients hij and sij are computed using equations (59)-(77) and (81)-(92) successively and normalized - by letting h11 = 1 and s11 = 1 - as (hij ) =

bx r23 ? by r13 = bx sin  cos + by sin  0 (100) This shows that this particular degenerate case corresponds to a constant dependence between two of the baseline components and two of the rotation angles: bx ; by ; ; , not involving bz and . This degenerate case can also be con rmed by using the algebraic equations in f and f 0 of (58). When d33  0, the three equations in f and f 0 of (58) reduce to only one equation. Consequently, there is no unique solution. In the standard case of robot and biological vision, only a vergence angle is allowed to be variable, so

= =0

(101)

If the two cameras are 'frozen' to have equal principal distances f = f 0, the unique principal distance f is then solvable. This is a special case of equation (94). Numerical tests show that by setting  to a tiny nonzero angle (e.g. 1=), it is suciently robust to avoid this degenerate case, so the two dierent focal lengths can be solved.

5 Numerical Examples The correctness of this solution has been proved through symbolic manipulation programs in Maple, and also con rmed through controlled numerical tests and real image tests. Here we provide a controlled numerical example, and a real image application. The error analysis of this solution is not included in this paper because it will make another subject of its own importance and it will require more eorts to complete.

Example 1: A Controlled Numerical Test

In this example, we rst generate 32 homologous image point pairs from a set of controlled parameters: (f; f 0) = (7:439; 12:568) (102)

(bx ; by ; bz ) = (1:000; 0:727; 0:493) (103) (; ; ) = (0:457; 0:286; 0:743) (104) The generated coordinates (x; y) and (x0 ; y0 ) are bounded by jxj; jyj  2f; jx0j; jy0 j  2f 0 (105)

We then add noise to the image coordinates with the noise bound of about 2 pixels relative to an image size of 1000  1000. From the generated data set, the coplanarity matrix D is solved, using an algorithm described in [Pan et al, 1995b], as: ! ?0:00998671 0:00789889 0:08631612 0:00228314 ?0:00783312 ?0:20676858 D = ?0:12584158 0:03336187 ?0:96572173 (106)

While D itself is normalized according to (50), its determinant is suciently close to zero jD j = 1  10?13 (107)

(sij ) =





1:00000000

0:05383093

1:00000000 0:50328232 0:12071670

0:01264379 0:00199945 0:00195932

0:01542610

?0:41745278 ?0:02247187 0:00002681 ?0:08937991 ?0:00481140 0:00579978 ?0:00004993 ?0:00042206 ?0:11009606  10??55 0:00013836 ?0:00040739 0:12789937  10 ?0:12512231  10?5 ?0:00004474 ?0:10398490  10?6 ?

? ?

?0:00010572 ?0:69631602  10?5

?0:94956915  10??78 ?0:40945670  10?7 0:13725297  10



 (108)

0:00001516 (109) (110)

From equations (80), three non-negative real roots of q are solved as

(157:4204842687; 157:4204841696; 157:4204841448) (111)

we then take the median of these roots, and obtain f 0 and f from equations (79) and (78) as (f; f 0 ) = (7:410857315; 12:54673201) (112)

With D , f and f 0, the special coplanarity matrix A is then solved via equation (57) as ! ?0:00998671 0:00789889 ?0:00687957 0:00228314 ?0:00783312 0:01647988 A= 0:01698070 ?0:00450176 ?0:01038611

(113)

From here on, we apply the approach described in [Pan 1995b] for determining relative orientation parameters: baseline vector (bx ; by ; bz ) and rotation matrix R and angles , , :

(bx ; by ; bz ) = (0:01651035; 0:01199541; 0:00809724) (114)

R=

0:70687652 0:64927117 ?0:28066446 ! ?0:51491528 0:74438371 0:42515309 0:48496169 ?0:15601232 0:86050701 (115)

(; ; ) = (0:45889460; 0:28448632; 0:74294648) (116) The baseline vector is normalized to:

(bx ; by ; bz ) = (1:0; 0:72653920; 0:49043440) The relative errors of this solution are 0 (
ff ;
ff0 ) = (0:38%;

0:17%; )

(117) (118)

(
bbyx ;
bbyy ;
bbzz ) = (0:0%; 0:06%; 0:52%) (119) (
 ;
;


) = (0:41%; 0:53%; 0:01%) (120)

Example 2: Surface Reconstruction From Two Real Images

In this example, two real stereo images of size 512  512 are rst matched using a general image matching algorithm described in [Pan, 1996]. Fig.2 - 3 show the matched images on the level 64  64. Actually, matching can be done down to the pixel level. In this example, we select about 64 pairs of homologous image points from the level 128  128, and then compute the 7 parameters of general relative orientation. Through forward triangulation, we compute the 3D surface mesh, and display the image-mapped surface in Fig.4, where the surface is viewed from left to right relative to the original image. Figure 4: Image-mapped surface on the level 128  128

6 Conclusions In this paper we have presented the kernel part of a novel solution to the general relative orientation of two images, a problem of recovering two principal distances and ve relative orientation parameters, using only homologous image points. We have shown that the two principal distances are fully determined by sixth-order algebraic equations whose coecients are fourth-order polynomials of the coplanarity matrix elements; and these sixth-order algebraic equations are solvable in a direct closed form. The degeneracy of general relative orientation is also treated in depth. The approach is pure algebraic. The correctness of all the direct closed-form equations has been proved by Maple symbolic manipulation programs, and also con rmed by numerical tests with both synthetic and real stereo images. Figure 2: Left image with matched mesh on the level 64  64

Acknowledgements Prof. Mike Brooks and Dr. Garry Newsam of Adelaide, Prof. W. Forstner of Bonn, Prof. A. Grun of Zurich, Prof. Li Deren of Wuhan, Prof. C.S. Fraser of Melbourne, Ilkka Niini from Helsinki, Prof. I.A. Harley of London, and Faugeras's Group in INRIA, France, provided helpful comments, suggestions, or literature to the early stage of this work. Dr. Danny Gibbins spent much of his time, implementing the single camera calibration algorithm without which I would not have been able to test the solutions described in this paper using real image data. Dr. Du Huynh provided some relevant Maple programs. Paul Connolly and David Burnes helped me in precise measurement of our camera calibration frame by using a theodolite.

References

 Abdel-Aziz, K. (1971): Direct linear transformation

Figure 3: Right image with matched meshes on the level

64  64

from comparator coordinates into object space coordinates in close-range photogrammetry. Proc. ASP/UI Symposium on Close-Range Photogrammetry, Urbana, Illinois, 1971, pp. 1-8.  Brandstatter, G.(1991): Zur relativen Orientierung projecktiver Bundel. Zeitschrift fur Photogrammetrie und Fernerkundung, 59 Jahrgang, Heft 6, 199-212.  Brandstatter, G. (1996): On critical con gurations of projective stereo correlation. International Archives of Photogrammetry and Remote Sensing (IAPRS), Vol. XXXI, B3, pp. 77-81, 1996, Vienna.

 Bruss, A.R. and Horn, B.K.P. (1983): Passive naviga-

 Niini, I. (1993): Solution of the interior orientation





     

         

tion. Computer Vision, Graphics, and Image Processing 21(1), 1983, 3-20. Deriche, R., Zhang, Z., Luong, Q.-T., and Faugeras, O.D. (1994): Robust recovery of the epipolar geometry for an uncalibrated stereo rig. Proc. ECCV'94, 1994, pp. 567-576. Faugeras, O.D. (1993): Three-Dimensional Computer Vision: a geometric viewpoint. MIT Press, 1993. Faugeras, O.D., Lustman, F. and Toscani, G. (1987): Motion and structure from motion from point and line matches. Proc. 1st ICCV, 1987. Faugeras, O.D., Luong, Q.-T. and Maybank, S.J. (1992): Camera self-calibration: Theory and experiments. Proc. ECCV'92, 1992, pp. 321-334. Finsterwalder, S. (1899): Die Geometrischen Grundlagen der Photogrammetrie. Jahresbericht der Deutschen Mathematiker-Vereinigung, Band VI, Teubner Press, Leipzig, 1899. Finsterwalder, S. (1932): Die Hauptaufgabe der Photogrammetrie. Bayer. Akad. der Wissenschaften Munchen, Sit.-ber. der math.-phys. Kl. 1932, pp. 115-131. Fourcade, H.G. (1926): A new method of aerial surveying. Transactions of the Royal Society of South Africa, XIV(1):93-112, 1926. On page 49 of this issue, reproduced is a part of a letter of 5th June, 1903, written by Fourcade to A. Laussadat (a founder of photogrammetry), mentioning the theory of relative orientation. The theory was presented on 16th August, 1905 at a meeting of the British Association for the Advancement of Science. Hartley, R.I. (1992): Estimation of relative camera positions for uncalibrated cameras. Proc. ECCV'92, 1992, pp. 579-587. Hartley, R.I. (1995): On the defence of the eight-point algorithm. Proc. ICCV'95, 1995, pp. 1064-1070. Heisse, O. (1863): Die cubische Gleichung, von welcher die Lolung des Problems der Homographies von M. Chasles abhangt. Journal f. Reine und Angew. Math. 62, 1863, 188-192. Kruppa, E. (1913): Zur Ermittlung eines Objektes aus zwei Perspektiven mit inner Orientierung. Sitz.-Ber. Akad. Wiss., Wien, Math. Naturw. Kl (Abt. IIa. 122), 1913, 1939-1948. Horn, B.K.P. (1990): Relative orientation. IJCV, 4, 1990, 59-78 Huang, T.S. and Faugeras, O.D. (1989): Some properties of the E matrix in two-view motion estimation. IEEE T-PAMI 11(12), 1989, 1310-1312. Khlebnikova, T.A. (1983): Determining relative orientation angles of oblique aerial photographs. Mapping Science and Remote Sensing, 21(1), 1983, 95-100. Longuet-Higgins, H.C. (1981): A computer algorithm for reconstructing a scene from two projections. Nature, 293(10), 1981, 133-135. Longuet-Higgins, H.C. and Prazdney, K. (1980): The interpretation of a moving retinal image, Proc. Roy. Soc. London B 208, 1980, pp. 385-397. Maybank, S.J. and Faugeras, O.D. (1992): A theory of self-calibration of a moving camera. IJCV, 8(2), 1992, 123-151.

            

without control points. The Photogrammetric Journal of Finland, 13(2):78-84, 1993. Niini, I. (1994): Relative orientation of multiple images using projective singular correlation. International Archives of Photogrammetry and Remote Sensing, vol. 30, part 3/2, 1994, pp. 615-621, Munich. Pan, H.-P. (1996): General stereo image matching using symmetric complex wavelets. SPIE Proceedings Vol 2825, Wavelet Applications in Signal and Image Processing. Denver, August, 1996, pp. 697-721. Pan, H.-P., Brooks, M.J. and Newsam, G.N. (1995a): Image resituation: initial theory. Proc. Videometrics IV, Philadelphia, USA, 1995, SPIE Proceedings 2598, pp. 162-173. Pan, H.-P., Huynh, D.Q. and Hamlyn, G.K. (1995b): Two-image resituation: practical algorithm. ibid, 1995, pp. 174-190. Schmid, H. (1954): An analytical treatment of the orientation of a photogrammetric camera. Photogrammetric Engineering, XX, 1954, 765-781. Schut, G. (1955/56): Analytical aerial triangulation and comparison between it and instrumental aerial triangulation. Photogrammetria, XII, 1955/56, 311-318. Stefanovic, P. (1973): Relative orientation - a new approach. ITC Journal, (ITC, Enschede, Netherlands), 1973, 417-448. Sturm, R. (1969): Das Problem der Projektivitat und seine Ansendung auf die Flachen zweiten Grades. Math. Ann., 1969, 1:533-574. Thompson, E.H. (1959): A rational algebraic formulation of the problem of relative orientation. Photogrammetric Record, 3(14), 1959, 152-159. Torr, P. (1995): Motion Segmentation and Outlier Detection. PhD thesis, Department of Engineering Science, University of Oxford, 1995. Tsai, R.Y. and Huang, T.S. (1984): Uniqueness and estimation of 3-D motion parameters of rigid bodies with curved surfaces. IEEE T-PAMI, 6, 1984, 13-27. Wang, Z.Z. (1979): Principles of Photogrammetry, Surveying and Mapping Press, Beijing, China, 1979, (English version is available). Weng, J., Huang, T.S. and Ahuja, N. (1993): Motion and Structure From Image Sequences, Springer-Verlag, 1993. Zhang, Z., Deriche, R., Faugeras, O.D. and Luong, Q.T. (1995): A robust technique for matching two uncalibrated images through the recovery of the unknown epipolar geometry. Arti cial Intelligence, 78:87-119, 1995.