A New Approach to Solving Kruppa Equations for Camera Self-Calibration Cheng Lei1, Fuchao Wu1, Zhanyi Hu1, H. T. Tsui2 1
National Laboratory of Pattern Recognition, Chinese Academy of Sciences, Beijing 100080, China {leic, fcwu, huzy}@nlpr.ia.ac.cn
Abstract In this paper, we propose a new approach to solving the Kruppa equations for camera self-calibration. Traditionally, the unknown scale factors in the Kruppa equations are eliminated first, leading to a set of nonlinear constraints. Instead, we determine the scale factors by a Levenberg-Marquardt (LM) optimization or Genetic optimization technique first. Then, the camera’s intrinsic parameters are derived from the resulting linear constraints. Extensive simulations as well as experiments with real images verify that the above technique is both accurate and robust. Keywords: Kruppa Equations, Camera Self-Calibration
1. Introduction Camera calibration is an essential step to retrieve the Euclidean 3D information from 2D images. Since its first introduction by Maybank and Faugeras [1] in 1992, camera self-calibration has become an important research direction in Computer Vision due to its flexibility. The Absolute Conic (AC) (or its dual quadric), whose image (IAC) (or the dual of IAC) is invariant with respect to the camera’s rigid motions, can be regarded as a virtual calibration object and used for self-calibration. As the first reported approach for camera self-calibration in [2], the Kruppa equations, which are derived from the epipolar geometry of AC, are solved for identifying the dual of IAC, from which the camera intrinsic parameters are retrieved. Recently, the Kruppa equation based approaches for camera self-calibration have been losing favor due to the fact that the solutions are difficult and also not robust. The main problem in solving the Kruppa equations is due to the unknown scale factors existing in them. In the literature, most approaches to solving the Kruppa equations [3, 4, 5], all try to eliminate the unknown scale factors first. This process leads to a set of non-linear constraint equations. Then the camera’s intrinsic parameters are determined by numerically solving a system of quadric equations or by non-linearly minimizing of a cost function deduced from those non-linear equations. While in this paper, we propose a novel
2
Department of Electronic Engineering, Chinese University of Hong Kong, Shatin, Hong Kong
[email protected] approach to solving the Kruppa equations. Specifically, the usually eliminated unknown scale factors in the Kruppa equations are first determined via the traditional LM method or a Genetic Algorithm. Then, with the estimated scale factors, the camera’s intrinsic parameters are retrieved linearly.
2. A New Formulation of the Kruppa Equations Based Camera Self-calibration 2.1 A Brief Review of Kruppa Equations In this paper, the camera is assumed to be of the pinhole model. So the image formation process from a 3D point X = ( x, y, z ,1)T to a 2D image point m = (u , v,1)T in homogeneous coordinates can be expressed as: (1) m ≅ K [R | t ]X where
γ u0 f v v0 0 1
fu K =0 0
is the camera calibration matrix, R
and t are the rotation matrix and translation vector from the world frame to the camera frame respectively, and “ ≅ ” means equality up to a non-zero scale factor. Given two images taken at two different positions with the intrinsic camera parameters fixed, from the epipolar geometry of AC, the following relation can be obtained: (2) FCFT = s[e′]× C[e′]T× Equation (2) is the classical Kruppa equation in matrix form, where C = KK T (the dual of IAC), s is an unknown positive scale factor, F and e ′ are the fundamental matrix between the two images and the epipole in the second image respectively. [e′]× is an antisymmetric matrix 0 − e3′
e2′
− e 2′ e1′
0
the vector e′ as: [e ′]× = e3′ 0 − e1′ .
derived from
Traditionally, by eliminating the scale factor s , one can obtain the following equations: ( FCF T )11 ([e′]× C[e′]T× )11
=
( FCF T )12 ([e′]× C[e′]T× )12
=
( FCF T ) 22 ([e ′]× C [e ′]T× ) 22
(FCF T )13 ([e′]× C[e′]T× )13 =
( FCF T ) 23 ([e ′]× C [e ′]×T ) 23
= =
( FCF T )33 ([e ′]× C [e ′]×T ) 33
(3)
Equations (3) provide the non-linear constraints used for self-calibration in traditional approaches.
1051-4651/02 $17.00 (c) 2002 IEEE
2.2 Problem Reformulation Given n images (assuming the degenerate cases have been excluded, and the intrinsic camera parameters are kept fixed), we can obtain at least n Kruppa equations: F CF T = s [e′ ] C [e′ ]T 1 1× 1 × 1 1 F2CF2T = s2 [e2′ ]× C [e′2 ]T× l T T FnCFn = sn [en′ ]× C [en′ ]×
6 there are totally C3n 6 × 6 sub-matrix Aisub ( sV ), i = 1,m, C36n ,
(4)
vector c = (c1 , c 2 , c 3 , c 4 , c5 , c 6 ) T , the constraints on C from (4) can be reformed as a homogenous equation system: A3n×6 ( sV )c = 0 (5) with sV = [ s1 , s 2 , m , s n ] denoting a vector of the n scale factors. Although one Kruppa equation can only provide two independent constraints on c, for considerations of robustness, three out 6 constraints are used as in [6]. Therefore n Kruppa equations give rise to a 3n × 6 coefficient matrix A3n×6 ( sV ) . (For the specific form of the matrix A, please refer to the Appendix.)
Under the ideal condition, the rank of the coefficient matrix A should be 5. Since the entries of the matrix A are functions of the unknown scale factors s1, s2 ,..., sn , the rank-5-ness property of matrix A provides constraints over s1, s2 ,..., sn . This allows us to determine the scale factors and also the matrix A by a non-linear optimization method first. Then with the matrix A known, the intrinsic camera parameters can be retrieved linearly. The proposed steps for solving the Kruppa equations are summarized as: (1) Determine the scale factors by an appropriate nonlinear optimization technique; (2) Linearly retrieve the camera intrinsic parameters using the determined scale factors. 2.3 Implementations of Non-linear Optimization for Searching the Optimal Scale Factors
In this subsection, we shall investigate two kinds of implementations of our new approach. In each implementation, it needs to set the initial values of the tobe-optimized scale factors. For this, we have a practiceverified approximation equation as follows:
where
σ 1Fk σ 2Fk ′
(σ [e k ]× ) 2
whose
determinants
are
indicated
as
det(Aisub ( sV )),
i = 1, , C 36n . Then by minimizing the norm of a C36n × 1
c1 c2 c3 By denoting the symmetric matrix C = c2 c4 c5 as a c3 c5 c6
k sinit ≈
regarding to our specific problem, we try to minimize the norm of a vector whose coordinates are the determinants of all the 6× 6 sub-matrix of A3n×6 ( sV ) with respect to n scale factors s1, s2 ,..., sn . Specifically for matrix A3n×6 ( sV ) ,
(6)
[e ′ ] σ iFk ( σ k × ) is the i-th singular value of
fundamental matrix Fk ( [ e′k ]× ) . 2.3.1 Implementation based on LM A matrix M ranks r means that the determinants of all M’s (r + 1) × (r + 1) sub-matrix will equal to zero. Therefore
vector [ det( A1sub ( sV )), , det( A sub ] with respect to sV , 6 ( sV ) C3 n
that is,
C36n sub Min ∑ (det( Ai ( sV sV i =1
)) 2
, we can get the optimal scale
factors sV = [ s1 , h , s n ] . For this minimization task, the LM method is used, in which the necessary partial derivatives of det( Aisub ( sV )) with respect to s j are computed as: ∂ det( Aisub ( sV )) ∂A sub ( sV ) ) = det( Aisub ( sV )) ⋅ trace (( Aisub ( sV )) −1 ⋅ i ∂s j ∂s j
(7)
2.3.2 Implementation Based on GA Beside the former implementation, we also implemented our new technique based on a Genetic Algorithm (GA). In this implementation, GA is used to find the optimal scale factors such that the sixth singular value of the coefficient matrix A is minimized, which is another criterion to enforce the rank-5-ness of the coefficient matrix A. The procedure undertaken is outlined as follows: First, we initialize a size-fixed population using the k estimated initial scale factors s init ( k = 1, n ) according to equation (6) as the initialization seeds. Each individual (chromosome) in the population is encoded as an n-tuple (n: the number of the Kruppa equations or the scale factors) of scale factors. When initializing a chromosome, k each gene s gen of it is set as a random value within the k interval of [ 1 − ∆ init , 1 + ∆ init ] sinit , where ∆init is called the initialization radius.
In GA, it also needs a fitness function used to evaluate whether an individual is fitting or not. Since each chromosome corresponds to a specific group of scale factors ( s1 , s2 ,, sn ) , which in turn can determine a coefficient matrix A3n×6 ( sV ) , in order to minimize A’s sixth singular value σ 6A ( sV ) , we choose the fitness function as to maximize ln(1/ σ 6A (s1, s2 ,, sn )) . Furthermore, given a coefficient matrix A3n×6 ( sV ) , in order to obtain a reasonable camera calibration matrix K by Choleskey factorization, the matrix C corresponding to the solution vector c in (5) should at least satisfy the requirement of positive-definite-ness. So in our GA optimization process, we deliberately punish the individual whose corresponding solution is not qualified by dividing its original fitness value by a factor of 10. Here by “qualified” we mean that corresponding matrix C is
1051-4651/02 $17.00 (c) 2002 IEEE
positive-definite and the intrinsic parameters obtained by Cholesky factorization are reasonable, such as the skew should not be too large, the focal length should be positive and so on. We call this process as qualification. ln(1 / σ 6A (s1, s2 ,l, sn )) Qualification passed 1 A ln(1 / σ 6 (s1, s2 ,l, sn )) Qualification not passed 10
Hereafter, we name our two implementations based on LM and GA as LM-S and GA-S respectively. 3.1 Experiments with Synthetic Images
First, the synthetic images are generated by projecting the 3D object points randomly distributed within a sphere centered at (0,0,600) with a radius of 400. The image size is 960 × 640. The camera intrinsic parameters are set as: f u = 800, f v = 720, u0 = 510, v0 = 300, γ = 0.25. The number of the matches is 100. For GA implementation, population size is 100, the crossover probability is 85%; the mutation probability is 10%. The initialization radius ∆init equals to 0.5. The coarse mutation radius ∆cmut is 0.1
(8)
After the whole population is initialized, the population will evolve from generation to generation by performing genetic operations such as reproduction, crossover, and mutation with some preset probabilities to search for the optimal group of the scale factors. More specifically, the selection scheme we use is the classical "roulette wheel" selection scheme. As for the crossover operation, the traditional "single-point" crossover scheme is used. To perform the mutation operation, the scale factor corresponding to one of the gene of the selected chromosome will be multiplied by a random factor ranging from 1 − ∆ mut to 1 + ∆ mut , where ∆ mut is called mutation radius. In particular, we further adopt a coarseto-fine mutation strategy, in which the mutation radius is set as a larger value for exploring at first and will be gradually reduced to a smaller value for exploiting.
f and the fine mutation radius ∆mut is 0.01.
To assess the robustness of our algorithm to random noise, the projected image points are perturbed by the zero-mean Gaussian noise with the noise level increasing from 0.0 to 2.0 pixels in a step of 0.2 pixels. At each noise level, 100 experiments are carried out. From the simulation results shown in Fig.1 and 2, we can see that, GA-S performs better due to its better ability to search a wider parameter space and better guidance mechanism in the course of search. As for LM-S, at low noise level, it can converge to the ground truth of the scale factors accurately and quickly, with the calibration result accordingly very accurate. This proves our technique is feasible. But its drawback is that it does not guarantee to converge to a reasonable calibration result every time at the higher noise levels. By adopting some active constraints in the LM optimization to guide the negative gradient direction to point to the feasible region and multistart strategy, its performance is improved significantly.
When the stopping criterion is satisfied, the evolution will be terminated and the individual having the highest fitness value in the last generation will be extracted. That is, an optimal group of scale factors are found and will be used to linearly calibrate the camera’s intrinsic parameters. 2.4 Linear Calibration With the optimal scale factors fed from the nonlinear search engine implemented by LM or GA, matrix C in (2) can be determined by solving equations (5) linearly. Then the camera calibration matrix K can be extracted by Cholesky factorization [10].
3.2 Real Image Experiments We have developed an automatic tri-images based 3D reconstruction system that integrates our new calibration technique (downloadable via www.ee.cuhk.edu.hk/~clei/) and conducted extensive experiments with real images for calibrations and reconstructions.
3. Experiments The LM and GA implementations of our proposed technique are both tested in the following experiments. 20
700
Mena of skew factor γ
800
800
700
600
600
500 0.0
500 0.0
500
700
15
LM-S GA-S
900
LM-S GA-S
10
600
LM-S GA-S
0 -5 -10
LM-S GA-S
400
5
Mean of u 0
LM-S GA-S
Mean of fv
Mean of fu
900
Mean of v0
1000
1000
500
300
200
400
-15
0.4
0.8
1.2
1.6
2.0
0.4
0.8
1.2
1.6
2.0
-20 0.0
0.4
0.8
Noise Level (Pixel)
Noise Level (Pixel)
(a) Mean of f u
1.2
1.6
2.0
300 0.0
0.4
0.8
Noise Level
(c) Mean of γ
(b) Mean of f v
1.2
1.6
100 0.0
2.0
0.4
0.8
1.2
1.6
2.0
Noise Level
Noise Level (Pixel)
(d) Mean of u0
(e) Mean of v0
Fig. 1 Mean of the intrinsic parameters calibrated by two implementations
40
20
60 40 20
0.4
0.8
1.2
1.6
Noise Level (Pixel)
(a) Variance of f u
2.0
0 0.0
0.4
0.8
1.2
1.6
2.0
80
LM-S GA-S
80
60
40
40
0.4
0.8
1.2
1.6
2.0
0 0.0
(b) Variance of f v
(c) Variance of γ
LM-S GA-S
60
40
20
0.4
Noise Level (Pixel)
Noise Level (Pixel)
80
60
20
20
0 0.0
100
LM-S GA-S Variance of u0
80
60
0 0.0
LM-S GA-S
Variance of Skew factor γ
100
100
100
120
LM-S GA-S Variance of fv
Variance of fu
80
Variance of v 0
100
0.8
1.2
1.6
2.0
0 0.0
Noise Level (Pixel)
(d) Variance of u0
Fig. 2 Variance of the intrinsic parameters calibrated by two implementations
1051-4651/02 $17.00 (c) 2002 IEEE
0.4
0.8
1.2
1.6
2.0
Noise Level (Pixel)
(e) Variance of v0
In Fig.3 and 4, some calibration and reconstruction results of the conducted experiments are shown, from which we can see that with the calibrated camera intrinsic parameters, the scene is reconstructed successfully and accurately. Due to the limited paper space, many other experiments and more detailed comparison to the approach of [6] is omitted.
system, whose promising performance verifies the robustness and accuracy of our new technique. Acknowledgements The work is supported by the National Grand Fundamental Research 973 Program of China under grant No. G1998030502 and the Hong Kong RGC Grant under grant No. CUHK 4310/98E.
References [1] S.J. Maybank and O.D. Faugeras, A theory of selfcalibration of a moving camera, International Journal
of Computer Vision, 8(2): 123--151, 1992 [2] O. D. Faugeras, Q. Luong, and Maybank, Camera selfcalibration: Theory and experiments, In proceedings of European Conferencce on Computer Vision, LMCS 588, pp. 321-334, Springer-Verlag, 1992. [3] Q.-T. Luong and O. Faugeras, Self-calibration of a moving camera from point correspondences and fundamental matrices, International Journal of Computer Vision, 22(3), pp.261-289, 1997. [4] C. Zeller and O. Faugeras, Camera self-calibration from video sequences: Kruppa equations revisited, Research Report 2793, INRIA, February 1996. [5] G. Xu and Noriko Sugimoto, Algebraic derivation of the Kruppa equations and a new algorithm for self-calibration of cameras, Journal of Optical Society of America, 16(10):2419-2424, October, 1999. [6] M. I. Lourakis and R. Deriche, Camera self-calibration using the singular value decomposition of the fundamental matrix: From point correspondences to 3D measurements. Research Report 3748, INRIA Sophia-Antipolis, Aug. 1999. [7] Y. Ma et al., Kruppa equation revisited: its renormailization and degeneracy, In Proceedings of ECCV’2000, June, Dublin. [8] Z. Zhang, R. Deriche, O. Faugeras, and Q.-T. Luong, A robust technique for matching two uncalibrated images through the recovery of the unknown epipolar geometry, Artificial Intelligence, 75(1-2):87--120, 1995. [9] L. Quan, Invariants of six points and projective reconstruction from three uncalibrated images, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 17, pp. 34-46, 1995 [10] Press: R. Hartley and A. Zisserman, Multiple view geometry in computer vision, Cambridge Univ. Press, 2000. [11] D. E. Goldberg, Genetic algorithms in search, optimization and machine Learning, Addison Wesly, 1989.
Fig.3 For Valbonne Church image sequence ( 512× 768)
LM-S GA-S
fu
675.26 664.75
fv
643.47 666.68
u0
v0
γ
232.62 255.73
307.21 346.99
-12.4 -0.85
Fig.4 For an indoor image sequence ( 1024 × 768)
LM-S GA-S
fu
1267.28 1223.59
fv
1218.62 1222.37
u0
v0
γ
653.24 607.16
457.09 425.63
-11.8 0.21
4. Conclusions A new approach to solving the Kruppa equations for camera self-calibration is proposed. Different from the previous approaches, the unknown scale factors in the Kruppa equations are first determined using a LM or a GA technique. And then the camera intrinsic parameters can be linearly extracted. Based on this new technique, we have developed an automatic 3D reconstruction
Appendix: For n fundamental matrices, the coefficient matrix A3n×6 ( sV ) can be written as: L1(3×6) − s1R(13×6) , k A3n×6 ( sV ) = L(3×6) − sk R(k3×6) n − s Rn L (3×6) n (3×6)
with
Lk(3×6)
(Fk11)2 2Fk11Fk12 2Fk11Fk13 (Fk12 )2 = Fk11Fk31 (Fk12Fk31 + Fk11Fk32) (Fk13Fk31 + Fk11Fk33) Fk12Fk32 (Fk21)2 2Fk21Fk22 2Fk21Fk23 (Fk22)2
0
R(k3×6) = 0
(ek′3)2
0
0
(ek′3)2
− 2e′k2e′k3
ek′2ek′3
− (ek′2 )2 − 2ek′1ek′3
− ek′1ek′3
ek′1ek′2
0
0
0
(Fk13)2 (Fk13Fk32 + Fk12Fk33) Fk13Fk33 2Fk22Fk23 (Fk23)2
(e′k2 )2 . 0 (e′k1)2
1051-4651/02 $17.00 (c) 2002 IEEE
2Fk12Fk13
