FULL PAPER International Journal of Recent Trends in Engineering, Vol 2, No. 1, November 2009
Reconstruction of 3D Plane using Min-Max Approach Gaurav Gupta, Balasubramanian Raman, and Rama Bhargava Department of Mathematics Indian Institute of Technology Roorkee, Roorkee-247667, INDIA
[email protected] {balafma,rbharfma}@iitr.ernet.in Abstract- In this paper, a new approach for the reconstruction of a 3D plane by solving the geometry of two perspective projections of the plane is presented. It is assumed that the position and the orientation of one camera are known with respect to the other one. The correspondence between the object space and image planes is established using the collinearity equation. The projection of the original plane into two image planes results in two set of discrete points from which the min-max points of horizontal and vertical axes are obtained. A set of four 2D points for each plane, is obtained from these corresponding points. Then four points in 3D space are obtain from the corresponding sets of 2D points using the collinearity equations and the correspondence (extrinsic and intrinsic parameter) between the respective pair of projections. The equation of the plane obtained from these four points is the equation of reconstructed plane in 3D space. This work, deals with the digitized and normalized image planes for both general and perfect stereo cases and considering various noisy cases in the reconstruction process. The angle and distance between the reconstructed and the original plane in 3D space have been used as the criterion for the measurement of the error. The results of this study are useful for the design of a stereo-based imaging system and for the best reconstruction with minimum error. Results for planes are presented using simulation studies. Simulation studies have been conducted to observe the effect of noise on errors in the process of reconstruction. Keywords: Plane Reconstruction, Collinearity Equations
detection technique and then Hough Transform (HT) applied to extract the straight-line boundary edges. The planar region determined by the intersections of the edges and the missing pixels depth information of a region is determined by the 3D plane equation in the world coordinate system. In this case, the range data in the world coordinate system is projected back into the image coordinate system using a pixel-to-pixel projection algorithm. Rother [7] presented a new linear method for the linear relationship between cameras and 3D features (points, lines and planes) simultaneously from multiple perspective views by solving a single linear system and called it Direct Reference Plane (DRP) method. The author has assumed that a real or virtual reference plane is visible in all views. The main contribution of his paper is that lines and cameras, as well as, planes and cameras also have a linear relationship whereas it is well known that the projection relationship between uncalibrated cameras and 3D features is non-linear in the absence of a reference plane. Consequently, all 3D features and all cameras can be reconstructed simultaneously from a single linear system, which handles missing image measurements naturally. Author has also done an experimental comparison, using real data, of different reference plane and non-reference plane reconstruction methods. For difficult reference plane scenarios, with point or line features, the DRP method is superior to all compared methods. Finally, author has presented an extensive list of reference plane scenarios, which shows the wide applicability of the DRP method.
I. INTRODUCTION Stereo vision is one of the effective methods to estimate depth and structural parameters of 3D objects from a pair of 2D images [1, 2]. Plane reconstruction continues to be a part of active research area as proven by many authors. Point and line reconstruction are sub-problems of plane reconstruction. Balasubramanian et al. [3, 4, 5] gave a methodology for the reconstruction of a line in 3D space as well as 3D quadratic curves, and also their error analysis using various techniques. Kumar et al. [4] proposed a reconstruction methodology for quadratic curves using non-digitized image planes then Sukavanam et al. extended problem with digitized and normalized image planes by considering various noisy cases and the analysis of the error in the reconstruction process. They used the angle between the reconstructed and the original quadratic curves in 3D space as the criterion for the measurement of the error. Zhang et al. [6] proposed a model based algorithm to fill the missing range information of a planar region in the depth map of an image obtained from a commercial stereo vision system, Digiclops, a fast three-camera module range measurement device. Initially, a binary edge image of the reference image extracted by the embedded confidence edge
Kanazawa and Kanatani [8] discussed the problem of reconstructing a planar surface by observing known multiple feature points are coplanar in the scene. The authors presented a direct method for reconstructing a planar surface by applying the principle of maximum likelihood estimation based on geometric constraints and a statistical model of image noise. They have reconstructed the 3D position of the surface accurately and its reliability is also computed quantitatively by doing numerical simulation. Chumerin and Hulle [9] proposed an approach for disparity plane estimation and its conversion into the ground plane. In this approach, they have estimated the ground plane parameters estimation and tracking by the moving observer. The method was based entirely on stereo disparity information. During estimation, they used an iterative fitting procedure. Hence the method remains relatively fast and can be easily optimized for realtime purposes. They assumed only the presence of textured dominant planar surface in the lower part of the screen, which is not severe restriction in real-world scenarios. A drawback of the method in which they mentioned that the approach rely heavily on the density and the accuracy of the disparity map estimation.
174 © 2009 ACADEMY PUBLISHER
FULL PAPER International Journal of Recent Trends in Engineering, Vol 2, No. 1, November 2009 Simond [10] presented a feature-based method for the reconstruction of the structured road plane in urban traffic conditions. The method consists of extracting then tracking features (points, lines) from the road and estimate the homography induced by the plane between two poses. They segmented the road plane and also extract the coplanar features in stereo images. As the proposed method copes with the dense traffic conditions: the free space required (first ten meters in front of the vehicle) is slightly equivalent to the security distance between two vehicles. They have also given an example of reconstruction of the road plane. Fraundorfer et al. [11] presented a novel method which is able to recover scene planes of arbitrary position and orientation from oriented images using homographies. The method can even be seen as a segmentation method, which segments the images into planar regions. It would therefore be desirable to incorporate the fusion of several models from different viewpoints into a true 3D model of the scene. They reconstructed the planar regions using only sparse, affineinvariant sets of corresponding seed regions. These regions are iteratively expanded and refined using plane-induced homographies. They presented the experimental results on synthetic data which show the high accuracy of the reconstruction and demonstrated that the reconstruction method can cope with large baseline changes. Infantino et al. [12] discussed the problem of reconstructing architectural scenes from multiple photographs taken from arbitrary viewpoints. To obtain a detailed model of a scene, they have used a map as a source of geometric constraints. Their modeling system uses one or more pairs of uncalibrated images of a façade and for each pair the user indicates line, point or plane correspondences between images and the map. They supposed that images have at least one planar structure as a facade for exploiting the planar homography induced between world plane and image to calculate a first estimation of the projection matrix. They have improved the estimations by using three constraints; point to point correspondences, point to line correspondences and line to line correspondences between geometric entities of image and map. They have used these constraints to calibrate the cameras and recover the projection matrices for each viewpoint. Triangulation is used to recover 3D models of the scene and to visualize new viewpoints. Their approach needs minimal a priori information about the camera being used. They have designed a working system and implemented to allow the user to interactively build a model from uncalibrated images from arbitrary viewpoints and a simple map. Kaucic et al. [13] presented a linear method for computing a projective reconstruction from a large number of images. The planar homographies are used between views to linearize the computation of the camera matrices. To develop relationships between the position vectors of all the cameras at once, they used constraints (based on the fundamental matrix) and trifocal tensors. The resulting sets of equations are solved using a SVD. They have processed all of the images simultaneously, as in the Sturm-Triggs factorization method. However there is not necessary that all points be visible in all views. In this paper, a plane reconstruction approach in 3D space from two perspective projections is presented. Unlike most of the work that has been done to reconstruct a plane
from two images, we explicitly employ the minimum and maximum values and propose a new approach, called the min-max approach for a robust and accurate reconstruction of a plane. The advantage of this method is that it overcomes the correspondence problem that occurs in pairs of projections of the plane. To obtain the information of three dimensional positions in the presence of noise is a crucial task in the field of Computer Vision. Simulation studies have also been conducted to observe the effect of noise on errors. We deal with the methodology of reconstruction of a plane in 3D space from two arbitrary perspective projections. The task of reconstructing plane in 3D space from two perspective images is very difficult when we do not know the correspondence between points in the given image planes. Hence in order to reconstruct the plane, we assume that the position and the orientation of one camera is known with respect to the other camera. The correspondence between the object space and image planes is established using the collinearity equation. Two sets of discerete points are obtained by projecting the original plane into two image planes. From these two sets of discerete points the minimum and maximum (min-max) points of horizontal and vertical axes are obtained for each plane. Using these min-max points and the collinearity equations, points in 3D space are obtained. The equation of reconstructed plane in 3D space is obtained from these 3D points. Relevant mathematical formulations and analytical solutions for obtaining the equation of the reconstructed plane are given. Results are useful as estimation of 3D object structure, reconstruction and making of polyhedral models of architectural scenes i.e. objects with planar shapes, straight lines and points are used as features for reconstruction. Performance analysis of the method of reconstruction described in this section is based on simulation studies. II. COLLINEARITY EQUATION The imaging set-up using two cameras is shown in figure 1. Let P1 and P2 be the first and second image planes of the pair of cameras C1 and C2 respectively. Let the position and the orientation of one camera be known with respect to another and both have a common field of view. Let O1 xyz be the rectangular cartesian frame of reference with its origin O1 at the center of projection of one of the cameras C1 . A point W in 3D space, with co-ordinates ( X w , Yw , Z w ) with respect to the frame of reference at C1 , is viewed by both the cameras C1 and C2 . Let O2 x ′y ′z ′ be the second rectangular cartesian co-ordinate system, not necessarily parallel to OXYZ system, with its origin O2 at the center of projection of the second camera C2 . Let the co-ordinates of the second camera C2 with respect to O1 be ( xd , y d , z d ). Let
P1 ( X 1 , Y1 , f1 ) and P2 ( X 2 , Y2 , f 2 ) be the corresponding pair of projections of point W on the pair of image planes I1 and I 2 respectively. Let f1 and f 2 be the focal lengths of the first and the second cameras respectively. Let the perspective projections of a plane π in 3D space be a pair of projeted planes 175
© 2009 ACADEMY PUBLISHER
π1
and
π2
, on the first and second
FULL PAPER International Journal of Recent Trends in Engineering, Vol 2, No. 1, November 2009 image planes respectively as shown in figure 1. The problem is to reconstruct the 3D plane π from the pair of its images
π1
and π 2 . The relation between the coordinates of the point W ( x w , y w , z w ) and that of the image point
P ( X 1 , Y1 , f 1 ) is given by the perspective equation:
X 1 = f1
xw , zw
Y1 = f1
yw zw
III. RECONSTRUCTION TECHNIQUE In order to reconstruct a plane in 3D from two perspective projections, knowledge of the following input parameters is necessary. 1. The set of pixel coordinates for the first (left) plane π 1 on the first (left) image plane.
Pl : ( x li , y li ); i = 1,2.....m .
(1)
2. The set of pixel coordinates for the corresponding plane π 2 on the second (right) image plane.
Pr : ( x ri , y ri ); i = 1,2.....n . 3. The parameters of the imaging setup and perspective geometry f1 , f 2 , x d , y d , z d and (α i , β i , γ i ), i = 1,2,3 . The algorithm to find the correspondence between pixels (points) and then reconstruct a plane is given below. 1. Find ( x min , y min ) and ( x max , y max ) in both the image planes. Let Pl1 → ( x min , y min ) , Pl 2 → ( x min , y max ) ,
Pl 3 → ( x max , y max ) , Pl 4 → ( x max , y max ) , Pr1 → ( x min , y min ) , Pr 2 → ( x min , y max ) , Pr 3 → ( x max , y max ) and Pr 4 → ( x max , y max )
Figure 1: Imaging Setup
The 3D co-ordinates of point W ( x w , y w , z w ) with respect to second camera C 2 , is given by ⎛ x′ ⎞ ⎛ cos α 1 ⎜ ⎟ ⎜ ⎜ y ′ ⎟ = λ ⎜ cos α 2 ⎜ z′⎟ ⎜ cos α 3 ⎝ ⎠ ⎝
cos β 1 cos β 2 cos β 3
cos γ 1 ⎞⎛ ( x w − x d ) ⎞ ⎟⎜ ⎟ cos γ 2 ⎟⎜ ( y w − y d ) ⎟ (2) cos γ 3 ⎟⎠⎜⎝ ( z w − z d ) ⎟⎠
2. ( Pli , Pri ) for i = 1 to 4 will be the corresponding pairs. 3. Using triangulation, reconstruct four points in 3D space with the help of four pairs of corresponding points. 4. Reconstruct the plane using at least three reconstructed 3D points.
In the above equations α i , β i , γ i (i = 1,2,3) are the respective direction cosines of the axes of O 2 x ′y ′z ′ with
IV. ERROR ANALYSIS IN THE PROCESS OF RECONSTRUCTION
respect to O1 xyz system and can be expressed in α , β and
γ (Eulerian angles). λ is a scale factor between the two
reference frames and without loss of generality this is considered to be 1, in the present work. Using Equation (2), the relation between the object space point W ( x w , y w , z w ) and the image point P2 ( X 2 , Y2 , f 2 ) is given by the perspective equations [1, 2]: X 2 = f2
( x w − x d )cosα 1 + ( y w − y d )cosβ1 + ( z w − z d )cosγ 1 ( x w − x d )cosα 3 + ( y w − y d )cosβ 3 + ( z w − z d )cosγ 3
The criterion used for computing the reconstruction error is the angle and distance between the original and the reconstructed plane. Noise with Gaussian distribution is added to the pixel coordinate values of the projection of the planes on both the image planes which disturbs the location of the pixels forming a projection of the plane on the image plane, simulating the effect of noise. The noise level is characterized by the variance, σ, of the Gaussian distribution. Let the equation of the original and reconstructed planes be
Ai x + Bi y + Ci z + Di = 0, for i = 1,2
(3) ( x − x d )cosα 2 + ( y w − y d )cosβ 2 + ( z w − z d )cosγ 2 Y2 = f 2 w ( x w − x d )cosα 3 + ( y w − y d )cosβ 3 + ( z w − z d )cosγ 3
Therefore, the cosine of the angle between the two planes is given by cos(δ ) =
(4) Equations (1),(3) and (4) are the collinearity equations for a pair of arbitrary perspective views.
A1 + B12 + C12
A2 2 + B2 2 + C 2 2
,
where δ is the reconstruction error. Using simulation studies, the error is estimated for different combinations of 176
© 2009 ACADEMY PUBLISHER
A1 A2 + B1 B2 + C1C 2 2
FULL PAPER International Journal of Recent Trends in Engineering, Vol 2, No. 1, November 2009 the geometry of the imaging setup, the parameters of the planes and the level of noise are added to the image planes. Table1. The accuracy table for numerical simulation (a)
Perfect stereo case
Rcons -truction
In absence of noise With normalization
Perfect Stereo Horizontal plane A D No No Error Error 1.4 No 211eError 014
In presence of noise
(b)
Tilt plane A
D
2.242
1.4043
2.242
1.4043
V 0.0001 0.001 0.01 0.1
No Error
A 3.0095 6.1995 19.371 44.850
D 2.8248 6.6179 19.691 38.367
General case Rconstruction
General Horizontal plane
In absence of noise With normalization
In presence of noise
Tilt plane
A
D
A
D
2.037
2.599
2.304
0.927
2.024
2.640
2.304
0.927
V 0.000001 0.0001 0.001 0.01 0.1 0.2 1 2 10
A 2.036 2.039 2.032 1.942 1.852 1.928 1.972 1.894 1.969
D 2.599 2.610 2.580 2.599 2.523 2.492 2.601 2.666 2.599
V
A
D
0.0001 0.001 0.01 0.1
2.697 6.807 18.40 45.94
2.502 7.120 21.61 37.15
V. RESULTS AND DISCUSSIONS A new reconstruction approach for reconstructing a plane in 3D space from two perspective views without the requirement of point-to-point correspondence is proposed. The correspondence between the object space and image planes is established using the collinearity equations. Two sets of discerete points are obtained by projecting the original plane into two image planes. From these two sets of discerete points the minimum and maximum (min-max) points of horizontal and vertical axes are obtained for each plane. Using these min-max points and the collinearity equations, points in 3D space are obtained. The equation of reconstructed plane in 3D space is obtained from these 3D points. For horizontal and tilt plane reconstruction both general and rectified (perfect stereo) cases are considered. In perfect stereo case, epipolar lines become collinear and parallel to the horizontal axis of the image, due to which a 2D search problem converts into a 1D search problem. Error analysis has also been carried out by adding Gaussian noise in both two-dimensional reference images of planes. The noise level is characterized by the variance of the Gaussian distribution. For rectified case, there is no error in horizontal plane reconstruction process for all three categories: without noise, with noise and with normalization whereas, in general case some error is present. In case of tilt plane, when the value of variance is increasing, the error increases gradually. Furthermore, we can see that the errors are not more than 3 without adding noise in all categories. In case of normalization, we have achieved better results. The accuracy table for numerical simulation is shown in table 1. We observe that our approach works well even in the noisy cases. Results are demonstrated in figures 2 and 3.
Note-* A-Angle, D-Distance, V-Variance
(a)
(b)
(c)
Figure 2: Results of plane reconstruction in case of perfect stereo (a) original horizontal plane (b) Reconstructed horizontal plane in absence of noise. (c) Reconstructed horizontal plane in presence of noise.
(a)
(b)
(c)
Figure 3: Results of plane reconstruction in case of perfect stereo (a) original tilt plane (b) Reconstructed tilt plane in absence of noise. (c) Reconstructed tilt plane in presence of noise (variance 0.001).
177 © 2009 ACADEMY PUBLISHER
FULL PAPER International Journal of Recent Trends in Engineering, Vol 2, No. 1, November 2009 CONCLUSIONS In this paper, a new algorithm for 3D plane reconstruction is proposed using min-max approach. The main contributions of this work are the simulated results for plane reconstruction in 3D space. Error analysis is also done by adding Gaussian noise. Experimental results show better performance for perfect stereo case. For general case, some error is still there. In future, we plan to implement this approach in various real-life problems such as robotic vision problems and city modeling from two images to reconstruct the various planar objects like buildings, park etc..
[13] R. Kaucic, R. I. Hartley, N. Y. Dano, “Plane-based Projective Reconstruction.”, In Proceedings of ICCV'01, Vol I, pp. 420 -427, 2001.
ACKNOWLEDGMENT We gratefully acknowledge the financial support of Indian Space Research Organization (ISRO), Ahemdabad, India through the Project scheme ISR-295-MTD to carry out this research work. REFERENCES [1]
R. J. Schalkoff, Digital Image Processing and Computer Vision, John Willey and Sons Inc. [2] Y. Shirai,Three Dimensional Computer Vision, Springer-Verlag, 1985. [3] R. Balasubramanian, D. Sukhendu, and K. Swaminathan, “Simulation studies for the performance analysis of the reconstruction of a line in 3D from two arbitrary perspective views using two plane intersection methods.” International Journal of Computer Mathematics, Vol. 80, pp. 39–51, 2003. [4] S. Kumar, N. Sukavanam, and R. Balasubramanian, “Reconstruction of quadratic curves in 3D using two or more perspective views: simulation studies.” In Proceedings of the SPIE Vision Geometry XIII IS&T/SPIE International Symposium on Electronic Imaging, San Jose, California, USA, Vol. 6066, pp. 182–191, 2006(January 15-19). [5] N. Sukavanam , R. Balasubramanian and S. Kumar, “Error estimation of quadratic curves in 3D space”, International Journal of Computer Mathematics, 84(1), pp. 121-132, 2007. [6] Zhang J. Chen, Jagath Samaranbandu, “Planar region depth filling using edge detection with embedded confidence technique and Hough Transform”. In Proceedings of the International Conference on Multimedia and Expo, Vol 2, pp. 89 – 92, 2003. [7] Carsten Rother, “Linear Multi-View Reconstruction of Points, Lines, Planes and Cameras using a Reference Plane”. In Proceedings of the Ninth IEEE International Conference on Computer Vision, vol.2. pp. 1210-1217, 2003(Oct 13-16),. [8] Yasushi Kanazawa, Kenichi Kanatani, “Direct reconstruction of planar surfaces by stereo vision”, Proceedings of IAPR Workshop on Machine Vision Applications(MVA ), Kawasaki, Japan, pp. 285-288, 1994(Dec 13-15). [9] N. Chumerin, and Van Hulle, M.M., “Ground Plane Estimation Based on Dense Stereo Disparity”. The Fifth International Conference on Neural Networks and artificial intelligence,Minsk, Belarus, pp. 209213, 2008(May 27-30). [10] N. Simond, “Reconstruction of the road plane with an embedded stereo-rig in urban environment”, IEEE Intelligent Vehicle Symposium, Tokyo, Japan, pp. 70–75, 2006(13-15 June),. [11] F. Fraundorfer, K. Schindler, H. Bischof, “Piecewise planar scene reconstruction from sparse correspondences. Image and Vision Computing”. 24(4), pp. 395-406, 2006. [12] I. Infantino, R. Cipolla and A. Chella, “Reconstruction of architectural scenes from uncalibrated photos and maps.”, IAPR workshop on Machine Vision Applications, The University of Tokyo, Japan, pp. 427430, 2000(Nov. 28-30).
178 © 2009 ACADEMY PUBLISHER
