Omnidirectional Stereo Vision with a Hiperbolic Double Lobed Mirror Eduardo L. L. Cabral, José C. de Souza Junior and Marcos C. Hunold
[email protected],
[email protected],
[email protected] Dept. of Mechatronic Engineering Escola de Engenharia Mauá Escola Politécnica of University of São Paulo, Brazil Instituto Mauá de Tecnologia - SP, Brazil
Abstract This paper presents a compact panoramic stereo vision system based on a double lobed mirror with hyperbolic profile. As hyperbolic mirrors ensure a single viewpoint the incident light rays are easily found from the points of the image. The geometry of the double lobed mirror naturally ensures matched epipolar lines in the two images of the scene. These two properties make the double lobed mirror especially suitable for panoramic stereo vision because range estimation becomes simple and fast. This is a great advantage for real-time applications. The proposed mirror is useful for mobile robot navigation, surveillance, and machine vision where fast and real-time calculations are needed.
1. Introduction Panoramic imaging is an important tool in the areas of mobile robots and machine vision. Among the many possible types of panoramic vision systems an attractive approach is to mount a single fixed camera under a mirror covering a hemisphere. These systems are normally referred as catadioptric. The advantages of a catadioptric system are the compactness, the wide angle of view, and the fast acquisition rate. However, a disadvantage is that the images acquired have a relatively low spatial resolution. The shape of the mirror determines the image formation model of a catadioptric panoramic system. Different mirror shapes have been used and tested [1, 2, 3]. A catadioptric system based on a hiperbolic mirror and a perspective (pinhole) camera forms an omnidirecional vision system with a single effective viewpoint. This characteristic is especially suitable for stereo imaging because the incident light rays are easily determined from the points of the image. Another important tool for mobile robots is stereo vision. Stereo vision allows determining the range to objects within the field of view of the system. Panoramic stereo vision is ideal for a mobile robot since the field of
view is 360o, thus requiring a single sensor instead of several sensors with more limited range or one sensor with moving capacity. Omnidirectional systems based on hyperbolic mirrors have been used for stereo vision as reported in [4, 5]. However, such systems were developed based on two mirrors and two cameras. Besides these systems are not compact they have the problem of requiring precise positioning of the cameras and the mirrors to guarantee alignment of the images. Panoramic stereo based on a double lobed mirror was first suggested in [6], and a resolution invariant double lobed mirror was developed in [3]. A double lobed mirror is essentially a coaxial mirror pair, where the centers of both mirrors are collinear with the camera axis, and the mirrors have a profile radially symmetric around this axis. This arrangement has the great advantage to produce two panoramic views of the scene in a single image. Thus, it is extremely compact and naturally ensures the alignment of the two images of the scene. In this paper a catadioptric stereovision system based on a double lobed mirror with hyperbolic profiles in both lobes is developed. The single viewpoint and the natural alignment of the images ensuring matched epipolar lines both in the omnidirectional and panoramic images make this system ideal for fast real-time stereo calculations. A catadioptric stereovision system based based on a constant resolution mirror, such as in [3], do not allow fast 3D stereo reconstruction since it requires solving a nonlinear differential equation to obtain the mirror shape function and its slope at the reflection point.
2. The omnidirectional stereo vision system The omnidirectional stereo vision system is composed of a conventional perspective (pinhole) camera aligned vertically (coaxial) with the double lobed mirror. Due to the rotational symmetry of the system only the radial cross section of the mirror need to be considered. Figure 1 presents a schematic of the hyperbolic double lobed mirror showing two incident and reflective light rays at the two lobes.
0-7695-2128-2/04 $20.00 (C) 2004 IEEE
A perspective camera together with a hyperbolic mirror form a wide-angle imaging system with two centers of projections, located at the hyperbole focal points. In Fig. 1 points F0 and FI are the two focal points of the inner lobe hyperbole, and points F0 and FE are the focal points of the outer lobe hyperbole. All incident light rays on the inner lobe pass through focal point FI and is reflected passing the other focal point F0. Similarly, all the incident light rays on the outer hyperbole pass through focal point FE and is reflected passing the other focal point F0. The inner and outer hyperbole focus F0 must coincide with the camera focal center.
Given a point P in space as shown in Fig. 1, two light rays emitted from this point reach the mirror surface, one on the inner lobe and other on the outer lobe. The incident light ray with the elevation angle φI reaches the inner mirror lobe at point MI and projects onto the image at radius ρI. The incident light ray with the elevation angle φE reaches the outer mirror lobe at point ME and projects onto the image at radius ρE. The equation for any incident light ray on the surface of the inner mirror lobe is
z = −rcotanφ I + z FI ,
zE
z
Outer lobe
FE FI
z = −(r − rFE )cotanφ E + z FE ,
MI
Inner lobe
φE
θI
φI
λ
F0
λ Image plane
ρE ρI
P
tan θ = ρ f = rM z M ,
r rE
Figure 1. Schematic of the hyperbolic double lobed mirror showing the incident and reflective light rays.
The optical axis of the inner lobe hyperbole is aligned with the camera axis, while the optical axis of the outer hyperbole is not aligned with the camera axis. The profile of the outer lobe is a hyperbole rotated around point F0 by an angle λ. The rotation of the hyperbole of the outer lobe is necessary to ensure the same field of view with the same resolution of the images generated by both lobes. A hyperbolic mirror is defined in the mirror coordinate system centered at focal point F0 as
z =c+
a b2 + r 2 , b
(3)
where (rFE, zFE) are the coordinates of the focal point FE. For any light ray reflected both from the inner and outer mirror lobes the following equation holds
Camera optical center
θE
(2)
where zFI is the z coordinate of point FI. Similarly, for any incident light ray on the outer lobe surface,
ME
f
2.1. Image formation model
(1)
where c, b and a are parameters of the mirror, with the restriction that c 2 = a 2 + b 2 . These parameters are different for the inner and outer lobes. Equation (1) is valid for the inner hyperbole described in the camera coordinate system (F0-rz), and for the outer hyperbole described in the outer lobe coordinate system (F0-rEzE).
(4)
where f is the lens focal distance, θ is the camera vertical angle, (rM, zM) are the coordinates of a point on the mirror surface, and ρ is the image radius. The coordinates of any point lying on the inner mirror surface must obey the inner lobe profile given by eq. (1). Also, the coordinates of any point lying on the outer mirror surface must obey the outer hyperbole equation given in the F0-rEzE coordinate system by eq. (1). Equation (5) gives the coordinate transformation from the camera to the outer lobe coordinate systems.
r cos λ z = sin λ
− sin λ r ( E ) , cos λ z ( E )
(5)
where (r ( E ) , z ( E ) ) are the coordinates of a point described in the outer lobe coordinate system.
3. Calculation of range Given a point P in space as shown in Fig. 1, its coordinates (rP, zP) are obtained by solving simultaneously the equations of the incident light rays from P to each mirror lobe reflection point (MI and ME).
rP tan φ I [tan φ E (2c I − z FE ) − rFE ] 1 = . z P (tan φ E − tan φ I ) z FE tan φ E − 2c I tan φ I + rFE (6) Based on first order optics, the elevation angles (φI and
0-7695-2128-2/04 $20.00 (C) 2004 IEEE
φE) are computed directly from the camera vertical angles (θI and θE) and from the slopes of the mirror shapes at the reflection points (tanαI and tanαE) as: φ = θ + 2atan(α ) .
(7)
From eq. (1) and (4) the slope of a hyperbolic mirror can be derived to give tan α =
a 2 ρ (cf + a u ) dz , = 2 dr b fa u + a 2 cρ 2
The two lobes are designed to have the same field of view, which is described by the minimum and maximum elevation angles of the scene (φmin and φmax). Figure 2 presents a schematic of the double lobed mirror showing the incident light rays with the minimum and maximum elevation angles and the corresponding reflected rays.
(8) FE
T
where u = [ρ, f] , and ||u|| is the norm of vector u. Equations (7) and (8) are used for the inner and outer mirror lobes to calculate φI and φE. For the inner lobe the image coordinates are given by uI = [ρI, f]T. For the outer lobe the image coordinates uE = [ρE, f]T must be first transformed to the outer lobe coordinate system ( uE(E ) ) using the inverse of the transformation given by eq. (6). When uE( E ) is substituted into eq. (8) the slope of the outer lobe described in the F0-rEzE coordinate system is calculated ( tan α E( E ) ). The slope of the outer mirror in the camera coordinate system may then be obtained as:
α E = α E( E ) + λ .
5. Calculation of the mirror profile
(9)
To calculate the coordinates of a point in space, using eq. (6) through (9), is straightforward once the image radius of corresponding points in the inner and outer images are known.
4. Epipolar constraint
∆zIE
φmax φmax
FI
φmin φmin
Figure 2. Schematic of the double lobed mirror showing the incident light rays with the maximum and the minimum elevation angles.
To achieve best resolution the projection of the mirror has to occupy the whole image. To achieve equal resolution in both mirror lobes the two images must have the same radius range, i.e., ( ρ E , max − ρ E , min ) = ( ρ I , max − ρ I , min ) ,
A fundamental problem in stereo vision is establishing the matches between image elements in the two images. The epipolar constraint reduces the problem of finding corresponding points in the two images to a 1D search. The epipolar constraint for catadioptric systems with hyperbolic mirrors has been studied in [7]. For hyperbolic mirrors, corresponding points must lie on conics (epipolar curves). However, when the hyperboles are vertically alignment the curves reduce to radial lines, and once the image is projected onto a cylinder (panoramic image) the epipolar lines become parallel. The arrangement of the double lobed mirror naturally ensures the vertical alignment of the two hyperboles, and since only one image generates the two views of the scene, the doubled lobed mirror also ensures matched epipolar lines both in the omnidirectional and panoramic images (see Figure 3). Matched epipolar lines are desirable because efficient search methods can be used to perform stereo correspondence, thus, allowing fast and real-time 3D reconstruction.
(10)
where, ρI,max, ρI,min, ρE,maxn and ρE,min are the maximum and minimum image radius of the inner and outer lobes. The known parameters for the mirror design are the elevation angles φmin and φmax, the lens focal distance, the maximum image radius (ρmax), the axial distance between focal points (∆zIE), and the maximum external radius of the mirror. With the exception of ρmax, these parameters are chosen based on the application requirements. The axial distance between the focal points FI and FE, which is the distance between the two view points of the stereo system, is chosen to obtain the desired reconstruction accuracy for a given maximum distance. Geometric restrictions provide three relations between the inner and outer hyperbole parameters cI and cE, the angle λ, and the coordinates of focal points FI and FE. 2c I = z FI ; 2c E = rF2E + z F2E ; tan λ = rF
E
z F . (11) E
Equations (10) and (11) together with the expressions for the incident and reflected light rays with the maximum
0-7695-2128-2/04 $20.00 (C) 2004 IEEE
and the minimum elevation angles on the inner and outer mirror lobes form a system of non-linear equations. The solution of this system results in the shapes (parameters) of the inner and outer mirror lobes.
5.1. The mirror design A double lobed mirror with hyperbolic profile was designed and manufactured. The mirror was designed for a camera with a maximum image radius equals to 4.84mm and equipped with 12mm lens. The mirror has a field of view that encompasses elevation view angles of the scene between 30o and 105o. Table 1 presents the main parameters of the mirror. Table 1. Main parameters of the mirror
Parameter aI cI rI,min rI,max
Value 70.725mm 76.792mm 3.26mm 17.51mm
Parameter aE cE
λ
rE,max
Value 76.729mm 82.795mm 5.08o 34.00mm
Figure 3(a) shows a raw omnidirectional image captured by the double lobed hyperbolic mirror and Fig. 3(b) shows the panoramic image obtained by projecting the omnidirectional image onto a cylinder. The two images of the scene have the same radial range in the omnidirectional image or, the same height in the panoramic image, as required for best resolution. In Fig. (3), three epipolar lines are draw both in the omnidirectional and in the panoramic images to show the perfect matching between theses lines in the images of the two scenes.
6. Conclusions A reflective double lobed mirror with hyperbolic profile for panoramic stereo imaging was designed and manufactured. The double lobed hyperbolic mirror may be used to give range measurements for the entirely panorama around the camera with the benefits of stereo vision with the need for only one image. As hyperbolic mirrors ensure a single viewpoint the incident light rays are easily found from the points of the image. This property and the perfect alignment of the images ensuring matched epipolar lines make this mirror especially suitable for real-time stereo calculations.
7. Acknowledgments The authors acknowledge the financial support given by Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP) to develop this work.
Figure 3. (a) Omnidirectional image capture by the double lobed mirror. (b) Panoramic image obtained by projecting the omnidirectional image onto a cylinder. Dashed lines are the epipolar curves.
8. References [1] J.G. Gaspar, C. Deccó, J. Okamoto Jr., and J.S. Victor, “Constant resolution omnidirectional cameras”, In OMNIVIS'02, Copenhagen, 2002. [2] K. Yamazawa, Y. Yagi, and M. Yachida, “Omnidirectional imaging with hyperboloidal projection”, In IEEE/RSJ Int. Conf. Intell. Robots and Systems, 1999. [3] T.L. Conroy and J.B. Moore, “Resolution invariant surfaces for panoramic vision systems”, In IEEE ICCV'99, pp. 392-397, 1999. [4] J. Gluckman, S. K. Nayar, and K. J. Thoresz, “Real-time omnidirectional and panoramic stereo”, In Proc. of Image Understanding Workshop - IUW’98, 1998. [5] H. Koyasu, J. Miura, and Y. Shirai, “Recognizing moving obstacles for robot navigation using real-time omnidirectional stereo vision”, J. of Robotics and Mechatronics, Vol. 14, No. 2, pp. 147-156, 2002. [6] M.F.D. Southwell, A. Basu, and J. Reyda, “Panoramic stereo”, In ICPR, pp. 378-382, Vienna, 1996. [7] T. Svoboda, T. Padjdla, V. Hlavác, “Epipolar geometry for panoramic cameras”, In ECCV'98, pp. 218-231, Freiburg, 1998.
0-7695-2128-2/04 $20.00 (C) 2004 IEEE
