Automatic detection of planar contours from uncalibrated images Houda Chabbi† , Lukas Theiler‡ and Stéphane Paris †
MISL, Mobile Information System Laboratary of the EIA-FR, Fribourg, Switzerland ‡
DIUF, Department of Informatics of the University of Fribourg, Switzerland
Automatic Laboratory of Cooperative Systems, University of Metz, Ile du Saulcy, 57000 Metz, France
E-mail:
[email protected],
[email protected],
[email protected] Abstract The present paper describes an original technique developed for detecting and matching unknown planar structures between a pair of uncalibrated images. Problems related to bad segmentation or to occlusion are automatically solved in most cases. The match process works hierarchically on pertinent point which provide local consistency and on contours which ensure the global consistency. After match, contours are splited into planar contour according to collineations. Tests made over different pairs of images demonstrate the ef£ciency of our approach. Keywords : uncalibrated matching, planar matching, contours splitting, collineation, hierarchical matching.
1. Introduction Despite their large applications in diverse £elds like mosaicing, reconstruction and self-calibration, the automatic detection of planar structures in images remains a dif£cult task. In fact, the main planar detection techniques, used in existing applications methods, are generally based on the use of an “a priori” knowledge. In the present work, our objective is to automatically detect and match unknown planar surfaces between a pair of uncalibrated images assuming that a scene, containing several planar surfaces, is observed under different viewpoints by an uncalibrated camera. Each camera performs a pure perspective projection allowing the use of projective geometry information like collineations or semi-local invariants [1]. However, the latter require contours to be £tted by curves (e.g. splines) and are very sensitive to noise. A 3D planar contour projected onto two image planes, delivers a pair of 2D contours. The planarity property induces a geometric relation between this pair of 2D contours, called a collineation. Since occlusion and bad segmentation makes planar surface projections look different between two images, perfect matches are dif£cult to obtain. Therefore,
we propose to proceed into two main stages. The £rst stage concerns a hierarchical contour matching process generating a consistent set of contour pairs. The second stage extracts planar correspondences from the previous set using collineation information. An overview of the approach is illustrated in Fig. 1. Image pair Chains Contours ++ Pertinent points Pertinent points
Primitive segmentation
Pairs of chains + Pairs of pertinent points
Hierarchical matching process First step
Planar split prediction
Collineation re-estimation
Input: Pertinent points Technic: LMedS
Input: Contours Technic: Levenberg-Marquart Second step
Contour splitting Input: Contours
Pairs of planar chains
Figure 1. The planar contour matching process. A most close technique to ours, is the approach presented by Chesi et al. [3] where they proposed a process for planar chain matching using collineations. The main difference with our approach is that they assumed that a reference chain is known without any occlusion. Then using collineations, they automatically selected from images the appropriate correspondent to the reference chain. Another category of approaches close to our are the techniques which use “multiple” primitives (i.e. pertinent points and contours) to perform contours matching[4, 6]. But, besides the use of the planarity constraint, compared to these works which over split contours, our approach keeps and matches the longest part possible between the contours. The paper is organized as follows. In section 2 the hierarchical contour matching process is presented. Section 3 presents the planar split prediction while section 4 explains
1051-4651/02 $17.00 (c) 2002 IEEE
the re£ning of this prediction. At the end, split of contours is performed and delivers pairs of planar contours as explained in section 5. Before concluding, results obtained on real images are presented and commented in section 6.
N = (n1 , n2 , n3 ) are the unknowns. In our approach, the fundamental matrix is recovered using an eight-point algorithm [5] and the whole set of matched pertinent points in two images, obtained from the £rst stage of our process. Initialization:
2. The hierarchical contour matching process
1 pair of chains (chi,chj) + their pairs of pertinents points Lij u=1
{
Figure 2. A non ambiguous multiple match between one contour (left image) and 3 others (right image). Pairs of pertinent points are labeled with the same number according to each contour of the right image.
3. Planar split prediction As shown before, the computation at the £rst step of our matching process delivers a set S of non ambiguous multiple contour matches. Collineations are used to detect planar contour pairs from this set S. A collineation H can be estimated using the fundamental matrix F , the epipole e = (ex , ey , et ) and at least three pairs of matched points (mu , nu ) using the following equations: ∼ nu ∼ (−[e ]× .F ) − e .N t and H.mu = H= (1)
No
Category (i)
Card(Lij)> 2
Compute a collineation Hij using Lij Equation (1) + SVD
{
A contour is a linked chain of edge points. To match contours between a pair of images, a reduced number of pertinent points are used to characterize each chain. Pertinent points are projective invariant primitives, which lay on contours. In our application, they correspond to corners or to in¤ection points of the contours. The contour matching process between two images is based on a hierarchical approach. A same generic relaxation technique is used to deliver £rst a reliable match of the pertinent points and second a consistent set S = {(chi , chj , Lij ), . . .} where each (chi , chj ) is a contour pair and Lij = {(pu , qv )/pu ∈ chi and qv ∈ chj } is its related list of matched pertinent-points (details of the technic can be found in [2]). Each contour pair hypothesis shares at least three matched pertinent points. Due to the differences of the contours segmentation related to an image pair, multiple correspondences of contours can appear in S. However, the set S is a set of non ambiguous multiple matches in the following sense: let (chi , chj , Lij ) and (chi , chk , Lik ) be two pairs of matched contours in S. Although they seem to share the same chain chi , the pair (chi , chj ) is not ambiguous with the pair (chi , chk ) since they do not share any pair of pertinent point. An example of a non ambiguous multiple match is shown in Fig. 2.
Category (ii)
If planar Lij Yes
u = u+1 (u) Lij = Lij \ Lij
Equation (2) Category (ii) (u)
Select a "good" collineation: Hij Lij Hij
(u) (u)
LMedS
= Lij = Hij
Lij (u) = { Pairs in Lij that verify the collineation} Equation (2)
Card(Lij (u) )> 2 Yes
(chi, chj, Lij
(u)
, Hij
(u)
)
Figure 3. The prediction split algorithm. To predict splits for a contour pair (chi , chj ), an iterative algorithm (illustrated in Fig. 3) based on the set Lij is used. Let us present the different stages of an uth iteration step. According to its related set Lij , the pair (chi , chj ) is classi£ed into one of the following categories : (i) A problematic contour pair. This is the case for contour pairs that have less then 3 matched pertinent points in Lij since no collineation is computable. (ii) A planar contour pair hypothesis. A match of this category veri£es the following equation ∀(pl , qk ) ∈ Lij , Hij (pl ) ∼ = qk
(2)
where Hij is the collineation obtained when using the whole set of pertinent points pairs ∈ Lij and equation (1). Note that, at this stage, contours chi and chj could be non planar, generally due to a merge between different contours, as illustrated by the kangaroo case of Fig. 2 which embodies one planar part. A pair of this category has to be split according to the collineation information. (iii) A decomposable contour pair. The set Lij has more than three pertinent point pairs but does not verify equation (2). Note again that contours chi and chj could be two non planar matched contours but they possibly embody one
1051-4651/02 $17.00 (c) 2002 IEEE
or more planar portions. This is illustrated in Fig. 2 by chi ={kangaroo,giraffe,lion} and chj ={giraffe,lion}. A speci£c action is related to each category. Each match of category (i) is removed from S. A match of category (ii) suggests a unique possible split induced by the collineation (u) Hij and delivers the following 4-tuple (chi , chj , Lij =
can be, in addition, poorly located. To compensate the de£ciency of pertinent points when computing the collineation, we further carry out a minimization procedure based on contours.
(u)
Lij , Hij = Hij ). When encountering one of these two categories the iterative algorithm stops. A problem remains for each match of category (iii). Let us assume that the initial contour pair embodies at least one (u) planar homologous sub-contours. Then, there exists Lij a subset of Lij corresponding to these sub-contours which (u) can induce a collineation Hij verifying (2). Our purpose is then to determine complementary subsets of Lij and their respective collineations in order to suggest relevant splits. One possible approach is to perform a search in the space of possible estimates of H generated from the data. Since this space can be large, we propose to randomly chose a subset of data to be analyzed. This idea suggests that among the set Lij there is a proportion τ of outliers matches (it must be less than 50%) regarding the relevant collineation. With respect to our speci£cation, it turns out that the least-medianof-squares (LMedS) method is the best method to perform this task. Fig. 4 shows a graphical result of our approach for one pair among the matches of Fig. 2. Collineations are estimated only using three pertinent points but for aesthetical reason results show the effects of the estimated collineation on the whole set of pixels of the considered contour.
H
Figure 5. Collineation effects: initial (red) and after re-estimation (green). The second example shows the result of the minimization over a bad initialization.
Since no identi£cation, and then no point-to-point correspondence, of homologous planar subparts is available, equation (1) can not be directly used to re-estimate the (u) (u) collineation Hij . Therefore, Hij is re-estimated using the entire contour chi from the £rst image and the intensity gradient image for the second image 1 . The re-estimated collineation will be denoted by H. The idea is to constrain the transformed contour H(chi ) to best £t a contour in the second image which should be a part of chj . This constraint is translated into maximizing E the sum of the squared gradient over the contour H(chi ): ∇(H(cu ))2 E=
(1)
cu ∈chi
where cu = (xu , yu , 1) is a point on the contour chi (including pertinent ones) and H(cu ) = (xhu , yuh ). Let us de£ne e u = ∇(H(cu ))2 then
H (2)
eu = ∇(xhu , yuh ) = (G2x [xhu , yuh ] + G2y [xhu , yuh ]) Figure 4. A split proposition obtained by our approach for one pair of the matches showed in Fig.2.
4. Collineation re-estimation using contours The previous step delivers for each initial contour pair (chi , chj , Lij ), one or more possible split prediction. Each split is depicted by the following 4-tuple: (u) (u) (u) (chi , chj , Lij , Hij ). Since, each collineation Hij is computed only using three pertinent points pairs, a bad estimation of this collineation is highly possible. Re-estimating (u) (u) Hij using the whole set Lij is not of a real bene£t since (u)
Lij generally contains a reduced number of pairs which
where Gx and Gy are respectively ∂I/∂x and ∂I/∂y. Maximizing E allows to recompute the three parameters (n1 , n2 , n3 ) of H (see equation (1)). To perform the maximization, we use the Levenberg-Marquart non-linear maximization algorithm. This has the advantage of not requiring any identi£able pairs of matched points over the homologous subparts of the contours. This algorithm requires the computation of the partial derivatives of ei with respect to the parameters (n1 , n2 , n3 ) and needs an initial guess for these parameters. The initialization is made using the val(u) ues of Hij obtained by the planar prediction step. The Levenberg-Marquart approach converges in few iterations. It delivers a collineation which generally better links the 1 We
assume that length(chi ) < length(chj ) otherwise we consider (u)
(u)
the pair (chj , chi , (Lij )t , (Hij )−1 )
1051-4651/02 $17.00 (c) 2002 IEEE
planar subparts of a pair (chi , chj ). Fig. 5 illustrates the bene£t of using contour information.
5. Contour splitting The last stage performs the contour split and delivers (u) (u) for each 4-tuple (chi , chj , Lij , Hij ) a point-to-point subpair of planar contours. The split is performed according (u) to the collineation information Hij . The contour chi is
that the result of our matching process depends essentially on the number of pertinent point per contour. They do not need to be dense or accurate but we need at least 3 pertinent points per contour. Our approach has been successfully tested on real pair images. In future, this system have to be completed by regrouping planar contour pieces coming from the same 3D planar contour and by merging into the same plan all possible coplanar contours.
(u)
projected into a contour, denoted by Hij (chi ), of the sec(u)
(u)
ond image using Hij . Points of Hij (chi ) corresponding to points of chj represent pairs of points which satisfy the (u) collineation. Computing Hij (chi ) ∩ chj can deliver several sub-contours. But not all sub-contours are pertinent. Let us note that the continuity property and the length of the detected corresponding portions are relevant to determine portion of planar pairs. That why we only keep the most rel(u) evant one i.e. the longest connected part ∈ Hij (chi )∩chj . (u)
(u)
Then to (chi , chj , Lij , Hij ) we associate the (u)
(u)
(u)
(u)
following 4-tuple (chi , chj , Lij , Hij ) (u) (u) •chi = M axconnected (Hij (chi ) ∩ chj ) (u) (u) −1 •chj = (Hij ) (chij )
where
6. Experimental results The proposed framework was applied to real pair of images. Due to the lake of place, we only present here a unique result in Fig. 62 . Note that the example presented in Fig. 2 and used along the paper is part of the pair (zoo6,zoo7). This part is stressed by a rectangular boxe in each image and shows the £nal stage of our approach (the contour splitting) over this example. Corresponding matched planar pairs share the same color after the split. Occlusion problems are solved by the split process. In the zoo sequence, the giraffe contour is never well segmented; in each image of the sequence, its related 2Dcontour is almost merged with another contour, not always the same: the lion or the parrot or the kangaroo. Despite this problem of bad segmentation, the giraffe is always well detected and matched.
7. Conclusion To sum up, lets present the bene£ts of the approach. Problems related to occlusion are solved by the cut process. Problems related to bad segmentation have no effect during the match and are solved by the cut process. It is obvious 2 More detailed results could http://www.eif.ch/∼chabbi/matching_result.html
be
found
Figure 6. First pair: Images of zoo6 and zoo7. Second pair: the initial set of chains. Third pair: the £nal set of matched chains.Boxes in the images delimit the example presented in Fig.2.
References [1] S. Carlsson, R. Mohr, T. Moons, L. Morin, C. Rothwell, M. V. Diest, L. V. Gool, F. Veillon, and A. Zisserman. Semilocal projective invariants for the recognition of smooth plane curves. IJCV, 19(3):211–236, 1996. [2] H. Chabbi and S. Paris. A generic relaxation technique for hierarchic stereo chain matching of uncalibrated images. In ICPR00, pages 746–749, Septembre 2000. [3] G. Chesi, E. Malis, and R. Cipolla. Automatic segmentation and matching of planar contours for visual servoing. In ICRA, pages 334–338, April 2000. [4] R. Deriche and O. Faugeras. 2-D Curve Matching Using High Curvature Points: Application to Stereo Vision. In ICPR, Atlantic City, NJ (USA), pages 240–242, 1990. [5] R. Hartley. In defense of the Eight-point Algorithm. PAMI, 19(6):580–593, June. 1997. [6] M. Xie. Cooperative strategy for matching multi-level edge primitives. Image and Vision Computing, 13(2):89–99, 1995.
1051-4651/02 $17.00 (c) 2002 IEEE
