Corner Guided Curve Matching and its Application ... - Semantic Scholar

Corner Guided Curve Matching and its Application to Scene Reconstruction Ying Shan, Zhengyou Zhang Microsoft Corporation, One Microsoft Way, Redmond, WA 98052-6399, USA Email: fyshan, [email protected] Abstract

nitude reaches the local maxima along the gradient vector of the image intensity surface. While it is not difficult to maintain the consistency of edge points between different views, it is extremely difficult to keep consistency of curves, especially their endpoints. In addition to some common factors such as occlusion that may affect the consistency of both corners and edges, the edge linking process, which is used to produce curves from individual edge points, may introduce a significant number of inconsistent curves between different views. All above makes curve matching a more difficult problem.

Corners and curves are important image features in many vision-based applications. Corners are usually more stable and easier to match than curves, while curves contain richer information of scene structure. In previously work, corners are often used to recover the epipolar geometry between two views, which is then used in curve matching to reduce the search space. However, information of the scene structure contained in this set of matched corners is ignored. In this paper, we present a curve matching algorithm that is guided by a set of matched corners. Within a probabilistic framework, the role of the corner guidance is explicitly defined by a set of similarity-invariant unary measurements and by a similarity function. The similarity function provides stronger capability of resolving matching ambiguity than the epipolar constraint, and is integrated into a relaxation scheme to reduce computational complexity and improve accuracy of curve matching. Experimental results clearly demonstrate the benefit of integrating corner matches into the curve matching procedure.

Two central problems in many vision-based applications are the camera motion recovery and the scene geometry reconstruction. Corner features are usually used to compute the camera motion because of reliability of their matching. However, they are not suitable for the scene reconstruction. First, the set of corners is usually very sparse, and only provides a very coarse representation of the scene structure. Second, corners are usually not available along curves, but the curves contain very rich information of the scene structure. The problem can be illustrated by the left image of Fig. 6, where the reconstructed scene is represented by a Delaunay triangle mesh textured by the original reference image. The vertexes of the triangles are matched corners. Each triangle is considered as a flat surface, and therefore, the Delaunay triangles can span across the surface boundaries, making visual appearance of the scene structure unrealistic. This suggests us to include curves in scene structure recovery.

Keywords: Stereo vision, Model acquisition, Curve matching, Corner matching, Probabilistic relaxation.

1 Introduction Corners are 2D image points where the image intensity surface has the highest curvatures. The popular method for corner detection is Harris’ corner detector [1]. Corners thus detected have the nice property that they are usually stable. That is, corners across views corresponding to the same physical point in the 3D world can usually be consistently detected despite different perspective distortions. Reliable corner matching techniques are already available. In fact, with the help of techniques such as robust estimation [2] and gradient disparity limit [3, 4], it is possible to obtain one hundred percent rate of correct matching for most real applications. Curves in this paper are referred to chains of 2D image points. These points, known as edge points, are usually obtained through edge detection, where the gradient mag-

In curve matching, epipolar geometry has been successfully used to reduce the search space. The epipolar constraint states the fact that, given a point p in one image, its corresponding point in the other image must lie on a line lp , known as the epipolar line. All curves intersecting with lp could be candidate matches for the curve on which point p lies. In some earlier work on curve matching [4, 5], the epipolar lines are horizontal scan lines. When the image pair is taken by cameras with more general configuration [2], the epipolar geometry is usually computed from a set of matched corners. The recovered epipolar geometry can 1

age). There are N curves in the left image, and M in the right. The curves in the left image form the object space, denoted by A = fa 1 ; : : : ; aN g. The curves in the right image form the model space, each labeled as ! j . We wish to match the object to the model. We therefore assign to each object ai a label i , which may take as its value any of the M + 1 model labels that form the set = f! 0 ; !1 ; : : : ; !M g, where !0 is the null label used to label curves for which no match in the right image is appropriate. Three sets of indices are defined for convenience as N0  f1; : : : ; N g, Ni  fj j j 2 N0 ; j 6= ig, and M0  f0; : : : ; M g. The unary measurement set is defined as X = fxi j i 2 N0 g, where xi is the unary measurement vector for the object a i . Unary measurements include intensity similarity, and will be addressed in section 3. The binary measurement set for the object a i is defined as Ai = fAij j j 2 Ni g, where Aij is the vector of binary measurements between objects a i and aj . Binary measurements include distance between two curves. To follow the traditional notation, we will use the upper-case P to denote the probability of an event, the lower-case p to denote the probability density function, and N v (; ) to denote the Gaussian probability density function of a random vector v with the mean  and the covariance matrix .

then be used in curve matching. It is natural to ask the question whether the matched corners set can be more useful for curve matching. The answer is definitely yes. In fact, the matched corner set gives a one-to-one mapping from a corner point p in one image to a corner point p 0 in the other image, which is stronger than the point-to-line mapping given by the epipolar constraint. The mapping given by the matched corner set will be referred to as the corner matching constraint. The major difference between the epipolar constraint and the corner matching constraint is that the latter is sparse and is only defined on the points in the matched corner set. In general, since the criterion and the process used to detect corners and edge points are different, these two types of features are unlikely to be overlapped. It is therefore improbable to expect that a corner point lies on a curve. Nevertheless, it is reasonable to assume that corner points can usually be found near a curve. Since object surfaces are usually continuous at least in a small neighborhood, local affinity or similarity constraint can be used to describe the relationship of a corner and a curve. In this paper, we present an algorithm that exploits the corner matching constraint and the local similarity constraint to provide stronger guidance for curve matching. The algorithm is described within a probabilistic relaxation framework, and the role of the corner guidance is explicitly defined by a set of similarity-invariant unary measurements and a similarity function. The unary measurements may also include conventional quantities such as the intensity in the neighborhood [6], orientation [7], and the local shape characteristics [8]. We also propose a set of uncertainty measurements for the unary measurements. These measurements are related to the distance from a corner to a curve, and are used to control the impact of corner guidance on curve matching. The similarity function between two curves is then defined in this high dimensional measurement space. The paper is organized as follows. Section 2 introduces a probabilistic relaxation framework and the notion of combined unary measurements. The combined unary measurements and the similarity function for corner guided matching are discussed in Section 3. Section 4 provides experimental results with real scenes. Section 5 concludes the paper and discusses the future work.

2.2 A framework of probabilistic relaxation With the above notation, the matching problem becomes that of finding for object a i a model label !i with the highest probability given the measurements X , and A i :

P (i = !i j X ; Ai ) = !max P ( = ! j X ; Ai ) 2 i 

According to [9], we have

P (i = !i j X ; Ai ) i = !i ) = P P (iP=(!=i j!xi )jQx()Q i  i (i = ! ) ! 2

(2)

where the support function is given by

Q(i = ! )

=

Y X

j 2Ni ! 2

P (j = ! j xj )

p(Aij j i = ! ; j = ! )

(3)

where the first item in the summation is the similarity function, and the second is the compatibility function. Based on (2), we update the labeling probability in a relaxation scheme according to the following iterative equation P (n+1) (i = !i ) (n) (n) (i = ! ) i = P P P((in=) (!=i )Q (4) ( n ) ! ) Q (  = i  i ! ) ! 2

2 A probabilistic relaxation framework The probabilistic relaxation framework given in this section is based on the nice work of [9] on structural matching. We also adopt most of its notation for the convenience of comparison reading.

2.1

(1)

Notation

Curve matching is modeled as a labeling problem from the object space (left image) to the model space (right im2

where n is the iteration number, and P (0) (i = !i ) = P (i = !i j xi ). The iteration process terminates if the change in the probabilities is less than a predefined small value or the maximum number of iterations is reached. Details of the relaxation scheme, binary measurements, and compatibility function are not the main concern of this paper. The reader is referred to [9, 10, 11] for more details. As can be seen from (3) and (4), the computational complexity of the relaxation procedure is proportional to N Sn Sc2 , where Sn is the average number of neighboring curves and Sc is the average number of candidate curves in the right image for each curve a i in the left image. It is obvious that if all other objects are regarded as neighbors and all models are regarded as candidates, relaxation will be extremely slow. To reduce the computational complexity, only objects within a certain distance to a i are selected. It is, however, more difficult to reduce the number of candidate curves. Epipolar constraint is used for this purpose. Besides, using the algorithm developed in this paper, it is possible to further reduce the number of candidate curves, e.g., by a factor of 6, which means that the relaxation can be accelerated by a factor of 36. As a result of reducing the number of candidate curves, matching ambiguity is also reduced, resulting more accurate curve matching.

=

Y X j 2Ni ! 2

P (j = ! j x(j ) )

p(Aij j i = ! ; j = ! )

(5)

3 Combined unary measurement and similarity function for corner guided matching The impact of the corner matching constraint can be naturally reflected in the similarity function in (5). The unary measurements discussed in this section are those related to the matched corner set. Other unary measurements are also possible, but are not the concern of this section. In order to efficiently compute combined unary measurements, we approximate a curve by a set of line segments connected by equally separated seed points. Seed points in the right image are obtained by intersection the epipolar lines of the left seed points with the curve in question. This approximation is accurate enough provided that an edge chain is broken at high curvature points, and that seed points are not very far from each other. A corner is said to be the neighbor of a curve if its distance to the curve is less than a predefined threshold. The distance from a corner to a curve is defined by the minimum distance from the corner to the midpoints of all line segments on the curve. The combined unary measurements for each line segment, to be described shortly, are computed. They are then grouped together in an optimal way to form the combined unary measurements for curves.

2.3 Adapted framework with combined measurements In our case, the unary measurements depend on information in both object and model space (i.e., left and right image). Therefore, we should consider measurements for () combined object-model pairs. Let x i ( 2 M0 ) be the combined unary measurement defined for the pair of the ith object and the th model. The unary measurement of object ai , xi , is then itself a set of combined unary measurements, () i.e, fxi j  2 M0 g. In order to adapt the framework for working with the combined measurements, consider the similarity function in (3) first. Since the event  j = ! does not depend on ( ) the combined unary measurements other than the x j , we have

3.1

Similarity function

According to Bayes’ theorem, the similarity function in (5) can be written as

P (i = ! j x(i) ) () ^ = P p(xi j(i) = ! )P (i^= ! ) ! 2 p(xi j i = ! )P (i = ! )

(6)

where P^ (i = ! ) is the prior probability equal to a prefixed value  if  = 0 (i.e., no match), and to (1  )=M 0 otherwise. The unary measurements are used in (6) to compute P (i = ! j x(i) ). It is then used to initialize the relaxation () scheme, that is: P (0) (i = ! ) = P (i = ! j xi ). The (  ) likelihood function p(x i j i = ! ) will be given later. It will be clear soon that our unary measurement is similarity invariant. As a result, there is no need to introduce a motion related term as in [9] for the computation of the similarity function.

P (j = ! j fx(j ) j 2 M0g) = P (j = ! j x(j ) ) substituting the above equation into (3), (3) into (2), (2) can be rewritten as

P (i = !i j X ; Ai ) (i ) ( = ! ) i i = P P (i = !i j xi )Q ( ) ! 2 P (i = ! j xi )Q(i = ! )

3.2

Combined unary measurements for line segments

We assume that the mapping from a neighborhood in one image to the corresponding neighborhood in the other can be approximated by a local similarity transformation. Consider a pair of line segments and a pair of corners illustrated

and (3) as

Q(i = ! ) 3

k i

θ

where K is the number of the neighboring corners, consider the following measurement vector

p2

p2

'

q

ω

p1

k α

b q

ez = 1 z1 +


+ K zK

p1 '

left image

2 e2 = 12 12 +


+ K2 K

right image

i = [i2

in Fig. 1. The similarity transformation x 0 = sR x + t can be computed from (p 1 ; p01 ) and (p2 ; p02 ), as described in Appendix A. We then compute b = sR  q + t, and form



3.3

j =1

j 2 ] 1

p(x(i) j i = ! ) = p(ez j i = ! ) = Nez (0; e2 I2 )

(7)

where v = b q0 . Ideally, if the local similarity transformation is valid and the point coordinates are noise-free, we have z = 0. These conditions are not satisfied in practice, and we assume that the components of z are i.i.d, and have the same standard deviations  , i.e., z = N z (0;  2 I2 ), where I2 is the 2  2 identity matrix. There are several advantages of the unary measurements z given in (7).



K X

Equation (8) is an unbiased estimate of the measurement error with minimal variance. The likelihood function in (6) is then given by

a random vector z as the follows



(9)

We then select j to minimize (9), which yields

Figure 1. Configuration of a pair of line segments and a pair of corners in both images

z = [v(x) ; v(y) ]>

(8)

PK where k ’s are coefficients and k=1 k = 1. It is obvious that e z is also a joint Gaussian with ez = Nez (0; e2 I2 ), where e is given by

'

(10)

() = ez is the combined unary measurements.

where xi

3.4 The distance between segments and the standard deviation

The standard deviation  of vector z in (8) should be a function of the distance d between the corner and the line segment in question. This is because the local similarity assumption becomes weaker when d becomes larger. The function that we use is

Measurements are invariant under the similarity transformation. The set of unary measurements is complete. Under similarity transformation, a pair of line segments and a pair of corners have 6 equations, and there are 4 degrees of freedom for the similarity transformation. Therefore, the number of invariants is equal to 2, and we have the same number of measurements. Measurements are evaluated in the same physical space, namely, they are quantified in pixels in the second image. This means that we only need one  for all the measurements, whose computation is given in section 3.4.

(d) =

p [(1 )(1 e

d2 2

(11) ) + ] 2 where  = min (W; H ), W and H are respectively the im-

age width and height,  is a positive scalar that controls the range within whichp the guidance from neighboring corner is effective, and = 2 0 = where  0 is the desired standard deviation when the neighboring corner is very close to the segment under consideration. Several observations can be made for the function defined in (11):

 When d is small, (d)  0 which is the desired standard deviation when neighboring segments are p close.  When d is much larger than  , (d)  = 2. In this case, Nz (0;  2 I2 )  1=2 , indicating that the

Combined unary measurements for curves and the likelihood function

Now consider a pair of curves  i and !. For each neighboring corner of  i, the closest segment is selected, and measurement zk as defined in (7) is computed. The rationale of choosing the closest line segments is that the similarity transformation better applies to a small neighborhood than to a larger one. For the vector set fz j j j 2 [1; : : : ; K ]g,

 4

probabilistic density function approaches to a uniform distribution, and provides no information about the similarity between the pair of line segments. The intermediate value of  (d) varies smoothly between the above two extreme cases.

Figure 2. Matched corner set used as guidance for curve matching As a consequence of the second point above, when all the distance measurements fd j j j 2 [1; : : : ; K ]g are much larger than  , the likelihood function in (10) is approximately a constant 1=2 . As a result, if all the other corners are far from the current curve  i , it is not difficult to see () from (6) that P ( i = ! j xi ) = P^ (i = ! ) This is in accordance with our intuition that the labeling probability can not be improved when the guidance from neighboring corners is weak. The adjustment of the parameter  depends on the type of the scene in question. If the scene is close to the camera, there is a severe perspective distortion, and  needs to be small enough to discard the support from curves far away. On the other hand, if the scene is far from the camera,  could be large in order to include support from as many neighboring curves as possible.

matching constraint is able to give a reasonable result even the matched corner set is sparse. Curves with similarity values larger than 0.01 are regarded as candidate matches. Sc is now 4.4, which is 6 times less than the one given by the epipolar constraint. If we use the conventional curve matching without the corner matching constraint, 71 curve matches are found, which are displayed by the same color in Fig. 4. The unary measurement used here is the average difference between the orientations of corresponding seed points, and the orientation of a seed point is the direction of its gradient vector. A pair of curves on the upper-left corner of the image is mismatched. When we integrate the corner matching constraint, 73 curve matches are found, which are displayed in Fig. 5. We can see that the pair of mismatched curves in Fig. 4 is now correct. Moreover, two other pairs missed, also in the upper-left corner in Fig. 4 are found. This demonstrates that the corner matching constraint can improve the accuracy of curve matching. Edge points on the matched curves are then used to reconstruct the scene in 3D. The result is given in the right picture of Fig. 6, which should be compared with the left picture (the 3D reconstruction only from corner matches). Obviously, the visual appearance of the reconstruction with matched curves added is much better. Fig. 7 shows an image pair of an outdoor scene displayed with the matched curves obtained by using the corner matching constraint alone. The S c ’s for the epipolar line constraint alone and the corner matching constraint combined are 32.1 and 5.8, respectively. The latter is almost 6 times less than the former. We integrated the corner

4 Experimental results Experimental results with two pairs of real images are presented in this section. The algorithm is implemented based on a curve matching program we developed earlier. Fig. 2 shows an image pair of an indoor scene together with matched corners. Epipolar geometry is computed from these matched corners, and is used to find the candidate curves in the right image. There are 157 and 164 curves in the left and right images. The average number of candidate curves Sc is 25.1 if we only use the epipolar constraint. Fig. 3 shows the initial set of matched curves obtained only with the corner matching constraint. A pair of curves whose similarity value (computed from (6)) larger than 0.5 is selected as a match and is indicated in the figure with the same color. It can be seen in Fig. 3 that the corner 5

Figure 3. Initial set of matched curves obtained only with the corner matching constraint

Figure 4. Result of a conventional curve matching method without the corner matching constraint matching constraint with the above mentioned conventional method, and found 301 curve matches. Fig. 8 shows the results, where all 3 bad matches are marked with pink arrows for reader’s convenience.

is explicitly defined by a set of similarity-invariant unary measurements and by a similarity function. The similarity function provides stronger capability of resolving matching ambiguity than the epipolar constraint, and is integrated into a relaxation scheme to reduce computational complexity and improve curve matching accuracy. Experimental results clearly demonstrate the benefit of integrating corner matches into the curve matching procedure.

5 Conclusions and Future Work In this paper, we have presented a curve matching algorithm that is guided by a set of matched corners. Within a probabilistic framework, the role of the corner guidance

We currently rely on the local similarity and affinity in 6

Figure 5. Result of a conventional curve matching method with the corner matching constraint

Figure 6. Scene reconstruction with matched corners only (left) and matched curves (right)

References

relaxation, which are of course not valid in presence of occlusions. An interesting future work is to consider more robust local support measures.

[1] C. Harris and M. Stepens. A combined corner and edge detector. In 4th Alvey Conference, pages 147–152, 1988. [2] Z.Zhang, R. Deriche, O. Faugeras, and Q. T. Luong. A robust technique for matching two uncalibrated images through the recovery of the unknown epipolar geometry. Artificial intelligence journal, 78:87–119, 1995.

A The similarity transformation A similarity transformation is of the following form:

x0 =



c s





s t1 c x + t2



(12)

[3] P. Burt and B. Julesz. A gradient limit for binocular fusion. Science, 208:615–617, 1980.

If we have two pairs of matched points (p 1 ; p01 ) and (p2 ; p02 ), fc; s; t1; t2 g can be solved linearly. Acknowledgment: We thank Rick Szeliski for his subpixel edge detection code and P. Anandan for his encouragement of this work.

[4] Stephen B Pollard, John E. W. Mayhew, and John P Frisby. PMF: A stereo correspondence algorithm using a disparity gradient limit. In John E. W. Mayhew and John P. Frisby, editors, 3D model recognition from stereoscopic cues, pages 11–25. MIT, 1991.

7

Figure 7. Initial set of matched curves obtained only with the corner matching constraint

Figure 8. Result of a conventional curve matching method with the corner matching constraint [5] Stephen B Pollard, John E. W. Mayhew, and John P Frisby. Implementation details of the PMF stereo algorithm. In John E. W. Mayhew and John P. Frisby, editors, 3D model recognition from stereoscopic cues, pages 33–41. MIT, 1991.

572, 1992. [9] William J. Christmas, Josef Kittler, and Maria Petrou. Structural matching in computer vision using probabilistic relaxation. IEEE PAMI, 17(8):749–764, aug 1995.

[6] Cordelia Schmid and Andrew Zisserman. Automatic line matching across views. In CVPR’97, pages 666–671, 1997.

[10] R.A. Hummel and S.W.Zucker. On the fundations of relaxation labeling process. IEEE Trans. PAMI, 5(3):267–286, may 1983.

[7] G. Medioni and R. Nevatia. Segment-based stereo matching. CVGIP, 31:2–18, 1985.

[11] J. Kittler and E.R. Hancock. Combining evidence in probabilistic relaxation. Int’l J. Pattern Recognition and Artificial Intellegent, 3:29–51, 1989.

[8] Nasser M. Nasrabadi. A stereo vision technique using curvesegments and relaxation matching. IEEE PAMI, 14(5):566–

8