Correspondences of Point Sets using Particle Filters - CiteSeerX

Correspondences of Point Sets using Particle Filters Rolf Lakaemper, Shusha Li, Marc Sobel Temple University, Philadelphia, PA [lakamper,shusha,marc.sobel]@temple.edu

Abstract The paper shows how Particle Filters can be used to establish visually consistent partial correspondences between similar features in unrestricted 2D point sets representing shapes. Given an update rule, the PF system has the advantage that global constraints can be learned. We motivate and define the update rule for the given task and show its superior performance in comparison to a globally unrestricted approach.

1. Introduction For many years, finding correspondences between visual features of pairs of shapes has been the focus of computer vision research. Solutions involving shapes with closed boundaries have been particularly impressive, e.g., [1], [3], [2]. In [4], we introduced Particle Filtering (PF) to solve a minimization problem to find correspondences between features of shapes and shape parts, represented by polygonal boundaries. The most important feature of that approach was to distinguish between local and global constraints of the correspondence problem. This idea leads to elegant implementation of the global constraints, since the iterative multiple hypothesis PF approach allows the system to learn these from the feedback obtained from current correspondence hypotheses. [4] describes the PF system as a general framework in which the importance of correspondences are defined via: (i) a matrix representing local constraints, and (ii) an update rule which iteratively generate a second matrix characterizing global constraints. The update rule implements the aforementioned feedback cycle. Figure 1 illustrates this idea. The main contribution of this paper is to demonstrate how the versatile PF tool can be utilized to find partial correspondences between unordered point sets, representing e.g. partial shapes with inner structures. We carry out the new task by redefining the update rule for global constraints, hence only the constraints are adjusted in comparison to an otherwise identical system

978-1-4244-2175-6/08/$25.00 ©2008 IEEE

Figure 1.

Global and local constraints influence the PF process. In our approach, the current particle and the local constraints are fed back to update the global constraints: global constraints are learned during the PF process.

to [4]. We will give a brief introduction to the PF system, motivate and define the update rule. We will also discuss the role of global constraints in the algorithms performance.

2

Related Work

The dynamic programming (DP) approach to shape feature correspondence detection defines the correspondence problem as one of minimizing an energy or cost function depending only on the local feature constraint matrix, e.g. [6] Shape feature correspondence detection with Dynamic Programming. The global constraints, in e.g., [6] consist in order preservation which is implicit in the algorithms structure, and hence, not easily changeable. In contrast to dynamic programming, which has fixed global constraints, our’s are learned in particle filter settings (PF). This makes DP a preferred approach for relatively simple tasks, e.g. correspondence of closed boundaries, while the PF system shows its advantages in problems requiring more complex global constraints. The process of finding correct correspondences can be seen as a labeling process. The features of one shape correspond to the labels; the features of the other have

to be labeled. In 1976, Rosenfeld et al. [5] introduced the technique of relaxation labeling (RL) to provide a solution to this class of problems. Since then, relaxation labeling has often been applied to this problem. [7]. RL is a gradient descent method which guarantees convergence towards some local optimum. It is an iterative, deterministic approach, highly dependent on the initial correspondence probability matrix. In this terminology, our approach can be viewed as sequential, non-deterministic, multiple hypotheses relaxation labeling. Sequential, because of the aforementioned prediction step, which assigns a single label unchangeably to a data point. PF has the advantage, in partial shape matching, of enhancing strong local feature properties, while RL subsumes them, enhancing the global labeling structure. This leads to unrobust behavior with respect to outliers, which, in contrast, are solved by the PF system (see e.g. fig.8). Belongie et

5

5

10

10

15

15

20

20

25

25

30

30

35

35

40

40

45

45

50 20

30

40

3. Correspondence Problem and Particle Filtering This section will give a brief overview over the PF approach. Details can be found in [4]. Let S1 = {v1 , .., vn },S2 = {u1 , .., um } be two shapes consisting (respectively) of sets of n and m points in R2 . We state the correspondence problem as an optimization problem: Given a measure q(vi , uj ) ∈ R characterizing the quality of correspondences between each given pair of points (vi , uj ) ∈ S1 × S2 , we want to find the configuration gˆ = {(vi1 , uj1 ), .., (vik , ujk )} (consisting of k correspondences) that maximizes the target function: w(g) = Qk l=1 exp (q(vil , jl )). In our setting, each configuration g, or more strictly gi , defines a single particle. q ded3 d2 d1

v1

u1

v2

u2

v3

v4

v1

v2

v3

v4

u1

1

0

0

0

u2

0

1

0.5

0

u3

0

0

0.7

1

u3

Figure 3.

50 10

section 4 shows the advantage of our approach.

50

10

20

30

40

50

x 10 11

Updating weights based on a single established correspondence and different distances, see text for explanation. The values in the table are computed using function u(d1, d2), see also fig.4.

5 10 10

9

15

8

20

7

25

6 5

30

4

35

3 40 2 45

1

50 10

20

30

40

50

Figure 2.

Correspondence weight matrices of the point sets in fig.8. Top left: L, the local constraint matrix, reflecting correspondence based on shape-context. Top right: global matrix Gt in iteration t = 10 for particle fig.8, right, reflecting the neighborhood structure based on established correspondences. Bottom: Qt = L ⊙ Gt .

al. [1] establish correspondences between point sets by optimizing a ’shape-context’ measure, which they introduce. The optimization task based on shape-context is solved without global constraints using the ’Hungarian method’. They achieve impressive results in matching entire shapes with one another. This is a consequence of the fact that the shape context measure already contains regional information, implying global structures. However, this approach fails when partial shapes are matched with one another. We also use shape context to create the local constraint matrix, but replace the Hungarian method with our PF method, which provides the global constraints. In a comparison between the two,

scribes a mapping S1 × S2 → R, represented by a real valued n×m matrix Q = [qij ]. Starting with empty particles (no correspondences), the update of each particle (note: please do not confuse this with the global constraint update)consists of augmentation by a single correspondence, randomly drawn from a distribution defined by Q. During the PF process, Q or, more strictly Qt = L ⊙ Gt (⊙=element-wise multiplication) is generated as an element-wise product of two n × m matrices L and Gt , representing (the consequently separated) local and global constraints, see fig.2. While L is a fixed matrix, computed off-line and based on local feature correspondences, Gt is generated anew at each iteration t of the PF process. In our experiments, we computed L based on the shape context distance measure defined in [1]: intuitively, a high value of lij in L describes a small distance between the shape context of vi and uj , which is a ’good (local) fit’. Computation of Gt is the ’update of global constraint’,as referred to in section 1 and fig.1. Keep in mind that each particle gi creates its own matrix Gt ; we omit the more strict notation Git for improved readability. The global constraint update to adjust the PF system to solve partial-point-set matching task is based on

σ(d1, d2) = k min (d1, d2) + σmin U is defined as: U(d1, d2) = exp



−|d1−d2|2 2σ(d1,d2)2



k and σmin determine the influence of neighborhood nearness: k determines the transition between ’close’

100 1.0 u(d1,d2)

80 0.5

60 40

0 0

20

40

20 d2

60

80

100

0

d1

1 0.9 0.8 0.7 u(d1,d2)

the following simple idea: points close to each other in S1 should be mapped to points similarly close to one another in S2 , and vice versa. This means that we preserve neighborhoods of (already established) correspondences. We note that this does not pose any constraint on points further from already established correspondences. Figure 3 illustrates the motivation: vi and uj are points of two shapes S1 , S2 in R2 . d1,2,3 denote the (Euclidean) distances between v1 and v2,3,4 as well as the respective distances between u1 and u2,3 . We assume an established correspondence (v1 , u1 ), and call v1 and u1 the ’seed points’ for the update. Following the aforementioned idea, the update values G(vi , uj ) should be computed using the difference between their distances to the seed points. Only those point pairs (vi′ , u′j ) should be assigned a low value (close to 0.0), where a) at least one point of {vi , uj } is close to its seed, and b)the distances of vi and uj to their respective seeds is different. All other pairs should be assigned a value closer to 1.0 (remember: the final correspondence matrix Q for the PF process consists of element-wise multiplication of L and Gt , hence a value of 1.0 in Gt effectively means the correspondence has no influence on the global constraint). The table fig.3, right shows update values G(vi , uj ) in accord with the motivation: G(v2 , u2 ) = 1 since d1 = d(v2 , v1 ) = d(u2 , u1 ) and both distances d(., .) are relatively small. G(v4 , u3 ) = 1 for a different reason: both distances are large (= d3 ), hence no statement can be inferred. G(v3 , u2 ) < G(v3 , u3 ) although |d2 − d1 | = |d2 − d3 | since min(d1 , d2 ) < min(d2 , d3 ): a correspondence weight between two points which are both relatively far away from the established points should be less (value closer to 1.0) influenced. The 0.0 values for (v1 , ·) and (·, u1 ) guarantee one to one correspondences. Motivated by this example, we will now derive the update rule for the global constraint matrix Gt → Gt+1 at iteration t. The initial n × m matrix G0 consists entirely of entries with value 1.0(=no global constraint yet). In the PF process, in each iteration t a single correspondence (vit , ujt ) =: (v ′ , u′ ) is established. This correspondence (v ′ , u′ ) defines the seed points for the update. We define a function U : R2 → R that takes as input the Euclidean distances d1 = d(vi , v ′ ), d2 = d(uj , u′ ) of points of S1 and S2 to their respective seeds v ′ , u′ and returns a scalar update value. with

0.6 0.5 0.4 d1=0 d1=30 d1=60 d1=90

0.3 0.2 0.1 0 0

20

40

60

80

100

d2

Figure 4. Update function U (d1, d2). Top: 3d view, bottom: u(d1,d2) for 4 different fixed values d1. The graphs correspond to the colored lines in the left figure. and ’distant’, while σmin determines the initial term of ’closeness’ (note that we do not deal with hard but soft definitions of ’close’ and ’distant’ here). Using different parameters k and σmin for each particle makes the approach scaling invariant. U is a 1-normalized Gauss distribution having a parameterized standard deviation σ(d1, d2). Figure 4 illustrates U, which models the behavior suggested by the idea: stronger differences in points closer to their seeds have a higher impact (=lower value), while more distant points have less influence. Using U(d1, d2) and the seed points v ′ , u′ , the update Gt → Gt+1 is defined by: Gt+1 (i, j) = Gt (i, j)U(d(vi , v ′ ), d(uj , u′ )) In the next iteration t + 1, the PF process selects a new correspondence based on the distribution in Qt = L ⊙ Gt+1 . Values reflecting point correspondences which are inconsistent with the neighborhood system imposed by the established correspondences in a particle converge towards 0.0 fast. The PF will not update (=establish a new correspondence) a particle further if either all points in at least one shape are covered (we

only allow one to one correspondences) or only correspondence values below a threshold Tmin are present in Qt (this is equal to the well known technique of matching badly corresponding points to a dummy point or ’sink’).

4. Experiments: Partial Shape Matching

Shape 2

Shape 1

alization we also assume that the indices of the points are sorted such that each version of shape 1 appears as subsequent points in shape 2 (in reality and in the experiment the columns are randomly shuffled). The matrix L then consists of 2 identical half-matrices, see fig.5. Since [1] optimizes using the Hungarian algorithm, each single correspondence offers 2 solutions, one in each half. The Hungarian will choose one of these randomly, leading to a wrong correspondence system as shown in fig.5,c). In contrast, the global constraints of the PF bias the correspondence selection process early on to a single part (=one half matrix), leading to the configuration in 5,d). The reason for the few outliers is that PF leads to a near optimal solution only. However, the outliers can simply be detected and eliminated in a post processing step. For the following experiments, we used the data set shown in fig.6.

10 20 30

Figure 6.

40 50 10

20

30

40

50

60

70

80

90

100

Top row: chinese words. bottom row: Our data consists of 200 random samples from a skeletonized version of the top row. We will find the partial match between (top row in reading order) a,b; c,d; e,f.

0.2

1

0 0.8 −0.2 0.6

−0.4 −0.6

0.4

−0.8 0.2 −1 0

−1.2 0

0.5

1

1.5

2

0

0.2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

Figure 5.

Partial matching. In reading order: a) The setting: shape 2 contains 2 identical versions of shape 1. b) Matrix L, consisting of 2 identical half matrices. c) correspondences achieved by approach of [1], d) our PF correspondence. The few outliers can easily be eliminated by simple post processing.

0.2 1 0 0.8 −0.2 −0.4

0.6

−0.6

0.4

−0.8 0.2

In our experiments, we mainly compare our approach to Belongie et al. [1]. [1] provides a well known feature descriptor, ’shape context’, which is also the basis for our local constraint matrix L. Therefore we only differ in the optimization strategy: while [1] solves without further constraints, our PF system adds global constraints. This comparison shows the impact of the global constraint, but will also motivate the need for the multiple hypothesis PF framework to handle that constraint. For better intuition, we start with a rather theoretical experimental setting. Here shape 2 consists of 2 exact copies of shape 1, separated sufficiently such that the (local) shape context is not influenced. For better visu-

−1 0 −1.2 0

0.5

1

1.5

2

0.4

0.6

0.8

1

1.2

0.8

1

1.4

1.6

1 0.9

0.6

0.8

0.4

0.7 0.2 0.6 0

0.5

−0.2

0.4

−0.4

0.3 0.2

−0.6

0.1 −0.8 0 −1 0.5

1

1.5

2

0

0.2

0.4

0.6

1.2

Figure 7.

Partial matching of data in fig.6 based on local descriptor ’shape context’. Left column: approach of [1], using the Hungarian method. Right column: our PF approach.

Shape context [1]is a regional feature descriptor, using a radial histogram to describe the density distribu-

1

1

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0

[2]

0

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

Figure 8.

PF outlier detection: left: weak particle, right: strong particle. Configurations like (left) are an effect of a simplified neighborhood description. Multiple hypotheses like in PF solve the problem.

tion of a circular symmetric neighborhood defined by the radius-parameter. The radius is crucial: if it is too small, the descriptor becomes too weak, leading to ambiguities unsolvable for the Hungarian method. If it is too big, it becomes too global, and is therefore not usable for partial shape correspondences. The effect can be seen in fig.7, left column: The correspondences fail to find a match consistent with the part, but find the optimal solution on the entire shape. Our additional global constraint solves this problem: using a rather small radius, the additional global descriptor assists the resulting weak local description. See fig.7, right column. However, a simple global descriptor like the one derived in this paper can fail, if a wrong correspondence is chosen in an early iteration: a wrong neighborhood hypothesis is learned. This problem is solved by the multiple hypothesis PF framework: the (wrong) particle shown in fig.8,left, which results from a prematurely learned local minimum, is dismissed during the PF process in favor of the stronger hypothesis (which developed later), shown in fig.8,right. Since we assumed similar scale of parts and the complete object, all particles used the same set of parameters k and σmin . This kept the number of particles relatively low, only 10 particles were used. The radius for the shape context was set to 0.2 for the PF experiment. We tested different radii for the approach of [1].

5 Conclusion and Outlook We presented a way to model global constraints in connection with the PF system of [4], to establish partial correspondences of point sets with inner structures and showed the resulting superior performance in comparison to a globally unrestricted system on partial matching of chinese characters. The described constrained can be extended to 3D in a straightforward manner, which will be part of future work.

References [1] S. Belongie, J. Malik, and J. Puzicha. Shape matching and object recognition using shape contexts. IEEE Trans.

[3]

[4]

[5]

[6]

[7]

Pattern Analysis and Machine Intelligence, 24:705–522, 2002. L. Gorelick, M. Galun, and A. Brandt. Shape representation and classification using the poisson equation. IEEE Trans. Pattern Anal. Mach. Intell., 28(12):1991– 2005, 2006. Member-Eitan Sharon and Member-Ronen Basri. F. Mokhtarian, N. Khalili, and P. Yuen. Estimation of error in curvature computation on multi-scale free-form surfaces. Int. J. Comput. Vision, 48(2):131–149, 2002. Rolf Lakaemper and Marc Sobel. Correspondences between parts of shapes with particle filters. proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), Anchorage, USA, June 2008. A. Rosenfeld, R. A. Hummel, and S. W. Zucker. Scene labeling by relaxation operations. IEEE Trans. Syst., Man, Cybern., SMC-6(6):420–433, 1976. C. Scott and R. Nowak. Robust contour matching via the order-preserving assignment problem. IEEE Transactions on Image Processing, 15(7):1831–1838, 2006. Yefeng Zheng and David Doermann. Robust Point Matching for Nonrigid Shapes By Preserving Local Neighborhood Structures. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(4):643–649, April 2006.