Improved Robust Point Matching with Label ... - Semantic Scholar

Improved Robust Point Matching with Label Consistency Roshni Bhagalia† , James V. Miller† and Arunabha Roy∗ † GE

Global Reseach, Niskayuna, NY USA; ∗ GE Global Reseach, Bangalore, India. ABSTRACT

Robust point matching (RPM) jointly estimates correspondences and non-rigid warps between unstructured point-clouds. RPM does not, however, utilize information of the topological structure or group memberships of the data it is matching. In numerous medical imaging applications, each extracted point can be assigned group membership attributes or labels based on segmentation, partitioning, or clustering operations. For example, points on the cortical surface of the brain can be grouped according to the four lobes. Estimated warps should enforce the topological structure of such point-sets, e.g. points belonging to the temporal lobe in the two point-sets should be mapped onto each other. We extend the RPM objective function to incorporate group membership labels by including a Label Entropy (LE) term. LE discourages mappings that transform points within a single group in one point-set onto points from multiple distinct groups in the other point-set. The resulting Labeled Point Matching (LPM) algorithm requires a very simple modification to the standard RPM update rules. We demonstrate the performance of LPM on coronary trees extracted from cardiac CT images. We partitioned the point sets into coronary sections without a priori anatomical context, yielding potentially disparate labelings (e.g. [1,2,3] → [a,b,c,d]). LPM simultaneously estimated label correspondences, point correspondences, and a non-linear warp. Non-matching branches were treated wholly through the standard RPM outlier process akin to non-matching points. Results show LPM produces warps that are more physically meaningful than RPM alone. In particular, LPM mitigates unrealistic branch crossings and results in more robust non-rigid warp estimates. Keywords: labeled point matching, robust point matching, group labels, label entropy, non-rigid warps

1. INTRODUCTION Finding the optimal nonrigid transformation (warp) and correspondences between two candidate point-sets is a challenging problem in computer vision. The robust point matching1 (RPM) algorithm has been used effectively to jointly compute both the warp and correspondences between pairs of unstructured point-sets. However, a vast number of computer vision algorithms involve labeled or attributed point-sets; e.g., points (or landmarks) on the surface of the brain can be grouped into the four cortical lobes or points drawn from the surface of a face can be labeled as belonging to the different facial features: nose, lips, cheeks etc. In such cases, we want the estimated warp and correspondences to be label consistent, i.e., points from the nose in both datasets (if present) should get mapped onto each other. Enforcing label consistency by injecting this additional attribute information in the point matching algorithm will not only result in more realistic warp estimates but will also make the algorithm more robust to spurious noise induced matches. Studholme et al.2 show that including label information can improve warp estimates obtained using intensity-based image registration. We extend the RPM objective function using a Label Entropy (LE) term. We define LE in a way that quantifies the degree of the match between distinct labels in the two point-sets and encourages correspondences and warp estimates that are label consistent. The resulting Labeled Point Matching (LPM) algorithm translates to a very simple modification to the standard RPM update rules. LPM allows the two point-sets to have an unequal number of distinct labels, with each point-set containing possibly different label identifiers. Groups of non-matching labels are treated wholly through the RPM outlier process similar to the mechanism used to handle non-matching points in standard RPM. This work is not being considered and has not been submitted for publication or presentation elsewhere. Authors can be reached at {bhagalia, millerjv, arunabha.roy}@research.ge.com

Results comparing the performance of LPM and RPM to match labeled point-sets belonging to coronary trees extracted from cardiac CT data show that LPM produces warps that are more physically meaningful than RPM alone. Specifically, LPM was able to effectively mitigate unrealistic branch crossings and produced more robust non-rigid warp estimates.

2. METHODS We begin the LPM formulation by reviewing the RPM objective function with a few minor changes in nomenclature in preparation for the extension to LPM in Section 2.1. Consider two labeled, un-matched point-sets: {xi }N i=1 Q K with P distinct labels {ap }P p=1 and {vj }j=1 with Q distinct labels {bq }q=1 . Let M be the unknown N + 1 × K + 1 matrix defining the correspondences between the two point-sets. The extra M are used to PK+1row and column inP N +1 account for outliers. M has elements {mji }j=K,i=N , s.t. m ∈ [0, 1], m = 1, ∀i and ji ji j=1,i=1 j=1 i=1 mji = 1, 3 ∀j. We limit our presentation to warps defined by Thin Plate Splines (TPS), S(v; D, W ) = v.D + φ(v).W , parameterized by unknown affine transform matrix D and warp coefficient matrix W . RPM finds the unknown matrix M and the TPS warp S(.; D, W ) by using softassign4 and deterministic annealing to minimize the following objective function: ERPM (M, W, D) =

N X K X

mji kxi − S(vj ; D, W )k2 + λ1 trace(W T ΦW ) + λ2 trace[(D − I)T (D − I)]

i=1 j=1

−ǫ

K N X X

mji − τ

i=1 j=1

K N X X

−mji log(mji ),

(1)

i=1 j=1

where, τ , ǫ, λ1 and λ2 are RPM tuning parameters. Φ is a square matrix related to the TPS basis function. To minimize (1), RPM iterates between keeping M (alternately S) fixed and solving for S (alternately M ) until convergence. For further details on the RPM method cf. Chui et al.1 The above RPM formulation discards all the label information associated with the two point-sets, viz. {ap }P p=1 and {bq }Q . Consequently, the resulting match is purely a function of the geometric locations of the point-sets. q=1 Constraining our point matching scheme to enforce label consistency will result in more realistic matches and aid in mitigating the effect of noise in the point-sets’ geometric information. In the following section, we design a metric that encourages label consistency and extend (1) to develop the LPM algorithm.

2.1 Labeled Point Matching At any iteration of the RPM algorithm mji is a function of how well point vj from one point-set maps onto point PN +1 xi from the other point-set. Further noting that i=1 mji = 1 and assuming one-to-one mappings between points in both sets (after excluding outliers), we can think of mji as the probability that point vj will be matched to point xi . We define a measure lqp ∈ [0, 1], of the likelihood that the sub-group under label bq in one point-set will be matched with the sub-group under label ap in the other point-set, as: lqp =

N X K X

Ip (i)Iq (j)mji ,

∀p = 1, 2, . . . P ; q = 1, 2, . . . Q,

(2)

i=1 j=1

where the indicator functions I.(.) are defined as:  1 if xi has label ap , Ip (i) = 0 otherwise

and

Iq (j) =



1 if vj has label bq , 0 otherwise.

(3)

The larger the value of lqp the better the match between sub-groups labeled bq and ap respectively. We can now define our Label Entropy metric. To encourage label consistency, i.e., to nudge all points with a given label

bq , q = 1, 2 . . . Q to map onto points belonging to a single label or smallest possible subset of distinct labels from q=Q,p=P {ap }P p=1 , the set {lqp }q=1,p=1 should have low entropy. That is, the ‘Label Entropy’ (LE) given by LE =

Q P X X

−lqp log(lqp ),

(4)

p=1 q=1

should be minimized. Finally, adding this term to the objective function in (1) yields our extended Labeled Point Matching (LPM) cost function: K N X X

ELPM (M, W, D) =

mji kxi − S(vj ; W, D)k2 + λ1 trace(W T ΦW ) + λ2 trace[(D − I)T (D − I)]

i=1 j=1

−ǫ

N X K X

mji − τ

i=1 j=1

N X K X

−mji log(mji ) + β

Q P X X

−lqp log(lqp ),

(5)

p=1 q=1

i=1 j=1

where β is an additional tuning parameter of the LPM algorithm. The new LE term is completely independent of the TPS warp S. That is, the standard RPM warp estimate update rule remains unchanged. The iterative update equations to compute correspondence matrix M can be derived by keeping S fixed and differentiating (5) with respect to the elements of M . Solving the resulting system of N K equations in N K unknowns yields the following update rules; for each (j, i) such that Ip (i) = Iq (j) = 1: mji = cji ×

X K N X

Ip (i)Iq (j)cji

i=1 j=1

 βτ −1

where, cji = exp



 kxi − S(vj )k2 + ǫ + β +1 − τ

(6)

and the indicator functions I.(.) are given by (3).

3. RESULTS To compare the performance of RPM and LPM, we derived labeled point-sets from coronary trees for six different phases of a single cardiac CT exam. Distinct labels were assigned to points within different segments of each extracted coronary tree. The assigned labels had no a priori context, i.e., each tree had potentially distinct label identifiers without corresponding anatomical segment names. Furthermore, the number of segments identified in each tree were disparate, i.e., some trees had extra branches missing from other trees.

1 − Fraction of Label Crossings

Thirty different point-set parings were generated from the six labeled point-sets, by successively treating each point-set as the ‘target’ and mapping the remaining five point-sets to this target. The point-set pairs were pruned using an automated method prior to matching to reduce the number of obvious outliers. Point correspondences 1 0.8 0.6 0.4 0.2 0 0

LPM RPM 5

10 15 Crossover Threshold

20

25

Figure 1. Comparison between the number of label consistent matches obtained from RPM and LPM.

and warps between the thirty pruned point-set pairs were estimated using both RPM and LPM. To ensure a fair comparison between the two methods, the same tuning parameter settings of ǫ, λ1 , λ2 , initial annealing temperature τ0 and the annealing temperature reduction rate were used for both RPM and LPM. K Final correspondences were estimated as {vj 7→ xψ(j) }K j=1 with labels {Attrib(vj ) 7→ Attrib(xψ(j) )}j=1 , where ψ(j) = arg maxi mji . Label bq was assumed to correspond to label ap∗ = mode {Attrib(xψ(j) ); j s.t. Attrib(vj ) = bq }. Points labeled bq that did not map onto points labeled ap∗ were treated as ‘cross-over’ points and bq was said to have a label crossing if more than ‘Crossover Threshold’ percent of its points were cross-over points. Fig. 1 shows what fraction of label correspondences did not contain crossings for RPM and LPM across all thirty trials, for six crossover threshold values. LPM results in significantly more label consistent matches than RPM.

(a) Original point-sets a1

a2

a3

a4

(b) RPM: Matched point-sets a5

a6

a7

a1

a8

0.2

b1

a2

(c) LPM: Matched point-sets a3

a4

a5

a6

a7

a8

b1 0.25

b2

b2 0.15

0.2

b3

b3

b4

b4

0.15

0.1

b5

b5

b8

0.1

b6

b6 0.05

b7

b8

b9

b9

b10 b11

b10 b11

0

b12

b7

0.05

b12

(d) RPM: Correspondance matrix

(e) LPM: Correspondance matrix

(f) RPM: Matching connections

(g) LPM: Matching connections

Figure 2. Comparison between the label consistency and warp estimates for point matches obtained from RPM and LPM.

The final point matches obtained using both methods for one candidate point-set pair are depicted in Figs. 2(b) and 2(c). The warp estimated by LPM is more ‘well-behaved’ than that obtained using RPM. To highlight the effect of LE on M , the correspondence matrices in Figs. 2(d) and 2(e) are re-ordered such that matrix elements

corresponding to points in sub-group bq , q = 1, 2 . . . Q are sorted by point coordinates and grouped together along the rows and elements corresponding to sub-group ap , p = 1, 2 . . . P are sorted and grouped along the columns. For ease of visualization, the re-ordered matrices are partitioned into grids of Q × P sub-matrices, {µqp }Q,P q=1,p=1 . P Thus, label bq is less prone to cross-over points if subset {µqp }p=1 is sparse, with many null sub-matrices and very few diagonal sub-matrices. In particular, bq will have a one-to-one mapping to ap∗ if µqp∗ is a diagonal sub-matrix and µqp ≈ 0; ∀p 6= p∗ . RPM shows several many-to-many label correspondences in M (cf. Fig. 2(d) ), while LPM shows more one-to-one label correspondences (cf. Fig. 2(e)). Similarly the point-matching connections in Figs. 2(f) and 2(g) indicate that RPM has many more label crossings than LPM.

4. CONCLUSION We have developed a method to find unknown correpondences and warps between pairs of labeled point-sets, while maintaining label consistency. Our Labeled Point Matching (LPM) algorithm can simultaneously estimate label correspondences, point correspondences, and a non-linear warp. LPM was developed by extending the standard RPM objective function using a Label Entropy term; the resulting algorithm translates to a very simple modification to the standard RPM update rules. Results show that LPM can effectively mitigate unrealistic branch crossings and results in more robust non-rigid warp estimates than RPM. Finally, we note that improvement in LPM is a function of the accuracy of the point-sets’ labeling strategy; lables that capture group associations incorrectly will degrade the performance of LPM.

REFERENCES [1] Chui, H. and Rangarajan, A., “A new point matching algorithm for non-rigid registration,” Computer Vision and Image Understanding 89(2-3), 114–141 (2003). [2] Studholme, C., Hill, D., and Hawkes, D., “Incorporating connected region labelling into automatic image registration using mutual information,” IEEE Workshop on Mathematical Methods in Biomedical Image Analysis 0, 00–23 (1996). [3] Bookstein, F. L., “Principal warps: thin-plate splines and the decomposition of deformations,” IEEE Transactions on Pattern Analysis and Machine Intelligence 11(6), 567–585 (1989). [4] Gold, S. and Rangarajan, A., “A graduated assignment algorithm for graph matching,” IEEE Transactions on Pattern Analysis and Machine Intelligence 18(4), 377–388 (1996).