Elastic Shape Registration using an Incremental Free Form Deformation Approach with the ICP Algorithm Hossam Abdelmunim Computer & Systems Engineering Department, Faculty of Engineering, Ain Shams University, Cairo 11517, Egypt. Email:
[email protected]
Abstract—Shape registration is one of the most challenging problems in computer vision and medical imaging. The process is affected by the way the shape is represented and the form of transformation used to move the source shape. Our paper handles the elastic shape registration by combining the incremental free form deformation (IFFD) with the point-based registration technique using the sum of least squares method. The iterative closest point (ICP) algorithm is used as a criteria to establish point correspondences in each level of the IFFD framework. The free form deformation (FFD) is well known in the literature and works by forming a lattice of control points that can move and hence deform the domain grid points smoothly and uniformly under some shape constrains. The control lattice resolution is increased step by step to achieve a satisfactory deformation of the source shape to exactly match the target boundaries. Our point-based registration is based on least squares that measure the Euclidean distance between source and target boundaries in addition to the shape constrains. The minimization gives a closed form solution of the lattice control points positions. Promising results will be demonstrated for closed and open shapes and structures. The approach can also work for structures that contain multiple parts without any problems. Keywords-Shape Representation; Shape Anatomical Structures; ICP Algorithm;
Registration;
I. I NTRODUCTION Shapes registration [1] is an important complex problem in computer vision, computer graphics and medical imaging. It has been handled in different manners in many applications like shape-based segmentation, shape recognition, and tracking. The shape registration problem is formulated such that a transformation that moves a point from a given source boundary to another target shape according to some dissimilarity measure [2], needs to be estimated. The dissimilarity measure can be defined according to either the curve or the entire region enclosed by the curve. The iterative closest point algorithm was proposed in [3]. The approach is based on finding the correspondence based on the minimum distance criterion. Different shape registration methods based on this technique are provided
Aly A. Farag Computer Vision and Image Processing Laboratory, Electrical & Computer Engineering Department, University of Louisville, KY 40292, USA. Email:
[email protected]; URL: www.cvip.uofl.edu
in the literature (e.g. [4]). It is used in registering either 2D or 3D objects. Different shape registration approaches were proposed in the literature [5], [6], [7]. These approaches suffer from various problems, including scale variations and dependence on initialization. Also local deformations can not be covered. Image boundaries are used for registration in [8]. Images are registered using the iterative closest point algorithm to match these boundaries(shapes). A hierarchical B-spline technique is used to match these surfaces. The algorithm depends mainly on the number of points that represent the source and target images in addition to the number of control points which create a huge matrix equation system. The solution of such equations is sometimes impossible. In [9], internal and external contours of an object are extracted from the image to be matched with a database to mark joints of the human body. Edge detectors are important to extract the features for matching in such cases. A shape registration framework was proposed in [2] using the signed distance map representation. A global transformation which includes homogeneous scale, rotation, and translation, is used to align planar shapes. The local deformations are handled, by minimizing sum of square differences of the shapes representation. The form of local deformations does not give the desired local correspondences. In [10], a 3D shape-segmentation approach was proposed which included a shape registration process. A shape model was built from a set of training shapes using signed distance functions (the conventional representation). A level set function evolved, thus minimizing the shape alignment and the intensity gray level energies. Using a simple global transformation with homogenous scales creates a problem when gathering training shapes from different patients scans. The registration process fails when the target shape requires inhomogeneous scales. An interesting shape registration approach was proposed recently in [11] using maximization of mutual information. Images of signed distance maps of the registered shapes are involved. An affine transformation is used to handle the global registration problem through a gradient descent
optimization technique. The local deformations are represented by the incremental free form deformations method. The control points minimize a sum of squared differences energy. Using such a method makes the problem complicated since a large number of control points may be required to cover detailed local deformations. The gradient descent optimization may have a problem with these large number of variables associated with different levels of the IFFD. Vector level set functions are used to represent and register shapes in [12]. The vector components represent the vector projections from any point in space to the nearest point on the shape boundary. A positive sign is used to mark points inside the shape while outside points are given negative values. A simple dissimilarity measure is used to handle the problem of inhomogeneous scaling in the shape registration framework used to solve the shape-based segmentation problem. In this paper, we introduce an elastic shape registration technique using the iterative closest point algorithm. The motion of curves to their targets is described by, the incremental free form deformations. Control points are calculated by a closed form solution for energy minimization as a point-based registration problem. Sum of least squares and smoothing constrains are used to formulate an energy function which is quadratic in terms of control points deformations. We will show that the multi level resolution control lattice approach gives exact and accurate correspondences compared to the single high resolution level case. The minimization of the energy function results in a linear system of size dependence only on the number of control points, which makes the solution possible and reasonable. Different synthetic and real shapes registration will be demonstrated to illustrate the efficiency of the approach. The rest of this paper is organized as follows: Point-based registration is formulated in Sec. II, a prelude to the coarse to fine strategy is given in Sec. III, a quick overview of the ICP algorithm is demonstrated in Sec. IV, the main algorithm steps are listed in Sec. V, experimental results and validation are in Sec. VI, and conclusion in Sec. VII. II. P OINT- BASED S HAPE R EGISTRATION Our objective is to find a function that gives the point correspondences between the two given domains (source and target). Let us define the 2D shape registration as follows: A map Cs (p) : [0, 1] ∈ R → R2 defines a planar source curve with a parameter p. The target is defined by Ct (p) : [0, 1] ∈ R → R2 . The cartesian coordinates of the point vector can be defined by C(p) = [x y]T where 0 ≤ x ≤ X and 0 ≤ y ≤ Y . Assume that Ct (p) is the corresponding point of Cs (p) (the criteria for finding correspondences will come in the following sections). The output will be a C 0 function f : R2 → R2 with f (Cs (p)) = Ct (p), ∀p ∈ [0, 1]. Different interpolation functions have been proposed to
handle this problem [8]. We choose the free form deformation F F D model, based on B-splines [13], [14], which is a powerful tool for modeling deformable objects and has been previously applied to the tracking and motion analysis problems. The basic idea is to deform the shape by manipulating a mesh of control points. The resulting deformation controls the shape of the object and produces a smooth and continuous transformation. Consider a lattice of control points P = Pm,n ; m = 1, . . . , M. n = 1, . . . , N , each point on the source shape will have the following form of deformation: L(p) =
3 3
Bk (u)Bl (v)δPi+k,j+l
(1)
k=0 l=0
where δP = δPm,n is the control point deformation, i = (x.(M − 1)/X) + 1, j = (y.(N − 1)/Y ) + 1, u = x.M/X − (x.M/X), v = y.N/Y − (y.N/Y ), and the spline basis functions (B) are defined in [14]. So the cubic B-spline is used as an approximation function for our interpolation problem. We propose the following energy to measure the difference between the deformed contour and its target points based on the Euclidean distance: E(δP) =
0
1
Cs (p) + L(p) − Ct (p)2 dp
(2)
Also, we need to avoid any undesired distortion in the shape and preserve the regularity of the registration grid flow. Another weighted term (by λ ∈ R+ ) is added for smoothness constrain as follows [15]: E(δP) =
1 0
Cs (p) + L(p) − Ct (p)2 dp + λ + Lpp 2 )dp
1 0
(Lp 2
(3) where Lp and Lpp are the first and second derivatives respectively of the deformation vector with respect to the parameterizations p. The above objective function is required to be minimal to calculate the control points locations and hence the deformation field at each point in the domain. Gradient descent and calculus of variation are used to optimize the given function as follows: ∂E ∂δP
=2
1 0
1 ∂L ∂L (L(p) − C(p))T ∂δP dp + 2λ 0 (LTp ∂δPp T ∂Lpp + Lpp ∂δP )dp
(4) where C(p) = Ct (p) − Cs (p). A detailed illustration for the local deformation derivatives are found in [11]. By setting the above equation to zero, we get a linear system of the control points coordinates: 1 T ∂L 1 T ∂Lp 1 T ∂L 0 (C(p)) ∂δP dp = 0 L ∂δP dp + λ 0 (Lp ∂δP (5) ∂Lpp + LTpp ∂δP )dp
The left hand side is free from the coordinates of the control points while the other side is linearly a function of these unknowns. The following linear equation holds: ΨΘ = Λ
(6)
where Θ = [δPx1,1 , . . . , δPxM,N , δPy1,1 , . . . , δPyM,N ]T and x, y are the coordinates of the embedding space. The other matrices elements are calculated as follows: 1 0
∂L (Lr,c )T ∂θ dp + λ r
Λr =
1 0
1 0
∂L
p T ((Lr,c p ) ∂θr T ∂Lpp + (Lr,c pp ) ∂θr )dp
∂L (C(p))T ∂θ dp r
4
x 10 2.5
(7) 2
(8)
where θr ∈ Θ and Lr,c stands for the coefficient of the control point vector calculated from the cubic spline interpolation. Row and column are represented by r and c respectively. The resulting matrix equation size depends on the number of control points and the space dimensions. Size of the data points of the curve does not have any impact on the matrices sizes which guarantees its computational efficiency (this is a huge advantage over the work proposed in [8] where number of linear equations is equal to number of data points). Also note that the smoothing constrains do not add extra load to the solution of the matrix equation. III. A C OARSE TO F INE S TRATEGY (I NCREMENTAL F REE F ORM D EFORMATION ) The above objective function is required to be minimized by moving the control points to get the correct correspondence over shape boundaries. A very small error can be achieved when using a high resolution control lattice because the number of degrees of freedom increases. However this is not enough. Such sudden movement will result in unnecessary cross overs of the domain grid lines and the registration process will be meaningless. This will result in changing and corrupting the object topology. A better way is to move the grid step by step towards the target. To illustrate the problem, we formed a contour which represents sides of a square (this will stand for the source). Another contour is obtained by rotating the first one by an angle of π/2 (this will stand for the target). Correspondences between the source and target shapes are established by the rotation effect. A control lattice is constructed and computed using the above formula for the point-based registration. When using a relatively high resolution of 12 × 12 lattice, foldovers occur as shown in Fig. 1 (top left image). To avoid this, a coarse to fine strategy is used (equivalent to the incremental free form deformations used in [11]). We start with a resolution of 4 × 4 and solve for the deformation. Iteratively we increase the resolution as 5 × 5, 6 × 6, and so on and so forth. In each step, the control points positions
1.5 Energy
Ψr,c =
1
0.5
0
5
10
15
20
25 30 Lattice Resolution
35
40
45
Figure 1. Grid local deformation when using a control point resolution of 12 × 12 is given in the top left image while multi level deformation to the right. The graph gives the energy calculated at each level of resolution where any value N in the x-axis means a resolution of N × N . The steady state energy value is 1.6918e−12 .
are computed and the contour moves to the new position until a satisfactory error distance is obtained. The result is smooth and the correspondence is achieved accurately. The final grid deformation is shown in the right image. This process handles extremely well the error and gives impressive infinitesimal energy function and smooth grid deformations at the same time. IV. I TERATIVE C LOSEST P OINT (ICP) A LGORITHM The iterative closest point (ICP) algorithm is a widely applied method for the registration of two data sets of points. For two Euclidean point sets, assume that the source point set is Cs and the target point is Ct . The source point set is assumed to change or deform according to a certain known form of transformation and the target will remain fixed. The ICP algorithm can be described as follows [3]: 1) Initialize transformation parameters. 2) For every point in Cs determine its closest point in Ct . 3) Compute the transformation parameters that minimize the sum of square differences between the correspondences.
(a)
(b)
(1)
Figure 3. Two shapes registration comparison: The source is given in the first row to the left shown in red while the target in blow. The use of the ICP gives directly the points correspondences shown in the right image. Our resulting dense point correspondences are demonstrated in the bottom.
(2)
4) Based on the new parameters, update the source points Cs to the new positions. 5) If the change in the source points is less than a certain threshold terminate, otherwise go to step #2 Transformation parameters will represent the way that the source data set moves to be registered to the target. Since we are focusing on elastic motion, all shapes are assumed to be globally registered. Only local deformations will be handled in this paper. The conventional ICP drawbacks are well known. For elastic registration, it is difficult to find a deformation function to keep the topology of the registered objects. Another issue is that the closest point is not unique. According to [8], a coarse to fine strategy using the B-splines overcomes these problems for covering the local deformations. Below we will use a similar algorithm with our closed form solution for the point-based registration problem.
(3) 8 7
Average Error per Pixel
6 5 4 3 2 1 0
V. A C OARSE TO F INE S TRATEGY WITH THE ICP 10
15
20 25 Lattice Resolution
30
35
40
Figure 2. Illustration of point-based registration example: (1.a) MRI image, (1.b) Deformed image, (2.a) Source edge points, (2.b) Target edge points, (3.a) Final deformed image grid, and (3.b) The checkerboard image. The average error per pixel is plotted versus the resolution level and given at the bottom graph.
Now we will illustrate the whole algorithm for shape registration. Assume that Nxi × Nyi is the resolution of the control point lattice initially denoted by i. The resolution at any time will be Nx × Ny . The basic steps will be as follows: 1) Set Nx = Nxi and Ny = Nyi 2) Construct a control lattice of size Nx × Ny and initialize its point deformation vectors to zeros. 3) For each point in Cs , determine its corresponding closest point in Ct .
4) Solve Eq. 6 to get the new deformation of each control point and hence calculate its new position. 5) Based on the new lattice, update the source points Cs by calculating the new deformation field using Eq. 1. 6) Set Nx = Nx + 1 and Ny = Ny + 1. 7) Check the stopping criteria. Either the objective function goes below a certain threshold or a number of maximum resolution levels is reached, otherwise go to step #2. VI. E XPERIMENTAL R ESULTS We start our experimental results by demonstrating an example for the point-based registration algorithm. We deformed an MRI image by constructing a 12 × 12 lattice of control points over its grid. The control points are moved randomly in a certain direction resulting in deforming the image grid points together as shown in Fig. 2 (1.b). Point correspondences are constructed by taking the edge contours of the original and deformed images (point illustration is given in (2.a) and (2.b)). So each point is associated with its target priori because the control points new position are already known. Now, the situation is reversed. Given the edge contours of the source and target images, we need to calculate the deformation grid using our point-based algorithm. Our approach successfully covers the deformation at each point with the coarse to fine strategy. A final average pixel error of 0.2 is achieved. As we notice from the graph on the bottom, the average error dramatically gets reduced when increasing the resolution of the control lattice. The deformed image is gathered with its target in a checkerboard illustration as shown in (3.b). Having a closer look at the ventricles shows that the deformation is very accurate since, the two parts coming from different images have no transition. Edges are connected along the squares in all parts of the checkerboard image. To show the difference between our approach and the conventional ICP algorithm ([3], [4]), we demonstrate the registration of two humanoid models in Fig. 3. The ICP fails to establish correct point correspondences. Our approach does the job efficiently because the whole grid moves slowly and step by step towards the target. In all experiments, shapes are priori registered globally using the approach developed in [12] for shape global registration. Different scales, rotations, and translations are estimated for each case. Different elastic shape registration experiments are demonstrated in Fig. 4. The algorithms demonstrated in Sec. V is used for registration. In all experiments, an initial lattice resolution of 7 × 7 is established and then the level is increased until a satisfactory deformation is achieved. Examples (1) to (4) are dedicated to teeth deformations. The first registration example does not have complicated local difference details between the given shapes. This does not require a high level of resolution. More complicated differences in the tags and roots areas
(1)
(2) (a)
(b)
(c)
(d)
Figure 5. Different elastic registration examples of open shapes (source contour is given in red, target contour is drawn in green, and deformed contour is shown in blue): (a) Initial position, (b) Intermediate stage of point correspondences, (c) Final correspondences, and (d) Final grid deformation.
(1)
(2) (a)
(b)
Figure 6. Different elastic registration examples of shapes containing multiple parts: (source contour is given in red, target contour is drawn in green, and deformed contour is shown in blue) two different brain structures are given: (a) Initial position (b) Final correspondences.
can be noticed in examples (2), (3), and (4). Large number of resolution levels are used to successfully achieve these local deformations. Another more complicated deformation example is shown for the fish in (5). Investigating the point correspondence in each case, we find that: our algorithm gives exact physical dense correspondences and the grid deformations do not have crossovers which shows the necessity of the coarse to fine strategy and the smoothness constrains. Our approach does not only work for closed shapes but it can also handle open structures registration problem efficiently as shown in Fig. 5. Another issue is the registration of shapes that include multiple parts. The registration of two brain structures, for example, is shown in Fig. 6. Open and multiple parts shapes are parameterized in different ways. End points of the open shape are different C(0) = C(1) while in closed shapes C(0) = C(1). For the multiple parts case, each individual curve is parameterized independently then gathered together to handle the entire shape.
(1)
(2)
(3)
(4)
(5) (a)
(b)
(c)
(d)
Figure 4. Different elastic registration examples of closed shapes (source contour is given in red, target contour is drawn in green, and deformed contour is shown in blue): teeth examples are given from (1) to (4) and a fish example is shown in (5). (a) Initial position, (b) Intermediate stage of point correspondences, (c) Final correspondences, and (d) Final grid deformation.
VII. C ONCLUSION AND F UTURE R ESEARCH We have proposed an efficient and simple approach for the elastic shape registration problem. Source and target shapes are represented explicitly and registered to each other by a hierarchical B-splines deformation algorithm. A closed form solution of the control points is obtained to minimize an energy function. The solution does not need huge amount of processing since the linear system size is a function of the number of control points only. The algorithms start with a small size control lattice which is increased step by step to handle the deformations. Exact physical correspondences are achieved through the demonstration of different teeth registration examples. Roots and tags deformations are covered efficiently. Another complicated example is given for the fish registration. The approach can work with open structures in addition to structures having multiple objects without difficulty. Only 2D examples are demonstrated but future work will be geared towards considering 3D applications. Image registration is of great interest and will be considered for future research. R EFERENCES [1] R. Veltkamp and M. Hagedoorn, “State of the Art in Shape Matching,“Technical Report, UU-CS-19999-27, Utrecht University, Sept. 19999. [2] Nikos Paragios, Mikael Rousson and Visvanathan Ramesh, “Matching Distance Functions: A Shape-to-Area Variational Approach for Global-to-Local Registration,“European Conference in Computer Vision, Copenhagen, Denmark, Jun. 2002. [3] P. Besl and N. Mckay, “A Method for Registration of 3-D Shapes,“IEEE Tr. on PAMI, 14(2):239–256, 1992. [4] Z. Zhang, ”Iterative Point Matching for Registration of Freeform Curves and Surfaces,” Ph.D. International Journal of Computer Vision, vol. 13, no. 2, pp. 119152, 1994. [5] I. Cohen and I. Herlin, “Curve Matching Using Geodesic Paths,“In IEEE CVPR, pp 741–746,Santa Barbara, USA, 1998. [6] A. Fitzgibbon, “Robust Registration of 2D and 3D Points Sets,“In Proceeding of The British Machine Vision Conference (BMVC), vol. 2, pp 411–420,University of Manchester, UK, 2001. [7] D. Kozinska, O. Tretika, J. Nissanov, and C. Ozturk, “Multidimensional Alignment Using the Euclidean Distance Transform,“Graphical Models and Image Processing, pp 6:373– 385,1997. [8] Z. Xie and G. E. Farin, “Image Registration Using Hierarchical B-Splines,“IEEE Transaction on Visualization and Computer Graphics, Vol. 10, NO. 1, 2004. [9] G. Mori and J. Malik,, “Recovering 3D Human Body Configurations Using Shape Contexts,“PAMI, VOL. 28, NO. 7,pp. 1052–1062, July 2006.
[10] M. Rousson, N. Paragios and R. Deriche. ”Implicit Active Shape Models for 3D Segmentation in MRI Imaging,” Medical Image Computing and Computer Assisted Intervention (MICCAI), Part 1, pp 209–216, Saint-Malo, France, September 2629, 2004. [11] Xiaolei Huang; Paragios, N.; Metaxas, D.N., “Shape registration in implicit spaces using information theory and free form deformations ,“TPAMI, Vol. 28, Issue 8, Aug. 2006 Page(s):1303-1318. [12] H. E. Abd El Munim and A. A. Farag, “A Shape-Based Segmentation Approach: An Improved Technique Using Level Sets ,“ Tenth IEEE International Conference on Computer Vision (ICCV), Beijing, China, Oct. 17-20,2005, pp. 930-935. [13] T. Sederberg and S. Parry, “Free-Form Deformation of Solid Geometric Models,“in ACM SIGGRAPH, 1986, pp. 151160. [14] D. Rueckert, L. Sonoda, C. Hayes, D. Hill, M. Leach, and D. Hawkes, “Nonrigid Registration Using Free-Form Deformations: Application to Breast MR Images,“IEEE Transactions on Medical Imaging,vol. 8, pp. 712721, 1999. [15] Tim McInerney and Demetri Terzopoulos, ”Deformable Models in Medical Image Analysis: A Survey,” In Medical Image Analysis, 1(2):91108, 1996.
