Region-Based Segmentation via Non-Rigid Template Matching Kinda Anna Saddi1,2 , Christophe Chefd’hotel1 , Mika¨el Rousson1 and Farida Cheriet2 1
Siemens Corporate Research, Princeton NJ, 08540, USA. Ecole Polytechnique de Montr´eal, Montr´eal Qc, H3T 1J4, Canada.
2´
[email protected]
Abstract We propose a new region segmentation method based on non-rigid template matching. We align a binary template to an image by maximizing the likelihood of intensity distributions within a region of interest and its background. The intensity model and the corresponding a posteriori distributions are estimated and updated throughout the alignment. The geometric deformation of the template is based on a fluid registration model. Unlike contour-based segmentation techniques, this registration framework allows for a global regularization of the template variations. This enables the segmentation of irregular shapes while avoiding leaks. We apply our method to the segmentation of the liver in computed tomography images, a challenging task due to the high inter-patient variability in the shape of this organ. We show that our segmentation results are equivalent or superior in accuracy to results obtained using existing techniques based on 3D shape models.
1. Introduction Image segmentation is the process of partitioning an image into different regions. The goal is generally to obtain a higher-level description of the image content. For instance, in medical imaging, the segmentation of anatomical structures is a key element for computer-aided diagnosis and image-guided therapies. A large number of segmentation algorithms have been developed, most of them are derived from energyminimization principles. For example, variational methods have been used to obtain algorithms based on contour evolutions. Such techniques are derived from the minimization of cost functions, and yield partial differential equations (PDE) that characterize the evolution of a contour describing the segmented region [14]. In this framework, level set methods have been introduced, where contours are rep-
978-1-4244-1631-8/07/$25.00 ©2007 IEEE
resented implicitly [19]. They allow topological changes during the segmentation process. In this context, cost functions based on statistical region models have proved very efficient. The contour evolves by fitting statistical intensity models within the segmented object and its background, rather than fitting the contour to local edge information. Chan and Vese [3] proposed to use the average intensity to distinguish different regions, based on the Mumford-Shah segmentation model [18]. Statistics of the regions are estimated continuously throughout the evolution. A more general statistical formulation using a maximum likelihood principle was also presented by Zhu and Yuille [23]. In these algorithms, the regularity of the segmented region is obtained by penalizing contour points with high curvature during the evolution. This often translates into a mean curvature term in the corresponding evolution equations. More robust global constraints have been proposed to deal efficiently with noise and the lack of strong edges. These approaches incorporate additional information based on shape models, and require a modeling or training step before the actual segmentation takes place. These ideas find their root in the active shape models first introduced by Cootes et al. [6]. Leventon et al. [16] proposed to incorporate shape information into a contour evolution algorithm. However, modeling shape information is challenging. To model a shape accurately, a large number of training samples should be considered, and the model might still be incomplete due to local shape changes specific to each segmented object. Alternatively, segmentation approaches combined with registration techniques have been proposed. Image registration consists of finding an optimal transformation that aligns two images. This process is designed to maximize a similarity measure [13]. In this case, the idea is to register a pre-segmented image (the atlas) to a study image. Segmented structures of the atlas are then mapped to the study using the estimated transformation [11]. This method
is independent of the shape and appearance of the regions to be segmented. Another strategy is to combine atlas segmentation and image registration. Yezzi et al. [22] solved this problem sequentially using active contours. However, Pohl et al. [20] proposed a joint model describing the relationship between atlas registration, intensity correction, and image segmentation. In this work, we propose a novel approach where we non-rigidly align a binary template of the region of interest to an image. The likelihood of intensity distributions inside and outside the template is used as similarity criterion. The probability density functions required in the statistical formulation are estimated continuously throughout the template matching process. We deform the initial template using different deformation models ranging from rigid to high-dimensional dense transformations. Our non-rigid registration is based on a fluid model, where a deformation field is computed by composition of small displacements. This captures global and local deformations of the template that yields the desired segmentation. In contrast to [22], we do not combine the segmentation and registration, but instead we directly derive the segmentation energy with respect to the deformation. This approach has a number of attractive properties. First, this method is unsupervised, no prior extraction of information of intensity or shape variation is needed. Second, unlike contour-based techniques, this framework offers a more global regularization of the shape over the entire image domain (regularization on the deformation). This enables the segmentation of irregular shapes while avoiding leaks, a problem often observed with surface evolution techniques. We validate our method on medical computed tomography (CT) images. We focus on liver segmentation, a challenging problem due to weak boundaries between the liver tissue and surrounding organs. Most of the existing liver segmentation methods try to address this problem using 3D shape models [12, 15]. However, building a good prior shape model is difficult because of large inter-patient variability. With our approach, we take into account the general shape of the liver using only one template, and we rely on the large capture range and robustness of our region-based fluid matching algorithm to address these issues. This paper is organized as follows. In Section 2, we review the probabilistic formulation of region-based segmentation models and sketch our new template registration framework. In Section 3, we describe the non-rigid registration algorithm and its implementation. Experimental results on medical images are shown in Section 4. We compare our results with segmentations obtained using an interactive technique. In Section 5, we highlight the key differences between this work and prior art. Finally, we conclude in Section 6.
2. Statistical formulation of region-based segmentation Let Ω ∈ Rd be open and bounded, and I : Ω → R be the image to be segmented. We assume that Ω is composed of N independent regions Ωi . We define pi (I(x)) as the probability density function of a random variable modeling intensity values I(x) in Ωi . Given this model, the optimal partition can be obtained using a maximum likelihood principle, and minimizing the following energy proposed by Zhu and Yuille [23]: E({Ωi , . . . , ΩN }) =
N Z X i=1
− log pi (I(x))dx.
(1)
Ωi
There are different strategies to minimize this energy and obtain a region segmentation. Here, we consider two methods: first, the known active contour approach using a level set representation (Sec. 2.1); second, our new template registration formulation using registration techniques (Sec. 2.2). In the following, we only consider the two-phase segmentation problem, where the domain Ω is partitioned into the object Ωin and its background Ωout . Let pin and pout correspond to the probability density functions (PDFs) of intensities in the object and its background.
2.1. Active contour In the level set framework, Ωin and Ωout are defined as the interior and exterior of an evolving contour C represented as the zero level line of a signed function φ : Ω → R, i.e. C = {x ∈ Ω | φ(x) = 0}. Now, let H(φ) be the Heaviside function, 1, if φ ≥ 0, H(φ) = (2) 0, otherwise. Following the notation used in Chan and Vese [3], the energy to be minimized is expressed as follows: Z E(φ) = − [H(φ) log pin (I(x))+ Ω
(1 − H(φ)) log pout (I(x)) + ν|∇H(φ)|] dx, (3) where ν|∇H(φ)| is a regularization term on the contour length. This energy is typically minimized by gradient descent. For this purpose, one defines a time dependent family of functions t 7→ φ(t) solutions of the following PDE: pin (I(x)) ∇φ ∂E(φ) = δ(φ) log + νdiv , (4) ∂t pout (I(x)) |∇φ| that corresponds to an evolution of the contour C. The first term accounts for the intensity likelihood for each region.
The second term is the regularization of the contour. In practice, the Heaviside H and its derivative the Dirac delta function δ are implemented using smooth approximations.
2.2. Registration Framework We reformulate Equation 3 in this new framework, where IT ◦ ψ can be seen as H(φ) in the last section, and obtain the following energy to be minimized: Z E(IT ◦ ψ) = − [(IT ◦ ψ) log pin (I(x))+ (5) Ω (1 − IT ◦ ψ) log pout (I(x))]dx. Since we want to find an optimal transformation ψ, we derive the last energy and obtain the following gradient descent direction: pin (I(x)) ∂E(IT ◦ ψ) = ∇(IT ◦ ψ) log . (6) ∂ψ pout (I(x)) In non-rigid registration, deriving this energy according to a high-dimensional transformation results in a vector field v. To guarantee a well-posed problem, this vector field has to be regularized. For this purpose, different techniques have been proposed. The fluid-based approach proposed by Christensen et al. [5] has the advantage of capturing large deformations. It uses a viscous fluid deformation model and solves the registration problem using a partial differential equation. Variations of the fluid method have been proposed. They aim at improving its computational efficiency [2, 7] or providing a geometric formulation of the method in terms of flows of diffeomorphisms [1, 17, 21]. In this work, we use a Gaussian filtering that can be seen as an instance of the fluid-approach [5]. Comparing Equation 4 and 6 we notice significant similarities. As mentioned previously, H(φ) corresponds to IT ◦ ψ. In addition, δ(φ) corresponds to ∇(IT ◦ ψ). In the active contour model the regularizing term applied to the mean curvature penalizes irregular shapes of the contour, whereas in the registration framework the regularization is applied to the transformation domain.
2.3. Statistical estimation In region-based segmentation using active contours (Sect. 2.1), the probability density functions of pin and pout are dynamically updated during the contour evolution. These PDFs are based on the intensity histogram of the image I. Given an intensity value i, the corresponding PDFs are computed using the following formulas: Z 1 pin (i) = H(φ)δ(I(x) − i)dx, (7) |Ωin | Ω Z 1 pout (i) = (1 − H(φ))δ(I(x) − i)dx, (8) |Ωout | Ω
where |Ωin | and |Ωout | are the volumes of the corresponding domains. Similarly, in this work, we compute the PDFs of the object and the background during the registration process (Sect. 2.2). The template image IT determines the inside of the region of interest and its background, where H(φ) is replaced by IT ◦ ψ in Equations 7 and 8. This yields to: Z 1 (IT ◦ ψ)δ(I(x) − i)dx, (9) pin (i) = |Ωin | Ω Z 1 (1 − IT ◦ ψ)δ(I(x) − i)dx. (10) pout (i) = |Ωout | Ω
3. Non-rigid registration Let us consider the problem formulated in Section 2.2. We wish to find an optimal high-dimensional transformation, therefore we build a sequence of transformations (ψk )k=0,...,+∞ , by composition of small displacements [4], ψk+1 = ψk ◦ (ψid + αvk ),
ψ0 = ψid ,
(11)
where ψid is the identity transformation and vk is a velocity vector field that follows the gradient of the cost functional to be minimized. Here, vk is obtained by computing the variational gradient of the cost functional given in Equation 6. We regularize the gradient vk using a fast recursive filtering technique. This approximates a Gaussian smoothing [8] that has proven very efficient in practice. The previous iterative scheme (Eq. 11) is repeated until convergence, and can be seen as the discretization (via Taylor expansion) of the transport equation in the Eulerian frame: ∂ψt = −Dψt · v, ψ0 = ψid , (12) ∂t where Dψt stands for the Jacobian matrix of ψt . In Christensen’s work, the velocity field is obtained from the gradient of an image similarity measure and is the solution of an elliptic PDE defined by a linear operator L = a∇2 + b∇ · ∇. This operator, borrowed from fluid dynamics, guarantees the spatial regularity of v. Note that large deformations are possible because the regularization is applied to the velocity rather than the deformation (Dupuis et al. [9] detail the suitable regularity conditions on the velocity field to generate a diffeomorphism). This non-rigid registration is initialized with a rigid transformation. We use the hill climbing method, a local search optimization technique, for the rigid registration.
3.1. Implementation The method just described is summarized in the following Algorithm:
Algorithm 1 Non-rigid template registration algorithm Input: I, IT and σ Output: ψk final transformation 1: Rigid initialization 2: Set ψ0 = ψid 3: while k < max iter or convergence not reached do 4: Compute pin and pout (Eq. 9 and 10) in (I(x)) 5: Compute vk = ∇(IT ◦ ψk ) log ppout (I(x)) (Eq. 6) 6: Regularize vk with a Gaussian filter, vk = Gσ ∗ vk 7: Set α to insure small displacements 8: Update ψk+1 = ψk ◦ (ψid + αvk ) (Eq. 11) 9: Update k = k + 1 10: end while
The region statistics are computed dynamically as the algorithm iterates. The probability density functions pin and pout are based on intensity histograms (Sect. 2.3) and are described by Equations 9 and 10. We consider a smooth version of the template IT where values vary between 0 and 255 (contour at 127). The regularization of the vector field vk is performed using the Deriche’s recursive filter [8]. This fast filter approximates a Gaussian filter (as proposed by D’Agostino et al. [7]), where σ is the standard deviation. We choose this filter because it has a linear complexity with respect to the number of voxels. The algorithm stops when the maximum number of iterations is reached, or when the cost functional stops decreasing. The previous algorithm is embedded in a coarse-to-fine strategy. Where coarser solutions are propagated back up in the multi-resolution hierarchy. This reduces the computational cost by working with less data at lower resolutions. This also allows for the recovery of large displacements (improved capture range), and avoids local minima.
4. Experimental results To validate our approach, we evaluate the performance of the algorithm on CT images of the abdomen. We focus on liver segmentation, a challenging problem because the discrimination between the liver tissues and other adjacent organs is difficult. We segment five datasets using our algorithm and compare the results to a ground-truth obtained with an interactive segmentation technique based on the random walker algorithm [10]. This semi-automatic technique requires an interactive specification of seeds for the object and its background. The segmentation can be corrected by adding additional seeds or removing existing ones, in order to obtain a segmentation that can be considered as the ground truth. In this work, we use a manually segmented liver as a template. Our algorithm is automated by first aligning the
(a) Alignment of the centers of grav- (b) Alignment after rigid initializaity. tion.
(c) Segmentation using the proposed (d) Segmentation using the random template registration algorithm. walker technique.
Figure 1. Contour superimposition of different stage of the template registration algorithm in liver segmentation.
center of gravity of the image and the template. This initialization is specific to the liver, and is a good starting point for the rigid registration. Finally, we perform the non-rigid matching. Figure 1 shows different stages of the template registration algorithm evolution. Figure 1(a), 1(b) and 1(c) show the first alignment based on the centers of gravity, the alignment after the rigid registration and the alignment after the non-rigid registration (the recovered segmentation) respectively. Figure 1(d) shows the result obtained with the interactive segmentation (random walker). To quantify the segmentation accuracy, we used 3D measures to compare shapes as described in [15]. We compare the segmentation to the ground-truth. We consider the relative symmetric volume difference between two shapes calculated using the following formula : v(S, S 0 ) = 1 −
2|VS ∩ VS 0 | , |VS | + |VS 0 |
(13)
where VS denotes the volume of the segmented shape. We also consider three symmetric surface distance measures, the mean surface distance, the root mean square distance and the area of deviation. Given two surfaces S and S 0 we define the distance d(x, S 0 ) between a point x on a surface S and the surface S 0 as minx0 ∈S 0 kx − x0 k2 where
k · k2 denotes the Euclidean norm. The distance measures are chosen to be symmetric when exchanging S with S 0 . The mean distance dmean is calculated using: dmean (S, S 0 ) = Z Z 1 0 d(x, S )dS + d(x, S)dS , |S| + |S 0 | S S0
(14)
where |S| denotes the area of the surface. The root mean square distance drms , is similar to dmean , although it measures the average of squared distances, so that negative values do not cancel positive values.
(a) Template Image
(b)
(c)
(d)
drms (S, S 0 ) = Z 21 Z 1 p d(x, S 0 )2 dS + d(x, S)2 dS . |S| + |S 0 | S S0 (15) Finally, the area of deviation gives a local measure of the deviation of the shape. It represents the relative area on which the deviations are larger than some threshold t : 1 dr (S, S 0 ) = |S| + |S 0 | Z Z H(d(x, S 0 ) − t)dS +
(16) H(d(x, S) − t)dS ,
S0
S
where H is the heaviside function. The results obtained by evaluating the liver segmentation (using our approach) to the ground-truth (using the interactive random-walker) are presented in the following table:
1 2 3 4 5
Figure 2. Liver segmentations obtained with the non-rigid template registration method. (a) represents the binary template image, (b), (c) and (d) represent segmentations of different datasets.
v(%) 4.7 4.0 6.5 6.2 5.5
dmean (mm) 1.6 1.4 2.4 1.8 2.0
drms (mm) 2.9 2.5 3.6 3.1 3.3
dr 5.5 7.7 13.0 11.2 6.5
Table 1. Results showing different measures between the template registration and the random walker segmentations.
We used 3 level of multi-resolution in the coarse-to-fine strategy, and a regularization parameter σ equal to 3.0. The presented results are compared to results obtained with statistical shape model segmentations, as presented by Lamecker et al. [15] and by Heimann et al. [12]. We notice that our results are similar or better than the model based ones, depending on the images. We compared our method to statistical shape models because they are the state-of-theart in liver segmentation and are much less sensitive to weak boundaries than plain-vanilla region-based algorithms.
Lamecker et al. obtained, approximately, a volume overlap of 7.0%, a mean distance of 2.3, a root mean square distance of 3.1 and finally an area of deviation of 15.3% for distances larger than 4mm. Here, we acheive a volume overlap of 5.38%, a mean distance of 1.84, a root mean square distance of 3.1 and an area of deviation of 8.76% also for distances larger than 4mm. However, we tested our approach on a limited number of images. We plan to segment more images in order to give a more accurate evaluation of the template registration approach. Figure 2 shows an example of different shapes (segmentations) recovered using the template registration. We observe that the algorithm succeeds in capturing very dissimilar shapes. In addition to a global regularization on the shape, our non-rigid template registration method has the advantage of preserving the topology of the liver even for large shape variations. Computational times for the presented segmentation vary between 30 and 60 seconds. But the computational time depends on the desired accuracy of the segmentation. If one desire to have a segmentation at a high resolution (image resolution) computational time will increase significantly. The proposed template matching algorithm can also be applied to segment other organs. For example, we segment
the kidney in CT images 3. In this case, a first initialization is required to start the rigid alignment. Then, the non-rigid registration is performed and yields a final segmentation. Figure 3(a) shows the first initialization and Figure 3(b) the final segmentation.
expense of higher computational cost. Finally, the proposed method is easy to implement. In particular there is no need to reinitialize a distance function as with level set methods.
6. Conclusion
(a) First manual initialization
(b) Final segmentation
Figure 3. Contour superimposition of kidney segmentation using the template registration algorithm.
5. Discussion In this section, we highlight the key differences between this work and prior art, in particular the techniques proposed in [3, 23] where a similar energy is used. 1. Rather than evolving a contour/surface, we deform the embedding space. This relies on the following assumption: we know approximately the shape of the structure of interest, and only local non-rigid deformations are required to obtain a final segmentation (template has first been aligned rigidly). Since the regularization is applied to the deformation, one can segment irregular shapes while avoiding leaks that would be difficult with surface evolution techniques. We decorrelate the regularization from the intrinsic geometry of the template (e.g. curvature of the contour). 2. Compared to image-to-image non-rigid registration, the template matching uses a region-based (global) image criterion by opposition to local intensity comparisons. Here, we make the assumption that an organ has a shape close to the template in contrast to strong correlation between image intensities at each voxel. Also, there is no need for prior extraction of information of intensity or shape variation, so the template matching can be considered as an unsupervised method. 3. The template matching uses efficient implementation techniques from non-rigid registration, such as the multi-resolution strategy. This also allows us to choose the accuracy of the segmentation easily. The accuracy can be increased to voxel precision, of course at the
We have presented an approach for image segmentation based on registration techniques. We aligned a binary template to an image by maximizing the likelihood intensity distributions within the region of interest and its background. This general approach sheds a new light on the link between segmentation and registration, which becomes nothing more than two alternatives to optimize the same functional cost under different constraints. Future work will include an extensive validation of our method on much larger data sets for the liver and different organs. Extensions to multi-phase segmentations and non-medical applications would also be of great interest.
References [1] M. F. Beg, M. I. Miller, A. Trouv´e, and L. Younes. Computing large deformation metric mappings via geodesic flows of diffeomorphisms. International Journal of Computer Vision, 61(2):139–157, February 2005. [2] M. Bro-Nielsen and C. Gramkow. Fast fluid registration of medical images. Visualization in Biomedical Computing, 3:267–276, 1996. [3] T. Chan and L. Vese. Active contours without edges. IEEE Transactions on Image Processing, 10(2):266–277, 2001. [4] C. Chefd’hotel, G. Hermosillo, and O. Faugeras. Flows of diffeomorphisms for multimodal image registration. In Proceedings of the IEEE International Symposium on Biomedical Imaging, pages 753–756, 2002. [5] G. E. Christensen, R. D. Rabbitt, and M. I. Miller. Deformable templates using large deformation kinematics. IEEE Transactions on Image Processing, 5(10):1435–1447, 1996. [6] T. F. Cootes, C. J. Taylor, D. H. Cooper, and J. Graham. Active shape models : their training and application. Comput. Vis. Image Underst., 61(1):38–59, 1995. [7] E. D’Agostino, F. Maes, D. Vandermeulen, and P. Suetens. A viscous fluid model for multimodal non-rigid image registration using mutual information. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 541–548, 2002. [8] R. Deriche. Recursively implementing the gaussian and its derivatives. In Proceedings of the International Conference on Image Processing, pages 263–267, Singapore, September 1992. [9] P. Dupuis, U. Grenander, and M. Miller. Variational problems on flows of diffeomorphisms for image matching. Quarterly of Applied Mathematics, LVI(3):587–600, 1998.
[10] L. Grady, T. Schiwietz, and S. Aharon. Random walks for interactive organ segmentation in two and three dimensions: Implementation and validation. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, volume 2, pages 773–780. Springer, 2005. [11] S. L. Hartmann, M. H. Parks, P. R. Martin, and B. M. Dawant. Automatic 3D segmentation of internal structures of the head in MR images using a combination of similarity and free form transformations part II, validation on severely atrophied brains. IEEE Transactions on Medical Imaging, 18(10):917–926, 1999. [12] T. Heimann, I. Wolf, and H.-P. Meinzer. Active shape models for a fully automated 3d segmentation of the liver - an evaluation on clinical data. In Proceedings of the International Conference on Medical Image Computing and ComputerAssisted Intervention, volume 2, pages 41–48, 2006. [13] D. Hill, P. Batchelor, M. Holden, and D. Hawkes. Medical image registration. Physics in Medicine and Biology, 26:R1– R45, 2001. [14] M. Kass, A. Witkin, and D. Terzopoulos. Snakes: Active contour models. International Journal of Computer Vision, 1(4):321–331, 1988. [15] H. Lamecker, T. Lange, and M. Seeba. Segmentation of the liver using a 3D statistical shape model. ZIB Preprint 04-09, 2004. [16] M. Leventon, E. Grimson, and O. Faugeras. Statistical shape influence in geodesic active contours. In Proceedings of the Conference of Computer Vison and Pattern Recognition, pages 316–322, 2000. [17] M. I. Miller and L. Younes. Group actions, homeomorphisms, and matching: A general framework. International Journal of Computer Vision, 41(1-2):61–84, 2001. [18] D. Mumford and J. Shah. Optimal approximations by piecewise smooth functions and associated variational problems. Comm. Pure Appl. Math, 42:577–684, 1989. [19] S. Osher and J. Sethian. Fronts propagating with curvature dependent speed: Algorithms based on hamilton-jacobi formulations. Journal of Computing Physics, 79:12–49, 1988. [20] K. M. Pohl, J. Fisher, J. J. Levitt, M. E. Shenton, R. Kikinis, W. E. L. Grimson, and W. M. Wells. A unifying approach to registration, segmentation, and intensity correction. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 310– 318, 2005. [21] A. Trouv´e. Diffeomorphisms groups and pattern matching in image analysis. International Journal of Computer Vision, 28(3):213–221, 1998. [22] A. Yezzi, L. . Zllei, and T. Kapur. A variational framework for integrating segmentation and registration through active contours. Medical Image Analysis, 7(2):171–185, 2003. [23] S. C. Zhu and A. L. Yuille. Region competition: Unifying snakes, region growing, and bayes/MDL for multiband image segmentation. IEEE Transaction and Pattern Analysis and Machine Intelligence, 18(9):884–900, 1996.
