170
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 1, JANUARY 2012
Integrating Segmentation Information for Improved MRF-Based Elastic Image Registration Dwarikanath Mahapatra and Ying Sun, Member, IEEE
Abstract—In this paper, we propose a method to exploit segmentation information for elastic image registration using a Markovrandom-field (MRF)-based objective function. MRFs are suitable for discrete labeling problems, and the labels are defined as the joint occurrence of displacement fields (for registration) and segmentation class probability. The data penalty is a combination of the image intensity (or gradient information) and the mutual dependence of registration and segmentation information. The smoothness is a function of the interaction between the defined labels. Since both terms are a function of registration and segmentation labels, the overall objective function captures their mutual dependence. A multiscale graph-cut approach is used to achieve subpixel registration and reduce the computation time. The user defines the object to be registered in the floating image, which is rigidly registered before applying our method. We test our method on synthetic image data sets with known levels of added noise and simulated deformations, and also on natural and medical images. Compared with other registration methods not using segmentation information, our proposed method exhibits greater robustness to noise and improved registration accuracy. Index Terms—Combined registration and segmentation (CRS), labels, Markov random fields (MRFs), natural and medical images, object of interest (OOI), simulated deformations.
I. INTRODUCTION
S
EGMENTATION information is expected to improve registration as the identification of the object improves the accuracy of feature extraction. Registration corrects rigid motion and elastic deformations by finding an optimal transformation such that the transformed image is as close as possible to a reference image. Prior models of an object also enhance the registration process [1]. Incorporating segmentation information in registration is the most relevant for medical applications where the focus is on a single organ, and segmentation is preceded by registration. Most existing approaches to elastic (or nonrigid) registration use smoothly varying curve equations. By designing appropriate cost functions, existing elastic registration frameworks have successfully registered a wide variety of images. Popular techniques include elastic models [2], fluid flow methods [3], [4], optical-flow-based methods [5], thin plate splines [6], free-form deformations (FFDs) using B-splines [7], [8], and radial basis functions [9]. Manuscript received March 01, 2010; revised September 12, 2010 and April 16, 2011; accepted July 06, 2011. Date of publication July 25, 2011; date of current version December 16, 2011. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. D. S. Taubman. The authors are with the Department of Electrical and Computer Engineering, National University of Singapore, Singapore 117576 (e-mail: dmahapatra@nus. edu.sg;
[email protected]). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TIP.2011.2162738
Recently, Markov random fields (MRFs) have been also used by formulating the elastic registration of natural and medical images as a discrete labeling problem [10]–[12]. Shekhovtsov et al. [10] use blocks of pixels in formulating the MRF energy function to compensate for nonrigid deformations in synthetic and real images. Smoothness was imposed based on the relative displacement between neighboring pixels. Labels can be also defined for subpixel displacements that make recovered displacement field’s smoothness to be closer to that of curve-based registration methods. In [11], the objective function is based on MRFs, and a dense deformation field is defined using a registration grid and interpolation. This enables dimensionality reduction. Mahapatra and Sun [12] use saliency information to identify corresponding regions in a pair of precontrast- and postcontrast-enhanced kidney perfusion images and elastically register them using MRFs. Simple segmentation techniques such as clustering, thresholding, adaptive thresholding, or watershed methods [13] do not work well on challenging cases having low spatial resolution, noise, or intensity inhomogeneity. Other popular segmentation methods for 2-D or 3-D images include snakes [14], active contours [15], level sets [16], and graph cuts [17]. Each method has its own distinct approach to segmentation. Except [17], which assigns labels (object or background) to different pixels, all the other works can be classified as “curve evolution” techniques that deform a pre-initialized curve to match object boundaries under the influence of external and internal forces. Boykov and Funka-Lea [17] use MRFs and graph cuts to segment natural and medical images where seed points are used to build model histograms for the object and the background. Joint registration and segmentation techniques achieve registration and segmentation simultaneously over a pair of images. In [18], an active contour framework was used to interleave level set segmentation with a feature-based registration method, thus successfully segmenting and registering portal images to computed tomography scans, whereas partial differential equations were used in [19] and [20]. Wyatt and Noble [21] combine MRFs and Bayesian estimation for joint registration and segmentation, where the use of MRFs is limited to the segmentation step. In [22], an expectation-maximization method is proposed for the simultaneous estimation of image artifacts, anatomical label maps, and mapping from an atlas to the image space for brain images. Ashburner and Friston [23] propose a probabilistic framework using a mixture of Gaussians for image registration, tissue classification, and bias correction for brain magnetic resonance imaging (MRI). A multistage approach involving filtering, edge detection, and segmentation in an iterative framework was used in [24]. An earlier version of this work appeared in [25]. There are other methods that use registration information
1057-7149/$26.00 © 2011 IEEE
MAHAPATRA AND SUN: SEGMENTATION INFORMATION FOR IMPROVED MRF-BASED ELASTIC IMAGE REGISTRATION
for segmentation and vice versa. In atlas-based registration, an image is registered to an atlas or a clearly identified object using a level set framework [26]. Likewise, including shape information in active contour segmentation requires a model of shape variation where the accurate registration of training shapes is a critical component [27], [28]. Leventon et al. [29] also use shape information in the curve evolution process. Cremers et al. [30] minimize an energy functional that includes the shape energy and the Mumford–Shah energy. Other works use local deformation [31] and prior shapes [32] in a variational framework. The major challenges in combining registration and segmentation in one framework are to ensure convergence and prevent estimates of registration or segmentation parameters adversely affecting each other. Defining appropriate energy functions to include relevant information for both registration and segmentation without bias is equally important. A. Our Contribution We propose a MRF-based method to integrate segmentation information for the registration of an object of interest (OOI) in a pair of images. The cost function is optimized using graph cuts. The motivation in using MRFs, i.e., the method suitable for discrete labeling problems, is that discrete optimization techniques such as graph cuts can be used, which are fast and give a global minimum or a strong local minimum (close to global minimum) [33], [34]. The registration problem is formulated as one of labeling, and each label defines the joint occurrence of displacement vectors (for registration) and segmentation class. The cost function is a combination of the mutual dependence of registration and segmentation information at every label. The final labels denote the displacement vector and the segmentation class of each pixel, and a multiresolution graph-cut (MRGC) technique reduces the computation time. We test our method on synthetic images with B-spline-based simulated elastic deformations and added Gaussian noise. Our method is also applied on natural and medical images exhibiting real elastic deformations. A mask indicating the OOI is first defined on the floating image, which is rigidly registered to the reference image. The mask is mapped to the reference image and is used to obtain the reference intensity distribution parameters for the segmentation of the reference and floating images. Subsequently, the OOI is registered using our method. The rest of this paper is organized as follows: Section II gives details about our MRF formulation. Section III details our experiments and results. In Section IV, we discuss the advantages of our method and finally conclude with Section V. II. THEORY A. Joint Registration and Segmentation In registration, the objective is to match each pixel in the floating image to the most similar pixel in the reference image, and the similarity metric depends on the type of images being used. The displacement field is regularized to give a smooth deformation, and the smoothness constraints depend upon the registration framework. For B-spline [7] and other curve-based registration methods, curve gradients are used as smoothness constraints. In [10], Shekhovstov et al. used MRFs for nonrigid registration. MRFs are used for discrete labeling problems that make it challenging to impose smoothness in registration as in-
171
dividual pixels do not always have integer displacements. However, a smooth solution is obtained by constraining the relative displacement between neighboring pixels to be within a specified range so that they have similar displacements. Since the smoothness formulation is not based on the boundary properties of the object being registered, it may result in unexpected deformations of the registered image, particularly at object boundaries. Smoothness criteria based on object features help to overcome this shortcoming. In our method, the user identifies the OOI, which is registered using segmentation information and is related to previous works on joint registration and segmentation. The first work on joint registration and segmentation [18] used an active contour framework. A piecewise-constant segmentation model in [16] is used, which favors a curve that yields the least total squared error approximation of the image by one constant inside the curve and another constant outside the curve. The constants are the mean intensity values inside and outside the curve. An initial curve is drawn on the reference image, and the corresponding curve on the floating image is given by , where is the registration mapping between the images. For each image, a separate energy function is defined, which are then combined such that the final joint energy function depends upon and . The curve is updated based on the gradient of , and a curvature term based on the curve length penalty is added to regularize the curve evolution. In the final solution, the value of gives the registration mapping the floating image to the reference image, and gives the boundary of the segmented object in the reference image. The object in the . Thus, by minifloating image can be segmented from mizing one energy functional relating two images, optimal registration and segmentation parameters are obtained. In joint registration and segmentation, the estimate of one set of parameters (that of registration) should not adversely affect the other set of parameters (of segmentation), i.e., they balance each other. The use of active contours has certain positives in combining registration and segmentation. It is easy to calculate derivatives of the real-valued curve and evolve it toward the final solution. Transformation maps a real-valued number to another real-valued number ensuring that, while evolves, its mapped version is also close to the desired object. In [18], results are shown for rigid transformations where the type of mapping (i.e., linear) is known but the parameters have to be determined. A major disadvantage of the active contour framework is the possibility of the curve being trapped in local minima and the high convergence time. The position of the initial curve also influences the final solution. Although this minor obstacle can be overcome by employing a standard segmentation technique to obtain the initial curve, the possibility of getting trapped in local minima still persists. The number of iterations can be reduced by using graph cuts, thus also decreasing computation time. Graph cuts is based on maximum-flow approach and is very effective in finding the global minimum or a strong local minimum of discrete MRF energy formulations [33]–[35]. However, a number of issues have to be addressed in using segmentation information for MRFbased registration such as the following: 1) Registration and segmentation energies have to be combined such that there is no bias for a particular term. 2) The mutual dependence of registration and segmentation has to be factored in the objective func-
172
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 1, JANUARY 2012
tion. In discrete labeling problems, we do not know the kind of mapping function similar to (i.e., linear or quadratic), which determines the position of the segmented object in the floating image and whose derivatives impose smoothness. Therefore, including the mutual dependence is crucial. 3) The smoothness energy must regularize the solution based on registration and segmentation information. In the subsequent sections, we describe MRFs and how we use them to integrate segmentation information for registration.
in the objective function and depends upon the type of images being registered. If there is no intensity change between and , a normalized sum of squared difference (SSD) of intensities is used, i.e.,
B. MRFs The energy function of a second-order MRF takes the following form:
(3) where is the number of pixels in the block. Equation (3) always returns values between 0 and 1 since the image intensities are normalized to lie between 0 and 1. For an image pair exis a function of the gradient hibiting intensity change, orientation information and is given by (4)
(1) denotes the set of pixels, denotes the label of pixel , and is the set of neighboring pixel pairs. For our method, the label of each pixel gives the displacement vector ; and the segmentation class, i.e., denotes the displacement along the two axes, and denotes the segmentation class ( denotes the object, and denotes the background). The labels of the entire set of pixels are denoted by . is a unary data penalty function derived from observed data that measures how well label fits pixel . is a pairwise interaction potential that imposes smoothness and measures the cost of assigning labels and to neighboring pixels and . We define as a function of both registration and segmentation information. The optimization scheme for (1) using graph cuts is discussed in Section II-B-3. Next, we discuss how each term of the energy function is used to combine registration and segmentation information. 1) Data Penalty Term: assigns a penalty to pixel taking on a particular label . is defined as the sum of two penalty terms. The first term is a function of gradient or intensity information and, by itself, is suitable for registration. The second term includes the mutual dependence of registration and segmentation in the penalty term. The following issues are considered in combining the two penalty values: 1) none of the values bias the overall data penalty, i.e., they have the same dynamic range (the difference between maximum and minimum values is the same); 2) the individual terms are robust for their specific purposes; and 3) the combination of the two terms truly captures the mutual dependence of registration and segmentation. We refer to the block centered at pixel as . Let denote the intensity of the th pixel of block in the floating image , and let denote the corresponding pixel intensity in the reference image . is denoted as , and the intensity The pixel block at of its th pixel in is given by . The corresponding gradients are denoted by and . Gradient has two components along the two axes, i.e., . Note that the data penalty is for each pixel but is calculated over a pixel block for robustness. The segmentation class denotes the object, and denotes the background. The data penalty term is given by where
(2) Here, is a parameter that decides the relative contribution of the two terms. incorporates registration information
Edge information has been successfully used to register contrast enhanced images [36] and is a robust feature in the face of is a normalized metric that gives values intensity changes. is a between 0 and 1, with 0 indicating a perfect match. Since penalty, its value is low for greater similarity between the pixels. The second penalty term is a function of the mutual dependence of the segmentation class and the displacement vectors. When a mask is defined around the OOI, the area inside (outside) the mask is used to get the intensity distributions of the object (the background) modeled as mixtures of Gaussians or nonparametric Parzen windows. The intensity value of each pixel is fit into these distributions to get their probability of belonging to each class, which is then used to determine the penalty value. Let denote the posterior probability of pixel in the floating image belonging to the object, and let denote its probability of belonging to the background. The probability of pixel in the reference image belonging to the object is given by , and is its probability of belonging to the background. Thus, is given by
(5) The aforementioned formulation gives a low penalty in case both posterior probabilities favor the same class. In case the individual probabilities favor separate segmentation classes, their combination gives the final penalty. For medical data, the value of in (2) was empirically fixed at 0.8 as it gave the best segmentation accuracy when compared with manual segmentation. However, for synthetic images, best segmentation results are obtained for . 2) Pairwise Interaction Term: The pairwise interaction is used to regularize the solution and combines smoothness due to the displacement vector and the segmentation class. It is defined as and and otherwise. (6) is a spatially varying weight that depends upon the intensity difference between the neighboring pixels of the floating image and is given by (7)
MAHAPATRA AND SUN: SEGMENTATION INFORMATION FOR IMPROVED MRF-BASED ELASTIC IMAGE REGISTRATION
is a weight empirically determined that gives the best tradeoff between the accuracy and the smoothness of the deformation field. When neighboring pixels have the same segmentation class , they may have similar displacements because pixels on the same object tend to move together. Thus, we constrain the maximum relative displacement between the pixels to be pixels. The relative displacement threshold for similar pixels is the maximum distance between neighbors in an eightneighborhood system. Consequently, this threshold constrains that the maximum relative displacement between similar pixels units. This ensures the continuity of the cannot be more than deformation field and has been used in the previous work [10]. If neighboring pixels have different segmentation classes , then they can have vastly different displacements since pixels on different objects may differently move. The threshold for dissimilar pixels was experimentally determined. The value of 3 gave the best tradeoff between the smoothness of the deformation field and the registration accuracy. The deformation alone is field smoothness was judged from visual results. the smoothness term for MRF-based segmentation [17] and includes smoothness due to object boundary characteristics. The importance of (6) is explained in Section III-H. 3) Optimization Using Graph Cuts: Pixels are represented as nodes in graph , which consists of a set of directed edges that connect two nodes. The edge weight between two neighboring nodes is the smoothness term, whereas the data penalty term is the edge weight for links between nodes and label nodes (terminal nodes). Edges between pixel nodes are called -links, whereas edges between pixel nodes and terminal nodes are called -links. The optimum labeling set is obtained by severing the edge links in such a manner that the cost of the cut is minimum. The cost of a cut is the sum of the weights of the severed edges. For labels, there are terminal nodes. The number of nodes is equal to , i.e., the number of pixels. In our method, , where is the number of the number of labels is possible displacement vectors and there are two segmentation classes for every displacement vector. For every node, the data penalty term is determined as a function of the corresponding floating image and the reference image. The optimal labeling is obtained via a series of -expansion moves. Pixel nodes, along with three terminal nodes, are shown, and two cuts indicate the severing of edges such that each pixel is assigned one label. Details of graph construction and optimization can be found in [35]. Note that, although each pixel is a node in the graph, the data penalty is derived from a block of neighborhood pixels for robustness. To increase the registration speed, a multiresolution implementation of graph cuts is used. MRGCs have been previously used for faster optimization in segmentation [37]. We employ one coarse step and two fine steps, details of which are given below: 1 Coarse step: Three displacement labels are defined along - and -axes with a step size of 2 between each label, i.e., and . Each displacement label has two segmentation labels making it a total of labels. The purpose of this step is to increase registration robustness by recovering the residual
173
translation motion after rigid registration. Therefore, the pixel block size is set to 11 11. The maximum displacement that is recovered is 6 pixel units. The displacement labels are used to transform the image and get an intermediate image for further estimation of the displacement labels. 2 First fine step: We search over a neighborhood of 3 pixels along each axis. Here, the labels correspond to displacements of 1 pixel (i.e., and pixels) making it a total of labels. The purpose behind the high displacement range was to compensate for any erroneous estimation of displacement labels. When we assign a step size of 2 between labels, there is a chance of wrong labels being assigned to a pixel as low-resolution images can give misleading low-level information. 3 Second fine step: In the final step, the search neigh2 pixels. Each label is equivborhood is over alent to a displacement of 0.5 pixels (i.e., and pixels), and the total number of labels is . The purpose behind using this large number of labels is the same as given in step 2. C. Extension to 3-D The extension of our method to 3-D is straightforward. The available medical 3-D data is in the form of a number of slices making up the volume. The gradient information at a pixel is calculated from its neighbors in its immediate 3-D neighborhood, i.e., 26 neighboring elements. The OOI is chosen from a slice as before, and the volume of interest (VOI) is determined by including the corresponding pixels from five slices before and after the chosen slice (total of 11 slices). The VOI is now used for determining intensity distributions for the area inside and is calculated in the same manner as that outside the mask. for 2-D using the inner product of gradients vectors. The posterior probability depends upon the intensity of the voxel. Thereis the same as in (5). The number of fore, the formulation of possible displacement vectors increases to a power of 3, each associated with two segmentation classes. The maximum distance . Therefore, the relative disbetween two neighbors in 3-D is placement threshold for neighbors with the same segmentation to , i.e., class is changed from and and otherwise. (8) is given in (7). For neighbors with different segmentation classes, their relative displacement is still a maximum of three units. The block sizes used for penalty calculation is ( for coarse optimization). For the multiresolution optimization in 3-D, we adopt the following steps: 1 Coarse step: Three displacement labels are defined along , each axis, i.e., , and . The total number of la. bels is
174
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 1, JANUARY 2012
2 Fine step:
, , and . The labels. total number of labels is 3 Second fine step: , , and . The total labels. This number of labels is arrangement was necessary because optimization for a large number of labels was time consuming. Our algorithm is summarized in Algorithm 1. We make the following observations about our algorithm: 1 In most registration applications, subpixel displacements are observed. A label set over all possible subpixel displacements is very large, and optimization takes up a lot of time. Therefore, a coarse-to-fine approach is employed for obtaining the correct displacement labels. The final optimization step is aimed at correcting subpixel displacements. 2 The data penalty formulation can be changed depending upon the type of images being registered. For images with no change in intensity, a normalized SSD of intensities is used, whereas for perfusion images, a contrast-invariant gradient-based measure is used. 3 The mutual dependence of registration and segmentation is factored into the data penalty by combining posterior probability values of the segmentation class from and . The smoothness term is also a function of both registration and segmentation information. This helps in avoiding misregistration at object boundaries. Algorithm 1 Registration Framework Require
and
1:Identify the OOI in . Rigidly register to , and use the user specified mask to estimate intensity distributions for the and . object and the background in 2:For every pixel in and all labels, calculate the data penalty as a function of registration and segmentation information using (3)–(5). 3:Calculate smoothness based on registration and segmentation information using (6). 4:Minimize energy function using multiscale graph cuts • determine coarse displacement labels at a spacing of 3 pixels (a block size of 11 11). • determine fine displacement labels at a spacing of 1 pixel (a block size of 5 5). • determine subpixel displacement at a spacing of 1/2 pixel (a block size of 5 5). 5:From the final labeling, nonrigidly register obtain the segmented OOI.
to
, and
III. EXPERIMENTS AND RESULTS Here, we investigate the performance of our proposed method. Results are shown for synthetic images with simulated deformations, natural, cine cardiac, and perfusion images.
Our method is called combined registration and segmentation (CRS). We also register the images without using segmentation information in the formulation (referred as MRF), i.e., (9) where
and
is given by otherwise.
(10)
The registration accuracy of CRS is compared with that of MRF and also the Demons algorithm [5]. The Demons algorithm is based on the principal of the optical flow where gradient information is used to deform the floating image. The interaction between the reference and floating images (based on gradient information) is calculated at each iteration. The optimal , fluid reguparameters for Demons was 100 iterations, , and no diffusion regularization. All the larization with experiments were carried out under different degrees of added noise. A. Synthetic Images Synthetic images containing simple shapes such as triangles, squares, rectangles, cylinders, and circles were created. The number of objects in any image was between one and four. The intensity of each object in the image was different but constant. Elastic deformations on the images were simulated using B-splines [7], the Gaussian noise added to it, and an initial mask defined around the OOI in the floating image. For registering the multiple objects in an image, we have two approaches. In the first approach, each object is individually registered. The number of segmentation classes in this approach is two, i.e., the object and the background. In the second approach, the mask should include the objects to be registered. The intensity distribution of the area inside the mask will have multiple modes (peaks), and each mode with a mean value greater than zero belongs to one object (assuming that the background has zero intensity value). After adding noise to the image, the peaks in the intensity histogram are smoothed. We assume a Gaussian mixture model (GMM) for the intensity distribution inside the mask and determine the different parameters corresponding to each object, i.e., the mean and the variance. The number of Gaussians is equal to the number of objects in the mask plus one (for the background). This is determined by finding out the number of distinct peaks in the mask’s intensity histogram by thresholding. The deformed noisy floating images are now registered to the original noise-free reference images using three registration methods, namely, CRS, MRF, and Demons. Demons are used to compare the performance of our method with other popular techniques. B-splines, which is another popular registration method, has been already used for simulating deformations and is therefore not used for comparison. The , where is the noisy images are represented by noise-free image and is Gaussian noise of zero mean and variance varying from 0.01 to 0.1. The image intensities are between 0 and 1. An 8 8 grid of control points was used to simulate elastic deformation on images whose dimensions were in the range
MAHAPATRA AND SUN: SEGMENTATION INFORMATION FOR IMPROVED MRF-BASED ELASTIC IMAGE REGISTRATION
175
Fig. 1. Registration results for synthetic images. (a) Reference image; (b) floating image; and registered images using (c) Demons, (d) MRFs, and (e) CRS. The : , and all the objects in the image were individually registered by defining a mask around them. added noise is equivalent to
= 0 08
of (70–90) (80–110) pixels. The spacing between the control points varied from 8–13 pixels depending upon image dimensions. The perturbations of the control points took random but known values between 6 pixels. From 2-D B-spline equations [7] and the new position of control points, the image is deformed, and the actual displacement of every pixel relative to the original image is known. 1) Mutual Influence of Registration and Segmentation: The images were all in grayscale. Fig. 1(a) shows the reference image without noise, and Fig. 1(b) shows the floating image with intensity changes, added noise, and simulated deformations. Approximate masks corresponding to the different shapes are drawn manually. Fig. 1(c) shows the registered image using the Demons method followed by the registered images using MRF and CRS [see Fig. 1(d) and (e), respectively]. For testing purposes, eight of such images with ten sets of deformation parameters and four noise levels for each image are used, giving us a total of 320 image pairs. The figures are for . For the set of images shown in Fig. 1, the added noise was . Each object in the image (circle, equivalent to cylinder, and rectangle) is separately registered using CRS. Although we use a GMM to estimate the probability density function of the selected mask, it is found that registering objects individually is more accurate than registering all of them simultaneously because a mask encompassing a single object gives a more accurate intensity distribution. All our subsequent results are for individual registration. Our method was robust . The performance of the three up to noise levels of . For Demons, methods is comparable up to noise of the average registration error increases above 1.0 pixel when , whereas, for MRFs, the error is above 1.0 pixel when . The corresponding threshold for CRS is 0.085. Fig. 3(a) shows the registration error in pixels for the three methods with a change in . In spite of intensity changes, we are able to successfully register the image pair if we use gradient information in the registration penalty. Although MRF performs better than Demons, the registered images show inaccurate registration due to the absence of segmentation information. The contribution of segmentation information in registering noisy images is evident from the registered image using CRS where all the shortcomings of the Demons method and MRF are overcome. The average registration errors for different values of are shown in Table I. 2) Accuracy of Segmentation: Aside from displacement vectors, the labels of the CRS method also give the segmentation class of each pixel. In a separate step, the floating images are segmented using a conventional graph-cut approach (described
TABLE I MEANS AND STANDARD DEVIATIONS OF REGISTRATION ERRORS FOR SYNTHETIC IMAGE DATA SETS AT DIFFERENT NOISE LEVELS. VALUES ARE IN PIXELS
in [17]). This method is denoted as GC. In GC, seed points belonging to the object and the background are identified, which are used to form reference intensity histograms for the object and the background. The penalty values (for the two labels) of each pixel is the negative log likelihood of its intensity, and the smoothness term is similar to the one in (7) with an additional weight that is inversely proportional to the distance between pixels. Note that the optimization for CRS and GC is performed using the graph-cut algorithm in [35]. The dice metric (DM) [38] is used to evaluate the similarity between manual denote the seg(expert) and automatic segmentations. Let be mented area (using CRS or any segmentation method), the area of manual segmentation, and be the intersection and . The DM is given as area between DM
(11)
DM takes values between 0 and 1, with 1 indicating a perfect match and 0 denoting no match. Generally, a DM higher than 0.80 indicates excellent agreement with manual segmentation [38]. Fig. 2 shows the visual results for segmentation of synthetic images using CRS and GC [Fig. 2(a) and (b), respectively]. The outlines of the segmented masks are shown in different colors to indicate that each object was separately chosen as the OOI. GC successfully extracts line segments in most noisy images. It falls short for circular boundaries in the presence of noise. Note that, for GC, we separately identify each shape as the object because of their different intensity characteristics. The displayed image . Inaccurate segmentation due to GC is observed has in the disk and in the curved area within the rectangle. By using registration information, CRS is able to accurately segment the images. They are compared with manual segmentation, and the results for different noise levels are shown in Table II. After registration using CRS, there is no separate segmentation step because the final labels also give the segmentation class. The segmentation accuracy degrades with increasing noise level. How-
176
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 1, JANUARY 2012
Fig. 2. Segmentation performance for synthetic images using (a) CRS and (b) graph cuts. (c) Inaccurate initial masks shown in different colors and (d) superimposed outline of segmented objects using CRS from masks in (c). TABLE II SEGMENTATION PERFORMANCE FOR SYNTHETIC IMAGES. AVERAGE DM VALUES FOR DIFFERENT NOISE LEVELS ARE SHOWN
although inaccurate, contain part of the objects, and Fig. 2(d) shows the final segmentation. Since the main purpose of the mask is to obtain the segmentation information of the OOI, the subsequent registration result was not hampered by the mask as long as we are able to identify the intensity distribution of the individual objects. B. Cine Cardiac MRI
Fig. 3. (a) Change in registration error (pixels) with increasing noise levels for the three registration methods. (b) Change in DM values for CRS and graph cuts. The x-axis shows the variance of added noise, and y -axis shows (a) average registration error in pixels and (b) DM values.
ever, the DM value is above 80% for CRS and GC for . , DM goes below 80%. Fig. 3(b) shows the When change in DM values for CRS and GC with increasing . Robustness to Initial Segmentation: The purpose of the mask is to get intensity distributions for the OOI and the background. However, CRS is not sensitive to the choice of initial mask. In Fig. 2(c), we show the initial masks in different colors, which,
With the help of the synthetic images, we show that, by including segmentation information, CRS is more robust to noise than MRF and Demons. The segmentation accuracy is also improved over a conventional graph-cut method. Here, we show results for registering the left ventricle (LV) in the cine cardiac MRI in Fig. 4. The cine MRI is characterized by large deformations of the LV and nearly no intensity change. Since the intensity change can be ignored, the registration energy is a function of intensity difference as in (3). The first frame is taken as the reference frame . Since large deformations are observed over the course of the image acquisition process, registering a frame directly to the reference frame increases the number of labels and, hence, the computation time. Therefore, the frame is first deformed using the displacement field of the previous registered frame in the sequence and is subsequently registered to . This approach gives a good initial position to start and reduces the number of labels. 1) Effects of CRS: A mask is drawn over the OOI in the is rigidly registered to using gradient floating image . information [36] to recover any translation motion. Thus, the is also determined, enapproximate location of the OOI in suring that we obtain the intensity distribution parameters for both floating and reference images. There is no available ground truth for this elastic registration. Since the LV is close to a uniform circle, it is difficult to identify unambiguous landmarks. However, the LV boundary can be used to evaluate registration accuracy. The outline of the LV in and are manually drawn by experts (for evaluating segmentation accuracy). After registration, the displacement field is used to deform the LV in . For each point on the deformed contour of , the distance to its nearest point on the LV in is calculated. If a point of the deformed contour exactly matches the corresponding point in , the distance will be zero. The mean distance for all points of the deformed contour gives the error values. Fig. 4 shows different frames (floating images) of a typical cine cardiac data set where the LV is contracting. The red contours show the LV boundary of the reference frame deformed using the displacement field. The blue contours show the original LV boundary in the reference frame. This is to illustrate the
MAHAPATRA AND SUN: SEGMENTATION INFORMATION FOR IMPROVED MRF-BASED ELASTIC IMAGE REGISTRATION
177
Fig. 4. Results for registration of cine cardiac images. The boundary of the LV (in reference image) is deformed using (red) the obtained motion field and is overlaid on the floating image. First row shows results when using CRS, second row has results using MRFs, and third row shows results for the Demons algorithm. (Blue line) Outline of the LV in the reference frame. This gives an idea of the degree of deformation recovered using CRS.
degree of motion that was recovered in each method. The deformed boundary superimposed on the floating image gives an idea of registration accuracy. The first row shows results for the proposed CRS method, the second row shows results for MRF, and the third row shows registration results for the Demons algorithm. The maximum displacement error of the LV with respect to the reference frame before the registration of cine cardiac images was 11 pixels, and the average error was 7.1 2.3 pixels. This high value shows the large amount of motion observed in cardiac images. After registration, the average registration errors are 0.7 0.4 pixels using CRS, 1.1 0.4 pixels for MRF, and 1.2 0.6 pixels for Demons. The corresponding maximum errors are 1.2, 1.6, and 2.3 pixels. By using segmentation information, we are able to achieve better registration in those cases where MRF and Demons show less than optimal performance. CRS, MRF, and Demons perform equally well for those images where there is good contrast at the object boundaries. The advantage of CRS is seen for medical images with poor contrast at object boundaries. Noise and low contrast are common for the cine cardiac MRI, and therefore, it is desirable to use segmentation information to improve registration. We determined the segmentation accuracy of each image of the data set by calculating DM values with expert manual segmentations as reference. The average DM value using CRS was 95.3%, and the corresponding value for GC was 93.4%. As a matter of fact, cine cardiac images are easy to segment because of the good contrast between the OOI and the background. Even in such a situation, CRS achieves better segmentation than GC. C. Cardiac Perfusion MRI being a Cardiac perfusion images are registered with function of gradient information as in (4) to overcome intensity
changes. The cardiac images were acquired on a Siemens Sonata magnetic resonance (MR) scanner following the bolus injection of a Gd–diethylenetriamine-penta-acetic-acid contrast agent. mm , The pixel spacing ranges from and the entire scan consists of 60 frames. The contrast agent flows into the right ventricle (RV), then into the LV, and, finally, into the myocardium. The acquired data sets were all in 2-D. In Fig. 5, we show results for registration and segmentation using our method. Fig. 5(a) and (b) show the reference and floating images, respectively. Their difference image before registration is shown in Fig. 5(c), and the difference image after registration using MRF is shown in Fig. 5(d). The corresponding difference image after registration using CRS is shown in Fig. 5(e). In the displayed images, the LV cavity was chosen as the OOI. The use of segmentation information results in a much-improved registration performance. This is evident from a close examination of the area corresponding to the LV and RV in the difference image. The difference images are such that the dark areas correspond to negative intensity values, whereas the gray areas correspond to nearly zero differences. The reference image is subtracted from the floating and registered images. The area corresponding to the LV should be elliptical. Before registration, we observe deviations from the elliptical shape due to elastic deformations. MRF is able to recover a significant amount of deformations, but the smoothness of the registered RV can be improved. This is achieved by using segmentation information in CRS. CRS achieves better registration of the RV and the LV. The registration accuracy was determined in a manner similar to that of cine cardiac images by manually identifying the LV contours and calculating the distance between registered and unregistered contours. The average error values over ten data sets are 3.1 1.2 pixels before registration, 0.6
178
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 1, JANUARY 2012
Fig. 5. Registration and segmentation result for cardiac perfusion images. (a) Reference image; (b) floating image; difference images (c) before registration, (d) after registration using MRFs, and (e) after registration using CRS; (f) segmented mask using CRS; and (g) segmented mask using GC.
Fig. 6. Results for registration of 3-D liver perfusion images. Each row shows results for different slices. The first column is the reference image, followed by the floating image in the second column. The third column shows the difference image before registration, followed by the difference image after registration using CRS in the fourth column.
0.3 pixels after registration using CRS, 1.1 0.3 pixels using MRF, and 1.0 0.2 pixels using Demons. Fig. 5(f) and (g) show the outline of the segmented LV in green using CRS and GC, respectively. The floating image has low contrast and poses difficulty in accurate segmentation using GC. With the use of registration information, CRS is able to give a more accurate segmentation when compared with expert manual segmentation. It is interesting to note that, for images where the LV had high contrast with respect to the background, the segmentation accuracy of CRS and GC was approximately the same. CRS showed better performance for low contrast images. The DM values for cardiac perfusion images were 87.8% using GC, and 90.1% using CRS. D. 3-D Registration Results Here, we show results for registering 3-D liver perfusion data. The data sets comprised of 65 volumes corresponding to 65 sampling instances. Each volume consists of 28 slices. The slice thickness was 4 mm, and the pixel spacing was 1.6 mm. Each slice’s dimension was 110 80. Preliminary tests were conducted on two data sets. Fig. 6 shows the results for 3-D registration. All volumes were registered to the first volume in the sequence. Each row shows a different slice of the volume to
illustrate the extent of deformations and also the effectiveness of CRS in registering these deformations in the face of noise and low resolution. The area covered by the liver changes with the slices, e.g., the last row, also shows the kidney, whereas the previous slices show only the liver. Translation motion is observed, along with elastic deformations at the base and the upper edge of the liver. These deformations could be corrected by our method. Note that registration was done for each volume using the method in Section II-C and not the individual slices. Slices are shown because the available data is in the form of slices and not surface data. To get error measures for the liver, we manually identify the outer boundary of the liver in each slice of the reference and floating volumes. The deformation field after registration is used to deform the liver in the reference volume. Then, the registration error between the liver in each slice of the reference volume relative to the corresponding slices of the floating volume is calculated in a manner similar to that of cardiac images. The average error after registration using CRS was 0.9 0.2 pixels. We also simulated deformations in 3-D on different volumes of the liver data sets using FFDs and recovered them by CRS. The average displacement error after recovering simulated deformations was 0.8 0.4 pixels. The DM values for CRS and GC were 91.2% and 89.3%, respectively.
MAHAPATRA AND SUN: SEGMENTATION INFORMATION FOR IMPROVED MRF-BASED ELASTIC IMAGE REGISTRATION
179
Fig. 7. Registration and segmentation performance for natural images. (a) Reference image; (b) floating image with mask outline; (c) difference image before registration; (d) difference image after registration using CRS; (e) segmented mask from floating image using GC; and (f) segmented mask using CRS.
E. Natural Images We also test our method on pairs of natural images that exhibit nonrigid motion between them. Fig. 7 shows results on a pair of images having relative motion of the hand. Fig. 7(a) and (b) show the reference and floating images, respectively, with the outline of the mask in blue. The difference image before registration is shown in Fig. 7(c), and the difference image after registration using CRS is shown in Fig. 7(d). Although only a part of the hand was chosen as the OOI, most of the motion has been recovered except for the minor movement of the fingers. The segmented mask of the OOI from the floating image using GC is shown in Fig. 7(e), and the mask obtained using CRS is shown in Fig. 7(f). Although the segmentation accuracy in this case is not very important, CRS with the help of registration information produces better segmentation than GC. F. Comparison With a Similar Method Our algorithm’s performance in relation to other methods combining registration and segmentation will give a better idea about its usefulness. Toward this end, we compared our algorithm’s performance with the “unified segmentation” (US) method described in [23]. We chose this method because of the easily available code as a part of the statistical parameter mapping (SPM 8) package. This method was specifically developed for 3-D brain data and uses a probabilistic framework for image registration, tissue classification, and bias correction combined within the same generative model. The model is based on a mixture of Gaussians and is extended to incorporate smooth intensity variation and nonlinear registration with tissue probability maps. We performed two sets of experiments to compare CRS and US. In the first set of experiments, we compared the performance of CRS and US on infant brain MR data. The original . To increase MR volumes were of dimension computation efficiency, we cropped the volumes to dimension . The resolution of the acquired data was mm/pixel. A total of five of such volumes were used. The number of segmentation classes were two, identifying brain and nonbrain regions. The deformation was simulated on each volume using B-splines. The original image was the reference, and CRS and US was applied to the deformed image (the floating image). For CRS, a small patch corresponding to brain and nonbrain regions was identified, and the intensity of the object patch was modeled using a mixture of two Gaussians because the brain area has different tissue types (white matter, gray matter, and
TABLE III QUANTITATIVE COMPARISON BETWEEN US AND CRS. REGISTRATION ERROR AND DM VALUES FOR CARDIAC PERFUSION AND BRAIN DATA ARE SHOWN
cerebrospinal fluid) with different intensities. We find that an accurate intensity model for the brain can be achieved by using two Gaussians. However, a single Gaussian is sufficient to model the background intensity. Subsequently, the energy for all pixels in the image is calculated, and graph cuts is used to determine the final labels. The number of labels are as given in Section II-B-3. For US, most of the volumes gave optimal results (in terms of segmentation) with the default parameters. In one volume, minor adjustments were required to get the final segmentation. Note that we calculate the DM values for the whole brain and not on individual white- and gray-matter areas. In the second set of experiments, we compared the performance of both methods on cardiac image data sets. The details for the perfusion data are given in Section III-C. US was applied with the default parameters. The results for both experiments are shown in Table III. The results by the US algorithm on the brain data shows better performance in terms of registration error and segmentation accuracy. CRS’s performance is slightly lower. This is to be expected because US is specifically designed for brain image data and includes steps (such as bias correction) that improve its performance. On the other hand, CRS is not designed to handle the challenges typical to brain data. However, CRS still shows an average registration error of around 1.3 mm, as compared with the 1.0-mm error of US. The DM values for CRS are also greater than 93% in most cases, whereas the corresponding values for US are around 96%. However, the opposite is observed for tests on cardiac images. Here, CRS performs better than US, although both methods show better performance than MRF and Demons. The brain area is manually identified in some slices. After registration is complete, the deformation field is used to map the deformed brain to the reference image. This registered contour is superimposed on the original slices and shown in Fig. 8. The images in the first row correspond to different slices of the brain volume. The red contour shows the outline of the deformed brain mask obtained using CRS, and the green contour is obtained using US. The registration error is determined from the distance between the deformed contours and the manually
180
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 1, JANUARY 2012
Fig. 8. Performance comparison between CRS and US. Results are shown for cardiac and brain images. (Red contour) Result obtained using CRS; (green contour) Results obtained using US. First row shows results for brain data, whereas second row shows results for cardiac data.
drawn contours (from ground-truth segmentation) on the reference image. The manually drawn ground-truth contour is not shown to maintain clarity in the image, but the brain boundary can be easily visually identified. It is evident that the contours from US are closer to the actual brain than CRS. The second row shows results for cardiac images where each image corresponds to different time points. Here, CRS shows better results than US, which is reflected on the closeness of the red contours to the LV cavity surface. G. Computational Complexity Under the assumption that and in (1) are constants independent of the image size , termination is achieved in cycles. These assumptions are quite reasonable, and in practice, the algorithm terminates in few cycles [35], and most of the improvements occur in the first cycle. All our experiments were carried out using MATLAB 7.5 on a personal computer a Pentium 4 processor with 3-GHz speed. Fig. 9(a) shows the computation time for fixed number of labels and the change in number of pixels using CRS. The plots also show the equivalent registration time taken by MRF and CRS. The total labels . This corresponds to a displacewere fixed at ment range of 6 pixels on each axis, and two segmentation classes for every possible displacement. The number of pixels was changed by increasing the image dimensions such that the ratio of rows to columns is approximately 4:3. If the labels are fixed and the number of pixels increased, we observe an almost linear increase in the computation time. It is interesting to note that, when the number of pixels are small, Demons converges faster than CRS, whereas with an increase in the number of pixels, it takes more time to converge. The same linearity is also observed for an image of fixed dimension (320 240) and increasing number of labels. Fig. 9(b) shows the variation in computation time (for CRS and MRF registrations) with an increase in the number of possible displacements. For MRF, this corresponds to the number of labels, whereas for CRS, the number of labels is twice the number of possible displacements (due to two segmentation classes). For a natural image of size 320 240 (shown in Fig. 7), the computation time for 169 labels was 55 s for CRS. The corre-
Fig. 9. Computational complexity analysis. The graphs show the computation time for (a) fixed labels and varying pixel size, and (b) fixed pixel size and varying number of labels.
sponding time for registration was 29 s for MRF and 77 s using Demons. The segmentation time using GC is less than 5 s on an average. The extra segmentation time is negligible, and although the combined time for MRF and GC is less than that of CRS, CRS provides greater registration and segmentation accuracy. For perfusion images of size 70 90 pixels and 121 labels (corresponding to a displacement range of 5 along both axes),
MAHAPATRA AND SUN: SEGMENTATION INFORMATION FOR IMPROVED MRF-BASED ELASTIC IMAGE REGISTRATION
181
volumes in our data sets were a collection of equally spaced slices. As a result, using curve-based approaches was difficult because of the absence of sufficiently continuous data. This being a common phenomenon for MRI data, our method could prove to be advantageous over active contours. We can register more than one OOI by interactively defining appropriate segmentation labels. Although modeling an OOI as a mixture of Gaussians may be simplistic in medical images, we can use nonparametric approaches such as Parzen windows to obtain accurate intensity distributions for a variety of objects. V. CONCLUSION
Fig. 10. Illustration of the importance of segmentation in smoothness formulation. (a) Reference image; (b) floating image; (c) registered image using MRF; and (d) registered image from CRS. The registered image using MRF shows undesirable deformations, which is not the case for CRS because segmentation information influences smoothness cost.
the average computation time was 33.4 s for the entire CRS procedure and 13.4 s for MRF. The time for Demons was 51.2 s. The segmentation time using GC was 3 s. H. Importance of Segmentation in Smoothness Formulation Equation (6) has been formulated such that the smoothness criteria take into account registration and segmentation information. This is particularly important for registration of low contrast structures. Relying on displacement labels or intensity information alone is not desirable. The importance of segmentation information in the formulation is illustrated in Fig. 10. Fig. 10(a) and (b) show the reference and floating images, respectively. Fig. 10(c) and (d) show the registered image using MRF and CRS, respectively. The registered image using MRF exhibits undesirable deformations at the kidney boundary, whereas CRS gives a smooth registration at the boundaries. This highlights the importance of segmentation information in formulating smoothness criteria for MRF-based registration. The results in Fig. 10 demonstrate that defining smoothness criteria based only on the relative displacement of neighboring pixels may not be adequate for the registration of complex medical data and the inclusion of segmentation information for smoothness formulation is important. IV. DISCUSSION Using an MRF-based approach provides us with certain computational benefits. Depending upon the range of displacements, the number of labels can be changed. For example, we do not expect a lot of motion along the -axis for 3-D perfusion data sets, and the number of labels along it can be reduced. In such data sets, the motion along the horizontal direction is limited, allowing for a further decrease in the number of labels. Since MRFs work on discrete labels and digital images are a collection of discrete pixels, faster implementation of the method is possible, making it suitable for clinical environments. The 3-D
We have proposed an MRF-based method to incorporate segmentation information in registration where a manually drawn initial mask of the object to be registered is used. Manual intervention is limited to defining a mask around the OOI, and there is no need for a large amount of training data sets. This makes the method applicable to different data types. The joint registration and segmentation method proposed in [18] used continuous-valued active contours that provided a framework to combine registration and segmentation in two images. The curve evolution has been influenced by gradient information, and smoothness has been imposed based on the curve length penalty. The active contour framework has the disadvantage of being trapped in local minima and having sensitivity to the initial curve. An optimization method based on maximum-flow techniques (graph cuts) can find a global minimum or a strong local minimum, independent of the initialization. However, it also presents us with some challenges. The mutual dependence of registration and segmentation information has to be considered because we do not know the type of mapping function, as in [18], between the discrete-valued labels. We have successfully overcome this by combining probabilities of joint occurrence of registration and segmentation labels from the reference and floating images. The results are less sensitive to the choice of initial mask. The problem has been formulated as one of finding the appropriate labels for each pixel such that they encode registration information, as well as segmentation class. The cost function depends on the observed data (intensity- or contrast-invariant edge information), the probability of labels conditioned on pixel intensity, and the smoothness of registration and segmentation labels. The final labels obtained by minimizing the cost function in an MRGC implementation give the displacement vector and the segmentation class for each pixel. By a coarse-to-fine graph-cut implementation, subpixel accuracy for registration and segmentation has been achieved. We have tested our method on synthetic images with different levels of added noise and deformed using FFD, and on natural and medical images having real deformations. Our results have been compared with an MRF-based registration method not using segmentation information and the Demons algorithm. In the case of synthetic images, we have observed that all methods perform equally well . Beyond up to a particular level of added noise that, a gradual decrease in the registration accuracy of the three methods has been observed with our method, i.e., CRS, having least registration error and being accurate up to a higher noise
182
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 1, JANUARY 2012
level . We first establish the importance of registration and segmentation aiding each other by calculating different error measures. CRS has been found to give the most accurate registration results compared with traditional registration schemes where no segmentation information is used. The segmentation labels from CRS exhibit greater accuracy than graph cuts for noisy images. An important feature of our method is the reduced computation time using an MRGC implementation. Experimental results also demonstrate the effectiveness of our method under intensity change and its robustness to initial segmentation. For OOIs with multivariate distributions (or multiple regions), the mask needs to enclose a part of all the regions. In registering a pair of cardiac perfusion images where the contrast between the LV and the myocardium is so low that it is difficult to differentiate their boundaries, the segmentation of the LV by graph cuts has a tendency to leak onto the myocardium. This is corrected to a great extent using CRS. It is important to note that the OOI needs to be defined manually or by a semi-automated method. Although results have been demonstrated for unimodal OOI and binary labels, the method can be easily extended to multivariate OOIs, multiple labels, and 3-D data. Preliminary tests on 3-D data show encouraging results, and further work would involve extensive testing and a faster implementation of the algorithm. REFERENCES [1] T. F. Cootes, A. Hill, C. J. Taylor, and J. Haslam, “Use of active shape models for locating structure in medical images,” Image Vis. Comput., vol. 12, no. 6, pp. 355–365, 1994. [2] R. Bajcsy and S. Kovacic, “Multiresolution elastic matching,” Comput. Vis. Graph. Image Process., vol. 46, no. 1, pp. 1–21, Apr. 1989. [3] G. E. Christensen, M. I. Miller, and M. Vannier, “3D brain mapping using a deformable anatomy,” Phys. Med. Biol., vol. 39, no. 3, pp. 609–618, Mar. 1994. [4] M. Bro-Nielsen and C. Gramkow, “Fast fluid registration of medical images,” in Proc. VBC, 1996, pp. 267–276. [5] H. Wang, L. Dong, J. O’Daniel, R. Mohan, A. S. Garden, K. K. Ang, D. A. Kuban, M. Bonnen, J. Y. Chang, and R. Cheung, “Validation of an accelerated ‘demons’ algorithm for deformable image registration in radiation therapy,” Phys. Med. Biol., vol. 50, no. 12, pp. 2887–2905, Jun. 2005. [6] C. R. Meyer, J. L. Boes, B. Kim, P. H. Bland, K. R. Zasadny, P. V. Kison, K. Koral, K. A. Frey, and R. L. Wahl, “Demonstration of accuracy and clinical versatility of mutual information for automatic multimodality image fusion using affine and thin-plate spline warped geometric deformations,” Med. Image Anal., vol. 1, no. 3, pp. 195–206, Apr. 1997. [7] D. Rueckert, L. I. Sonoda, C. Hayes, D. L. G. Hill, M. O. Leach, and D. J. Hawkes, “Nonrigid registration using free-form deformations: Application to breast MR images,” IEEE Trans. Med. Imag., vol. 18, no. 8, pp. 712–721, Aug. 1999. [8] Y. Zheng, J. Yu, C. Kambhamettu, S. Englander, M. D. Schnall, and D. Shen, “De-enhancing the dynamic contrast-enhanced breast MRI for robust registration,” in Proc. MICCAI, 2007, pp. 933–941. [9] G. K. Rohde, A. Aldroubi, and B. M. Dawant, “The adaptive bases algorithm for intensity based nonrigid image registration,” IEEE Trans. Med. Imag., vol. 22, no. 11, pp. 1470–1479, Nov. 2003. [10] A. Shekhovtsov, I. Kovtun, and V. Hlavác, “Efficient MRF deformation model for non-rigid image matching,” Comput. Vis. Image Understand., vol. 112, no. 1, pp. 91–99, Oct. 2008. [11] B. Glocker, N. Komodakis, G. Tziritas, N. Navab, and N. Paragios, “Dense image registration through MRFs and efficient linear programming,” Med. Image Anal., vol. 12, no. 6, pp. 731–741, Dec. 2008. [12] D. Mahapatra and Y. Sun, “Nonrigid registration of dynamic renal MR images using a saliency based MRF model,” in Proc. MICCAI, 2008, pp. 771–779.
[13] R. C. Gonzalez and R. E. Woods, Digital Image Processing, 2nd ed. Upper Saddle River, NJ: Prentice-Hall, 2002. [14] M. Kass, A. Witkin, and D. Terzolpopulos, “Snakes: Active contour models,” Int. J. Comput. Vis., vol. 1, no. 4, pp. 321–331, 1988. [15] M. Isard and A. Blake, Active Contours. New York: Springer-Verlag, 1998. [16] T. F. Chan and L. A. Vese, “Active contours without edges,” IEEE Trans. Image Process., vol. 10, no. 2, pp. 266–277, Feb. 2001. [17] Y. Boykov and G. Funka-Lea, “Graph cuts and efficient N-D image segmentation,” Int. J. Comput. Vis., vol. 70, no. 2, pp. 109–131, Nov. 2006. [18] A. Yezzi, L. Zollei, and T. Kapur, “A variational framework for joint segmentation and registration,” in Proc. MMBIA, 2001, pp. 44–51. [19] F. Wang, B. Vemuri, and S. Eisenschenk, “Joint registration and segmentation of neuroanatomic structures from brain MRI,” Acad. Radiol., vol. 13, no. 9, pp. 1104–1111, Sep. 2006. [20] J. An, Y. Chen, F. Huang, D. Wilson, and E. Geiser, “A variational PDE based level set method for a simultaneous segmentation and non-rigid registration,” in Proc. MICCAI, 2005, pp. 286–293. [21] P. P. Wyatt and J. A. Noble, “MAP MRF joint segmentation and registration,” in Proc. MICCAI—Part I, T. Dohi and R. Kikinis, Eds., 2002, vol. 2488, Lecture Notes in Computer Science, pp. 580–587, London, U.K.: Springer-Verlag. [22] K. M. Pohl, J. Fisher, W. E. L. Grimson, R. Kikinis, and W. M. Wells, “A Bayesian model for joint segmentation and registration,” NeuroImage, vol. 31, no. 1, pp. 228–239, May 2006. [23] J. Ashburner and K. J. Friston, “Unified segmentation,” NeuroImage, vol. 26, no. 3, pp. 839–851, Jul. 2005. [24] T. Song, V. Lee, H. Rusinek, S. Wong, and A. F. Laine, “Integrated four dimensional registration and segmentation of dynamic renal MR images,” in Proc. MICCAI, 2006, pp. 758–765. [25] D. Mahapatra and Y. Sun, “Joint registration and segmentation of dynamic cardiac perfusion images using MRFs,” in Proc. MICCAI, 2010, pp. 493–501. [26] A. Tsai, A. Yezzi, W. Wells, C. Tempany, D. Tucker, A. Fan, W. E. Grimson, and A. Willsky, “A shape-based approach to the segmentation of medical imagery using level sets,” IEEE Trans. Med. Imag., vol. 22, no. 2, pp. 137–154, Feb. 2003. [27] T. Cootes, C. Beeston, G. Edwards, and C. Taylor, “Unified framework for atlas matching using active appearance models,” in Proc. IPMI, 1999, pp. 322–333. [28] Y. Wang and L. Staib, “Boundary finding with correspondence using statistical shape models,” in Proc. CVPR, 1998, pp. 338–345. [29] M. Leventon, E. Grimson, and O. Faugeras, “Statistical shape influence in geodesic active contours,” in Proc. CVPR, 2000, pp. 316–322. [30] D. Cremers, F. Tischhauser, J. Weickert, and C. Schnorr, “Diffusion snakes: Introducing statistical shape knowledge into the Mumford-Shah functional,” Int. J. Comput. Vis., vol. 50, no. 3, pp. 295–313, Dec. 2002. [31] N. Paragios, “A variational approach for the segmentation of the left ventricle in cardiac image analysis,” Int. J. Comput. Vis., vol. 50, no. 3, pp. 345–362, Dec. 2002. [32] Y. Chen, H. D. Tagare, S. Thiruvenkadam, F. Huang, D. Wilson, K. S. Gopinath, R. W. Briggs, and E. A. Geiser, “Using prior shapes in geometric active contours in a variational framework,” Int. J. Comput. Vis., vol. 50, no. 3, pp. 315–328, Dec. 2002. [33] V. Kolmogorov and R. Zabih, “What energy functions can be minimized via graph cuts?,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, no. 2, pp. 147–159, Feb. 2004. [34] O. Juan and Y. Boykov, “Active graph cuts,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2006, pp. 1023–1029. [35] Y. Boykov, O. Veksler, and R. Zabih, “Fast approximate energy minimization via graph cuts,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 11, pp. 1222–1239, Nov. 2001. [36] Y. Sun, M.-P. Jolly, and J. M. F. Moura, “Contrast-invariant registration of cardiac and renal MR perfusion images,” in Proc. MICCAI, 2004, pp. 903–910. [37] H. Lombart, Y. Sun, L. Grady, and C. Xu, “A multilevel banded graph cuts method for fast image segmentation,” in Proc. ICCV, 2005, vol. 1, pp. 259–265. [38] C. Pluempitiwiriyawej, J. M. F. Moura, Y. L. Wu, and C. Ho, “STACS: New active contour scheme for cardiac MR image segmentation,” IEEE Trans. Med. Imag., vol. 24, no. 5, pp. 593–603, May 2005.
MAHAPATRA AND SUN: SEGMENTATION INFORMATION FOR IMPROVED MRF-BASED ELASTIC IMAGE REGISTRATION
Dwarikanath Mahapatra received the B.Tech. degree in electrical engineering from the National Institute of Technology, Rourkela, India, in 2006. He is currently working toward the Ph.D. degree in electrical and computer engineering at the National University of Singapore, Singapore. His research interests include medical image processing, computer vision, machine learning, and visual saliency.
183
Ying Sun (S’00–M’05) received the B.Eng. degree from Tsinghua University, Beijing, China, in 1998, the M.Phil. degree from Hong Kong University of Science and Technology, Kowloon, Hong Kong, in 2000, and the Ph.D. degree in electrical and computer engineering from Carnegie Mellon University, Pittsburgh, PA, in 2004. She was a Member of the Technical Staff in the Imaging and Visualization Department, Siemens Corporate Research, Princeton, NJ. Since 2007, she has been with the Department of Electrical and Computer Engineering, National University of Singapore, Singapore, where she is currently an Assistant Professor. Her research interests include medical image analysis, signal processing, and pattern recognition.
