2013 IEEE 10th International Symposium on Biomedical Imaging: From Nano to Macro San Francisco, CA, USA, April 7-11, 2013
AUTOMATIC CARDIAC RV SEGMENTATION USING SEMANTIC INFORMATION WITH GRAPH CUTS Dwarikanath Mahapatra∗ , Joachim M. Buhmann Department of Computer Science, ETH Zurich, Switzerland. ∗
[email protected] ABSTRACT We propose a fully automatic method for cardiac right ventricle (RV) segmentation using image features, context information and semantic knowledge using graph cuts. A region of interest (ROI) is first identified and pixels within it are assigned labels (RV or background) using Random forest (RF) classifiers and graph cuts. Semantic information obtained from the trained RF classifiers is used to formulate the smoothness cost. Use of context and semantic information contributes to higher segmentation accuracy than competing methods used on the MICCAI 2012 RV segmentation dataset. Index Terms— Automatic segmentation,MRI, Right ventricle, graph cut, semantic information
to fine segmentation and label propagation [6]. Graph cut (GC) based methods were used with convex relaxation and distribution matching [7], statistical principal components as shape priors [8] and region merging [9]. In this work we propose a method for RV segmentation (endocardium and epicardium) using low-level features (like intensity, texture and curvature), and high level context information. Random forest (RF) classifiers are trained to get probability maps of a pixel belonging to RV or background, and their log-likelihood serves as label costs for segmentation. Semantic information about the discriminative importance of each feature is obtained from the trained RF classifiers and used to design a novel weighted smoothness cost function. We describe our method in Section 2, present experimental resuts in Section 3 and conclude with Section 4.
1. INTRODUCTION Segmentation of the right ventricle (RV) has recently gained attention because of new findings that confirm the relationship between RV function and a number of cardiac diseases such as heart failure and RV myocardial infarction [1]. Manual segmentation of the RV is tedious and likely to suffer from inter-observer variations which necessitates the development of computer-aided segmentation methods. Such methods have to overcome the challenges of low resolution and noisy magnetic resonance (MR) images, complex deformations of the RV chamber, its highly variable crescent-shaped structure, and presence of papillary muscles. We propose a fully automated method that aims to segment the RV using a combination of shape information and learned image statistics within a graph cut framework. Cardiac left ventricle (LV) segmentation has been a widely researched topic [2], and increasing importance of the RV led to a RV segmentation challenge in MICCAI 2012. Previous registration-based methods used for RV segmentation had difficulty in defining a proper energy function to drive the curve evolution to the boundary because of the complex and variable RV shape [3]. Elbaz et al. [4] used active shape models (ASM) and inter profile modeling for RV segmentation while context information was used in [5] for segmenting the LV and RV. Recent methods from the MICCAI 2012 RV segmentation challenge include coarse
978-1-4673-6454-6/13/$31.00 ©2013 IEEE
1094
2. METHODS 2.1. Probability maps using RF classifiers A RF classifier [10] is used to obtain class probability values to be used in graph cut segmentation. In this section we describe different image features and our approach to extract semantic information from the learned RF classifier. Image Features: The mean and variance of intensity, texture and 2D curvature values are used as features. Texture maps are obtained using oriented Gabor filters at angles of {0◦ , 45◦ , 90◦ , 135◦ } and two scales (1, 0.5). These features give a 20 dimensional initial feature vector. Relative Context: As the relative arrangement of organs is constant (except for missing organs) one organ can provide contextual information about others through relative distance and orientation. Thus context is particularly important for medical images and can be used to obtain higher segmentation accuracy. Since context information depends on relative orientation and distance we sample regions at fixed directions from a pixel. Figure 1 (c) shows an illustration of the sampling scheme where the circle center is the pixel in question and the sampled point are identified by a red ‘X’. At each point corresponding to a ‘X’ we extract a 3 × 3 region and calculate the mean intensity, texture and curvature values. The
(a)
(b)
(c)
Fig. 1. (a) downsampled image with manual segmentation(red) and obtained segmentation (green); (b) original image with manual segmentation (red), upsampled mask (green) and ROI (yellow) (c) sampling diagram for context information. texture values were derived from the texture maps obtained at 90◦ orientation and scale 1. The ‘X’s are at distances of 3, 8, 15, 22 and the angle between consecutive rays is 45◦ . The values from the 32 regions are concatenated into a 96 dimensional feature vector. The final feature vector has 116 values. Equal number of samples from RV and non-RV pixels (from training images) are extracted and used to train a RF classifier. The trained classifier is used to generate probability maps for every pixel within the identified region of interest (described later) of the test image. Each voxel has 2 probability values corresponding to RV and background. 2.2. Graph Cut Segmentation The final segmentation is obtained by optimizing a second order Markov random field (MRF) energy function which is written as D(Ls ) + λ V (Ls , Lt ), (1) E(L) = s∈P
(s,t)∈N
where P denotes the set of pixels and N is the set of neighboring pixels for pixel s. The cost function is optimized using graph cuts [11]. λ is a weight that determines the relative contribution of penalty cost (D) and smoothness cost (V ). D(Ls ) is given by D(Ls ) = − log (P r(Ls ) + ) ,
(2)
where P r is the likelihood (or probabilities) previously obtained using RF classifiers and = 0.00001 is a very small value to ensure that the cost is a real number. Figure 2 shows the probability maps of a test image for RV endocardium and epicardium regions. Higher the probability for a class lower is the corresponding data penalty for that class. 2.2.1. Semantic Information for Smoothness Cost V ensures a smooth solution by penalizing spatial discontinuities. We formulate the smoothness cost by using semantic information from the trained RF classifier. The RF classifier
1095
returns a measure of the importance of each dimension in the feature vector to the classification task. Inspite of the multiple dimensional feature vector, the features can be classified into three types - intensity, texture and curvature. Note that the context information is a combination of the three. By aggregating the importance values of each feature category and normalizing them we obtain the relative importance of each feature in the classification task. This provides the necessary semantic information by quantifying the importance of each feature in classifying a voxel into different categories. Let the weight of the different features be wI (intensity), wT (texture) and wC (curvature), where wI + wT + wC = 1. The smoothness cost V is given by wI VI + wT VT + wC VC , Ls = Lt , V (Ls , Lt ) = 0 Ls = Lt . (3) where VI , VT , VC are the individual contributions to the smoothness by intensity, texture and curvature. VI is defined as (Is −It )2 1 , (4) VI (Ls , Lt ) = e− 2σ2 · s − t I is the intensity. VT and VC are similarly defined using texture and curvature. To choose the value of λ we adopt the following steps. We choose a small subset of the training data consisting of 10 patient volumes, and perform segmentation using our method but with λ varying from 0 to 1 in steps of 0.001. Based on the segmentation accuracy using Dice Metric (DM) we set λ = 0.02. After training of the classifiers we obtain wI = 0.26,wT = 0.30 and wC = 0.44. 2.2.2. Segmenting RV in subsequent frames After the RV is segmented in the first slice of the first frame of a patient sequence its outline is propagated to the next slice and serves as its ROI. The final segmentation is obtained using RF classifiers and graph cuts as described before and propagated to the next slice till all slices of the first frame are segmented. The sequence of images is such that the RV shrinks in consecutive images The segmentations of the slices are used as starting ROIs for the corresponding slice in the next frame to get the final segmentations. This procedure is repeated for all frames on the sequence till we get the segmentations for all slices at all time points. These set of steps is adopted for both end-diastole (ED) and end systole (ES) images. 2.3. Automatic ROI identification Outputting the probability of every image pixel is very time consuming. In order to reduce processing time we want to identify a region of interest (ROI) that would encompass the RV. To get the ROI we first downsample the image by a factor of 2. We segment the RV in the downsampled image using the steps described previously Note that for this step the RF classifiers are trained on downsampled training images to capture
DM HD
(a)
(b)
OurnC 81.2 13.2
OurnVI 89.7 8.1
OurnVT 87.2 9.2
OurnVC 89.9 7.9
M AH 87.4 9.0
Table 2. Quantitative measures for ED and ES segmentation accuracy. DM- Dice Metric in % and HD is Hausdorff distance mm
(c)
Fig. 2. Probability maps for (a) RV endocardium; (b) RV epicardium. HIgher values indicate greater likelihood of belonging to that region. Red indicates maximum probability while blue indicates zero probability. (c) final segmentation output; red- manual segmentation,green-our segmentation. characteristics specific to downsampled images, particularly context information. However other feature extraction steps remain the same including the distance of the sampled locations from the pixel. The RV segmentation so obtained is upsampled to get an approximate mask for the RV. In practice we take the maximum and minimum row and column co-ordinates, increase their range by ±15 pixels to get the new ROI. Further processing is now limited to the pixels within this ROI. Figure 1 (a) shows an example image alongwith the manual segmentation mask in red (both downsampled versions). The automatic RV segmentation of the downsampled image is shown in green. Figure 1 (b) shows the original image, manual annotation (in red), the upsampled mask and the new ROI in the original image. Comparing Figs1 (a),(b) we observe that even if the initial segmentation is faulty increasing the range of row and column co-ordinates by ±15 pixels gives a good ROI. 3. EXPERIMENTS AND RESULTS Cardiac MR examinations were performed at 1.5T (Symphony Tim, Siemens Medical Systems, Erlangen, Germany) using a eight-element phased-array cardiac coil and repeated breath-holds of 10 − 15 s. A total of 8 − 12 contiguous cine short axis slices were performed from the base to the apex of the ventricles. Sequence parameters were as follows: TR = 50 ms; TE = 1.7 ms; flip angle = 55; slice thickness = 7 mm; matrix size = 256 × 216; Field of view = 360 − 420 mm; 20 images per cardiac cycle; spatial resolution of 0.75mm/pixel [12]. There were 32 datasets and we use a leave-one-out strategy to evaluate our method. 3.1. Segmentation Results We present RV segmentation results of the following methods: Our - our proposed method; M AH - shape prior segmentation method of [13]. ZU L ([6]); N AM ([7]); GRO ([8]); M AI ([9]). Quantitative evaluation of segmentation performance is given in terms of Dice Metric (DM) and Hausdorff distance (HD) measures. DM gives a measure of over-
1096
lap between the reference manual segmentations and the algorithm segmentations. HD gives an idea on the distance between the the boundaries of the two segmentations. Tables 1,2 summarize the performance of all the above methods. The measures are a combination of ED and ES phases. Our gives results that are comparable to the best performing methods. Since we did not have access to the segmentations of the individual methods, we are unable to perform a detailed statistical comparison with the different methods. However the mean DM values by Our are higher and the mean HD values are lower than other methods. This indicates the superiority of our method. Furthermore, our method also shows better performance for ES images, which are more challenging than ED images. Compared to other methods the difference of the average DM and HD values between ED and ES images is quite low for Our indicating a more consistent segmentation of images from different phases. The consistently better performance can be attributed to two factors: 1) use of machine learning techniques to identify most discriminant features; and 2) incorporating semantic information into the smoothness cost. The importance of semantic information is further examined by different experiments, and subsequent statistical analysis. Table 2 summarizes the performance of our method under different conditions. OurnC - Our without context information from images for training the RF classifier; OurnV Our without semantic context in V . wI = wT = wC = 0.33; OurnVI - Our with wI = 0; OurnVT - Our with wT = 0; OurnVC - Our with wC = 0; The results show a significant reduction in segmentation accuracy without using context information. A t−test between the values of Our and OurnV gives pOur−OurnV < 0.02 indicating a large degree of difference between the results. Similar tests show significant improvement of Our compared to other methods. Figure 3 shows segmentation results for Patient 18 images obtained by Our,OurnC , OurnVC and OurnVT . The results reflect the values in Table 2. The low DM values for OurnC highlights the fact that context plays a very important role in our method. Although all the low level features are used, without context information it is very difficult to discriminate between RV endocardium and epicardium. For other methods although one low level feature is excluded, context information provides greater discrimination. Computational Cost: The average computation time for our entire method on a 256 × 216 image was 528 seconds. The automatic ROI identification stage including sub sam-
ED ES
DM 93(6) 85(8)
Our HD 6.7(4.2) 8.4(4.1)
DM 83(17) 72(27)
ZU L HD 9.7(7.8) 11.4(10.5)
N AM DM HD 67(19) 17.7(7.7) 48(25) 23.2(9.7)
GRO DM HD 83(15) 9.5(5.4) 69(23) 10.5(5.5)
M AI DM HD 92(5) 7.1(3.9) 83(16) 10.8(5)
Table 1. Quantitative measures for ED segmentation accuracy. DM- Dice Metric in % and HD is Hausdorff distance mm work for automatic whole heart segmentation of cardiac MRI,” IEEE Trans. Med. Imag., vol. 29, no. 9, pp. 1612– 1625, 2010.
Fig. 3. Segmentation results for Patient 18: (a) Our;(b) OurnC ; (c)OurnVT ; and (d) OurnVT . The manual segmentations are in red while the algorithm segmentations are in green. Areas of inaccurate segmentation are highlighted by yellow arrows. pling, feature extraction, classification, segmentation and upsampling to get the ROI took 224 seconds on an average. Further segmentation of the ROI took 304 seconds on an average, inclusive of the time taken for classification and segmentation. 4. CONCLUSION We have presented a novel method for RV segmentation that is fully automated and does not require any manual intervention. As part of our method we have developed novel context features to train a Random forest classifier and classify each pixel as RV or background. Additionally we incorporate semantic information from the RF classifiers into the smoothness cost to achieve higher segmentation accuracy. Experimental results on the MICCAI 2012 segmentation database show our method is highly accurate, robust and can efficiently segment images from all phases of a cine cardiac MRI sequence. It is also comparable to other algorithms on the same database and in many cases outperforms them.
[4] M.S. ElBaz and A.S. Fahmy, “Active shape model with inter-profile modeling paradigm for cardiac right ventricle segmentation,” in MICCAI, 2012, pp. 691–698. [5] D. Mahapatra, “Cardiac LV and RV segmentation using mutual context information,” in Proc. MICCAI-MLMI, 2012, pp. 201–208. [6] M.A. Zuluaga, M.J. Cardoso, and S. Ourselin, “Multi atlas fusion: Automatic right ventricle segmentation using multi-label fusion in cardiac mri,” in Proc. MICCAI RV Segmentation Challenge, 2012. [7] C.M.S. Nambakhsh, M. Rajchl, J. Yuan, T.M. Peters, and I. Ben-Ayed, “Rapid automated 3d rv endocardium segmentation in mri via convex relaxation and distribution matching,” in Proc. MICCAI RV Segmentation Challenge, 2012. [8] D. Grosgeorge C. Petitjean, S. Ruan, J. Caudron, and J. Dacher, “Right ventricle segmentation by graph cut with shape prior,” in Proc. MICCAI RV Segmentation Challenge, 2012. [9] O. Maier, D. Jimenez-Carretero, A. Santos, and M.J. Ledesma-Carbayo, “Right-ventricle segmentation with 4d region-merging graph cuts in mr,” in Proc. MICCAI RV Segmentation Challenge, 2012. [10] L. Breiman, “Random forests.,” Machine Learning, vol. 45, no. 1, pp. 5–32, 2001. [11] Y. Boykov and O. Veksler, “Fast approximate energy minimization via graph cuts,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, pp. 1222–1239, 2001.
5. REFERENCES [1] J.C. Matthews, T.F. Dardas, M.P. Dorsch, and K.D Aaronson, “Right sided heart failure: diagnosis and treatment strategies,” Curr. Treat. Options Cardiovasc, vol. 10, no. 4, pp. 329–341, 2008. [2] C. Petitjean and J-N. Dacher, “A review of segmentation methods in short axis cardiac mr images,” Med. Imag. Anal., vol. 15, no. 2, pp. 169–184, 2011. [3] X. Zhuang, K.S. Rhode, R.S. Razavi, D.J. Hawkes, and S. Ourselin, “A registration based propagation frame-
1097
[12] J. Caudron et al., “Cardiac mr assessment of right ventricular function in acquired heart disease: Factors of variability,” Academic Radiology, vol. 19, no. 8, pp. 991–1002, 2012. [13] D. Mahapatra and Y. Sun, “Orientation histograms as shape priors for left ventricle segmentation using graph cuts,” in In Proc: MICCAI, 2011, pp. 420–427.
