INTERACTIVE SEGMENTATION OF MEDICAL IMAGES USING ...

INTERACTIVE SEGMENTATION OF MEDICAL IMAGES USING BELIEF PROPAGATION WITH LEVEL SETS Yingxuan Zhu1 , Samuel Cheng2 , and Amrit Goel1 1

Department of Electrical Engineering and Computer Science, Syracuse University 2 School of Electrical and Computer Engineering, University of Oklahoma-Tulsa

ABSTRACT In this paper, we propose an interactive segmentation method to apply user information during the segmentation of a specific anatomic structure. This method is formulated to use belief propagation to minimize a global cost function according to local level sets. The propagation starts with one user labeled point, and iteratively extends the user information from the labeled pixel to its neighborhood by calculating the beliefs of the pixels in the same level as the labeled pixel. Since the segmentation relies on both local user information and global image features, it is less interrupted by noise, and works well even the target is not obvious to its neighbor. The promising segmentation results also show that our method is robust to the objects with high shape variation and inhomogeneous intensity value appearance. Index Terms— Interactive segmentation, level set method, belief propagation, user information, medical imaging I. INTRODUCTION Object identification and localization play important roles in image guided therapy (IGT), and both highly depend on image segmentation. Image segmentation delineates structures of the interested objects. An accurate segmentation in IGT is necessary for dosimetry, and it also relieves the burden of manual delineation. Segmentation methods in IGT can be separated into four groups, manual, semi-automatic, interactive and automatic. Manual segmentation is precise but time consuming, making it inconvenient in practice. Semi-automatic segmentation usually relies on manually parameters changing of specific target or different images, which may be hard to end users. In general, the results of automatic segmentation include multiple objects besides the targeted anatomic structures, e.g., Fig. 2 in [1]. In order to obtain a satisfactory result, the segmentation either happens in a specified area, i.e., narrowing down the target region to have as few object(s) as possible, e.g., Fig. 1 in [2]; or uses plenty of training data, such as atlas. Interactive segmentation is developed to obtain an accurate segmentation of a specific object in a small amount of time. Interactive segmentation restricts

the segmentation result to be a specified anatomic object by applying user information (e.g. a selected point inside the object) in the segmentation algorithm [3]. General segmentation methods have been extended to handle user information in interactive segmentation. Level set method is one of the most important methods of them. Level set method can easily handle topological changes by evolving contours according to different image features, e.g., edge and contrast. Moreover, the generated edges from level set method are close, which is good for segmenting medical objects [4, 5]. For example, Cremers et al. and Ben-Zadok et al. applied level set methods in interactive segmentation recently [2, 6]. Their methods level up/down the pixels around the user labeled pixels according to the distance between the user decided pixels and the unknown pixels, which work effectively in solo round shape object, but loosely delineate the objects with sharp corners or large curvature. Moreover, using distance as a hard constraint in the evolution of level sets may cause under-segmentation or over-segmentation problems, especially when the image has multiple objects, i.e., the evolution stops before the zero level set reaches the edges of target, or the evolution doesn’t stop even after the boundary leaks out the target. Instead of restricting the levels by distance, we consider propagating the user information from the selected pixel(s) to their nearby pixels according to some similarity criteria between these pixels with respect to level sets. To implement this idea, we apply belief propagation algorithm to spread out the user information, and use level set method to decide if the unknown pixels are similar to the user selected pixels, i.e., we apply BP to implement the distance as a soft constraint during finding out how far the evolution of level sets should extend the user information. In this paper, we use “target” to represent the anatomic structure which is going to be solo segmented out, such as the liver, and use “background” to represent the region outside the target in the image. Belief propagation (BP) is a probability approximation method which has been widely used in decoding and pattern recognition. Pearl provided a detail description of BP in [7]. BP is a message passing algorithm, where the belief of each variable is iteratively updated based on the information past from other variables, i.e., the system propagates local

estimates to find the best, global solution [8]. Different BP algorithms have been proposed, e.g., sum-product, maxproduct and Gaussian BP [9–11]. In the application of image processing and computer vision, BP algorithm propagates information throughout a graphical model by iteratively sending messages between neighboring nodes. For example, the max-product algorithm was applied to solve the Bayesian inference problem of stereo matching in [12]. An optimization scheme, priorityBP, had been proposed to use BP to calculate the confidence of assigning a specific label to a node [13]. Moreover, Cheng et al. proposed an image registration method which used BP to solve the optimization problem about distance constraints of the descriptors in two images [14]. We consider using BP to spread out user information with local level sets. After the user labels a point inside the target, the algorithm iteratively computes the belief of assigning each pixel to the target by comparing the levels between labeled pixels and unknown pixels. And during the propagation, level set method evaluates the similarity, i.e., level, of labeled points and their neighborhood. The algorithm can start by using as few as one user labeled point, which significantly reduces user’s load during segmentation. Since the propagation is determined by local level sets, so our algorithm can completely segment the target out even it has a low contrast with its neighbors. On the other hand, a global propagation of user information avoids the disturbing from noise or because of inhomogeneous intensity value appearance. Moreover, unlike prevalent level set based interactive segmentation approaches, our method not only relies on distance between user labeled points and their neighboring points, but also considers the structural similarity of them through level sets, so it is robust to high shape variation. II. INTERACTIVE SEGMENTATION USING BELIEF PROPAGATION WITH LEVEL SETS In this section, we propose an interactive segmentation method which uses belief propagation to extend user information according to level sets. We use discrete domain in the following formulation. Let u be a given two-dimensional image, and Ω be the domain of u. If x and y represent the location of a pixel in x and y direction, respectively, u(x, y) is the corresponding pixel intensity value. Moreover, b(x, y) represents the belief that a pixel at (x, y) belongs to the target. In order to reduce user’s load during segmentation, our algorithm start the propagation by as few as one user selected point in the target. Assuming the user labels are correct, the user information can be presented as:  +1, if (x, y) labeled as target, L(x, y) = (1) 0, otherwise. The level set method is used to find out the relation between user information and its neighbors. Assume u is

segmented into two parts, the target u1 and the background u2 , which is represented by a level set function f : Ω → R, such that  f (x, y) = 0,  the boundary, inside the target, f (x, y) > 0, (2)  outside the target, f (x, y) < 0. The Heaviside function  H and its Dirac measure δ are 1, if f ≥ 0 d defined as: H(f ) = and δ = df H(f ). 0, if f < 0 Considering the interactive segmentation combines the image features with user information, we define the following cost function: F = λU FU + λI FI ,

(3)

where FU and FI represent the energy functional decided by user information and image information, respectively. λU , λI > 0 are fixed parameters. Next, we propose our method about spreading the user information from a local pixel to the whole target according to level sets using BP. Assume i and j are points in the image, j is i’s neighbor, and i, j ∈ Ω. Let m represent the set of segmentation results which may include extrema for the cost function (3), mi ∈ [0, 1], and mi = 1 is for the pixel i inside the target, while mi = 0 indicates i outside the target. In our algorithm, m is decided by calculating the level set and BP. In level sets, to a specific pixel, its adjacent pixel has a better chance of being in the same level as it, i.e., mj −H(fi ) is more likely to be 0 when i and j become closer. So in the whole image, the cost function for FU is formulated as: XX FU = ((xi − xj )2 + (yi − yj )2 )|mj − H(fi )|, (4) i∈Ω j∈Ωi

where Ωi is the neighborhood of i, j ∈ Ωi . And the segmentation using image feature will be directly determined by the level set X FI = |mi − H(fi )|, (5) i∈Ω

f is computed by level set method. Because the page limitation, we refer the reader to relative literature, e.g., [1, 5], for more detail about level set method and variational formulation in image segmentation. The minimization of cost function (3) can be rewritten as min (F ) = max (exp(−λI FI ) exp(−λU FU )).

(6)

Moreover, (6) can be presented as max-product BP algorithm [11, 12, 14], ˆ = arg max exp(−λU FU (m)) exp(−λI FI (m)), m

(7)

m

which is equivalent to Y ˆ = arg max m m

i∈Ω,j∈Ωi

bU (mi , mj )

Y

i∈Ω

bI (mi ),

(8)

where bU (mi , mj ) = λU exp(−((xi − xj )2 + (yi − yj )2 )|mj − H(fi )|), bI (mi ) = λI exp(−|mi − H(fi )|).

(9) (10)

image, and we randomly select one point in “a”, as shown in blue in Fig. 1a. Fig. 1b shows that the user information is extending from the user labeled pixel to its neighborhood. And Fig. 1c presents the segmentation result, which indicates that “a” is fully segmented out from the other letters.

The overall algorithm is summarized as following steps: 1) Start from setting mi = 1 if i is the user selected points, and mi = 0 otherwise. And initialize f as  (0) f > 0, if pixel at(x, y) is labeled as target, (11) f (0) < 0, otherwise. 2) Calculate the beliefs for all pixels of u (0) a) Initialize messages µi,j (mj ) as a constant, n = 1. (n) b) Update messages µi,j iteratively for all i ∈ Ω (n) µi,j (mj )

← max bU (mi , mj )bI (mi ) mi Q (n−1) µj ′ ,i (mi ). (12) j ′ ∈Ωi \j

c) Compute beliefs for all i ∈ Ω: Y (n) (n) bi (mi ) ← bI (mi ) µj,i (mi ),

(a) Image with a point labeled in “a”

(b) During evolution

(c) Segmentation result

Fig. 1: Interactive segmentation of a synthetic image. Fig. 2 is a zoom-in display of neuronal nuclei segmentation. Assume the user labels a point near the boundary but inside one of the nuclei, Fig. 2a. Fig. 2b-2d show that the user information is propagated inside the target without leaking into the surrounding area though the labeled point is near the edge, i.e., the user information is extended according to the level sets instead of just distance.

(13)

j∈Ωi

(n)

m ˆi = arg max bi (mi ). mi

(14)

d) Go to 2b until n reaches the maximum number of iterations. 3) Let u′ represent the region which has bi > thr, where thr is a threshold value. For the pixels in u′ , calculate their level set f ′ by: ′

′

f ′(n ) = f ′(n −1) + △t(δ(f )((u′ (x, y) − u′2 )2 ′

+(u (x, y) −

u′1 )2 ))

(a) Labeled original image

(b) During evolution

(c) During evolution

(d) Segmentation result

(15)

whereu′1 and u′2 are the average intensity values inside and outside f ′ , respectively. In this paper, we only use contrast as the criterion in level set method, other criteria can be added to the level set function to represent different image features, e.g., gradient for edges. For more details about the criteria of level set methods, please refer to [1]. Update f using f ′ by fi = fi′ for all i which has bi > thr. 4) If f (n) = f (n−1) , stop and let the region which has fi > 0 be the target, otherwise set the pixels of H(fi ) > 0 to mi = 1, and go to step 2. III. EXPERIMENTAL RESULTS In this section, we present the experimental results of medical images. The user labeled pixels are indicated by blue points, and the targets are outlined by green lines in the results. Fig. 1 shows the interactive segmentation of a synthetic image. Assume we only want to segment letter “a” from the

Fig. 2: Interactive segmentation of a neuronal nuclei. The interactive segmentation result of left ventricle is presented in Fig. 3, where the left ventricle is in a “∩” shape. This example further validates that our algorithm can handle targets which are not in regular shapes. Figs. 4 and 5 compare the ground truth and the segmentation results obtained by using our method in liver and tumor segmentation, respectively. Fig. 4 shows a liver segmentation. Liver is one of the most difficult anatomic structures to be segmented out in medical imaging, because it

Further research includes extending the current algorithm for three dimensional segmentation; adjusting the level set criteria to manage special shapes; discussing the convergence and computation of current algorithm; evaluating the performance; adapting the algorithm to handle the image with multiple labeled points, inside or outside the target. R EFERENCES Fig. 3: Segmentation result of a left ventricle. is highly variational in shape, inhomogeneous in appearance, and connective with other anatomic structures. As show in Fig. 4a, the liver is in an “I” shape and adjacent to a low contrast structure in the CT image. Comparing the ground truth in Fig. 4b, we can see the liver is completely segmented out among a bunch of other anatomic structures. Moveover, Fig. 5 shows promising result of liver tumor segmentation, in spite of the low contrast between intensity values of the liver and the tumor.

(a) Segmentation result of liver

(b) Manual segmentation result

Fig. 4: Liver segmentation using our method.

(a) Segmentation result of tumor

(b) Manual segmentation result

Fig. 5: Tumor segmentation using our method. IV. SUMMARY In this paper, we propose an interactive image segmentation method based on BP and level set method. After user labels a point inside the target, our algorithm spreads out the user information according to level set using BP. Our method can start by using as few as one labeled pixel, which highly reduces user’s load in segmentation. The segmentation relies on the level sets of target, which are formulated with image features. The algorithm can be easily adjusted to handle special shapes by changing the criteria of level sets. Various experiment results validate the advantages of our method.

[1] Y. Zhu and K. Tieu, “Exploiting user labels with generalized distance transforms random field level sets,” in Proc. IEEE Int. Symp. Biomedical Imaging, 2010. [2] N. Ben-Zadok, T. Riklin-Raviv, and N. Kiryati, “Interactive level set segmentation for image guided therapy,” in IEEE Int. Symp. Biomedical Imaging, 2009. [3] S. D. Olabarriagaa and A. W. M. Smeuldersb, “Interaction in the segmentation of medical images: A survey,” Medical Image Analysis, vol. 5, pp. 127–142, 2001. [4] D. Mumford and J. Shah, “Optimal approximation by piecewise smooth functions and associated variational problems,” Communications On Pure & Applied Mathematics, vol. 42, pp. 577–685, 1989. [5] T. F. Chan and L. A. Vese, “Active contours without edges,” IEEE Trans. Image Processing, vol. 10, no. 2, pp. 266–277, 2001. [6] D. Cremers, O. Fluck, M. Rousson, and S. Aharon, “A probabilistic level set formulation for interactive organ segmentation,” in Proc. SPIE Medical Imaging: Image Processing, 2007. [7] J. Pearl, Probabilistic reasoning in intelligent systems: networks of plausible inference, Morgan Kaufmann, San Francisco, first edition, 1988. [8] C. M. Bishop, Pattern recognition and machine learning, Springer, 2006. [9] F. Kschischang, B. Frey, and H. Loeliger, “Factor graphs and the sum-product algorithm,” IEEE Trans. Information Theory, vol. 47, pp. 498–519, 2001. [10] Y. Weiss and W. T. Freeman, “On the optimality of solutions of the max-product belief-propagation algorithm in arbitrary graphs,” IEEE Trans. Information Theory, vol. 47, no. 2, pp. 736–744, 2001. [11] T. Werner, “A linear programming approach to maxsum problem: A review,” Tech. Rep., Czech Technical University, 2005. [12] J. Sun, N. Zheng, and H. Shum, “Stereo matching using belief propagation,” IEEE Trans. Information Theory, vol. 25, no. 7, pp. 787–800, 2003. [13] N. Komodakis and G. Tziritas, “Image completion using efficient belief propagation via priority scheduling and dynamic pruning,” IEEE Trans. Image Processing, vol. 16, no. 11, pp. 2649–2611, 2007. [14] S. Cheng, V. Stankovic, and L. Stankovic, “Improved sift-based image registration using belief propagation,” in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing, 2009.