Efficient Searching of Globally Optimal and Smooth ...

Efficient Searching of Globally Optimal and Smooth Multi-Surfaces with Shape Priors ⋆ Lei Xu1 Branislav Stojkovic1 Hu Ding1 Qi Song2 2 2 Xiaodong Wu Milan Sonka Jinhui Xu1 1

Department of Computer Science and Engineering State University of New York at Buffalo Buffalo, NY 14260, USA {lxu, bs65, huding, jinhui}@buffalo.edu 2 Department of Electrical and Computer Engineering University of Iowa Iowa City, IA 52242, USA {qi-song, xiaodong-wu, milan-sonka}@uiowa.edu

Abstract. Despite extensive studies in the past, the problem of segmenting globally optimal multiple surfaces in 3D volumetric images remains challenging in medical imaging. The problem becomes even harder in highly noisy and edge-weak images. In this paper we present a novel and highly efficient graphtheoretical iterative method based on a volumetric graph representation of the 3D image that incorporates curvature and shape prior information. Compared with the graph-based method, applying the shape prior to construct the graph on a specific preferred shape model allows easy incorporation of a wide spectrum of shape prior information. Furthermore, the key insight that computation of the objective function can be done independently in the x and y directions makes local improvement possible. Thus, instead of using global optimization technique such as maximum flow algorithm, the iteration based method is much faster. Additionally, the utilization of the curvature in the objective function ensures the smoothness. To the best of our knowledge, this is the first paper to combine the shape-prior penalties with utilizing curvature in objective function to ensure the smoothness of the generated surfaces while striving for achieving global optimality. To evaluate the performance of our method, we test it on a set of 14 3D OCT images. Comparing to the best existing approaches, our experiments suggest that the proposed method reduces the unsigned surface positioning errors form 5.44 ± 1.07(µm) to 4.52 ± 0.84(µm). Moreover, our method has a much improved running time, yields almost the same global optimality but with much better smoothness, which makes it especially suitable for segmenting highly noisy images. The proposed method is also suitable for parallel implementation on GPUs, which could potentially allow us to segment highly noisy volumetric images in real time.

1

Introduction

Efficient extraction of globally optimal multi-surfaces in volumetric images is one of the most challenging problems in imaging processing, especially in the presence of high noise, weak edges and high interacting surfaces. The problem arises not only in the segmentation problem of biomedical images (e.g., CT, MRI, Ultrasound, Microscopy, Optical Coherence Tomography (OCT)), but also in many other fundamental optimization problems such as surface reconstruction [8], data mining [5], ore mining [4], metric labeling [3], and radiation treatment planning [3]. In recent years, graph based methods with a global energy optimization property have attracted considerable attention in computer vision, such as minimum spanning tree [6], shortest paths and their 3D extensions [1], graph cuts [2] and graphs search framework [8, 12, 10, 9]. In those multiple surfaces detection tasks, the shape priors information plays a critical role since it provides a more local and flexible control of shape. In [11], Song et al. proposed a new detection method for optimal surfaces. Their optimality is controlled by a cost function using the voxel weights plus the shape-prior penalties and some geometric constraints on the connectivity of the surface. Their approach can compute globally optimal surfaces in polynomial time. However, since their cost function only considers the weight and the shape-prior penalties of the voxels, the resulting surface often contains ⋆

The research of the first three and the last authors was supported in part by NSF through a CAREER Award CCF0546509 and a grant IIS-0713489. The research of the other three authors was supported in part by the NSF grants CCF-0844765 and the NIH grants K25 CA123112 and R01 EB004640.

spikes and jaggedness in noisy regions. To fix this problem, Xu et al. proposed a new segmentation model which emphasizes both the global optimality and local smoothness [14]. The approach uses an upper-bounded (mean) curvature as a penalty to avoid spikes and achieves global optimality and smoothness simultaneously. A faster implementation of the method was also presented in [15]. However, since it does not include any shape prior information, the iterative local improvement based method could not segment multiple high interacting surfaces in the regions with high noise and weak edges. In this paper, we proposed a novel approach to combine the mean curvature and the shape-prior penalties into one objective function. Thus, we are able to segment high interacting surfaces in noisy image with much more smoothness.

2

Description of Problem

Given a 3D image I (see Figure 1) of size nx ×ny ×nz and with each voxel I (x, y, z) associated with a non-negative weight w (x, y, z) representing the inverse of the probability that I (x, y, z) is a boundary voxel, the problem is to find a set S of λ terrain surfaces such that each column of I intersects each surface Si at exactly one voxel. In simultaneously detecting λ (λ ≥ 2) distinct but interrelated surfaces, the optimality is also confined by their interrelations. Thus additional constraints are added to model the desired relations between the surfaces. For each (x, y) pair, let p(x, y) denote the voxel subset {I (x, y, z)|0 ≤ z ≤ nz } forming a column parallel to the z-axis. A surface interacting constraint is added to the column p(x, y) for each pair of the sought surfaces Si ¯ where δ and δ¯ are two specified surface interacting and Sj . For each p(x, y), we have δ ≤ Si (p) − Sj (p) ≤ δ, ¯ parameters for Si and Sj . Note that if δ · δ ≤ 0, two surfaces may cross each other at the same voxel. However, it does not happen in some medical image applications. Thus, we also required that (1) any pair of surfaces does not cross and (2) two interacting surfaces can not pass through the same voxel.

Fig. 1: Orientation of surface.

Fig. 2: Curvature at P.

Curvature captures the intrinsic geometric properties of surfaces and has been widely used for measuring smoothness [7, 13]. For 3D surfaces, there are two types of commonly used curvatures, mean curvature and Gaussian curvature. In previous work [14, 15], the mean curvature was approximated by using the curvatures along the x and y directions. One advantage of the approximation is the freedom to de-couple the curvature computation in the two directions. In our proposed approach, we adopt this idea to ensure the smoothness of surfaces. Each x and y curvature is the curvature of a 2D curve. For a 2D curve given explicitly as a function of |y ′′ | y = f (x), its curvature is κ = (1+y ′2 )3/2 . Intuitively, the curvature of a smooth curve at point P (see Figure 2) is defined as the inverse of its minimum curvature radius (i.e., the radius of the osculating circle at P ). Thus, we add κi (x, y, z) to the energy cost function where κi (x, y, z) is the mean curvature of Si at I (x, y, z). To incorporate the shape information, two kinds of shape constraints are enforced: the hard shape constraint and the shape-prior penalties. The hard shape constraint is defined as follows: ∆ip,q ≤ Si (p) − Si (q) ≤ ∆¯ip,q ,

¯ip,q are the shape constraint parameters between p and q for Si . The shape-prior penalties can where ∆ip,q and ∆ i i be expressed as fp,q (Si (p) − Si (q)), where fp,q is a convex function penalizing the shape changes of Si on p and q. The overall energy of Si takes the form: X X i c(Si ) = αw (x, y, z) + (1 − α)|κi (x, y, z)| + fp,q (Si (p) − Si (q)), I (x,y,z)∈Si

(p,q)∈N

where (1) α ∈ [0, 1] is a weighting constant and (2) Each column has a set of neighbors for a certain neighbor Pλ setting N , e.g., four-neighbor relationship. Thus, the overall energy of the set S of λ surfaces is i=1 c(Si ). Our goal is to find an optimal set of λ surfaces satisfying the surface interacting constraint and the hard shape constraint with minimum overall energy.

3

The Methods

We propose a graph-theoretic approach to solve this problem. To illustrate the idea more clearly, we first show how to search a single surface (i.e., λ = 1) in the 3D volumetric image in which there is no surfaces interacting constraint. After that, we present how to extend the idea from single surface searching to multi-surfaces searching.

(a)

(b)

(c)

(d)

Fig. 3: Reduce to the shortest path problem.

In the single surface searching problem, our main idea is to reduce the problem to a sequence of shortest path problems in 2D and 3D spaces. Our approach is based on the following three observations: (a) The curvature can be computed independently in the x and y directions (e.g., the x-curvature of I (x, y, z) depends only on voxels in the neighboring columns Col(x − 1, y), Col(x, y) and Col(x + 1, y)); (b) The curvature in one direction can be computed by using only a constant number (e.g., 3) of neighboring voxels. (c)The shape-prior penalties can be computed independently in the x and y directions by using two neighboring voxels. With these observations, we can extend the work in [14] and show that the single surface searching problem in an x-slice I (i, , ) (i.e., all voxels with the same x coordinate i) can be reduced to a shortest path problem in a graph G3x (i) and solved optimally, where each node in G3x (i) is a 3-tuple of voxels from three consecutive columns C(i, j − 1), C(i, j), C(i, j + 1) respectively. The cost of the resulting path includes the weight, the shape-prior penalties with the two neighbors in y-direction and the y-curvature of each voxel on the path. The running time for each slice is O(ny nz △3 (△ + log(ny nz △3 ))), where △ is the maximum value of height difference of two neighboring voxels which can be connected in the surface. Thus its value is directly related to the hard shape constraint. Generalizing this 2D technique by considering 3 x-slice I (i, , ), I (i − 1, , ) and I (i + 1, , ) at one time, the optimal surface can be obtained in polynomial time for those three slices (see Figure 3). In this partial surface, the cost for slices other than the first and last slices includes the weight, the mean curvature and the shape-prior penalties of each voxel on the surface.

(a)

(b)

(c)

Fig. 4: Stitching.

To form a complete surface, we follow the idea in [14] to first generate a set of partial surfaces, with each partial surface spanning a constant number (e.g., 3) of slices so that every pair of consecutive partial surfaces overlap at one slice. Then, we stitch those overlapping partial surfaces (see Figure 4). To stitch two partial surfaces in slice I (i, , ), we first fix the selected voxels in slices I (i − 1, , ) and I (i + 1, , ), and then convert the stitching problem into a 2D shortest path problem in graph G3x (i). Different from the single slice case, the cost function in this case includes the weight, the mean curvature and the shape-prior penalties of voxels in slice I (i, , ), as well as the adjustment of the x-curvatures and shape-prior penalties with two neighbors in x-direction for those selected voxels in slices I (i − 1, , ) and I (i + 1, , ). To further improved the obtained surface, we repeatedly apply the stitching algorithm on the surface with each time shifting the to-be-stitched slices by one slice. For instance, if the current round of stitching is on slices I (1, , ), I (3, , ), · · ·, I (2i − 1, , ), · · · , then in the next round, the stitching is on slices I (2, , ), I (4, , ), · · ·, I (2i, , ), · · ·. A nice property of stitching is that the cost of the surface will monotonically decrease in each round, and an optimal or near optimal solution can be finally obtained with enough number of rounds. Thus the obtained surfaces are iteratively improved through local computation, with each iteration strictly reducing the objective value. To solve the searching for multi-surfaces problem, we extend the method of searching for single surface problem discussed in the above section. Following the idea in [14], we first construct a set of feasible surfaces S1 to Sλ with less cost by either repeating the above procedure for single surface searching or direct applying the edge-disjoint shortest path algorithm. Note both the surface interacting constraint and the hard shape constraint have to be satisfied during this step. Once we obtain a set of feasible surfaces, we follow the same idea of single surface searching to do the stitching. For one iteration, the set of λ surfaces is stitched one by one similar to the single surface case. Again, the interacting constraint has to be satisfied during the whole process.

4

Experimental Results

The proposed algorithm was examined on 14 datasets of 3D OCT images (200 × 200 × 256) with the voxel size (6 × 6 × 2µm). The seven surfaces we try to find are shown in Figure 5(a). The average of the two tracings from two human experts were used as the reference standard. In our experiments, surfaces 1,6, and 7 with relatively strong boundaries were simultaneously detected without incorporating the shape prior penalty to the above energy cost function. The above approach with the convex penalties of the shape priors was then used to segment the remaining surfaces 2,3,4 and 5, which lack clear boundaries and have substantial interactions in between. To properly incorporate the shape prior information, we used the manual tracing results as the training datasets. The mean and standard deviation of how the z-value changes form column p to its neighboring column q (i.e., Si (p) − Si (q)) were learned for each surface. Let d¯p,q and σp,q denote the mean and the standard deviation

Table 1: Summary of Mean Unsigned Surface Positioning Errors

Surface curvature penalty 2 3.10 ± 0.41 3 6.54 ± 1.21 4 4.11 ± 0.96 5 4.34 ± 0.76 Overall 4.52 ± 0.84 1

1

no curvature penalty 1 4.90 ± 1.01 6.08 ± 0.98 5.59 ± 0.72 5.17 ± 0.91 5.44 ± 1.07

Mean ± standard deviation in µm.

of Si (p) − Si (q) for surface i, respectively. To allow for 99% of the shape changes from column p to column q (assuming a normal distribution), the two hard shape constraints were set as: ∆ip,q = d¯p,q − 2.6 · σp,q and ∆¯ip,q = d¯p,q + 2.6 · σp,q . For soft shape prior penalty, we employed a quadratic function, fp,q (x) = a(x − d¯p,q )2 , where a was a weighting coefficient. In our experiments, we choose a = 5 according to experiments on the training sets.

(a)

(b)

Fig. 5: (a) Original image; (b) Example result (green) and reference standard (red).

For validation, 7 datasets were used as the training data and the remaining 7 datasets were used for performance assessment. We mainly focused on segmenting surfaces 2,3,4 and 5 to show the effectiveness of incorporating the shape prior penalty. Our algorithm is implemented by C++ and LEDA (Library of Efficient Data types and Algorithms)-5.2. The experiments were conducted on a Linux workstation (2.4 GHz, 4GB memory). The segmentation performance with α = 0.75 is shown quantitatively in Table 1. Our method produced either more accurate or comparable segmentation results compared with the previous method without the curvature penalty [11]. Figure 5 (b) shows an example result (green) and the reference standard (red) on one 2-D slice from the 3-D volume.

5

Discussion and Conclusion

Compared to the previous methods, the graph-based method in [14] that utilize curvature in the objective function to ensure smoothness could not segment high interacting surfaces such as the layers 2,3,4 and 5 in

the OCT images. On the other hand, applying the shape prior to construct the graph on a specific preferred shape model allows easy incorporation of a wide spectrum of shape prior information. The method in [11] could simultaneously search globally optimal interacting surfaces (e.g., seven layers in the OCT images)with priors. However, the method suffers from the relatively slow running time and the non-smoothness in some regions. The novelty of this technique is the combination of the mean curvature and the shape-prior penalties into one objective function. Furthermore, the computation of the objective function can be done independently in the x and y directions. Thus, instead of using global optimization technique such as maximum flow algorithm, the iteration based method is much faster. Additionally, the utilization of the curvature in the objective function ensures the smoothness. In this paper, we present a novel approach incorporating curvature and shape-prior penalties to search multisurfaces in the presence of high noise and weak edges. To our best knowledge, this is the first paper combining the shape-prior penalties with curvature as the surface smoothness measurement. Compared with other graph-based methods computing the maximum flow of the graph, our algorithm has a much improved running time. Our approach not only provides more flexible control of the shape but also ensures the smoothness of surfaces as well. The proposed method is also suitable for parallel implementation on GPUs, which could potentially allow us to segment highly noisy volumetric images in real time.

References 1. Roberto Ardon, Laurent D. Cohen, and Anthony J. Yezzi. A new implicit method for surface segmentation by minimal paths: Applications in 3d medical images. In Energy Minimization Methods in Computer Vision and Pattern Recognition, pages 520–535, 2005. 2. Yuri Boykov and Vladimir Kolmogorov. Computing geodesics and minimal surfaces via graph cuts. In International Conference on Computer Vision, pages 26–33, 2003. 3. Chandra Chekuri, Sanjeev Khanna, Joseph Naor, and Leonid Zosin. A linear programming formulation and approximation algorithms for the metric labeling problem. SIAM J. Discret. Math., 18(3):608–625, 2005. 4. Denby and Schofield. Genetic algorithms: A new approach to pit optimization. Proc. 24th Int.Symp. on the Application of Computers and Operations Research in the Mineral Industry, pages 126–133, 1993. 5. Takeshi Fukuda, Yasuhiko Morimoto, Shimichi Morishita, and Takeshi Tokuyama. Data mining with optimized two-dimensional association rules. ACM Trans. Database Syst., 26(2):179–213, 2001. 6. Pedro F. Felzenszwalb and Daniel P. Huttenlocher. Efficient graph-based image segmentation. Int. J. Comput. Vision, 59(2):167–181, 2004. 7. Michael E. Leventon, W. Eric L. Grimson, Olivier Faugeras and William M. Wells III. Level Set Based Segmentation with Intensity and Curvature Priors, In IEEE Workshop on Mathematical Methods in Biomedical Image Analysis, 2000. 8. Kang Li, Xiaodong Wu, Danny Z. Chen, and Milan Sonka. Optimal surface segmentation in volumetric images-a graph-theoretic approach. IEEE Trans. Pattern Anal. Mach. Intell., 28(1):119–134, 2006. 9. Mona K. Garvin, Michael D. Abr` amoff, Randy kardon, Stephen R. Russell, Xiaodong Wu, and Milan Sonka. Intraretinal Layer Segmentation of Macular Optical Coherence Tomography Images Using Optimal 3-D Graph Search In em IEEE Trans Med Imaging, vol. 27(10), pp. 1495-1505, 2008. 10. Mona Haeker, Xiaodong Wu, Michael D. Abr` amoff, Randy Kardon and Milan Sonka. Incorporation of Regional Information in Optimal 3-D Graph Search with Application for Intraretinal Layer Segmentation of Optical Coherence Tomography Images. In Information Processing in Medical Imaging, pages 607–618, 2007. 11. Qi Song, Xiaodong Wu, Yunlong Liu, Milan Sonka, Mona Kathryn Garvin. Simultaneous searching of globally optimal interacting surfaces with shape priors. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2879–2886, 2010. 12. Xiaodong Wu and Danny Z. Chen. Optimal net surface problems with applications. In International Colloquium on Automata, Languages and Programming, pages 1029–1042, 2002. 13. Christopher L. Wyatt, Yaorong Ge, and David J. Vining. Segmentation in virtual colonoscopy using a geometric deformable model. In Computerized Medical Imaging and Graphics, vol. 30 (1), pages 17-30, 2006. 14. Lei Xu, Branislav Stojkovic, Yongding Zhu, Qi Song, Xiaodong Wu, Milan Sonka and Jinhui Xu. Efficient Algorithms for Segmenting Globally Optimal and Smooth Multi-surfaces. In Information Processing in Medical Imaging, pages 208-220, 2011. 15. Lei Xu, Branislav Stojkovic, Hu Ding, Qi Song, Xiaodong Wu, Milan Sonka and Jinhui Xu. Faster Segmentation Algorithm for Optical Coherence Tomography Images with Guaranteed Smoothness. In Machine Learning in Medical Imaging - Second International Workshop, Held in Conjunction with MICCAI 2011, pages 308-316, 2011.