An interactive editing framework for electron microscopy image

An Interactive Editing Framework for Electron Microscopy Image Segmentation Huei-Fang Yang and Yoonsuck Choe Department of Computer Science and Engineering Texas A&M University College Station, TX 77843-3112 [email protected], [email protected]

Abstract. There are various automated segmentation algorithms for medical images. However, 100% accuracy may be hard to achieve because medical images usually have low contrast and high noise content. These segmentation errors may require manual correction. In this paper, we present an interactive editing framework that allows the user to quickly correct segmentation errors produced by automated segmentation algorithms. The framework includes two editing methods: (1) editing through multiple choice and (2) interactive editing through graph cuts. The first method provides a set of alternative segmentations generated from a confidence map that carries valuable information from multiple cues, such as the probability of a boundary, an intervening contour cue, and a soft segmentation by a random walker. The user can then choose the most acceptable one from those segmentation alternatives. The second method introduces an interactive editing tool where the user can interactively connect or disconnect presegmented regions. The editing task is posed as an energy minimization problem: We incorporate a set of constraints into the energy function and obtain an optimal solution via graph cuts. The results show that the proposed editing framework provides a promising solution for the efficient correction of incorrect segmentation results.

1

Introduction

With the significant progress made in the image acquisition process, electron microscopy (EM) instruments are now able to image tissues at a nanometer resolution and thus produce large-scale data volumes, which require image analysis methods for connectomic reconstruction. Therefore, researchers have put in a great amount of effort in developing segmentation algorithms to extract neuronal morphologies from stacks of serial EM images [1,2,3,4,5,6]. A great majority of these existing segmentation algorithms are designed for the automation of the segmentation pipeline. However, fully automated segmentation algorithms sometimes yield incorrect results even when their parameters are optimally tuned, which is mainly because of their failing to capture all variations posed by the data sets to be processed. Thus, erroneous segmentations would inevitably arise, and they may require manual correction. G. Bebis et al. (Eds.): ISVC 2011, Part I, LNCS 6938, pp. 400–409, 2011. c Springer-Verlag Berlin Heidelberg 2011 

An Interactive Editing Framework

401

Whereas much attempt has been made to design segmentation algorithms, relatively little attention has been given to the development of editing tools [7]. Hence, editing software for segmented EM imaging data remains in its infancy [8], and more effort in this area is needed. In an effort to provide editing tools, in this paper, we present an interactive editing framework allowing the user to refine segmentation results with minimal user interaction. The framework includes two editing methods: (1) editing through multiple choice that provides a set of segmentation alternatives from which a user can select an acceptable one and (2) interactive editing through graph cuts that allows a user to edit segmentation results by interactively placing editing strokes. The first editing method aims to minimize user interaction by providing a set of potential segmentations. The editing process begins with combining multiple cues to generate a confidence map that indicates the degree of each pixel belonging to a specific label. By thresholding the confidence map using different values, a pool of segmentations are generated. The user can choose the most acceptable one, if available, among the automatically generated segmentations before starting any manual editing. Allowing the user to select the desired segmentation prior to manual editing will simplify the editing process. The second editing method utilizes recent advances in interactive segmentation algorithms. These algorithms take a few of the user inputs, either scribbles on foreground/background objects [9,10] or contour points belonging to an object’s boundary [11], and produce a segmentation result accordingly. Our editing tools are based on the interactive graph cuts algorithms [9], where the editing task is cast as an energy minimization problem. We incorporate a set of constraints into the energy function and then obtain an optimal solution via graph cuts. The remainder of this paper is organized as follows. Section 2 details the approach for generating the confidence map and a set of potential segmentations. Section 3 elaborates on the editing tool allowing the user to place editing strokes to refine segmentation results. Section 4 offers several results followed by conclusions in section 5.

2

Editing through Multiple Choice

Image segmentation is an ill-posed problem. A particular feature used or parameter choice for a segmentation algorithm strongly affects the quality of segmentations, thus many researchers have considered multiple segmentations for an image for the segmentation task [12]. Inspired by this, we generate a pool of candidate segmentations by thresholding a confidence map at different values. The user thus can choose the correct segmentation, if available, from these generated segmentations. 2.1

Cues for Confidence Map Generation

It is shown in [13] that the visual system integrates different attributes (e.g., luminance, color, motion, and texture) for contour localization because all

402

H.-F. Yang and Y. Choe

(a) Image

(b) PB

(c) RW

(d) IC(1)

(e) IC(2)

(f) Conf. map

(g) Seg. 1

(h) Seg. 2

(i) Seg. 3

(j) Seg. 4

Fig. 1. Generation of the confidence map and segmentation alternatives. (a) is the original image. (b), (c), (d), and (e) show the results produced by different cues. (f) is the confidence map generated by a linear combination of different cues. Note that a brighter color indicates a higher probability of belonging to an object. (g), (h), (i), and (j) are segmentations obtained by thresholding the confidence map at 0.4, 0.5, 0.6, and 0.7, respectively.

attributes play an essential role for a contour localization task. Motivated by this, we use multiple cues, analogous to different attributes in localization of contours, to generate a confidence map that indicates the degree of a pixel belonging to a label. These cues are: – Probability of boundary (PB). Boundary is an important cue for distinguishing different segments in the image segmentation task. This cue takes the output of a classifier that provides the posterior probability of a boundary at each pixel. The classifier is trained by combining local brightness, color, and texture features in order to accurately detect the boundary [14]. Figure 1(b) shows the probability of a boundary of the image in Figure 1(a). – Random walker segmentation (RW). In contrast to a hard segmentation, that is, a pixel belonging to either the foreground (1) or not (0) for the binary case, random walker segmentation [15] produces a soft segmentation, as shown in Figure 1(c), obtained by determining the probability a random walker starting from an unlabeled pixel reaches to the user pre-labeled pixels. The segmentation is obtained by assigning each point to the label that is of the largest probability. – Intervening contour (IC). The intervening contour [16] concept suggests that pixels on the two different sides of a boundary are more likely to belong to different segments. Given two pixels on an image, the affinity between them is measured as the maximum gradient magnitude (or other measurements) of a straight-line path between them, as shown in Figure 1(d) (IC1).

An Interactive Editing Framework

403

In addition to using gradient magnitude as a measurement, we also compute the affinity between pixels based on the probability of a boundary. Figure 1(e) (IC2) shows the result of considering the probability of a boundary as a measurement in computing an intervening contour cue. 2.2

Cue Combination

Before all ofthe individual cues are combined, their values are normalized to [0, 1]. Let P ckp  I) be the value at pixel p produced by cue k. The confidence value at pixel p is defined as a linear combination of all cues, which is    P (cp |I) = wk P ckp |I , (1) k

where wk is the relative importance of cue k, and sum of all wk is equal to 1. The weight for each cue was set empirically. The confidence map shown in Figure 1(f) is a combination of Figure 1(b) through Figure 1(e). Note that the brighter color indicates a higher probability of belonging to an object. 2.3

Generation of Segmentation Alternatives

To generate multiple segmentation alternatives of an image, the confidence map is thresholded at different values. Figure 1(g) through Figure 1(j) show the segmentation alternatives obtained by using the threshold values between 0.4 and 0.7. The user can choose the most acceptable segmentation, if one existed, from these alternatives before applying any editing. This reduces the amount of time for user interaction.

3

Interactive Editing through Graph Cuts

User interaction to correct erroneous segmentation results is key to providing accurate segmentations. We cast the interactive editing problem as the Maximum A Posteriori (MAP) estimation of Markov Random Field (MRF). The MAPMRF formulation minimizes a Gibbs energy function that is defined over the user input, presegmentation1, and image intensity. Graph cuts are then used to obtain the optimal solution to the energy function. 3.1

Editing Energy Function

The interactive editing aims to obtain a new segmentation that satisfies a set of constraints: the user input, presegmentation, and image data. Analogous to image segmentation, the interactive editing is a labeling problem which involves assigning image pixels a set of labels [17]. Consider an image I containing the set of pixels P = {1, 2, ..., M }, a set of labels L = {li , l2 , ..., lK }, 1

Here, following the definition from [10], we define presegmentation as the prior, incorrect, pre-existing segmentation.

404

H.-F. Yang and Y. Choe

(a)

(b)

(c)

(d)

(e)

Fig. 2. The difference between using the Euclidean distance and using the intervening contour cue in the unary potential. The blue and green scribbles in (c) indicate the foreground and background marks, respectively. (d) shows the penalty, based on the Euclidean distance, for changing the pixels classified as the background in (b) to the foreground. (e) shows the penalty by using the intervening contour cue. The color clearly shows transitions from black (low penalty) to gray (high penalty) when the intervening contour cue is used, and the transitions coincide with the object boundary.

the user input U = {up : p ∈ P, up ∈ L}, and a nearly correct presegmentation y = {yp : p ∈ P, yp ∈ L}. We seek a new optimal labeling x by means of minimizing an energy function given the user input U , presegmentation y, and image data D. The energy function is given as [18]:    Vp (xp | y, U, D) + Vpq (xp , xq | D) , (2) E (x | y, U, D) = p∈P

p∈P q∈Np

where Np is the set of neighboring pixels of p, Vp (xp | y, U, D) is the unary clique potential, and Vpq (xp , xq | D) is the piecewise clique potential. We incorporate the user input and presegmentation into the unary potential and impose the boundary smoothness in the piecewise potential. 3.2

User Input and Presegmentation Constraints

To solve Equation 2 via graph cuts, two terminal nodes, s and t, are added into the graph. The unary potential defines the weights between a node p and terminal nodes. Because a presegmentation is nearly correct, the new segmentation should be similar to the presegmentation after editing. Only the pixels with changed labels are penalized. The penalty for changing a label is defined as:  ∞ p ∈ Uf (3) wsp = ICb (p) p ∈ / Uf 

and wpt =

∞ ICf (p)

p ∈ Ub , p∈ / Ub

(4)

where Uf and Ub are the foreground and background labels, respectively, and ICf (p) and ICb (p) are the affinities between a pixel p to the nearest of the user labeled pixels Uf and Ub , respectively. The unary potential is similar to that

An Interactive Editing Framework

405

in [10] that suggests use of the Euclidean distance from a pixel to the nearest of the user labeled pixels; however, our work considers the intervening contour cue that is important for distinguishing different objects. Figure 2 demonstrates the difference in using Euclidean distance and the intervening contour cue as the unary potential. As can be seen from Figure 2(e), the color clearly shows transitions from black (low penalty) to gray (high penalty), whereas no such transitions are shown in Figure 2(d) in which the penalties are given based on the Euclidean distance. The transitions in Figure 2(e) coincide with the object boundary, where a cut is more likely to occur. 3.3

Image Data Constraint

Piecewise potential ensures boundary smoothness by penalizing neighboring pixels assigned different labels. Based on Potts model [9], it is given as: Vpq (xp , xq | D) = wpq · (1 − δ (xp , xq )) ,

(5)

where δ (xp , xq ) is the Kronecker delta, and wpq is a penalty for assigning two neighboring pixels, p, and q, to different labels, defined using a Gaussian weighting function:   2 (Ip − Iq ) 1 , (6) wpq = exp − · 2 2σ p − q where Ip and Iq are pixel intensities ranging from 0 to 255, p−q is the Euclidean distance between p and q, and σ is a positive parameter.

4

Results

We carried out the experiments on the Serial Block Face Scanning Electron Microscopy (SBFSEM) [19] image stack to demonstrate the effectiveness of the proposed segmentation editing methods. The volumetric larval zebrafish tectum SBFSEM data set contains 561 images, each of which has the size of 631×539 pixels. A major challenge in reconstructing neuronal morphologies from SBFSEM image data lies in segmenting densely packed cells that have blurred boundaries mainly resulting from the imperfect image acquisition process. As a result, user editing is key to resolving the boundary ambiguities, in which case a segmentation algorithm fails to produce a satisfactory result. Thirty incorrect segmentations produced by an automated segmentation algorithm were used in the experiments. A few examples of editing through multiple choice are shown in Figure 3. These examples demonstrate that the generated alternatives contain a few acceptable segmentations, showing that the confidence map obtained by integrating different cues is reliable. Figure 4 shows the examples of interactive editing through graph cuts. The first, second, third, and last columns of Figure 4 show the original images, incorrect segmentations, user edits, and editing segmentation results, respectively. The blue strokes are the foreground marks, and the green strokes are background marks. This editing

406

H.-F. Yang and Y. Choe

(a) Image

(b) Seg. 1

(c) Seg. 2

(d) Seg. 3

(e) Seg. 4

Fig. 3. Examples of generated segmentation alternatives. (a) Original images. (b)-(d) The generated segmentation alternatives by thresholding the confidence map at 0.4, 0.5, 0.6, and 0.7, respectively. As can be seen, the generated alternatives contain at least a few acceptable segmentations. This shows that the confidence map obtained by integrating different cues is reliable. As a result, generating segmentations based on the confidence map is able to provide reasonable segmentation alternatives from which the user can choose one.

tool gives the user flexibility of placing strokes on a few pixels and produces a segmentation that meets the user’s requirements. For quantitative evaluation, we applied Dice similarity coefficient (DSC) [20] and F-measure to measuring the similarity between two segmentations. DSC measures the amount of overlap between the obtained segmentation results and the ground truth. More specifically, letting Z be the set of voxels of the obtained 2|Z∩G| , segmentation results and G be the ground truth, DSC is given as DSC = |Z|+|G| where | · | is the number of voxels. A DSC value of 0 indicates no overlap between two segmentations, and 1 means two segmentations are identical. F-measure is R , where P = |Z∩G| and R = |Z∩G| are the precision and defined as F = P2P+R |Z| |G| recall of the segmentation results relative to the ground truth.

An Interactive Editing Framework

(a) Image

(b) Incorrect seg.

(c) User edit

407

(d) Result

Fig. 4. Examples of interactive segmentation editing. (a) shows the original SBFSEM images, (b) the incorrect segmentations produced by an automated segmentation algorithm, (c) the user edits, where the blue and green strokes represent the foreground and background marks, respectively, and (d) the editing results. The editing algorithm takes a number of user inputs, together with the incorrect segmentation and image data constraints, and computes a new segmentation accordingly.

When the editing through multiple choice method was applied to refining segmentations, around 15 out of 30 segmentation errors can be corrected (i.e., the user can obtain an acceptable segmentation from the generated alternatives). The DSC and F-measure for this method were 0.9670 and 0.9672, respectively. The interactive editing through graph cuts was used to correct all 30 erroneous Table 1. Quantitative evaluation results of the proposed methods. The DSC values of both methods are above 0.96, indicating the obtained editing results are highly overlapped with ground truth segmentations. Both methods yield high F-measure values as well.

Editing through multiple choice Editing through graph cuts

DSC 0.9670 0.9763

Precision 0.9767 0.9676

Recall 0.9578 0.9947

F-measure 0.9672 0.9810

408

H.-F. Yang and Y. Choe

segmentations. The DSC and F-measure for this method were 0.9763 and 0.9810, respectively. Table 1 summaries the quantitative evaluation results of the proposed framework. As we can see, the DSC values of both methods are above 0.96, indicating that the editing results are highly overlapped with the manual segmentations. Moreover, the editing through multiple choice method can correct more than 50% of the segmentation errors, minimizing the user intervention prior to applying manual editing.

5

Conclusions and Future Work

In this paper, we presented an interactive editing framework that allows the user to correct segmentation errors produced by automated segmentation algorithms. By thresholding a confidence map using different values, the proposed editing framework first obtains a pool of alternative segmentations from which the user can select the most acceptable result, aiming for minimizing user interaction. In addition, the editing framework includes an editing tool that the user can place editing marks on a few pixels to produce the desired segmentation result. The editing task is formalized as an energy minimization problem and incorporates a set of constraints, ranging from the user input and presegmentation to image data, into the energy function. Experimental results showed that the proposed editing framework provides a promising solution to the segmentation of SBFSEM image data sets. Future work includes using the knowledge of neuronal topologies to analyze the reconstruction results and to detect potential errors and thus pointing out these errors to the user for further correction, minimizing the amount of time required for manual proofreading of the reconstruction results. Acknowledgements. This work was supported in part by NIH/NINDS #1R01NS54252 and NSF CRCNS #0905041. We would like to thank Stephen J. Smith (Stanford) for the SBFSEM data and Timothy Mann for the valuable comments and proofread.

References 1. Andres, B., K¨ othe, U., Helmstaedter, M., Denk, W., Hamprecht, F.A.: Segmentation of SBFSEM volume data of neural tissue by hierarchical classification. In: Rigoll, G. (ed.) DAGM 2008. LNCS, vol. 5096, pp. 142–152. Springer, Heidelberg (2008) 2. Jain, V., Murray, J.F., Roth, F., Turaga, S.C., Zhigulin, V.P., Briggman, K.L., Helmstaedter, M., Denk, W., Seung, H.S.: Supervised learning of image restoration with convolutional networks. In: Proc. IEEE Int’l Conf. Computer Vision, pp. 1–8 (2007) 3. Jurrus, E., Hardy, M., Tasdizen, T., Fletcher, P., Koshevoy, P., Chien, C.B., Denk, W., Whitaker, R.: Axon tracking in serial block-face scanning electron microscopy. Medical Image Analysis 13, 180–188 (2009)

An Interactive Editing Framework

409

4. Kaynig, V., Fuchs, T., Buhmann, J.M.: Neuron geometry extraction by perceptual grouping in ssTEM images. In: Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 2902–2909 (2010) 5. Mishchenko, Y.: Automation of 3D reconstruction of neural tissue from large volume of conventional serial section transmission electron micrographs. J. Neuroscience Methods 176, 276–289 (2009) 6. Yang, H.F., Choe, Y.: Cell tracking and segmentation in electron microscopy images using graph cuts. In: Proc. IEEE Int’l. Symp. Biomedical Imaging, pp. 306–309 (2009) 7. Kang, Y., Engelke, K., Kalender, W.A.: Interactive 3D editing tools for image segmentation. Medical Image Analysis 8, 35–46 (2004) 8. Jain, V., Seung, H.S., Turaga, S.C.: Machines that learn to segment images: a crucial technology for connectomics. Current Opinion in Neurobiology 20, 653–666 (2010) 9. Boykov, Y., Jolly, M.: Interactive graph cuts for optimal boundary and region segmentation of objects in N-D images. In: Proc. IEEE Int’l. Conf. Computer Vision, pp. 105–112 (2001) 10. Grady, L., Funka-Lea, G.: An energy minimization approach to the data driven editing of presegmented images/volumes. In: Larsen, R., Nielsen, M., Sporring, J. (eds.) MICCAI 2006. LNCS, vol. 4191, pp. 888–895. Springer, Heidelberg (2006) 11. Mortensen, E.N., Barrett, W.A.: Intelligent scissors for image composition. In: SIGGRAPH 1995: Int’l conf. Computer Graphics and Interactive Techniques, pp. 191–198 (1995) 12. Pantofaru, C., Schmid, C., Hebert, M.: Object recognition by integrating multiple image segmentations. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part III. LNCS, vol. 5304, pp. 481–494. Springer, Heidelberg (2008) 13. Rivest, J., Cabanagh, P.: Localizing contours defined by more than one attribute. Vision Research 36, 53–66 (1996) 14. Martin, D.R., Fowlkes, C., Malik, J.: Learning to detect natural image boundaries using local brightness, color, and texture cues. IEEE Trans. Pattern Anal. Mach. Intell. 26, 530–549 (2004) 15. Grady, L.: Random walks for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 28, 1768–1783 (2006) 16. Leung, T., Malik, J.: Contour continuity in region based image segmentation. In: Burkhardt, H.-J., Neumann, B. (eds.) ECCV 1998, Part I. LNCS, vol. 1406, pp. 544–559. Springer, Heidelberg (1998) 17. Li, S.Z.: Markov random field modeling in image analysis. Springer-Verlag New York, Inc., Secaucus (2001) 18. Vu, N.: Image Segmentation with Semantic Priors: A Graph Cut Approach. PhD thesis, The University of California at Santa Barbara (2008) 19. Denk, W., Horstmann, H.: Serial block-face scanning electron microscopy to reconstruct three-dimensional tissue nanostructure. PLoS Biology 2, e329 (2004) 20. Dice, L.R.: Measures of the amount of ecologic association between species. Ecology 26, 297–302 (1945)