Extension of 2D segmentation methods into 3D by means of Coons-patch interpolation Ivo Wolf, Amir Eid, Marcus Vetter, Peter Hassenpflug, Hans-Peter Meinzer Deutsches Krebsforschungszentrum, Div. Medical and Biological Informatics B010, Im Neuenheimer Feld 280, 69120 Heidelberg, Germany ABSTRACT In medical imaging, segmentation is an important step for many visualization tasks and image-guided procedures. Except for very rare cases, automatic segmentation methods cannot guarantee to provide the correct segmentation. Therefore, for clinical usage, physicians insist on full control over the segmentation result, i.e., to verify and interactively correct the segmentation (if necessary). Display and interaction in 2D slices (original or multi-planar reformatted) are more precise than in 3D visualizations and therefore indispensable for segmentation, verification and correction. The usage of slices in more than one orientation (multi-planar reformatted slices) helps to avoid inconsistencies between 2D segmentation results in neighboring slices. For the verification and correction of three-dimensional segmentations as well as for generating a new 3D segmentation, it is therefore desirable to have a method that constructs a new or improved 3D segmentation from 2D segmentation results. The proposed method enables to quickly extend segmentations performed on intersecting slices of arbitrary orientation to a three-dimensional surface model by means of interpolation with specialized Coons patches. It can be used as a segmentation tool of its own as well as for making more sophisticated segmentation methods (that need an initialization close to the boundary to detect) feasible for clinical routine. Keywords: Interpolation, segmentation, Coons-patch, multi-planar reformation
1. INTRODUCTION Most image-guided procedures require the segmentation of organs, other anatomical entities, target regions or risk structures as an initial step. A major limitation for the acceptance of many advanced procedures is that a reliable segmentation of three-dimensional or even four-dimensional data (3D+time, e.g., in the case of cardiac applications) is very time-consuming. Reliability is essential for the clinical acceptance of a procedure and comprises two aspects. First, the accuracy should be as high as reasonably achievable. On the other hand, the segmentation result does not need to be more accurate than the precision of other components of the complete image-guided procedure, e.g., optical or magnetic tracking systems. Secondly, the segmentation – and, in consequence, the treatment of the patient with the image-guided procedure – should be possible in all cases accessible for expert human vision. Although automatic methods can reduce the required time for segmentation, they are in general not applicable in all cases and cannot guarantee to provide the correct segmentation. Thus, interactive methods are indispensable for clinical usage. Interactive segmentation, monitoring and, if necessary, the correction of the segmentation results are much easier to perform and much more reliable in two-dimensional image slices than in three-dimensional visualizations. Besides the simpler and more familiar handling of two-dimensional input devices and usually two-dimensional (non-stereoscopic) screens for visualization, the structures inside and outside of the segmented region can be visualized simultaneously with the regions boundary in two-dimensional image slices (compare Fig. 1, right). On the other hand, a slice-by-slice interactive segmentation of the acquired data is not only prohibitively timeconsuming for large three-dimensional data sets, but also involves the risk of inconsistent segmentations of Further author information: (Send correspondence to I.W.) I.W..: E-mail:
[email protected], Telephone: +49 6221 42 2327, Fax: +49 6221 42 2345 A.E.: E-mail:
[email protected] M.V.: E-mail:
[email protected] P.H.: E-mail: p.hassenpfl
[email protected] H.P.M.: E-mail:
[email protected]
Figure 1. Left: Risk of inconsistencies when segmenting on parallel slices. The segmentation was performed on the original transversal slices perpendicular to the displayed, reconstructed slice. Right: A consistent segmentation in a reconstructed frontal slice.
neighboring slices. Fig. 1, left, demonstrates that slice-by-slice segmentation even by experienced physicians may result in considerable inconsistencies. It is therefore desirable to have a method at hand that allows interactive segmentation on none-parallel slices (e.g., reformatted slices perpendicular to the orientation of acquisition as in Fig. 1, right) and reconstruct the three-dimensional object shape out of the two-dimensional, interactively created segmentation results. Furthermore, such a method has the potential to significantly reduce the number of slices that need to be segmented to obtain a given accuracy and thus accelerating the segmentation of three-dimensional data sets.
2. SEGMENTATION BY MEANS OF INTERPOLATION The acceleration of interactive segmentation methods by means of interpolation or extrapolation was studied by several authors, but is normally restricted to parallel slices. A standard method is shape-based interpolation.1 Treece et al. describe an extension of the shape-based interpolation approach for non-parallel slices.2 Intersecting contours are supported by this method, but according to the requirements of the area of application, for which the method was designed (freehand-3D-ultrasound), intersecting contours are treated as a special case that need to be tolerated. Best results are obtained for nearly parallel slices. An extension of the live-wire approach allows the reconstruction of a three-dimensional segmentation of nonparallel, two-dimensional segmentation results.3 A live-wire line is automatically calculated between every two neighboring intersection points of interactively created, non-parallel contours. Accordingly the interpolation of the interactively created contours is based on image data and thus is itself a segmentation process. This can be an advantage as well as a disadvantage, since inconsistencies similar to those in Fig. 1, left, can occur. The generation of interpolating surfaces from confining contours (thus without the usage of underlying image data) is part of computer aided geometric design (CAGD).4 Whereas B´ezier and B-spline methods create a surface based on a net of control points, which are partially not contained in the resulting surface, the contours used as input for Coons-patches or Gordon-Surfaces (a generalization of Coons-patches) are always part of the interpolated surface. Coons-patches or Gordon-Surfaces, respectively, are therefore a suitable basis for reconstructing the three-dimensional object shape from (the contours of) non-parallel, two-dimensional segmentation results, and thus for the method proposed in this paper.
Figure 2. Left: Consistent segmentation in a reformatted slice. The contour (light gray) of the segmentation in the displayed slice must intersect the segmentation results of the intersecting non-parallel slices (visible as white lines) in their endpoints. Right: 3D visualization of the resulting net of contours.
3. COONS-PATCH INTERPOLATION The input of the method is any number of two-dimensional segmentation results (provided as or converted to contours) on arbitrarily oriented, intersecting (plane) slices. The two-dimensional segmentation results may be constructed in any possible way. Besides simple manual contouring or polygon drawing, more sophisticated methods like live-wire/live-lane,3 phase-wire5 or active-contour models6 can be applied. The sole prerequisite is that the segmentation results are consistent with each other, i.e., when a voxel is defined as belonging to an object in one slice and is also part of another slice (which is, e.g., perpendicular to the first), it also has to be marked as a voxel of the object in the segmentation result of that slice (see Fig. 2, left). In other words, contours must not intersect the area enclosed by other contours. The construction of consistent segmentations can be facilitated by a graphical user interface that shows the segmentation results obtained in other orientation (white lines in Fig. 2, left) and, additionally, allows to place the next (control) point (e.g., for the live-wire method) exactly on the closest boundary point of an intersecting contour. Contours of one object must intersect at least one other contour of the same object. Thus, the input for the interpolation algorithm is a net with segments of the contours as non-linear edges and the intersection points as nodes (see Fig. 2, right).
3.1. Assembly of patch faces The object’s surface is divided into meshes (faces) by the initial net of contours. Each face will be covered by one Coons-patch, as described in the following sections. Each Coons-patch is constructed from the directly surrounding contour segments. Thus, before the actual interpolation can be performed, the contour segments defining the faces have to be identified. The net divides the object in disjunctive, connected compartments (see Fig. 3(a)). A valid assembly of contour segments, which encloses a single face, may not intersect a side of a compartment, as illustrated in Fig. 3(b). The maximum possible number of contour edges (equally to the number of nodes of the respective face) is equal to the number of contours N provided as input. Since we allow any number of contours, the assembled faces can have any number of edges.
C KVIII
KVII
KI K IX
K II
K III K XI
KVI
b2 n2
B
A
KX
K IV KV
n1
a
K XII
(a) compartments
(b) assembly of patch faces
Figure 3. (a) The sides of the compartments (KI -KXII ) defined by the net of contours may not be intersected when assembling a patch face (b).
3.2. Coons’s interpolation formulas The interpolation of the surface by means of Coons-patches is done patch-by-patch. Each Coons-patch is defined by the surrounding contour segments. Standard Coons-patches consist of two pairs of contour segments c1 (u), c2 (u) and d1 (v), d2 (v), u, v ∈ [0, 1], facing each other.4 The surface x(u, v) between the contour segments must fulfill the conditions x(u, 0) = c1 (u),
x(u, 1) = c2 (u),
x(0, v) = d1 (v),
x(1, v) = d2 (v).
The interpolation is done in three steps. First, two temporary surfaces rc (u, v) and rd (u, v) are calculated, each linearly interpolating one of the two pairs of contour segments (see Fig. 4(a) and 4(b)), as rc (u, v) = (1 − v)x(u, 0) + vx(u, 1) rd (u, v) = (1 − u)x(0, v) + ux(1, v). Secondly, a so-called error surface rcd (u, v) is calculated, which linearly interpolates between the four end points of the contour segments (see Fig. 4(c)) x(0, 0) x(0, 1) 1−v rcd (u, v) = 1 − u u . x(1, 0) x(1, 1) v Finally, the standard, linearly interpolated Coons-patch is calculated as the sum of the temporary surfaces minus the error-surface (see Fig. 4(d)) x(u, v) = rc (u, v) + rd (u, v) − rcd (u, v). Standard, linear interpolated Coons-patches are treated completely separately from each other, leading to discontinuous first derivatives between the patches. A solution to this problem is to use cubic Hermite (instead of
c 2 (u )
d1 (v) d 2 (v) c 1 (u )
(a) temporary surface rc (u, v)
(b) temporary surface rd (u, v)
c 2 (u )
d1 (v) d 2 (v) c 1 (u )
(c) error-surface rcd (u, v)
(d) linearly interpolated Coons-patch
Figure 4. A standard, linearly interpolated Coons-patch (d) is defined as the sum of the temporary surfaces (a,b) minus the error-surface (c).
linear) interpolation.4 The cubic Hermite polynomials Hi (t), i = 0..2, are H0 (t) = 2t3 − 3t2 + 1 H1 (t) = t3 − 2t2 + t H2 (t) = t3 − t2 H3 (t) = −2t3 + 3t2 . The first partial derivatives xn (u, v) with respect to n ∈ {u, v} perpendicular to the boundary of the patch are
interpolated from the (known) tangents of the contour segments at their endpoints (u, v ∈ {0, 1}) by∗ xu (0, v) = xu (0, 0)H0 (v) + xu (0, 1)H3 (v) xu (1, v) = xu (1, 0)H0 (v) + xu (1, 1)H3 (v) xv (u, 0) = xv (0, 0)H0 (u) + xv (1, 0)H3 (u) xv (u, 1) = xv (0, 1)H0 (u) + xv (1, 1)H3 (u). As in the linear case, two temporary surfaces hc (u, v) and hd (u, v) are calculated. The temporary surfaces can be written as4 hc (u, v) = x(0, v)H0 (u) + xu (0, v)H1 (u) + xu (1, v)H2 (u) + x(1, v)H3 (u) hd (u, v) = x(u, 0)H0 (v) + xv (u, 0)H1 (v) + xv (u, 1)H2 (v) + x(u, 1)H3 (v) and the error-surface hcd (u, v) accordingly as hcd (u, v) =
H0 (u) H1 (u)
x(0, 0) xu (0, 0) H2 (u) H3 (u) xu (1, 0) x(1, 0)
xv (0, 0) 0 0 xv (1, 0)
xv (0, 1) 0 0 xv (1, 1)
x(0, 1) H0 (v) H1 (v) xu (0, 1) xu (1, 1) H2 (v) x(1, 1) H3 (v)
.
The bicubic Coons-patch is combined in complete analogy to the linear Coons-patch as x(u, v) = hc (u, v) + hd (u, v) − hcd (u, v). These formulas refer to patches with four boundary contours. Similar formulas exist for three boundary contours. With barycentric coordinates x(u, v, w), u, v, w ∈ [0, 1], u + v + w = 1, the surface becomes x(u, v, w) = uh1 (u, v, w) + vh2 (u, v, w) + wh3 (u, v, w) with h1 (u, v, w) = x(u, v, 0)H0 (s) + x1 (u, v, 0)H1 (s) + x1 (0, v, w)H2 (s) + x(0, v, w)H3 (s) h2 (u, v, w) = x(0, v, w)H0 (t) + x2 (0, v, w)H1 (t) + x2 (u, 0, w)H2 (t) + x(u, 0, w)H3 (t) h3 (u, v, w) = x(u, 0, w)H0 (r) + x3 (u, 0, w)H1 (r) + x3 (u, v, 0)H2 (r) + x(u, v, 0)H3 (r).
3.3. Special patches The Coons’s interpolation formulas refer to patches defined by three or four boundary contour segments. Since we want to allow any number N ≥ 2 of two-dimensional segmentations as input for the interpolation, patches with n ∈ [2, N ] boundary contour segments (corresponding to the number of edges of a face) need to be constructed in our case (see Sect. 3.1). The case n = 2 can be treated analogous to the previous section. Since similar formulas do not exist for n > 4, we reduce the number of boundary segments to n = 4. As a rule for the combination of the initial contour segments, our current implementation optimizes for equal length of the resulting four boundary segments.
4. RESULTS We have applied the method to CT data sets and to three-dimensional echocardiographic data. A detailed reconstruction of a liver with 272 patches from 27 contours is shown in Fig. 5. The method yields a good approximation of the organ surface already for a very small number of initial contours. Fig. 6 shows the results of the reconstruction of a liver and a kidney from altogether ten and three segmented slices respectively. A first quantitative test for liver data sets (n = 5) with three initial contours in each of the three spatial directions showed an average agreement with manual segmentation of all slices (n ≈ 140) of 90 % and for five initial contours each of 93 %. The time needed for interpolation is neglectable for the mentioned number of contours. Even for large models (several hundred patches), the interpolation process takes less than 1 s on a standard PC (1 GHz). ∗
the second derivatives at the endpoints (so-called twist-vectors) are set to zero4
(a) front side
(b) back side
Figure 5. Interpolated surfaces of a liver consisting of 272 patches. Patches with n = 2, 3, 4 and n > 4 are displayed in different shades of gray.
(a) liver (10 contours)
(b) kidney (3 contours)
Figure 6. Results of the interpolation (a) of a liver with four transversal and three sagittal and frontal segmented slices (using pixel-based segmentation) and (b) of a kidney with one transversal, sagittal and frontal contour-based segmented slices.
5. DISCUSSION The more slices are provided for interpolation, the higher is the accuracy of the result. Therefore, by iterative application (segmenting additional slices where the interpolation results were not yet satisfactory), the proposed method can be employed as a fast, interactive three-dimensional segmentation method with adjustable accuracy. The contours provided as input are guaranteed to be included without changes into the three-dimensional model. This is an important feature of the method, since it is reasonable to assume that contours interactively created or corrected in two-dimensional slices represent the correct segmentation in the respective slice. The interpolated 3D model can be used to quickly obtain a 3D visualization or volumetric estimate. It also
turned out to be very useful for the selection of complex volumes-of-interest (VOI). These VOIs can be used, for example, as a preprocessing step of volume rendering or as a cut mask before or after the application of other, especially region-based segmentation methods. Furthermore, it can serve as a provider for good initializations for segmentation methods, which depend on initializations relatively close to the desired boundary, e.g., deformable models, level-set methods, etc. Thereby, the method may help to reduce the gap between clinical practice and many sophisticated segmentation methods.
6. CONCLUSION The described approach considerably reduces the time required for user interaction to obtain a rather good approximation of the object’s shape. Interaction on two-dimensional slices is precise and well-known to users. The concept of segmenting in two-dimensional, non-parallel slices therefore has the potential of a wide user acceptance. The proposed method is a promising approach as a segmentation tool of its own as well as for making more advanced segmentation methods (that need an initialization rather close to the boundary to detect) feasible for clinical use.
ACKNOWLEDGMENTS This work was supported in part by the Deutsche Forschungsgemeinschaft within the SFB 414 “Information Technology in Medicine – Computer and Sensor Supported Surgery”.
REFERENCES 1. G. Herman, J. Zheng, and C. Bucholtz, “Shape-based interpolation,” IEEE Comp. Graph. and Appl. 12(3), pp. 69–79, 1992. 2. G. Treece, R. Prager, A. Gee, and L. Berman, “Fast surface and volume estimation from non-parallel crosssections, for freehand three-dimensional ultrasound,” Med. Image Anal. 3(2), pp. 141–173, 1999. 3. A. Falcao and J. Udupa, “A 3d generalization of user-steered live-wire segmentation,” Med. Image Anal. 4(4), pp. 389–402, 2000. 4. G. Farin, Curves and Surfaces for CAGD: A Practical Guide, Academic Press, San Francisco, 1999. 5. L. O’Donnell, C. Westin, W. Grimson, J. Ruiz-Alzola, M. Shenton, and R. Kikinis, “Phase-based user-steered image segmentation,” in Proc. MICCAI, W. J. Niessen and M. A. Viergever, eds., Lecture Notes in Computer Science 2208, pp. 1022–1030, Springer, 2001. 6. T. McInerney and D. Terzopoulos, “Deformable models in medical image analysis: a survey,” Med. Image Anal. 1(2), pp. 91–108, 1996.
