Model-based Segmentation of 4D MR Images of the

Lehrstuhl fur Nachrichtentechnik

Friedrich - Alexander Universitat Erlangen - Nurnberg Prof. B. Girod and

Laboratoire de Traitement des Signaux E cole Polytechnique Federale de Lausanne Prof. M. Kunt

Model-based Segmentation of 4D MR Images of the Heart

Wolfgang Sorgel

Diploma Thesis 1996

ACKNOWLEDGMENTS This work has been done as a thesis project for a degree from Friedrich Alexander University Erlangen-Nurnberg. Supported by the European student exchange program ERASMUS, it was performed at the Signal Processing Laboratory (LTS) of the EPFL ( E cole Polytechnique Federal de Lausanne) in Switzerland. I would like to thank all the people who made this possible:

 Prof. B. Girod of the Telecommunications Institute of the Friedrich Alexander

University Erlangen-Nurnberg for the great opportunity to come to Switzerland and Prof M. Kunt of LTS for allowing foreign students to work at his laboratory.

 Vincent Vaerman, my supervisor at LTS for all support, guidance and the op-

portunity to work with him on this exciting subject. Thank you very much also to Dr. Jean-Marc Vessin for the valuable input during the regular ERASMUS students supervision and to all other faculty and stu at LTS for the friendly reception and helpful support.

 Erwin Keeve for the supervision of this work for the university in Erlangen and his valuable input.

 Special thanks go to Luc Bidaut at the Digital Imaging Unit and Dr. Dominique Didier, both with the University hospital at Geneva for providing the MR image data and valuable tips and information.

 Last but not least i would like to thankfully acknowledge all the support received from my parents during my stay here. ii

ABSTRACT This thesis develops a method to automatically segment the outer contours of the human cardiovascular system from 4D multi-slice magnetic resonance image (MRI) sets. The approach decomposes the problem into two stages: First, the heart is identi ed within the image and a template contour coarsely adapted to estimated position, size and pose. Then this initial contour is locally deformed using active contour models in a multi-step approach. The template contour can be either generated automatically or a provided model template is used. For localization of the heart a method based on the temporal gray level variance of the images for xed spatial location is developed. A con dence mask using spatial relations between the image slices is used to identify the heart in the variance images. The method is largely independent of eld of view, image perspective and MRI imaging technique. Only one input parameter, a coarse estimation of the maximal cross-sectional area of the heart, is required. For ne segmentation, an existing model-based active contour model is adapted. A smoothing term and a slice oriented multi-contour model are additionally introduced into the formulation. Using the active contour model, the ne segmentation is done by a multi-step approach based on dierent types of images with dierent snake performance parameters. This local deformation of the ane matched contours obtained from the localization algorithm yields accurate outlining of the heart at all slice levels and acquisition times. The resulting contours can then be used for modeling and visualization purposes.

iii

Contents LIST OF TABLES

viii

1 INTRODUCTION

1.1 Project Goals : : : : : : : : : : : : : : : : : 1.2 MR Medical Imaging : : : : : : : : : : : : : 1.2.1 MRI Techniques for Cardiac Images : 1.2.2 Papyrus Image File Format : : : : : 1.3 Overview Over Segmentation Methods : : : 1.3.1 Region-Based Methods - Clustering : 1.3.2 Local Feature Detection : : : : : : : 1.3.3 Template Matching : : : : : : : : : : 1.3.4 Active Contour Models - Snakes : : : 1.4 Approach to be Used : : : : : : : : : : : : : 1.4.1 Requirements for the Algorithm : : : 1.4.2 Outline of the Segmentation Process

2 CONTOURS and IMAGES

: : : : : : : : : : : :

2.1 Image Representation : : : : : : : : : : : : : : 2.2 Discrete Point Contours : : : : : : : : : : : : 2.2.1 Contour Features : : : : : : : : : : : : 2.2.2 Interpolation Using Spline Techniques

3 LOCALIZATION of the HEART

: : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : :

: : : : : : : : : : : : : : : :

: : : : : : : : : : : :

1 2 2 3 5 5 5 6 6 6 6 7

9

: 9 : 10 : 10 : 12

3.1 Introduction : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 3.2 Temporal Variance Images : : : : : : : : : : : : : : : : : : : : : : : : 3.2.1 Interpretation : : : : : : : : : : : : : : : : : : : : : : : : : : : iv

1

14 14 16 16

3.2.2 Thresholding : : : : : : : : : : : : : : : : : : : 3.2.3 Morphological Cleaning of the Binary Images : 3.2.4 Including Spatial Information: Using a Mask : : 3.2.5 Adapting the Algorithm to 2D+time Data Sets 3.3 Placement of an Initial Contour : : : : : : : : : : : : : 3.3.1 Initial Contours : : : : : : : : : : : : : : : : : : 3.3.2 Estimation of the Heart Pose : : : : : : : : : : 3.3.3 Ane Transform of the Initial Contour : : : : :

: : : : : : : :

: : : : : : : :

: : : : : : : :

: : : : : : : :

: : : : : : : :

: : : : : : : :

: : : : : : : :

: : : : : : : :

18 21 22 27 27 27 28 29

4 FINE SEGMENTATION of the HEART: Active Contour Models 32 4.1 Introduction to Active Contours: Why Snakes : : : : : : : : : : : 4.2 Active Contour Energy Formulation : : : : : : : : : : : : : : : : : 4.2.1 In uence of the Image: External Energy : : : : : : : : : : 4.2.2 In uence of the Model: Internal Energy : : : : : : : : : : : 4.2.3 Energy Regularization and Minimization : : : : : : : : : : 4.3 Slice Oriented 3D Contour-Model : : : : : : : : : : : : : : : : : : 4.4 Multi-step Approach to the Segmentation of MRI Heart Contours

: : : : : : :

: : : : : : :

32 34 34 37 41 44 48

5 FUTURE EFFORTS

55

6 CONCLUSIONS

58

APPENDIX

60

BIBLIOGRAPHY

78

5.1 Heart Localization : : : : : : : : : : : : : : : : : : : : : : : : : : : : 5.2 Active Contour Model : : : : : : : : : : : : : : : : : : : : : : : : : : 5.3 Outlook: What are the Results Used for ? : : : : : : : : : : : : : : :

New Classes and Methods for GSNAKE API : : : : : : : : : : : : : : : :

v

55 56 56

60

List of Figures 1.1 Bright-blood MRI Images : : : : : : : : : : : : : : : : : : : : : : : : 1.2 Morphologic 4D MRI Images : : : : : : : : : : : : : : : : : : : : : : : 1.3 Outline of the Segmentation Algorithm : : : : : : : : : : : : : : : : :

3 4 8

2.1 Examples for Interpolation Using Splines : : : : : : : : : : : : : : : :

13

3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10

Examples for Variance Images : : : : : : : : : : : : : : : : : : : : : : Variance Images after Thresholding : : : : : : : : : : : : : : : : : : : Thresholded Variance Images for Dierent Heart Size Estimation : : Automatic Determination of the Threshold : : : : : : : : : : : : : : : Eect of the Cleaning Operator : : : : : : : : : : : : : : : : : : : : : Raw Mask: Summation of all Thresholded Variance Images : : : : : : Mask for the Heart Region after Thresholding : : : : : : : : : : : : : Final Binary Mask for the Heart Region : : : : : : : : : : : : : : : : Masked Variance Images : : : : : : : : : : : : : : : : : : : : : : : : : Initial Contours adapted to the Variance Image by Ane Transform :

17 19 20 22 23 23 24 25 26 31

4.1 4.2 4.3 4.4 4.5 4.6 4.7

Edge Magnitude and Orientation : : : : : : : : : : : : : : : : : : : : Eect of the Smoothing in Internal Energy Calculation : : : : : : : : Snake Performance with Multi-slice Contour Model on Noisy Images : Snakes Minimized on Binary Variance Images : : : : : : : : : : : : : Snakes Minimized on Mean Images : : : : : : : : : : : : : : : : : : : Final Segmentation of the Actual Images : : : : : : : : : : : : : : : : Radial Contour Movement over Time : : : : : : : : : : : : : : : : : :

36 40 48 51 52 53 54

vi

5.1 3D Reconstruction of the Heart : : : : : : : : : : : : : : : : : : : : :

vii

57

Chapter 1 INTRODUCTION 1.1 Project Goals This work is part of a larger project concerned with processing of 4D magnetic resonance images (MRI) of the human cardiovascular system. The project goals are visualization, modeling and model-based coding of the heart. The input data is delivered in a standard Picture Archiving and Communication System (PACS) format, Papyrus [1] 1. All methods developed should be able to work automatically, using only information provided from the input and, if available, model data. Although we are primarily interested in 4D image sets, multi-slice image sets acquired at dierent times during the cardiac cycle, the treatment of pure 3D image acquired at one time and 2D image sets of the same spatial location at dierent times, should be considered. This thesis project is concerned with the segmentation of the outer contour of the heart from the image data. The resulting contours will then be used for visualization and modeling. Although no complete model exists yet, the segmentation algorithm should speci cally allow to use model based information. 1 Papyrus has been developed at the digital imaging unit of the university hospital in Geneva,

Switzerland, which also kindly provided our image data.

1

1.2 MR Medical Imaging Magnetic resonance imaging (MRI) is a relatively nouvelle, non-invasive and nonhazardous medical imaging technique [2, 3, 4]. The basic principle is to place the patient in a strong, homogeneous magnetic eld B0 and then excite the protons of the (mostly hydrogen) atoms in the dierent body tissues with radio frequency (RF) pulses perpendicular to B0. The RF pulse frequency is set such that the resulting proton spin alternation between lower and higher energy levels occurs at the intrinsic precessional frequency, the Larmor frequency, which is referred to as the resonance in MRI imaging. The eects which are actually measured and imaged are the relaxation of the proton spins after termination of the RF pulses. The weighting of dierent relaxation eects as well as dierent excitation techniques allow for a wide variety in contrast and visibility of dierent tissues.

1.2.1 MRI Techniques for Cardiac Images For the acquisition of 4D MR image sets, especially of moving organs such as the heart, fast MRI techniques are essential to obtain high quality images without movement artifacts within a reasonable time. Numerous acquisition technologies have been developed for this purpose [5, 2] to allow the acquisition of several images during one breath hold. The images we use have been kindly provided by the digital imaging unit of the Geneva university hospital. They were acquired by dierent ECG-gated multi-slice techniques, resulting in temporally synchronized image sets of relatively thick, widely spaced slices. The gradient echo methods used for this fast imaging normally deliver images with bright signal intensity resulting from blood. Examples for these so-called bright blood images are shown in gure 1.1. Unfortunately, the outer contours of the heart are not everywhere very well visible in these images: The inner walls of the ventricle often dominate. The visibility of these contours can be enhanced by spatial magnetic presaturation of the blood before it enters the imaged slice, resulting in morphologic images. However, the spatial resolution and SNR are also slightly decreased by this method. Figure 1.2 shows some examples of a transversal image set from a healthy person 2

Figure 1.1: a) Transversal Bright-blood Image, b) Long Axis Bright-blood Image acquired with this technique, in spatial (top - down) and temporal (left - right) order. This image set, consisting of 16 ECG synchronized images each taken at 17 coplanar spatial locations, with a resolution of 256  256 pixels in the image plane and 12 bit gray level resolution, was the main resource for this work. Unless otherwise noted, all subsequent examples are related to this sequence.

1.2.2 Papyrus Image File Format The MR images are delivered in Papyrus format [1], a medical image storage format compatible with the ACR-NEMA speci cations on image communication, developed at the digital imaging unit of the Geneva university hospital, Switzerland. Besides the image pixel data for one or more images, the images contain additional header information including patient information, acquisition information (e.g. imaging technique used, slice thickness), relationship information (e.g. patient orientation) and image presentation information (e.g. pixel size). Unfortunately, the inclusion of much useful information is optional and not routinely done by the acquisition hardware so that we often either have to rely on assumptions or user interaction at some points. Access to Papyrus les is facilitated by the availability of a toolkit [6] as function library, written in C. 3

Figure 1.2: Examples from Transversal Morphologic 4D Image Set 4

1.3 Overview Over Segmentation Methods Before outlining our approach to the segmentation problem, we give a short overview over possible segmentation techniques. Due to the fact that the slice thickness of our data is about one order of magnitude larger than the pixel size within the images and that we not always have multiple slice positions within the image sets, pure 3D, voxel-based approaches are principally ruled out.

1.3.1 Region-Based Methods - Clustering In region-based clustering methods, image pixels are clustered together in groups based on a local, statistical similarity measure. This measure may be derived from gray level, local gray level variance or other local features. Clusters are either split or merged (or both) until a criterion function on the similarity measure is minimized [7]. Then the cluster corresponding to the desired object has to be selected, implicitly de ning the outlining contour. Problems with these techniques include the choice of a suitable similarity measure which is relevant to the detection of the desired region and invariant over the possible range of the input data. With the wide variety of MRI techniques generally and also large dierences in appearance of pixels within the heart in one image, direct use of clustering techniques to extract and outline the heart is not regarded feasible. Further model aspects regarding the expected form of the heart can not well be combined with clustering. However, our localization approach described in chapter 3 implicitly uses some ideas from clustering.

1.3.2 Local Feature Detection Another low-level approach is the use of operators to detect local dissimilarities in the image. The most common of these operators are edge detectors such as the Roberts, Sobel or Prewitt operator or the Canny edge detector [8, 9]. Typically these operators deliver a multitude of non-connected edges which then have to be linked together and identi ed with object boundaries. The possible presence of other objects, spurious edges not corresponding to desired features and the presence of contour parts without border visible in MR images deter 5

us from directly using local contour features to solve our segmentation and outlining problem. Nevertheless we will use edge maps as part of the contour ne segmentation described in chapter 4.

1.3.3 Template Matching If models or templates of the desired contour are available, one can try to nd the desired object by nding the maximum of correlation of these templates with the image or an edge map. This is equivalent to a matched lter in communications and can be implemented using the generalized Hough Transform [10]. However this method is computationally expensive and we found it very prone to misidentify other structures in the image for the heart, despite well tting templates. Another problem is the principal diculty to detect a highly deformable object of possibly very dierent shape using rigid templates.

1.3.4 Active Contour Models - Snakes Snakes [11] are curves C with an associated non-negative energy functional E (C). Typically, the energy is composed of an external energy Eext, derived from the underlying image data, and an internal energy Eint expressing smoothness or model constraints. The energy terms are selected such that (local) minimization of E (C) lets C outline the desired contour. Besides energy formulation, initialization of the snake close to the target is critical for snake performance as the desired solution may not correspond to a global but to a local minimum of E (C), with other minima corresponding to non-desired features.

1.4 Approach to be Used 1.4.1 Requirements for the Algorithm The requirements for the contour segmentation algorithm to be developed can be outlined as follows:

 The method should not rely on the presence of any features in the images besides

the heart. It should further be independent of the orientation of the heart with 6

respect to the image plane. We further have to deal with cluttered scenes with other organs and features present, possibly resembling the heart in form, size or texture.

 The data delivered by our algorithm must be suitable for building of a 4D model of the heart. On the other hand, model information should be implicitly and explicitly usable for the segmentation.

 Besides the use of available model information, the method should autonomo-

usly work with the data from the PACS input les. If additional input data is required, it should be easy to estimate and provide by the medical personnel operating the MRI equipment or analyzing the images.

 The approach should be suited for 4D MR image sets but also must be open to include image sets with only one spatial slice position or one acquisition time.

 The method should be as robust as possible to achieve good results without

operator interaction in the presence of the noise and blur eects typical for dynamic MR images. It should be adaptable to the dierent image appearance due to dierent MRI techniques as well.

1.4.2 Outline of the Segmentation Process Our approach to the segmentation problem is split into two major steps as shown by gure 1.3: 1. Localization of the heart in the images, estimation of heart size and pose and adaption of an initial contour template by ane transform. Dierent techniques can be used for dierent types of image sets. In chapter 3, we introduce a method based on the temporal variance of the gray level images which is targeted at 4D image sets and also suitable for 2D+time image sets. 2. Fine segmentation of the heart contour using active contour models (snakes). A robust and exible multi-step procedure with model based internal energy will be introduced in chapter 4. A heart model can be used for the generation of the initial contours and determination / learning of some performance parameters for the second step. Furthermore, 7

Input Data (pixel data + additionl information)

Model

Global Localization of the Heart

Local Fine Segmentation of the Contours

Discrete Point Contours at Each Spatial and Temporal Location

Figure 1.3: Outline of the Segmentation Algorithm some knowledge about the anatomy of the heart and the MRI techniques are implicitly used throughout the algorithm. The dotted line in gure 1.3 indicates the possibility of using the results to build a model of the heart from the obtained contours which then allows to close the loop and perform completely model-based segmentation.

8

Chapter 2 CONTOURS and IMAGES In this chapter some general notational conventions regarding the representation of images, image sets and contours are introduced. Methods for assessing and interpolating contours are also presented.

2.1 Image Representation This work is based on 4-dimensional image sets and subsets thereof. A 4-dimensional image is de ned as a mapping I(p) : II 7! IR (2.1) with the image set domain II = fp = (x; y; l; t) : (x; y) 2 IE ; l = 1; 2; : : : ; L; t = 1; 2; : : : ; T g. This means we have a set of L
T images I(x) : IE 7! IR at L dierent normalized spatial locations l orthogonal to the image coordinates (x; y) and T different normalized discrete times t. The image points x = [x ; y]T of a single image are also de ned on the nite grid x 2 IE = f(x; y) : x = 1; 2; : : : ; X ; y = 1; 2; : : : ; Y g. However we explicitly allow values x 62 IE to be used here , with either rounding to the next valid pixel coordinate or a suitable interpolation. This will be exploited later to gain some sub-pixel accuracy. Throughout this work we will use the notation Itl(x) to denote a single image at time t (superscript) and location l (subscript). As special cases of a 4-dimensional image set, we can also treat 3-dimensional 9

image sets if either T = 1 or L = 1. This later case will also be referred to as a 2D+time image set. We further introduce the convention that all arithmetic operations, addition or multiplication for example, on images and image sets are interpreted pointwise, i.e if we write I = I1 + I2 then the gray level value I(x; y) is the sum of the corresponding values in I1 and I2 for all pixel coordinates (x; y).

2.2 Discrete Point Contours In the context of this work, a contour C = [c1; c2; : : : cn] is de ned as the ordered set of points ci = [ci;x; ci;y ]T 2 II = IR2. A contour is said to be closed if the last point cn is connected to the rst point c1. In this case, the selection of the starting point c1 is therefore arbitrary.In this work all contours contours are assumed to be closed and we introduce the convention that the indexes for the contour points are interpreted as modulo-n, i.e. cn+1 = c1 and c0 = cn.

2.2.1 Contour Features For globally describing properties of a contour, we compute features to indicate the location, size, orientation and aspect ratio. The location of a closed contour shall be given as n X C = 21L kci+1 ? cik2 (ci+1 + ci) (2.2) i=1

where L = Pni=1 kci+1 ? cik2 is the overall contour length and kvk2 denotes the lp2 vector norm. This de nition of c yields the center of gravity for a structure of homogeneous, straight rods connecting the single contour points, and is independent of the spacing of the contour points. Note that simply averaging the point coordinates would not yield this property. Let 3 2 3 2 (2.3) ui = ci ? c = 64 ui;x 75 := ri 64 cos i 75 sin i ui;y be the coordinate of each point i of the contour with respect to the center of gravity and u^Ti = [uTi ; 0]. Assuming again connection of neighboring points by straight lines 10

we can compute the area of the contour C as n X sC = 21 ku^i  u^i+1k2 (2.4) i=1 where a  b denotes the vector cross product. Note that (2.4) only delivers the correct result, if i+1  i 8 i 1 , i.e. if the polar representation (2.3) of the centered contour is ordered by nondecreasing angle . Although this condition may not hold true in all cases later, we can still use (2.4) as an approximation. Assuming the same conditions as for (2.2) we compute a covariance matrix ! i X 1 T T T T C = 6L kci+1 ? cik2 (2(uiui + ui+1ui+1 ) + ui+1ui uiui+1) ? C TC (2.5) i=1 which is equivalent to the moment of inertia in mechanics. Note that equations (2.2) and (2.5) may be easily derived as moments of a two-dimensional random variable u which is uniformly distributed on the border of a polygon formed by C. For the assessment of rotation and aspect ratio, we apply the Karhunen-Loeve Transformation [7, 12] to C to get

C = MTCC MC where

2 C = 64 C;1

(2.6) 3

0 7 5

0 C;2 is the diagonal matrix of the eigenvalues of C and

2 3 MC = 64 mC;11 mC;12 75 = [mC;1; mC;2]

?mC;12 mC;11

(2.7)

(2.8)

is the orthonormal matrix of the corresponding eigenvectors mC;1; mC;2. We note that MC is a rotation matrix indicating the rotation C of the contour major axes with respect to the coordinate axes, with mC;11 = cos C and mC;12 = sin C . The aspect ratio of the contour is determined by the ratio C;1 =C;2 of the eigenvalues. The measures derived above will later be used in an ane transform to globally adapt an initial contour to the images of the heart. 1 Jumps of  by 2 are disregarded here. We assume further right-handed orientation of the

contour.

11

2.2.2 Interpolation Using Spline Techniques In the process of adapting a contour to image data, it may be desirable to add or reposition points. While linear interpolation between neighboring points is possible here as well, we can not expect \smooth" contours from this technique. Instead spline interpolation [13, 14, 15] is used. For this purpose, we consider the two coordinates as separate tables of one dimensional function values fxigni=1 and fyigni=1 with uniformly spaced integer arguments i. Then two B-spline functions Bxd(s) and Bxd(s), s 2 (0; n] 2 of degree d are computed. The choice of d depends on the desired smoothness, since the resulting spline is a piecewise polynomial of degree d, it has d ? 1 continuous derivatives [13]. In our implementation, d = 2 (biquadratic interpolation) or d = 3 (bicubic interpolation) were used. Note that bilinear interpolation d = 1 is equivalent to connecting the contour points by straight lines, which is the assumption used in (2.2) - (2.5). Higher order interpolation could be used there as well, but then no closed expressions for C, sC and C can be given and numerical integration would have to be used. Our goal is now to interpolate a contour C with n contour points such that the new Contour C^ has n^ 6= n points. This can be done in two ways: 1. Interpolation keeping relative point spacing: For this case, the coordinates c^^i;x; c^^i;y of the interpolated contour points c^^i are given simply by c^^i;x = Bxd(^i nn^ ) (2.9) c^^i;y = Byd(^i nn^ ) for ^i = 1; 2; : : : ; n^ .

2. Often it is desired to obtain contours with at least approximately equal spaced points. However the contour deformation process introduced in chapter 4 as well as model based or interactively generated initial contours may result in contours with unequally spaced points. Thus we introduce an interpolation scheme to resample a contour C with n arbitrarily spaced points to a contour C^ with n^ equally spaced points:

2 Note that the input sequences need to be padded (partly repeated) to obtain splines really valid

8 s 2 (0; n] without adverse boundary eects.

12

Let Bx0 (s) = dBdsx and By0 (s) = dBdsy the rst order derivatives of our spline functions. The overall length of the spline-interpolated, closed contour is then given by Z q ^L = n Bx0 (s)2 + By0 (s)2 ds (2.10) 0 c = n^ =L^ . The and the desired spacing between the single contour points is L corresponding spline coordinates s^i for such points can be found by solving Z s^i q c Bx0 (s)2 + By0 (s)2 ds = (^i ? 1) L (2.11) 0

for s^i; 8 ^i 2 f1; 2; : : : ; n^ g. Numerical integration has to be used for the evaluation of (2.10) and (2.11), a simple rectangular approximation being sucient. The interpolated contour coordinates are then given as

c^^i;x = Bxd(s^i) c^^i;y = Byd(s^i)

(2.12)

similar to (2.9). d=1

d=1

d=2

d=3

d=2

d=3

Figure 2.1: Examples for Interpolation Using Splines Figure 2.1 shows some examples for interpolations of degree 1,2 and 3 (left to right) with unchanged (upper row) and equalized (lower row) point spacing. We note that higher degrees of interpolation tend to shrink the contour away from pronounced edges. In practical applications we use more contour points than in the example, thus this eect is not as pronounced as in gure 2.1. 13

Chapter 3 LOCALIZATION of the HEART In this chapter methods for localizing the heart in the images and placing a contour template close to the actual heart contour are described. A new approach suitable for 4D image sets based on the temporal variance of the image gray levels for xed spatial location is introduced. Spatial relationship is then used to derive a con dence mask for the heart region, excluding regions of high variance not corresponding to the heart. This allows then to extract binary images of the heart to which an initial contour is adapted by ane transform.

3.1 Introduction As explained in section 1.3.4, the active contour models (snakes) to be used for ne segmentation require to be initialized suciently close to the desired contour. This leads to the task of nding the approximate position and pose of the heart within the given image set and to adapt an initial contour accordingly. As for overall segmentation, several dierent approaches are possible, including

 Manual outlining of the contours by an operator;  Model-based approaches using spatial relation within the image scene such as

relative position of the heart with respect to the outer upper body contour or other easily detectable organs as lungs or spine; 14

 Region based approaches using texture and gray level information,  Template matching, implemented by the Generalized Hough Transform [10];  Combinations of these methods. As we want to develop a method which shall work without human intervention and for dierent image perspectives, the two approaches cited rst are ruled out1. Due to the large dierences in gray level and texture for dierent MRI techniques, approaches based on these features are not very promising either. Rigid Template Matching, proposed in [16], is computationally expensive and not well suited to detect such a highly deformable object as the heart. We have performed some experiments with an implementation of the generalized Hough Transform to con rm this assertion. Furthermore, success highly depends on the availability of suitable template contours. An example for a complex multi-step approach using Fuzzy Hough transform [17] and region growing is for example reported by Philip et al. in [18]. However, the algorithm only works for transversal images and relies on the presence of other body features besides the heart such as chest wall and the left lung in the images. Fuzzy Hough transform as an extension of this classic technique suitable for deformable objects could be used for localization of the heart in pure 3D image sets nevertheless. This has not been treated in this work as pure 3D images are not our main interest.

Outline of the Localization Algorithm To gain a perspective and eld of view invariant method for localization of the heart, we base our approach on the temporal variance of the gray levels at xed slice positions. The outputs are contours CGl globally adapted to the desired object for each slice location l. Each contour will then be used as initialization for the active contour models introduced in chapter 4. The idea outlines as follows: 1. For each spatial position l, a temporal variance image Vl is computed. 2. Each Vl is thresholded such that a number of pixels representing the maximal expected size of the heart is kept and morphological cleaning is performed on the result.

1 Manual, interactive outlining has been used for study purposes nevertheless.

15

3. A con dence mask for the heart location is built by

 Adding all variance images Vl ;  Thresholding the result by a xed threshold;  Selecting the largest object in the resulting binary image. 4. The thresholded variance images are pointwise multiplied with the mask to get a binary image for each slice. 5. An initial contour is adapted to the masked variance image by ane transform. Details are presented in the following sections.

3.2 Temporal Variance Images Let I be a given 4D image set with T images taken at each of L dierent slice positions. Then we can compute the elements of a set M of L temporal mean images at each slice position as T X Ml = T1 Itl (3.1) t=1

Then for each slice location l a variance image T X Vl = T ?1 1 (Itl ? Ml)2 t=1 is computed.

(3.2)

3.2.1 Interpretation We have found that within the outer contours of the heart, this variance has large values with high probability. Outside the heart generally smaller values are found, independent of imaging technique or perspective. This can be explained by the movement of the heart walls during the cardiac cycle and the associated heavy ow of blood. Figure 3.1 shows some examples for dierent slice positions. Position, size and orientation of the heart can be well seen. We especially note that in the transaxial images, the bordering between the heart and the vertebra at the back and the pericardial line at the chest is well pronounced, contrary to the actual images. 16

Figure 3.1: Examples for Variance Images

17

However it is also evident that there may be other areas of high variance in the image, caused for example by movement of the patient body or breathing during the image acquisition. Some movement artifacts may be reduced in the future by even faster acquisition techniques [5]. However, we still need to deal with variance peaks generated by other imaged organs, such as the extremities or internal organs like stomach or liver, as well as a reasonable amount of movement and imaging artifacts. Furthermore, the variance value is not constant within the heart as well: Blood vessels and the walls of the left ventricle generally show higher variance. These facts are taken into account by binarization for the further processing, as discussed below. An interesting side aspect of this work, aiming to localize and segment the outer heart contours, may be the direct use of the variance images for diagnostic purposes. Especially disturbed movement and blood ow in infarcted hearts may be well visible. The idea has been discussed with some physicians: Some more experience with images of pathologic organs has to be gained before nal conclusions can be drawn.

3.2.2 Thresholding We now proceed by thresholding the variance images. The purpose of this step is to obtain binary images where the heart is well outlined. Additional objects, resulting from the various noise sources described above, are tolerated at this step. Some experimentation showed that it is best to threshold the Vl in such a way that a number of pixels sH representing the expected maximal size of the heart is set to one, the rest to zero. sH is a value which can be easily estimated by a trained physician. If in doubt, sH should be overestimated. Furthermore the algorithm showed to be insensitive to miss-estimation of sH up to a factor of 2. Automatic detection of deviations and correction of sH seems possible. The threshold for binarization is found by rst computing an histogram

hVl (v) =

M N X X x=1 y=1

(v ? Vl (x; y))

(3.3)

and then nding the largest value vlT for which the inequality Z vlmax vlT

hVl (v)dv  sH

holds. 18

(3.4)

The thresholded variance images are then obtained by setting 8 > T T < T Vl (x; y) = >: 1 if Vl (x; y) > vl 0 else

(3.5)

Thus approximately the sH pixels with the largest variance values are kept in each image.

Figure 3.2: Variance Images after Thresholding Figure 3.2 exemplary shows some resulting images. It can be seen that the heart is well outlined in all images, with only few blank spots within. On the other hand there are also noise clusters present which may in some cases be even larger than the heart area. This necessitates the need for distinguishing this noise from the desired 19

Figure 3.3: Thresholded Variance Images for Dierent Heart Size Estimation object. To accomplish this, we use the spatial relation between the dierent slices to compute a binary con dence mask for the heart region. Figure 3.3 compares the results for dierent choices of sH : In the left images, nH = 2500, in the middle nH = 5000 and in the right images nH = 10000 pixels 2. We see the heart well outlined in all images. In the lower images, at a slice where the heart cross section is maximal, we note that some parts are missing for nH = 2500. Our algorithm will work with both extreme cases shown in gure 3.3, although they are close to the feasible limits and may require some human intervention for optimal results. Other thresholding strategies are possible. However we note that techniques based on bimodality of a histogram or the use of one identical threshold for all images do not deliver better results than the technique explained above. 2 We use pixels as unit because our image data did not provide pixel-size information. Future

acquisitions will.

20

Automatic Determination of the Threshold From gure 3.3 we observe, that the thresholded variance image contains only one large object for correctly chosen heart size. For underestimated heart size, the heart is broken up in several smaller parts, for larger objects, additional noise clusters are visible. We use this observation to suggest a method for automatic determination of nH and with it the threshold: First a lower bound nH;low and an upper bound nH;up are selected for the cross sectional heart size. Then the threshold vlT (nH;i) is computed according to (3.4) for dierent values nH;low  nH;i  nH;up, resulting in thresholded variance images for the dierent thresholds. For each image, the size of all connected objects is determined and from that the ratio of the size of the largest object to the sum over all other objects. The value nH;max where this ratio is maximal has been found to be a good indicator of the heart size. As the later processing is better suited to an overestimation of nH , we set nH = " nH;max with a multiplicative factor ". In our experience, 1  "  2 are suitable. Figure 3.4 show examples for the ratio of largest object size to size of all other objects plotted over the estimated heart area. For the image set used, nH  5000 was experimentally found to work optimally, with 2500  nH  1000 resulting in valid images, see gure 3.3. So we suggest to chose " = 1:25. Tests with more image data than we had available are needed however to consolidate this choice.

3.2.3 Morphological Cleaning of the Binary Images We also see that the thresholded variance images may contain small speckles of noise and small holes within the heart region. Although not absolutely necessary, we gain accuracy and robustness in the localization of the heart if these artifacts, often referred to as pepper-and-salt noise [19], are removed. This is done by the following cleaning operation: For each pixel (x; y) in VlT set 8 P > T 1 if >

r (x;y) Vl (u; v ) > Nmax < VlT (x; y) = > 0 if P r (x;y) VlT (u; v) < Nmin > : VlT (x; y ) else

21

(3.6)

Slice 4

Slice 8

6

4000

5 3000 4 3

2000

2 1000 1 0 0

5000

0 0

10000

5000

Slice 12

Slice 15

800

800

600

600

400

400

200

200

0 0

5000

10000

0 0

10000

5000

10000

Figure 3.4: Ratio of Largest Object Size to Sum Over All Other Objects for Dierent Heart Size Estimations (Thresholds) Here, r (x; y) is a neighborhood of radius r around pixel (x; y). Using the lp-1 vector norm, we consider (u; v) 2 r (x; y)

; if max((u ? x)2; (v ? y)2)  r2

For cleaning the VlT , we used r = 2 (yielding 24 neighbors for each pixel), Nmax = 15 and Nmin = 11. This parameter choice removes small objects of up to 10 pixels, erodes small protrusions and closes holes of up to 9 pixels. Figure 3.5 compares a cleaned image (right) with the original (left). Note that the images in gures 3.2 and 3.3 are cleaned as well.

3.2.4 Including Spatial Information: Using a Mask We start with the observation that the heart region is outlined in all slices l of the thresholded variance images VlT , whereas the noise is more randomly distributed within the images and other organs typically extend only over few slices. We use this to build a binary mask outlining the heart region: 1. The thresholded, cleaned variance images are added together to get a raw mask

Mr = which is shown in gure 3.6. 22

L X l=1

VlT

(3.7)

Figure 3.5: Eect of the Cleaning Operator: a) Before Cleaning, b) After Cleaning

Figure 3.6: Raw Mask: Summation of all Thresholded Variance Images

23

2. The raw mask image is then thresholded to get a binary mask image 8 > < Mb(x; y) = >: 1 if Mr(x; y) > rtL 0 else

(3.8)

The threshold ratio rt determines in which fraction of the single variance images a pixel has to be present to be considered in the mask. It was xed at rt = 0:25 in our implementation, but it is noteworthy that this value is rather uncritical for the results: In some experiments, 0:15  rt  0:5 was found to be valid. The result of this step is displayed in gure 3.7.

Figure 3.7: Mask for the Heart Region after Thresholding 3. The binary mask Mb now outlines the heart but still contains noise clusters as well. However we observe that the heart corresponds to the largest cluster in the image. Thus a connected component labeling is performed to distinguish the dierent objects within Mb and count the number of pixels. The labeling algorithm uses two passes over the image to nd, label and count the pixels in 4-connected neighborhoods, cfr. Rosenfeld in [8].

24

Let IH max be the set of all coordinates (x; y) of pixels within the largest cluster in Mb. Then we get the mask 8 > < M(x; y) = >: 1 if (x; y) 2 IH max 0 else

(3.9)

4. The mask M obtained in the last step is now cleaned again by the operator de ned in (3.6) to slightly dilate the outer contour and ll in small holes. The parameters used here are neighborhood radius r = 2, Nmax = 10 and Nmin = 7. This cleaning step is not strictly necessary and could be left out but was found bene cial for the robustness of the algorithm. The nal resulting mask M is shown in gure 3.8.

Figure 3.8: Final Binary Mask for the Heart Region We now use the mask M to extract the heart region out of the thresholded variance images VlT as VlH (x; y) = M(x; y)VlT (x; y) (3.10) The resulting binary images VlH are then used to estimate the position, size, aspect ratio and orientation of the heart and match an initial contour by ane transform as described in the next section. 25

Figure 3.9 shows the masked variance images for the same slice positions and threshold as in gure 3.2.

Figure 3.9: Masked Variance Images Optionally, the largest object of the single images VlH can be extracted by connected component labeling. This step was not found to be generally necessary but enhanced robustness in case of bad estimation of the heart size sH or too small threshold ratio rt. Experience with more dierent image sets is needed at this point. Another possible extension to increase robustness is to ll all \holes" in the images, using for example an algorithm using image inversion and labeling as suggested in [19] 26

3.2.5 Adapting the Algorithm to 2D+time Data Sets The algorithm described above does not strictly rely on a full 4D data set with L > 1. In the case of L = 1, the binary mask before selecting the largest object is identical to the thresholded variance image:

Mb = VlT In this situation only the labeling step and selection of the largest object are required to obtain the binary variance image of the heart, VH . However we must take care that the largest object really corresponds to the heart. This is not necessarily guaranteed in the case of single slice data. Additional information about the location of the heart, gained from image header information or user input may be necessary. A very simplistic approach that nevertheless has been found quite eective is to multiply the variance image V with a real valued a priori con dence mask Mp before thresholding. A two dimensional Gaussian function

Mp = e? 2s1H (x?p)T(x?p) has been found to be suitable. For the a priori mean p the middle of the image [X=2 ; Y=2]T can be used in absence of other knowledge. Doing so, we successfully detected the heart in all 18 available 2D+time image sets.

3.3 Placement of an Initial Contour 3.3.1 Initial Contours An initial contour CIl can be obtained by a variety of ways. The following possibilities were considered, realized and compared for the purpose of this work:

 Model-based initialization by intersecting a generic or speci c model-of the heart

at a slice position equivalent to the normalized position l as de ned in section 2.1. In later stages of the project this work is part of, model-based initialization will be used. Some experiments with a generic heart model in Silicon Graphics Open Inventor format (vertices arranged to triangles composing the object) have been conducted and showed that the approach is feasible. However, as no nal description format of the model exists yet, no nal implementation exists yet. 27

 Using the results of previous segmentations as initial contours. This can be

done with both spatially and temporally related contours, i.e. either the already extracted contour of a neighboring slice or a set of contours from a previously taken MR image set with comparable parameters.

 Automatic initialization as points on the unit circle, which is equivalent to model the heart cross sections as ellipses. This simple approach has been found to be surprisingly accurate and eective. Drawbacks are that there is no control on either inclusion or exclusion of smaller features, such as the aorta for transversal images and that in some situations, namely sagittal and long-axis views of the heart, ellipses are not the best model.

 Manual, interactive initialization by the user which has been used for experimental purposes as well.

Unless otherwise noted, the procedures described in the following are independent of how an initial contour CIl has been obtained.

3.3.2 Estimation of the Heart Pose To adapt an initial contour CIl to each of the the binary images VlH we now rst need to estimate the location, size, orientation and aspect ratio of the object from the images. This is done in equivalence to the procedure outlined for contours in chapter 2 and leads to measures equivalent to equations (2.4), (2.2), (2.7) and (2.8). The size is simply the number of nonzero pixels,

sVl =

X X Y X x=1 y=1

VlH (x; y)

The location, as expressed by the center of gravity, is computed as Y X X X 1 xVlH (x; y) Vl = s Vl x=1 y=1

(3.11)

(3.12)

with x = [x; y]T. For estimation of the orientation and the aspect ratio of the heart, we rst need to compute a 2  2 covariance matrix Y X X X (xxT ? Vl TVl )VlH (x; y) (3.13) VlH = s 1? 1 Vl x=1 y=1 28

of the masked binary variance images. We then apply the Karhunen-Loeve Transformation [7, 12] to write the covariance as where

VlH = MTVl Vl MVl ;

(3.14)

2 Vl = 64 Vl ;1

(3.15)

3 0 7 5

0 Vl;2 is the diagonal matrix of the eigenvalues of VH and l

3 2 MVl = 64 mVl;11 mVl;12 75 = [mVl;1 ; mVl;2 ]

?mVl;12 mVl;11

(3.16)

is the orthonormal matrix of the corresponding eigenvectors.

3.3.3 Ane Transform of the Initial Contour We can now give a sequence of ane transformations to match the initial contours CIl globally to the binary variance images. This is done separately for each slice position l. Note that the steps given sequentially below may be of course combined to one single ane transform in an actual implementation: 1. Transfer of the initial contours CIl to zero mean form (2.3) as Ul 2. Rotation of the result such that the contour major axes are coincident with the carthesian coordinate axes:

U^ l = MCIl Ul sign(det MCIl ) where the contour modal matrix MCIl is given by (2.8). The multiplication with the signum of the determinant ensures that the rotation is at most 45, avoiding a model contour, which could be assymetric, being ipped over. 3. Correction of the aspect ratio to get

3 2 U~ l = 64  0 75 U^ l

0 1

where

v u u  I  = t Vl ;1 Cl ;2

Vl ;2CIl ;1

29

is computed from the ratio of the contour and image eigenvalues given by (2.7) and (3.15). 4. Second rotation of the contour to re ect the orientation of the heart as given by the variance image modal matrix MVl . We get

U l = MVTl U~ l sign(det MVl ) 5. Contour scaling to the image dimensions: s Ul = sVl U l sU l Note that the contour area sU l needs to be computed after step 3 (correction of the aspect ratio) 3. 6. Finally a globally adapted contour CAl is obtained by adding the image center position Vl to each contour point ul;i. Note that in some cases where detailed model contours can be provided it can be advantageous to leave out the rotations (steps 2. and 4.) based on the eigendecomposition of the variances. Instead, a priori knowledge about the orientation of the heart, gained from the image le headers, can be used. The resulting contours CAl are now globally adapted to the heart position and are close enough to the heart outline to be a suitable input for the multi-step ne segmentation procedure detailed in chapter 4. Figure 3.10 shows some examples for the adaptation of model-templates (left) and automatically generated ellipses (right). The small black crosses mark actual contour points, the white connections have been obtained by bicubic spline interpolation. We see that both, model-contours and ellipses match the contours well with only few dierences. Notably, the model-contour ts better at the topmost slice. Further, the aorta is visibly excluded by the model-contours, whereas the ellipses intersect this vessel near the middle. This predictable exclusion (or inclusion) of smaller features is a further advantage of the use of model-contours. The model-contours used here have been obtained from an earlier segmentation, initialized by the operator. 3 Otherwise the scaling eect of step 3 has to be taken into account.

30

Figure 3.10: Initial Contours adapted to the Variance Image by Ane Transform 31

Chapter 4 FINE SEGMENTATION of the HEART: Active Contour Models In this chapter, we show how active contour models, also called snakes, are used for the ne segmentation of the heart contours. We assume that the target has been globally localized and contours CAl have been adapted accordingly. These provided contours now need to be locally deformed for nal segmentation. We rst introduce our formulation of a model-based active contour for single images. Then an extension for treating multiple related contours is presented. Finally we use the snakes in a multi-step approach to solve the ne segmentation problem of the heart contours for our 4D MRI dataset.

4.1 Introduction to Active Contours: Why Snakes Snakes have been rst introduced by [11] as a method for image segmentation and object outlining. A snake is a curve, either parameterized or de ned as discrete point contour C as in section 2.2, with an associated non negative energy functional E (C). The energy typically is composed of an external energy or image force Eext and an internal energy Eint expressing smoothness or model constraints. The snake is said to be minimized if its energy is (globally or locally) minimized. So the choice of E (C) is critical for having the minimized snake contour identical to the desired target contour, together with an initialization close to the target. Many improvements to the original snakes and their energy have been suggested, 32

often for applications to medical imaging problems [20, 21, 22, 23, 24, 25]. The theory of snakes is also related to curve propagation and level sets [26, 27] and can be combined with other segmentation techniques [28]. We use an active contour technique for the local ne segmentation of the heart for many reasons:

 Unlike in methods relying on edge detection with subsequent edge linking, snakes provide inherently connected and closed contours.

 Because of the high variability of MR images, region-based methods using split-

and-merge techniques based on local gray level or texture similarity are not very promising.

 We can incorporate model information into the internal energy formulation of the snake to allow the use of a priori information.

 We have developed a method in chapter 3 which very eciently solves the initialization problem for our application.

GSNAKE API as Implementation Basis Our snake implementation is based on the \gsnake" of K. F. Lai [29, 16, 30]. Main advantages for using this discrete-point based model include a model-based internal energy formulation. The publically available library GSNAKE API 1 [31], written in the C++ programming language [32], was used as an implementation basis for our work. It provides basic classes for matrix, image, contour and snake manipulation. More classes and methods where added and some methods modi ed for the purpose of this work. See the appendix for a short overview over all new or changed classes, methods. and utilities. Another feature of GSNAKE API is that basic model learning routines are already available which could be easily extended. 1 http://www.cs.wisc.edu/computer-vision/projects/gsnake.html

33

4.2 Active Contour Energy Formulation We now develop the energy formulation for a snake consisting of a single contour C on one image I. Later, in section 4.3, we will extend this model to cope with multiple contours on dierent slice levels. We specify the total snake energy

E (C) =

n X i=1

E (ci) =

n X i=1

i Eint(ci) + (1 ? i )Eext(ci)

(4.1)

of a snake C as a summation over the energy E (ci) of the single points ci. Here Eint(ci) is the internal energy of contour point (called snaxel) i, Eext(ci) the external energy and i 2 [0; 1] a regularization constant. A snaxel-based energy formulation is advantageous for minimization, however we explicitly note that E (ci) not only depends on snaxel i.

4.2.1 In uence of the Image: External Energy The external energy Eext(ci) is induced on the snake by the underlying image I(C) and is intended to pull the contour to desired features. Often, the magnitude image gradient values [11, 20], a function of the gradient [22] or a distance map depicting the distance of each pixel to the nearest edge point [21, 33] is used. More recently, energy formulations using both gradient magnitude and direction have been proposed (see for example [21, 22, 16]). We use an edge map (Pmag; P ) consisting of gradient magnitude and direction to derive the external energy. To compute this edge map, the image is rst convolved with a 3  3 Gaussian kernel. This step smoothes out some noise eects and also stretches out the edges, increasing the ability to attract distant snaxels. To achieve higher accuracy and better convergence properties, we can leave this step out for snakes already very close to the optimal solution (see section 4.4). Edge magnitude P^ mag and orientation P then are computed using a 3  3 edge operator such as the Prewitt or Sobel operator [8]. It is important to use an odd order operator as otherwise (using a Roberts operator for example) a bias would be introduced in the edge position. In our experience, the Prewitt operator as shown 34

below performed best: We rst compute two directional derivative images as 3 2 ? 1 0 1 77 66 6 Dx = I(C)  64 ?1 0 1 775

?1 0 1

2 66 1 Dy = I(C)  664 0

The edge magnitude is then given as

3

1 1 7 7 0 0 775 ?1 ?1 ?1

q

P^ mag(x; y) = Dx(x; y)2 + Dy (x; y)2

(4.2)

and the edge orientation as Pang(x; y) = tan?1(Dy (x; y)=Dx(x; y))

(4.3)

for 8 (x; y) 2 Ij!E . In the MR images of the upper body the heart contour is frequently not the dominant and most pronounced structure. On the other hand, there are many edges corresponding to spurious features and small details we are not interested in as well. We thus proceed with an histogram-based conditioning [31] of P^ mag with parameters set to enhance medium strong structures and suppress weak edges. For the conditioning, we rst compute a histogram hP^ mag (v) of the edge magnitude image P^ mag as in (3.3) and then estimate the cumulative distribution function as v X PP^ mag (v) = N1M hP^ mag (u) u=0

for all gray-levels v = 0 : : : vmax. This cumulative distribution is then mapped according to 8 > P (v) vlow ; if PP^ mag (v) < plow > < P^ mag plow PP^ mag (v)?plow PCmag (v) = > (vhigh ? vlow)( phigh ?plow )exp + vlow ; if plow  PP^ mag (v) < phigh > : (1 ? vhigh)( PP^ mag (v)?phigh )exp + vhigh ; if phigh  PP^ (v ) 1?phigh mag

(4.4)

35

and the nal edge magnitude for all (x; y) obtained as Pmag(x; y) = vmaxPCmag (P^ mag(x; y))

(4.5)

For the parameters, we experimentally found plow = 0:65, phigh = 1,vlow = 0, vhigh = 1 and exp = 3, to work best for our presaturated 4D image set. Adjusting the parameters of this conditioning also allows adaption to the results of various imaging techniques. This is important as MR image appearance varies considerably depending on which imaging technique is used (see the many examples in [2]). Note that plow = 0:0, phigh = 1,vlow = 0, vhigh = 1 and exp = 1 yields histogram equalization. Figure 4.1 shows an example for an edge map with magnitude (left) and orientation (right) after these processing steps.

Figure 4.1: Edge Magnitude and Orientation To take the edge direction into account, we rst compute the contour tangent ti at snaxel i as (4.6) ti = 21 ( kcci ?? cci?1k + kcci+1 ?? ccik ) i

i?1

i+1

i

and rotate ti by either 90 or 270 , dependent on contour orientation, to get an outward facing contour normal ni = [ni;x; ni;y ]T. This is then used to obtain an 36

external energy measure ! 1 1 Eext(ci) = n 1 + max (P ) Pmag(ci)[ cos(P (ci)) ; sin(P (ci))]
ni (4.7) IE mag Here we have also used the empirical knowledge that the inside of the heart appears brighter on MR images than the surrounding tissues, thus the gradient orientation Pmag(ci) faces inward at contour points ci. This results in Eext(ci) being close to 0 if ci is a strong edge point on an edge parallel to C with gray level values brighter in the inside of the contour. For weak edge points or disagreeing directions of edge and contour, the energy will be near 1=n. For strong edges, parallel to the contour but with reversed gray level gradient (outside brighter than inside), the energy can become larger than 1=n, resulting in a strong repelling eect of such edges on the snake. This was found very eective and essential to avoid the snake being trapped by strong edges not representing the outer contour of the heart, as for example the inner border of the lungs at the chest. As the edge map is de ned on the nite grid but snaxel coordinates are allowed to be real numbers, the need for interpolation arises. The simplest method is to simply round ci;x and ci;y to the next integer coordinates within the edge map. To achieve some more (sub-pixel) accuracy, we have also implemented a simple linear interpolation scheme which models the edge map as triangular patches with vertexes at the integer pixel positions. A drawback of this is the higher computation time.

4.2.2 In uence of the Model: Internal Energy The internal energy of the snake is derived from its contour shape and is intended to give the shape desired properties such as smoothness or similarity to a model. In the original publication of Kass et. al [11], the absolute value of the rst two local derivatives of the contour with respect to a curve parameter were used, in analogy to membrane and thin plate behavior in mechanics. To overcome shortcomings of this model, such as attraction by too weak edges, shrinking around corners, and to introduce model properties, many more formulations, for example adding an in ation force, have been suggested. Good references with applications in medical imaging include [20, 22, 34] 37

We follow and extend the approach of Lai [16, 30] for a model-based, ane invariant internal energy. Let ui = ci ? C represent the Contour C with respect to the contour mean (2.2). We then express each contour point as a linear combination 2 of its two neighbors as ui = iui?1 + iui+1 (4.8) with shape coecients i and i. These coecients can be computed by interpreting (4.8) as a linear equation system: 2 3 2 64 i 75 = 64 ui?1;x ui+1;x

i

ui?1;y ui+1;y

3 3?1 2 75 64 ui;x 75

(4.9)

ui;y

The indexes i are again modulo n-fashion de ned so that un+1 = u1 and u0 = un . By collecting equations (4.8) for all points and rearranging, we get a matrix shape equation 2 66 66 66 AUT = 6666 66 66 4

1 ? 1 0 0 ?2 1 ? 2 0 0 ?3 1 ? 3 ... . . . . . . . . . 0 0 0 ?n?1 ? n 0 0 0









32

3

?1 7 6 u1 7 76 7 0 77 66 u2 77 76 7

... 0 ... ... 1 ? n?1 ?n 1

77 66 77 66 77 66 77 66 75 64

... ...

77 77 = 0 77 7 un?1 775

(4.10)

un

with shape matrix A. The shape matrix has rank n ? 2. The shape equation is thus ful lled by all contours UT = BAT where BA is an orthonormal basis for the null space of A and T an arbitrary 2  2 transformation matrix. If C and either T or two arbitrary points ui; uj are known, a contour can thus be regenerated from A. ^ , (4.10) becomes AU^ T =
where
is a n  2 For a locally deformed snake U matrix of local coordinate deviations from the model contour, row wise conditioned on u^i?1 and u^i+1. With spur(A) denoting the summation over the main diagonal elements of a matrix A, we de ne the overall internal internal energy in form of a 2 This requires ui?1 and ui+1 to be linearly independent. This was always the case in our stu-

dies and we will assume this property throughout this work. Precautions have to be taken in an implementation nevertheless.

38

summation over the single snaxel internal energies as Eint(U^ ) =  1 spur(AU^ T R?1U^ AT ) ^) L(U

(4.11)

Here R = diag(12; 22; : : : ; n2 ) allows introduction of local weighting of the pixels and the average squared snaxel distance L(U^ ) = n1 Pni=1 ku^i ? u^i?1 k2 is used for normalization. In [16] it has been shown that this energy formulation is scale and rotation invariant. It is further translation independent as the equation deals with the contour in a centered form as de ned by (2.3). This energy is model-based as the shape matrix A represents the shape information of the contour U with E (U) = 0. Thus we can start with an initial contour obtained from a model. As the localization procedure described in chapter 3 is based on ane transform, it does not change the shape matrix of such an initial contour and we can start with zero internal energy. Then we have to deform the snake until the increase of internal energy is compensated by a larger decrease of external energy, ignoring weak local minima. This emphasizes the need for a good initialization.

Smoothing Term In practice we know that organs in medical images tend to have relatively smooth outer contours, corrupted by noise and possibly small details we might not be interested in. So it is desirable to introduce a smoothing term into the internal energy. First, we observe that the shape coecients for an ellipse are given by (4.12) i = i = 2 cos(21 =n) and further that an ellipse is perfectly smooth in the sense that it minimizes contour length for given area and aspect ratio. Thus we get a smoothing shape matrix AS with nontrivial elements given by (4.12). Let AC be the contour shape matrix as given by (4.10), computed for a model contour. Then we rede ne the shape matrix as a weighted average A =  AS + (1 ?  )AC (4.13) of the two shape matrices with weighting  2 [0; 1]. 39

0.4

0.2

0.0

0.6

0.8

1.0

Figure 4.2: Eect of the Smoothing in Internal Energy Calculation

40

Figure 4.2 shows examples for various smoothing factors  . In practice we found 0:2    0:4 to be most bene cial. As our internal energy formulation is scale invariant, there is no need to introduce an image data dependent parameter such as the in ation force in [20]. It should be noted that now A is generally of rank n and the contour equation is no longer homogeneous and regenerative. We now get AUT = e and the energy function can be interpreted as a least squares minimization of the error e.

4.2.3 Energy Regularization and Minimization Energy Regularization As formulated in (4.1) internal energy and external energy are combined to the snaxel energy E (ci) = i Eint(ui)+(1?i )Eext(ci) by a regularization factor i 2 [0; 1]. Large i close to 1 emphasize internal energy (model aspects), small i external energy (image forces). Several choices for this factor are possible:

 We can x i =  for all snaxels in advance and so decide whether the model

or the image aspects should be emphasized. This method, with emphasis on internal energy, was found to be well suited for initializations not close to the actual contour for all snaxels, such as the ane adapted contours resulting from the procedure described in chapter 3.

 A local minimax criterion is suggested in [29, 16], setting i = 1, if Eint(ui) > Eext(ci) and i = 0 otherwise. In our experiments we found this criterion to perform well only if the initialization is very close (less than about 3 pixels distance) to the actual contour, otherwise the snake was often attracted by spurious edges. However, small details are very well recovered by this regularization.

 The i could be chosen xed but dierent for each snaxel by a learning pro-

cess, taking local variability of the heart contour into account. However as more image data to actually build a contour model for the heart has not been available, this promising approach has not been further pursued yet. 41

 Some experiments have been made with a mixture model, combining xed i with the minimax approach to

E (ci) =  max(Eint(ui ); Eext(ci)) + (1 ? ) min(Eint(ui); Eext(ci))

(4.14)

with  2 [0:5; 1]. This promises to keep the advantage of the minimax approach which is that  needs not to be predetermined 3. At the same time, problems such as point accumulation around strong edges are avoided. The dierent approaches will later be used to optimize the multi-step approach described in section 4.4.

Energy Minimization To segment the desired contours with a snake, we now have to minimize the energy (4.1). As there is no closed formulation for E (C), this has to be done numerically. Further, Eext is not convex but displays many local minimax corresponding to suboptimal solutions so that gradient based methods such as gradient descent techniques can not be used. Hence exhaustive search methods have to be employed. In order to reduce the search space, we employ line search methods along the contour normals 3 !2 0 1 c ? c c ? c 1 75 ni = 2 kci ? ci?1 k + kci+1 ? cik 64 i i?1 i+1 i ?1 0 This means we allow each contour point ci only to be changed to c^i = ci + ini for some real i. The normals ni can either be xed for the complete minimization process or recomputed during the iterations. We found xed normals advantageous in most situations as the computational cost and the possibility of dis-ordering the contour points are reduced. Two algorithms have been used for minimization:

 Dynamic Programming [35, 36] which is a computationally ecient and stable

exhaustive search method over a nite set of predetermined possible solutions.

3 It is less critical to determine  than .

42

This method is well suited for localizing a minimum of the energy without being trapped by local minima. However it is computationally expensive, especially if a large number of search locations is used for accurate segmentation. Strati ed line search [16] partly overcomes this problem. Additionally, increased number of mutually dependent contour points, as it is the case with the 3D contour model presented in section 4.3, exponentially increases search complexity. As a consequence, convergence of the algorithm is slowed down. Partly neglecting snaxel dependencies, we found it still adequate for practical purposes, especially with wide search spacing to approximately locate the minimum of snake energy. The performance of dynamic programming is determined by the number of search locations for each snaxel nn and the spacing ns of these search locations along the normals ni.

 A random walk routine [37], which has been introduced for the minimization of

neural network objective functions and is similar to the metropolis algorithm [38, 36] is used for to nd an energy minimumnot bound to nite pixel locations. With a zero mean Gaussian random variable s of variance s2, we minimize a contour as follows: For each slice location l 1. Randomly select one snaxel cil 2. Generate s. 3. If E (cil + snil ) < E (cil ) let cil = cil + snil , else leave cil unchanged.

Performance is characterized by the variance s2 of s. This variance can be either held constant or decreased with increasing number of iterations in an annealing scheme. We used simple linear reduction from given s;max to s;min over a prescribed maximum number of iterations Nmax. Alternatively we could change each snaxel of the snake simultaneously and then compare the snake energy. While this method is guaranteed to converge with some annealing schemes [36], it turned out to be too slow for practical application. 43

We will employ these techniques such that either only one of the two is used for a minimization step or dynamic programming rst for coarse localization, followed by random walk for nal minimization for a coarse-to- ne approach.

4.3 Slice Oriented 3D Contour-Model All snake energy calculation and minimization techniques presented in this chapter thus far has been done for contours de ned in the two-dimensional space IR2. As our data not only includes two but frequently three spatial dimensions, we seek to extend our contour model accordingly. A priori, we require that the model is a compatible extension of the two dimensional contours and that it is suited to sparse, widely spaced sliced data sets. These requirements stem from the fact that we need to deal with 2D+time data sets or data sets with very few spatial locations l. So it is not feasible to employ a fully three dimensional model as the nite element balloon model of [39]. In the cited work, a CT image set of 118 slices of 128  128 pixels of the left ventricle of a canine heart could be used while our human MR image sets contain at most 17 slices. We start with two observations: 1. In (4.9), a contour point ui is represented as a linear combination of its two next neighbors ui?1 and ui+1 . While this choice may seem obvious and appealing, it is not the only one valid. In fact, every contour point ui could be represented as the linear combination of any two linearly independent contour points u, u to yield a valid shape matrix. Moreover, any linear combination of such shape matrices would yield a valid shape matrix A. All these dierent shape matrices have the same null space with orthonormal basis BA. For consistent regularization, we introduce the convention to normalize all rows of A such that ai;i = 1. 2. We further observe that by concatenating contour matrices U1; U2; : : : ; UL and arranging the shape matrices A1; A2; : : : ; AL accordingly, we obtain one single 44

shape equation

2 66 A1 0


66 0 A2 0 T AU = 666 .. . . . . . . 64 .

32

3 77 77 ... 77 66 ... 777 = 0 75 4 5 T

0 77 6 UT1 ... 77 66 UT 7 66 2

0 0


AL

UL

for L contours. These contours may each have a dierent number nl of points. Given that all contours are nontrivial 4, A then has rank PLl=1 nl ? 2L. Note that as the ane space of A only determines the coordinates within the slices, the slice positions l need to be stored separately. We now combine the two observations to obtain a shape equation in which each single contour point uil is not only determined by its neighbors within contour Ul but also by points in other contours. Speci cally, for our case with only one contour in each slice l, we now de ne a shape equation as 2 66 A1 A12 66 A21 A2 6 AUT = 666 0 A23 66 .. . . . 64 .

0 A23 ...




0 ...

0 ... ...

32 77 66 77 66 77 66 77 66 77 66 75 64

3 77 77 ... 777 = 0 77 T UL?1 775 T

UT1 UT2

(4.15)

AL?1;L?2 AL?1 AL?1;L UL 0 0 0 AL;L?1 AL where the points in each contour Ul are in uenced by the neighboring contours Ul?1 and Ul+1. We now introduce a regularization parameter  2 [0; 1] which determines the strength of the inter contour relation. Then the elements of the shape matrix A are computed as follows:

 The main diagonal elements of Al are set to ai;il = 1 for l = 1 : : : L.  The other nonzero elements of Al are set to il and  il according to (4.9).  For computing the i-th row of each minor diagonal submatrix Al;k (k = l  1) we rst de ne a neighborhood mil;k containing m snaxels uj1k : : : ujmk of contour 4 All points within a non-trivial contour are mutually linearly independent

45

k which have been selected as neighbors of snaxel i in contour Ul. Equally weighting all elements of mil;k , we then can write this snaxel uil as a summation over all possible and unique 2-snaxel permutations in contour k as

uil = 2 (mm?! 2)!

jm jm X?1 X =j 1 =+1

;k k ; il uk + il uk

(4.16)

;k k The ; il and il are again computed using (4.9). Summing up coecients at the single snaxels, we then get the j -th element of the i-th row of Al;k as

(1 ?  ) ( al;k i;j =

mX ?1 =j

k j; il +

m X =j +1

ilk ;j )

(4.17)

re ecting the in uence of snaxel j in contour k on snaxel i in contour l. Here 8 > < 1 if i = 1 or i = nl

=> : 2 else

accounts for each contour either having one or two adjacent contours. All other elements of Al;k not covered by (4.17) are set to zero. In our implementation we have selected a neighborhood mil;k of m pixels from Uk with minimal angle dierence jjk ? jl j for each pixel in Ul. The angle  is given as indicated by (2.3). This selection allows contours in dierent slice levels l with dierent number nl of elements.We further used m = 3 neighbors. The reason for this shape formulation is to allow the introduction of 3D model information into the internal energy with the ability to control the weight of this information. This allows for example to increase this measure for small spatial distance between the slices and to decrease it if only few slice locations are available. For our transversal image set of 17 slices, 0:3    0:4 was found to be suitable. The visible bene t of the model is an increased resistance against noise, spurious edges and partially missing edges which may result from the initialization procedure of chapter 3. In an actual implementation of the model, one can take advantage of the banded, sparse structure of A. As for the 2D case, precautions must be taken to ensure that all pairs of contour points used in (4.16) are in fact linearly independent. In case they 46

are not, one possible solution is to set the corresponding  = = 1=2 or ignore the point pair altogether. It should be noted that many more choices for the de nition of the neighborhood

mil;k are possible, notably solutions involving more than one neighboring contour. This opens for example the possibility to express a contour point in contour Ul as an average of its nearest neighbors in Ul=1 and Ul+1, introducing an inter-contour smoothing term. Also note that we assume the neighbor to be at dierent spatial slice level: mil;k should be adapted if multiple contours in the same slice are considered which is easily possible within the framework presented above. As the use of more basis points for each contour element increases the order of the Markov Random Field associated with the internal energy, energy minimization gets more costly compared to single contours. We thus apply the model only where necessary. Figure 4.3 compares the segmentation result on an image which has been heavily corrupted by additive Gaussian noise with and without use of the multi-slice contour model. The non-noisy neighboring slices and the initialization are identical in both cases. For both examples,  = 0:4 was used as smoothing strength, the strength of the inter-contour relation was set to  = 0:4 for the example using inter-slice relation. We see that the left contour in gure 4.3 follows the outline of the heart much better with only some minor, local deviations. The right example without use of inter-contour relation still retains the coarse form of the heart, due to the model based character of the internal energy, but does not follow the contour as closely and is more easily attracted by spurious edges. Note that the image used in gure 4.3 is the original image without added noise, the contour is hardly visibly anymore in the noisy image. Nevertheless the contour model proposed above showed advantages in terms of robustness also on non-noisy image sets.

47

Figure 4.3: Snake Performance With (left) and Without (right) Multi-slice Contour Model after Minimization on Noisy Images

4.4 Multi-step Approach to the Segmentation of MRI Heart Contours To increase segmentation robustness and computational eectiveness, we use a multistep approach for minimizing the snake energy, utilizing dierent images and snake performance parameter for each step: 1. In a rst step, the initial contours CAl globally adapted to the estimated pose of the heart as described in chapter 3 are minimized using again the binary variance images VlH for external energy computation. This step performs major deformations necessary to adapt the form of the initial contour CAl (as obtained in chapter 3) to the outline of the imaged heart. As we do not need adaption to ne details at this point, we emphasize internal energy, use a relatively low number of contour points and employ a relatively large search space. Table 4.1 shows performance parameters typically used, gure 4.4 images with minimized snakes. 2. We then use the temporal mean images Ml in a second minimization step. This step is aimed to closely follow the contour of the heart without having the snake 48

attracted by too small details such as blood vessels. The mean images Ml are used because small details and spurious edges are not present there, enhancing the robustness of the algorithm. The search space is still relatively large, the internal energy emphasized and the number of contour points low. The results are shown in gure 4.5 3. Now the single images Itl for all locations l and times t are used as external energy resource for the further minimization of the snakes resulting from the last step, tracking the heart wall motions. For optimization of search eciency, we start with one image Itl and extract contour Ctl, using the result of step 2 as initialization. This contour Ctl is then used as initialization on Itl+1 for extraction of Ctl+1, which is then used as initialization on the next image and so on. Experimentally, we found t  3=8 : : : 1=2T to be the time matching the mean contour best for the images we had available, i.e. a time in the middle of the cardiac cycle. More data is needed however to con rm this observation. As the contours already match closely, search space can be reduced. Figure 4.6 shows resulting contours on the actual images at time t = 0. This multi-step approach allows the use of generic initial contours (ellipse) which are then step by step adapted to the image features. It is ecient as the rst two steps with larger search region can be done with a fairly small number of contour points which then can be up-sampled before step 3 using the spline techniques described in section 2.2.2. The multi-step approach is a robust, fully automatic contour extraction method. Nevertheless, it allows an implementation in which the user can interact and correct the position of single points, using a mouse, if desired. We found this particularly useful after step 2 for the lowest and topmost slice locations. In practice, the use of multiple dierent images with adapted minimization parameters proved to be more ecient and reliable than a scale-space approach using Gaussian pyramid images. This is partly due to the fact that the heart contours often have low contrast compared to other features in the MRI images. 49

Depending on whether the initial contour was model-based or automatically generated, the shape matrix A (4.15) is held constant, changing only the smoothness factor  , or recomputed after each step. In the case of user interaction, A is always recomputed for each step to re ect the user's input. Table 4.1 shows typical choices for the relevant parameters. If two values are given, the rst one corresponds to automatic initialization as ellipse, the second one to the use of a model template. Parameter:

Step 1

Step 2

Step 3

(Bin. Variance Images) (Mean Images) (Single Images)

No. of Points nl  (4.1)  (4.13)  (4.17) Filter for Pmag nn ns s;2 max

30 - 40 0.65 /0.75 1.0 / 0.4 0.3 Gauss 9 3 4

30 - 40 0.65 0.3 0.3 Gauss 5 2 2

30 - 200 0.5 or  = 0:7 0 0 None 3 1 1

Table 4.1: Snake Performance Parameters for the Dierent Steps. Figures 4.4 - 4.6 exemplary show the resulting contours on the images for some slice locations after completion of the dierent steps. The tracking of the nal contour done by step 3 can be very well demonstrated by a computer animation displaying the images with contours projected on them as a movie. An other possibility to visualize the heart wall movement is to plot the radial movement of the extracted contour over time. Figure 4.7 shows this for the region of the left ventricle. The plot uses averaged and spline interpolated values over an arc length of =15, it is repeated three times. Plots like this show some resemblance to electro cardiograms (ECG), diagnostic use could be considered as well.

50

Figure 4.4: Snakes Minimized on Binary Variance Images

51

Figure 4.5: Snakes Minimized on Mean Images

52

Figure 4.6: Final Segmentation of the Actual Images

53

3

Deviation from Mean Position

2

1

0

−1

−2

−3 0

5

10

15

20

25 Time

30

35

40

45

50

Figure 4.7: Radial Movement of the Outer Contour of the Left Ventricle Over Time

54

Chapter 5 FUTURE EFFORTS The work presented in this thesis represents only the rst steps of the cardiac MRI processing project. The methods developed could only be tested on very vew images, so testing on more images with the subsequent re nement of the algorithms and inclusion of more sophisticated model aspects are the major parts of future eorts to be undertaken.

5.1 Heart Localization Future research on the temporal variance base localization of the heart could be done to further increase robustness. Especially the proposed method for estimation of the heart size sH (and so the threshold) from the binarized variance images has to be tested on more images and re ned as necessary. This gets very important for images in which the heart is not present at all but only other organs such as the stomach or the liver and the diaphragm as observed in the lowest slice of our image set. At this time, these slices require human interaction and are best manually discarded from the image set before segmentation. Also the use of dierent estimations of sH for dierent slice levels can be considered. With more test image sets, more detailed generic models for the heart than ellipses could be generated. This would be especially important for reconstruction from sagittal views in which the heart is not well modeled by an ellipse. In transaxial 55

views, one could consider a simple model with a \bump" either inwards or outwards at the expected location of the aorta to either deliberately include or exclude this vessel in the segmentation 1. Trials with manually generated initial contours have con rmed the feasibility of this idea. Finally, one could consider the use of a three dimensional model to derive the initial contours. This would require a 3D ane transform of this model (an ellipsoid in the simplest case) according to all image slices and then intersecting the model with the single image planes to get initial contours. However for robust and accurate adaption of such a model, the number of image slices should be larger than in our currently available data sets.

5.2 Active Contour Model Future Work As our image supply was very limited, future work should mainly

include experiences with the algorithm on more, dierent image sets to nally nd optimal performance parameters. Dierent shape matrix formulations for the multislice contour model have to be researched and methods for nding optimal, nonempirical solutions developed. More ecient algorithms for numerical minimization could be researched as well. This should nally lead to the availability of a model of the heart and derive parameters, such as local energy regularization, using learning methods. A further extension could be the use of multiple contours in each image slice to simultaneously outline several contours, for example outer heart contour and inner contour of left and right ventricle.

5.3 Outlook: What are the Results Used for ? The segmentation method developed in this thesis yields two dimensional contours of each slice image as discrete point sets. This data will be used in the future to build and adapt a 4D model of the heart as a basis for model-based coding, visualization and diagnostic purposes. Such a model, based on easier to segment tagged MRI data, has for example been presented in [40]. 1 Both cases are of diagnostic interest

56

In the context of this work, we have used the software package NUAGES 4.0 [41] to demonstrate the suitability of the extracted data for this purpose. The tool uses Delaunay triangulation to calculate surface patches and normals of a three dimensional model from slice oriented discrete point contours as extracted by the methods described in this work. This model can then be rendered and displayed, using packages such as Silicon Graphics OpenInventor. Figure 5.1 shows some examples of such three dimensional views which have been generated in fully automatic way without any user interaction from our 4D image set.

Figure 5.1: 3D Reconstruction of the Heart Other, more direct clinical uses of the data could include direct assessment of the gray level temporal variance images as an aid to diagnosis. More image examples, including acquisitions from patients with pathologic conditions which are not available at this time, are needed to evaluate this possibility.

57

Chapter 6 CONCLUSIONS Heart Localization We have developed a method for automatic localization of

the heart in 4D MR images based on the temporal variance of the image gray levels. Spatial information is used to compute a con dence mask for the heart location. The method is independent of the image orientation and perspective. The localization is done automatically: Only one input parameter, the estimated maximal size of the heart, has to be provided. The localization is fairly stable with respect to missestimation of this parameter. Automatic determination of this parameter is feasible and a method for doing it has been suggested. We have further shown how to adapt the algorithm to 2D image sets with only temporal but no additional spatial information available and how to take advantage of a priori information provided in the image les. The initial contour templates to be adapted to the heart by this algorithm can be either automatically generated or a model template can be used if available.

Active Contour Model We have adapted and extended a exible, model-based

active contour model to the requirements of our segmentation problem. The formulation uses a model-based internal energy but also introduces the possibility of using a smoothing term which does neither shrink nor in ate the contour. A fundamental extension allows simultaneous minimization of multiple contours with model-based mutual dependency. External energy consists of edge magnitude and 58

signed edge direction, using the knowledge that the interior of the heart appears brighter in MR images than surrounding tissues. We use the active contour model in a multi-step approach on dierent types of images (binary temporal variance images, temporal mean images and single actually acquired images) to increase robustness and eciency. A drawback, especially for the multiple contour model, is the high computational cost of numerical minimization.

59

APPENDIX New Classes and Methods for GSNAKE API This appendix contains short descriptions of classes and methods added to the GSNAKE API. For a more detailed description see the GSNAKE API reference manual [31] and the reference to the added methods.

ARRAY: Basis Class for IMAGE and MATRIX provides basic functionality for real two dimensional arrays. It serves as a base class for IMAGE and MATRIX to allow ecient matrix operations without the added overhead needed for images. Provided functions are

ARRAY

public : ARRAY(); ~ARRAY(); void reset(); init(int _row, int _col); void initAsGauss(); void initAsUnit(); void dump(char *header=NULL); void put(int m, int n, float val); float get(int m, int n); getRow() {return row;} getCol() {return col;} void print(); ARRAY *correlate (ARRAY *InputImg,int RowStep, int ColStep,char verbose); int fill(float val, int sx=0,int sy=0, int length=0, int height=0);

60

Syntax and synopsis are identical to the equivalent functions described for IMAGE in [31].

MATRIX: Matrix Manipulation Routines MATRIX

provides matrix operations for real arrays. It has the following structure:

class MATRIX : public ARRAY { protected: MATRIX * eigvals; MATRIX * eigvecs; public : ~MATRIX(); void reset(); init(short _row, short _col, char identity=0); swapRow(short i, short j ); MATRIX *MATRIX::cut(int sx,int sy,int height, int length); MATRIX *copy() {return cut(0,0,col,row);} MATRIX * transpose (void) ; MATRIX * operator+ (MATRIX& mtx) ; MATRIX * operator- (MATRIX& mtx) ; MATRIX * operator* (MATRIX& mtx) ; MATRIX * inverse (void) ; void computeEig(void); void computeEig22(void); MATRIX * getEigVecs(); MATRIX * getEigVals();};

Additionally to the ARRAY data, MATRIX contains pointers to its eigenvalue and eigenvector matrices.

Compute Eigendecomposition computes and stores the eigenvectors and eigenvalues. If a matrix is nonsquare, the largest square submatrix is used. computeEig22 is a version slightly optimized for 2  2 matrices. computeEig

Getting Eigenvalues and Eigenvectors and getEigVals return the eigendecomposition of a square matrix. If the eigenvalues have not been computed before, they are automatically computed (computeEig). The eigenvectors form an orthonormal matrix, the rows of the eigenvector matrix correspond to the respective elements in the eigenvalue matrix. getEigVecs

61

IMAGE: Process and Display Images Images are real valued arrays. Only extensions to GSNAKE API's IMAGE class are described here, see the reference manual [31] for a complete description. The data structure is as follows: class IMAGE : public ARRAY { friend class IMAGESEQUENCE; protected: double cg_row, cg_col; /* center of gravity */ public: XImage *ximg; };

Reading Image Data from a File read (char *filename,int row =256, int col =256);

reads images from a le. Currently, Gsnake API *.bin format, SUn raster format and raw 16 bit *.HD format are supported. For the reading of *.HD les, the lename must end on \.HD". read

Writing Image Data to a File int write(char *filename,int filetype = _ras) write

writes the image to a le of the speci ed format. New: *.HD les are supported.

Image operators IMAGE * transpose (void) ; IMAGE * operator+ (IMAGE& mtx) ; IMAGE * operator+=(IMAGE& mtx) ;

//unary addition

add2im(IMAGE* mtx) ;

//unary addition

IMAGE & operator+=(IMAGE*

mtx) ;

//unary addition

IMAGE * point_multi(IMAGE& mtx) ;

//pointwise multiplication

point_mul(IMAGE* mtx);

//unary pointwise multiplication

IMAGE * operator- (IMAGE& mtx) ; IMAGE * operator* (IMAGE& mtx) ;

//matrix multiplication

IMAGE * operator* (float mul) ;

//multiplication with scalar

The IMAGE operators do basic arithmetic operations.

62

Normalize the Gray-Level void normalize(float mag = GRAYLEVELRANGE); normalize mag.

normalizes the gray-level values of an image such that the maximal value is

ROI- Determination by Multiplication with 2D Gaussian void mult_gauss(double sigma=50,int mu_row=128, int mu_col=128); mult gauss

multiplies the image with a 2-dimensional Gaussian function 2 2 row col 1 f (i; j ) = p 2 e? 22 ((i?row ) +(j ?col ) ) 2 2

Thresholding an Image IMAGE *IMAGE::thresh(float th= GRAYLEVELRANGE/2.0);

sets all image values larger than th to one, all others to zero. A new object is returned. thresh

IMAGE

Thresholding an Image to Given Number of Pixels IMAGE *IMAGE::autothresh(int numpixels);

thresholds the image based on an histogram. The threshold is selected as the largest integer such that at least the numpixels brightest pixels are kept and set to one, the rest is set to zero. autothresh

Morphological Cleaning of an Image void IMAGE::clean(int mask_size=3,int thr_low=-1,int thr_up=-1); IMAGE *IMAGE::cleaned(int mask_size=3,int thr_low=-1,int thr_up=-1);

Morphological cleaning is used to remove pepper-and-salt noise in binary images. The center pixel of a quadratic mask of side-length mask size is set to 1 , if more than thr up pixels are set 0 , if less than thr low pixels are set Dependent on the arguments, this operator kept otherwise. can be used to clean, erode and dilate images. clean is used to allocate a new image and keep the original, cleaned overwrites the existing image without allocating a new one.

63

Label Components of Binary Image IMAGE *label(short **count,short *numlabel=NULL,short MAXLABEL=512);

performs a connected component labeling of a binary image, using a 4-connected neighborhood. The dierent objects are assigned labels from 1 to maxlabel, the number of pixels in each object is returned in count. label

Select one Single Object in the Image int IMAGE::mask(short value=1);

returns an image containing only the pixels of a given gray-level value in the referenced IMAGE. mask

Compute the Center of Gravity void computeCG(); computeCG

computes and stores the center of gravity of an image.

Getting Image Properties int getArea(); double getCgRow(void); double getCgCol(void); MATRIX* getCov();

computes and returns he sum of all pixel values which is the area for binary images. getCG returns the coordinates of the center of gravity and getCov computes and returns the 2  2 covariance of the pixel values. getArea

IMAGESEQUENCE: Container for IMAGES provides routines to access, read and write series of related, equally sized IMAGES. Further statistical measures such as mean and variance over all IMAGES are provided. It has the following structure: IMAGESEQUENCE

class IMAGESEQUENCE{ friend class IMAGE protected: IMAGE **images;

/* array pointers to images*/

64

int

numims;

/* number of images */

IMAGE *mean;

/* Mean Image */

IMAGE *variance;

/* Variance Image */

};

IMAGESEQUENCE Constructor and Destructor IMAGESEQUENCE() ~IMAGESEQUENCE()

The constructor initializes an image sequence with 0 elements, and set mean and variance to NULL. No images are initialized. The destructor frees the memory allocated to the image sequence data.

Resetting an IMAGESEQUENCE reset()

allows the reuse of an IMAGESEQUENCE object. All IMAGE objects of the sequence are reset and the series is reinitialized to 0 elements. reset

Initializing int init(int _numims);

allocates memory of type *IMAGE for the image pointers. The IMAGES themselves are not initialized. init

Putting an image in the sequence int putImage(IMAGE * im, int i)

puts an image at speci ed position in the image sequence. An existing image is overwritten. All images should have equal size ! putImage

Getting IMAGESEQUENCE row, col and numims int getNum() int getRow() int getCol()

returns the number of the images in the sequences ot row and column size of image matrices. get

65

Getting an IMAGE, the Sum of the Images, the Mean or the Variance IMAGE* getImage(i=0); IMAGE* getSum(); IMAGE* getMean(); IMAGE* getVariance();

returns a speci c image, the sum over all images, the mean gray-level value or the variance image. Sum, mean and variance are automatically computed upon the rst call. get

Thresholding the Image Sequence to Fixed Number of Pixels IMAGESEQUENCE autothresh(int numpixels);

thresholds all images to keep approximately the given number of brightest pixels which are set to 1, the rest is set to 0. autothresh

Reading Image Data from File int read(char *filename, int lower=0, int upper=0) int read(char *name_beg,char *name_end,int lower,int upper);

An image sequence in either Sun Raster format, binary format or raw 16 bit .HD format is read from disk. The rst syntax can only read *.HD les, with name composed as is composed as filename >> `` '' >> number >> ``.HD'', where number runs from lower to upper. The second syntax can handle all supported le formats and composes the lenames as tt name beg >> \2-digit-number" >> name end. The number is always brought to 2 digits, by leading zero if necessary.

Writing Image Data to File Writing IMAGESEQUENCES is done by getImage(i)->write(filename) for each single image if needed.

EDGE: Compute an Edge Map The only relevant extension to GsnakeAPI's EDGE class is the possibility for dierent edge operators to be used:

66

Calculating the edge map int compute(IMAGE *img,int verbose=1,double low_pct,double high_pct, double low_val, double high_val, double imm_pow, EDGEOPERATOR op=_PREWITT);

generates the edge gradient magnitude and orientation. Currently available edge operators are PREWITT, ROBERTS and SOBEL. compute

PYRAMID: Pyramid of Images for Scale Space Approach The only relevant extension to GsnakeAPI's PYRAMID class is the possibility for dierent convolution kernels and edge operators to be used:

Generating the pyramid generate(short level, int verbose=0,double low_pct=0.9, double high_pct=0.95, double low_val=0.2, double high_val=0.9, double imm_pow=1.0, EextTYPE=_EDGE, FILTTYPE=_GAUSS, EDGEOPERATOR=_PREWITT) generate creates a pyramid of Gaussian images by correlating the raw image with a spe-

ci ed kernel, and computes edge gradient based on the EextTYPE and EDGEOPERATOR speci ed. Note that the (nearly) non-smoothing kernel FILTTYPE UNIT should only be used for pyramids of top level.

SNAXEL: Single Contour Points Only extension to GsnakeAPI's SNAXEL class are presented here:

Computing Shape Coecients computeCoeff(SNAXEL *s1, SNAXEL *s2, double *ca, double *cb);

computes the shape coecient  and of the SNAXEL with respect to two given, arbitrary base snaxels. computeCoeff

67

Randomly Moving a Snaxel void deformRand(float std, double normalX, double normalY); deformRand moves

the snaxel by a random amount. The direction is determined by the vector [normalX, normalY], the amount by a zero mean Gaussian random variable with standard deviation std.

CONTOUR: Deformable Template Objects Contours are linked a linked list of snaxels with functions relevant for all points. Only extensions to GsnakeAPI's IMAGE class are described here, see the reference manual [31] for a complete description.

Compute Global Contour Data void computeCG(); void computeCov() double computeArea();

and stores the center of gravity of an image, computeCov the 2  2 covariance matrix and computeArea computes and returns the area cover by the contour. computeCG must be called before getCGRow or getCGCol are called the rst time. The respective values are accessed by computeCG computes

double getCgRow(void); double getCgCol(void); MATRIX *getCov(void); double getArea(void);

Interpolating a contour void *interpolate(int numpoints ,int degree = 2,int _equal=0); interpolate changes

the number of contour points to numpoints. The new point coordinates are computed using the B-Spline routines of Magic3.07 [15]. Relative point spacing is either equalized ( equal=1) or left unchanged ( equal=0). The contour is automatically reinitialized to the new number of coordinates.

68

Matching a Contour to an Image by ane Transform void initialFit(IMAGE* initimage);

matches the contour to the image (which should be a binary image containing one object) by ane transform. Contour position, size, orientation and aspect ratio are adapted, based on center of gravity and covariance of both image and contour and major axis transform of the covariances. initialFit

Getting the Shape Matrix MATRIX *getShapeMatrix(double strength=1.0, double smooth =0.0);

Arguments strength smooth

Fraction of intra-contour strength. Smoothing strength.

Returns Allocated shape matrix of successful, NULL otherwise.

Description computes and returns the contour shape matrix A for the speci ed parameters. Use strength 6= 1 only for GSNAKE3D objects.

getShapeMatrix

IMDATA: All Data for a 4D Image Set IMDATA contains

all image data for 4D, 3D or 2D+time data sets as one pointer to an IMAGE SEQUENCE for each location. Additionally all le header information is stored here. The data structure is as follows: class IMDATA{ protected: IMAGESEQUENCE **images;/* array with pointers to imagesequences*/ int numslices;

/* number of image slice positions */

int numtimes;

/* number of different times per slice */

double thickness;

/* slice thickness */

double *height;

/* 3rd coordinate, fixed for each contour*/

includes a modi ed version of the Papyrus header Papyrus2.h where all functions are declared external C. IMDATA

69

IMDATA Constructor and Destructor IMDATA() ~IMDATA()

The constructor initializes an image data set with 0 image sequences, images and height are set to NULL. No images or image sequences are initialized. The destructor frees the memory allocated to the image set data.

Resetting Image Set Data reset()

allows the reuse of an IMDATA object. All IMAGESEQUENCE objects of the set (and with it all single images) are reset and the series are reinitialized to 0 elements. reset

Initializing int init(int _numslices = 1, int _numtimes = 16);

allocates memory of type *IMAGESEQUENCE for the image sequence pointers. The IMAGES themselves are not initialized ! init

Initializing a 2D+time Series from Filename int init2D(char* filename);

initializes an IMADATA object for a single slice location using information from a Papyrus le. The IMAGES themselves are not initialized or loaded ! init2D

Reading Image Data from File int readpap2D(char *filename, int slice=0)

The image pixel data for a 2D+time series is read from a Papyrus le into the given slice position. To allow the use of multiple slices, the IMDATA object must be initialized before this function is called. If available in the Papyrus le, slice thickness and slice position is set. Reading of other data is implemented on an experimental basis but not used yet due to absence in all test les.

70

Putting data into image set void putNumSlices(int numslices); void putNumTimes(int numtimes); void putHeight(double _height,int slice); void putThickness(double _thickness); putImage(IMAGE * im, int slice,int time); putImageSequence(IMAGESEQUENCE *ims, int slice);

The put functions allow to directly put data into an IMDATA object. Note that they do not do any initialization.

Getting data from image set int getNumSlices(); int getNumTimes(); double getHeight(int slice); double getThickness(); IMAGE *getImage(int slice=0,int time=0); IMAGESEQUENCE *getImageSequence(int slice=0);

The get functions allow to directly get data from an IMDATA object. This way, the public functions of classes IMAGE and IMAGESEQUENCE can be used on data elements of IMDATA.

Computing a con dence Mask based on Variance IMAGE* getVarMask(IMAGESEQUENCE* vars_t, int area, double frac=0.25)

computes a con dence mask for the heart region based on the temporal variance of the image data. The binary variance images on which the mask is based are initialized and returned as well. For further processing, these images should then be masked, using IMAGE::mask(mask). getVarMask

GSNAKE3D: Active Contours for all Slice Levels implements a multi-slice contour model based on multiple, possibly related, GSNAKE objects. The structure is GSNAKE3D

class GSNAKE3D { friend class GSNAKE; protected: GSNAKE **snakes;

/* array of pointers to 2D gsnakes*/

71

int

numsnakes;

/* number of snakes/slices */

double avglen;

/* average intersnaxel distance */

double thickness;

/* slice thickness */

double *height;

/* 3rd coordinate, fixed for each contour */

CONTOURSTYLE style;

/* Style of contour */

double Eint;

/* Total internal energy of 3dsnake */

double Eext;

/* Total external energy of 3dsnake */

double Esnake;

/* Total 3dsnake energy */

double nu_inter;

/* strength of influence of neighbors */

MATRIX ***A;

/* Shape Matrix, composed of sub-matrices*/

GSNAKE3D Constructor and Destructor GSNAKE3D() ~GSNAKE3D()

The constructor initializes a multi-slice snake with 0 elements, and sets the snakes and shape-matrix to NULL. No single gsnakes are initialized. The destructor frees the memory allocated to the GSNAKE3D data. All single snakes and the shape matrix are reset.

Resetting a GSNAKE3D reset()

allows the reuse of an GSNAKE3D object. All GSNAKE and MATRIX objects of the snake are reset and the series is reinitialized to 0 elements. reset

Initializing int init(int _numsnakes = 1, CONTOURSTYLE _style = _2D); init allocates

memory of type *GSNAKE for the contour pointers and creates new GSNAKE objects. Pointers to the sub-matrices of the shape matrix and all other data are initialized as well. The GSNAKES and shape MATRIX themselves are not initialized.

Writing Contours to Single-Contour Files write2d(char* name, char* name2=".con", int lower =0);

interfaces to CONTOUR::write and so creates one contour le for each contour. The lenames are generated as `name >> `` %i'' >>name2 where the contour in the sequence, oset by lower. No leading zeros are used for the numbering. write2d

72

Writing Contours for Visualization write(char* name); write generates an output le in the NUAGES [41] le format.

This le can be processed using NUAGES for visualisation by Silicon Graphics OpenInventor.

Reading of a Single Contour readContour(char* filename, int position=0);

interfaces place in the stack. readcontour

CONTOUR::read

and places the read contour at the speci ed

Reading a Complete Set of Contours read (char *name1,char *name2=''.con'',int lower=1,int upper=17, CONTOURSTYLE _style=_2D); read interfaces to CONTOUR::read and loads upper-lower+1 contours from single les to

compose one GSNAKE3D object. The lenames are generated as `name >> `` %i'' >>name2 where the contour in the sequence, oset by lower. No leading zeros are used for the numbering. The GSNAKE3D object is automatically initialized.

Duplicating a GSNAKE3D GSNAKE3D *duplicate(GSNAKE3D *target=NULL); Duplicate creates

a duplicate of the referenced GSNAKE3D object. If target exists, only the contour data is copied, otherwise a target is initialized and all CONTOUR and PYRAMID data is copied.

Access GSNAKE3D Data short getNumSnaxel(void); double getHeight(int position); double getAvgLength(void); double getEint(void); double getEext(void); double getEsnake(void); GSNAKE* getSnake(int position); CONTOURSTYLE getStyle();

73

The get functions are used to access the data members of a GSNAKE3D object.

Putting Data into GSNAKE3D void putRawImg(IMAGE* rawim, int position=0); void putHeight(int position, double _height); void putStyle(CONTOURSTYLE _style); put

functions allow direct storage of data in a GSNAKE3D object.

Compute Average Snaxel Spacing double computeAvgLength();

This function needs to be called before Energy Computation as the average length is used for normalization.

Compute the Shape-Matrix void computeShape(double _nuinter = 0.0, double smooth_intra=0.0); computeShape computes the shape

matrix for the current Contour point positions. This means, with smooth intra = 0, the internal energy is reset to zero. If the style of a contour is set to 2D, no inter-contour relations are determined.

Compute one Non-Diagonal Submatrix of the Shape-Matrix MATRIX* getInterContourShape(GSNAKE* target, GSNAKE* source, double strength, int numneighbours); getInterContourShape computes one submatrix of

A describing inter-contour relation.

Only relations beween two contours are considered at one time.

Computing GSNAKE3D Energy double ESnake( short level, int verbose = 0 );

The overall energy of a GSNAKE3D object is computed and returned. Before computing the energy, a valid shape matrix has to be computed, using computeShape.

74

Compute Snaxel Energy double EInternal(SNAXEL *now, short slice, short position); double EInternal(SNAXEL *now); double ESnaxel( SNAXEL *sxptr, short slice, short position, BOOLEAN store = _FALSE, int verbose = 0 ); double ESnaxel( SNAXEL *sxptr, BOOLEAN store = _FALSE, int verbose = 0 );

These functions are used to compute the energy of a single snaxel. If position are known, the data should be provided for faster computation.

slice

and

Synopsis generate(int position=0, short level=1, int verbose=0, double low_pct=0.65, double high_pct=1.0, double low_val=0.0, double high_val=1.0, double imm_pow=3.0, EEXTTYPE = _EDGE, FILTTYPE = _GAUSS, EDGEOPERATOR=_PREWITT); generate(short level=1, int verbose=0,double low_pct=0.65, double high_pct=1.0, double low_val=0.0, double high_val=1.0, double im_pow=3.0, EEXTTYPE = _EDGE, FILTTYPE = _GAUSS, EDGEOPERATOR = _PREWITT);

generates Pyramid Images containing the edge map for either all levels of the GSNAKE3D object (second form) or one speci ed slice level ( rst form). EEXTTYPE is either EDGE (edge magnitude and direction), EDGEMAG (edge magnitude only) or INTENSITY (image intensity). The non-smoothing kernel UNIT should only be used for rst level pyramids. For explanation of the conditioning parameters, see [31]. generate

Minimization by Dynamic Programming minimize(int segmentSpacing = 5, int numSearchSegment = 5, int snaxelSpacing

= 5,

unsigned char blowup = 1, int verbose = 0, double theLambda = _DEFINED_LAMBDA, int showImg = 1, int img_Xoffset = 0, int img_Yoffset = 0,

75

int maxiterations = 20); minimize2d(int segmentSpacing = 5, int numSearchSegment = 5, int snaxelSpacing

= 5,

unsigned char blowup = 1, int verbose = 0, double theLambda = _DEFINED_LAMBDA, int showImg = 1, int img_Xoffset = 0, int img_Yoffset = 0 );

allocates an array of nite possible search locations and minimizes the snake energy within these locations by dynamic programming. minimize uses the multi-slice contour model while minimize2d interfaces to GSNAKE::minimize without use of this model. The syntax is identical to GSNAKE::minimize. minimize

Minimization by Random Movement of Snaxels minrand( float init_std =1.0,

/* initial standart deviation */

int maxiterations=1000, /* maximal number of iterations*/ int snaxelSpacing=5,

/* snaxel spacing */

unsigned char blowup=1, /* blow up ratio */ int verbose=5, double theLambda=_DEFINED_LAMBDA, int showImg=1,/* show image ? */ int img_Xoffset=0,/* offset from image */ int img_Yoffset=0); minrand_snaxel( float init_std =1.0,

/* initial standart deviation*/

int maxiterations=1000, /* maximal number of iterations*/ int snaxelSpacing=5,

/* snaxel spacing */

unsigned char blowup=1, /* blow up ratio */ int verbose=5, double theLambda=_DEFINED_LAMBDA, int showImg=1,/* show image ? */ int img_Xoffset=0,/* offset from image */ int img_Yoffset=0);

These methods use random deformation of the snakes along the contour normals for minimization. minrand deforms all snaxels at the same time. While convergance can be provenm it is very slow and thus not recommended. minrand snaxel only moves one snaxel

76

at one time, the decision whether the new position is kept or rejected is based on snaxel enery only. This is vey fast and has been found to eectively minimize energy.

77

Bibliography [1] H^opital Cantonal Universitaire de Geneve. Papyrus Image File Format, Version 2.3, 1991. [2] Ch. B. Higgins, H. Hricak, and C. A. Helms. Magnetic Resonance Imaging of the Body. Raven Press, 2. edition, 1992. [3] J. Kucharczyk and W.M. Chew. Basic principles. In Charles B. Higgins et al., editors, Magnetic Resonance Imaging of the Body, chapter 7, pages 127 { 135. Raven Press, 2. edition, 1992. [4] E. Moseley, Michael. Imaging techniques: Pulse sequences from spin-echo to diusion. In Charles B. Higgins et al., editors, Magnetic Resonance Imaging of the Body, chapter 9, pages 157 { 174. Raven Press, 2. edition, 1992. [5] D. Chien and R. R. Edelman. Fast magnetic resonance imaging. In Ch. B. Higgins et al., editors, Magnetic Resonance Imaging of the Body, chapter 10, pages 175 { 198. Raven Press, 2. edition, 1992. [6] H^opital Cantonal Universitaire de Geneve. Papyrus Toolkit User's Manual, Version 2.1 for Papyrus 2.3 speci cations, 1995. [7] R.J. Schalko. Pattern Recognition. John Wiley & Sons, 1992. [8] A. Rosenfeld. Image analysis. In M. P. Ekstrom, editor, Digital Image Processing Techniques, chapter 7, pages 257 { 287. Academic Press, 1983. [9] R.M. Haralick and L. G. Shapiro. Computer and Robot Vision, volume 1. Addison-Wesley, 1992. [10] D. H. Ballard. Generalizing the Hough Transform to detect arbitray shapes. Pattern Recognition, 13:111 { 122, 1981. [11] M. Kass, A. Witkin, and D. Terzopoulos. Snakes: Active contour models. In 1987 IEEE First International Conference on Computer Vision, pages 259 { 268, June 1987. [12] I.N Bronstein and K.A. Semendjajew. Taschenbuch der Mathematik. Harri Deutsch, 24. edition, 1989. 78

[13] W. Cheney and D. Kincaid. Numerical Mathematics and Computing. Brooks / Cole, 3rd. edition, 1994. [14] W.H. Press, S. A. Teukolsky, W. T. Vetterling, and B.P. Flannery. Numerical Recipies in C: The Art of Scienti c Computing. Cambridge University Press, 2. edition, 1992. [15] David Eberly. MAGIC: An Object oriented Library for Image Analysis, Version 3.07. University of North Carolina, Chapel Hill, 1996. [16] K. F. Lai. Deformable Contours: Modeling, Extraction, Detection and Classi cation. PhD thesis, University of Wisconsin-Madison, 1994. [17] K.P. Philip, E.L. Dove, D.D. McPherson, N.L. Gotteiner, W. Stanford, and K.B. Chandran. Automatic detection of myocardial contours in cine-computed tomographic images. IEEE Transactions on Medical Imageing, 13(2):241 { 253, June 1994. [18] K.P. Philip, E.L. Dove, D.D. McPherson, N.L. Gotteiner, W. Stanford, and K.B. Chandran. The fuzzy Hough Transform feature extraction in medical images. IEEE Transactions on Medical Imaging, 13(2):235 { 240, June 1994. [19] E.M. Brummer, R.J. Lewine, et al. Automatic detection of brain contours in MRI data sets. IEEE Transactions on Medical Imaging, 12(2):153 { 166, June 1993. [20] L. Cohen. On active contour models and balloons. CVGIP: Image Understanding, 53(2):211 { 218, March 1991. [21] P. Radeva and J. Serrat. Rubber snake: Implementation on signed distance potential. In Proceedings SWISS VISSION'93, pages 187 { 194, September 1993. [22] P. Radeva, J. Serrat, and E. Marti. A snake for model-based segmentation. ICCV, (8):816 { 821, 1995. [23] T. McInerney and D. Terzopoulos. Topologically adaptable snakes. ICCV, (8):840 { 845, 1995. [24] G. I. Chiou and J.-N. Hwang. A neural network-based stochastic active contour model (nss-snake) for contour nding of distinct features. IEEE Transactions on Image Processing, 4(10):1407 = 1416, October 1995. [25] F. Leymarie and M. Levine. Tracking deformable objects in the plane using an active contour model. IEEE Transactions on Pattern Recognition and Machine Intelligence, PAMI-15(6):617 { 634, June 1993. [26] V. Caselles, R. Kimmel, and G. Sapiro. Geodesic active contours. ICCV, (8):694 { 699, 1995. 79

[27] S. Kichenassamy, A. Kumar, P. Olver, A. Tannenbaum, and A. Yezzy. Gradient

ows and geometric active contour models. ICCV, (8):810 { 815, 1995. [28] S.C. Zhu, T.S. Lee, and A.L. Yuille. Region competition: Unifying snakes, region growing energy/bayes/mdl for multi-band image segmentation. ICCV, (8):416 { 423, 1995. [29] Kok F. Lai and Roland T. Chin. On regularization, formulation and initialization of the active contour models. In First Asian Conference on Computer Vision, pages 542{545, Osaka, 1993. [30] K. F. Lai and R. T. Chin. Deformable contours: Modelling and extraction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(11):1084 { 1090, November 1995. [31] K. F. Lai, S. Chan, C. W. Ngo, E. L. Ang, and K. W. Ong. GSNAKE API ver 1.0: Reference Manual. Information Technology Institute, Singapore and Nanyang Technological University, Singapore, September 1995. [32] Bjarne Sroustrup. The C++ Programming Language. Addison Wesley, 2. edition, 1991. [33] F. Leymarie and M. Levine. Simulating the grass re transform using an active contour model. IEEE Transactions on Pattern Recognition and Machine Intelligence, PAMI-14(1):56 { 75, June 1992. [34] A.J. Abrantes and J.S. Marques. A new algorithm for active contours. In Image Processing: Theory and Applications, Proceedings of IPTA '93, pages 91 { 94, June 1993. [35] A. A. Amini, T. E. Weymouth, and Jain R. C. Using dynamic programming for solving variational problems in machine vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-12(9):855 { 867, September 1990. [36] S. Z. Li. Markov Random Field Modeling in Computer Vision. Springer, 1995. [37] Norio Baba. A new approach for nding the global minimum of error function of neural networks. Neural Networks, 2:367 { 373, 1989. [38] R. J. Schalko. Arti cial Neural Networks. McGraw-Hill, 1996. [39] T. McInerney and D. Terzopoulos. A dynamic nite element surface model for segmentation and tracking in multidimensional medical images with application to cardiac 4D image analysis. Computerized Medical Imaging and Graphics, Special Issue on Cardiopulmonary Imaging, 19(1):69{83, January 1995. 80

[40] J. Park, D. Metaxas, A. A. Young, and L. Axel. Deformable models with parameter functions for cardiac motion analysis from tagged MRI data. IEEE Transactions on Medical Imaging, 15(3):290 { 298, June 1996. [41] Bernhard Geiger. Three-dimensional modelling of human organs and its application to diagnosis and surgical planning. PhD thesis, Ecole des Mines de Paris, France, 1993.

81