A strategy for automated multimodality image registration ...

A Strategy for Automated Multimodality Image Registration Incorporating Anatomical Knowledge and Imager Characteristics. Derek L.G. Hill, David J. Hawkes, Neff A. Harrison and Cliff E Ruff Division of Radiological Sciences, UMDS, Guy's Hospital, London, SE1 9RT, United Kingdom. This paper describes two methods for automating registration of 3D medical images acquired from different moralities. One uses dispersion in an intensity based feature space as a measure of mis-registration, together with knowledge of imager characteristics. The other uses anatomical knowledge of proximity and containment between associated structures to modify a distance transform for registration. Pre-registered training images are used to customise the algorithms for specific applications. Using stochastic optimisation techniques, we automatically registered MR and CT images of the head from three patients using one training set. In each case, the accuracy of registration was comparable to that obtained by point landmark registration. We present initial results for the modified distance transform in the same clinical application, and in a new application to combine angiographic data with the surface of the brain derived from MR.

1.0 Introduction Registration and combination of images from multiple modalities provides additional useful information by providing the means of accurately relating features in different images [ 1-11 ]. It is usually assumed that movement of structures between image acquisitions and therapy is small compared to the image resolution or the accuracy of therapy, so that the coordinate transformation between modalities can be limited to the 6 degrees of freedom of the rigid body transformation. This problem constitutes a special case of the more general matching problem (eg: matching images from different individuals, matching an individual's images with an atlas, matching images containing deformable structures such as the heart, gastro-intestinal tract and bladder). A satisfactory registration technique for routine clinical applications should ideally make use of anatomical information contained within the images, rather than external fiducial markers or frames which complicate the image acquisition protocols. The technique should require little or no changes to routine image acquisition protocols and the registration process should require minimal user interaction. Solutions to the rigid body registration problem normally consist of the following

stages:1. Calibration of the image acquisition systems so that voxel coordinates are accurately known. Image distortion correction is applied if necessary. 2. Identification of equivalent structures in the images to be registered. These might be points [5,7], lines [12], surfaces [1] or regions [19]. This stage is usually at least partially interactive, and often intensively so. 3. Use of an optimisation algorithm to determine the best registration transformation relating the equivalent structures. Except for the case of least

183

squares minimisation of the displacement between registered points, this is a non-linear problem, so iterative optimisation algorithms are used to find the best solution. 4. Rigid body transformation is assumed to be valid, and the transformation between the images is assumed to be the same as the transformation between the equivalent structures identified in the images. 5. In the absence of external markers, validation is performed by getting an expert observer to visually inspect the images once they have been transformed into a common coordinate frame. Several techniques have been proposed to solve the rigid body registration problem using anatomical features visible in the images. The "Head and Hat" algorithm [ 1] has been widely used for registering Magnetic Resonance (MR) and Positron Emission Tomography (PET) images. This technique relies on accurate delineation of corresponding surfaces. The technique they use for minimising the distance between the two surfaces results in many local minima in the search space. As a result the optimisation process frequently fails to deliver an adequate registration especially when surfaces are convoluted, poorly defined or incompletely overlapping, or where the surface chosen is distorted between acquisitions (eg: the skin of the head and neck). Woods [19, 20] observed that whilst a given tissue will have different intensities in different modalities, the variance of image intensity ratios between voxels of the same tissue type can be small when the images are correctly registered. In the neuro-MP-d PET registration application, in which his algorithm has been applied with considerable success [20], MR grey matter has a very different intensity from PET grey matter (similarly for white matter and CSF). However by calculating the variance of the ratios between voxel intensities for each intensity value of one modality (he uses MR) within the segmented brain, a goodness of fit criteria can be derived which gives the optimum answer when the images are correctly registered. Interactive point landmark based registration has found widespread acceptance and works well in the hands of experienced and careful users [5,13], but is time consuming and prone to user error. Other interactive protocols have been proposed [4] but are designed for very specific applications and lack general applicability. Recently the use of the distance transform has been reported for registration of multimodality images of the head [14, 15]. These papers report the use of the distance transform for registering equivalent surfaces. They do not take into account any anatomical knowledge on proximity of adjacent surfaces or the concept of containment. We have reported initial work in which anatomical knowledge was used to modify the distance transform [18] and this paper describes our recent progress on this technique. The requirement that equivalent structures are identified in the images to be registered can be a significant limitation. At best, only a small number of equivalent features are identifiable (typically approximately 10 landmarks, or a single surface). The vast majority of information in the images is being "thrown away" for the purposes of registration. The limitation of this is clearly illustrated by the manner in which validation is carried out. The expert observers who evaluate accuracy by visual inspection are not restricting their evaluation to the identified equivalent structures. They ensure that the appearance of the registered images is consistent with their knowledge of anatomy and their experience with the imaging modalities being registered. Thus, as well as checking that equivalent structures overlie each other, the observer might check depending on the region of the body being investigated - that the brain (from MR or

184

perhaps PET) lies inside the skull (from x-ray computed tomography - CT) but the scalp (from MR) lies outside the skull; bone marrow (from MR) lies inside cortical bone (from CT), no portion of the bladder (from PET) overlies the pelvic bone (from CT or MR); the heart (from PET) is surrounded by lungs (from CT); activation sites in functional studies (eg: PET) lie on grey matter rather than white matter. Two types of knowledge are being used in this validation: firstly anatomical knowledge about structures, their adjacency and containment (which structures lie inside / outside other structures); secondly knowledge about the physics of the imaging devices (imager characteristics) which defines the appearance of structures, in particular their intensities, in the different modalities. We propose that it is possible to use these types of knowledge for the purposes of image registration, and that this will provide more robust, more accurate, and more easily automatable registration algorithms. We have implemented two registration algorithms. The first is an adaptation of grey level correlation and the Woods algorithm [20] which exploits visual similarity between images. A feature space image is constructed from the intensity values of the two modalites to be registered. At registration, the feature space contains specific structure which disperses with mis-registration. Our algorithm registers the images by minimising a measure of this dispersion. The second algorithm is an adaptation to 3D hierarchical chamfer matching [15-17] which incorporates anatomical knowledge about structure adjacency and containment [18]. In this paper we describe these two methods, the manner in which knowledge is acquired from (registered) training data sets, how this knowledge is incorporated into the algorithm and how the validation of the algorithms has been carried out using image data which has been accurately registered using an existing algorithm, whose accuracy has been investigated [7, 27]. Finally, we discuss the way in which they could lead to automated image registration. The application used to demonstrate these algorithms is registration of MR, CT and angiographic images of the head.

2.0 Method The data representation used by the two algorithms is quite distinct, but training and optimisation components are very similar.

2.1 Registration by Minimisation of Feature Space Dispersion A feature space constructed from the intensities of registered images from different modalities can exhibit structure which disperses as the images are mis-registered. An appropriate measure of this dispersion can be used for registration of the images. We have investigated several measurements of features space dispersion. The most successful is the variance of intensity ratios (VIR) between voxels of selected tissues. This technique is an extension of the algorithm developed by Woods [19,20]. In our implementation, knowledge of imager characteristics is used to classify certain regions of the feature space image according to their tissue types. The dispersion with mis-registration of these regions in feature space is computed as the VIR and used in order to determine which tissues, as classified by intensity value, most tightly constrain the registration. A coordinate C in optimisation space is a six component vector consisting of the translation and rotation elements of the rigid body transformation. For each coordinate C, the intensity of each voxel in the reference image is compared with the range of val-

185

ues which were found from the training data to constrain the registration. If the voxel lies in one of the appropriate ranges, or bins, the coordinates of that voxel (in mm) is transformed using the registration transformation defined by C. If the calculated voxel is a valid one (ie: inside the image) the ratio of the reference intensity in one modality to the transformed voxel intensity in the other is calcdated, and the rolling variance and mean are calculated. A sum of the standard errors of the means for all the selected bins is calculated, and used as a measure of the goodness of fit (the objective function). 2.2 Modified Chamfer Matching. Structures with known anatomical relationships to each other (eg: adjacency, being inside, outside or 5ram lateral to etc) are delineated from the modalities to be registered. For the image from the reference modality (usually the one of higher resolution), the anatomical structures are binarised such that the inside of the structure has the voxel value 1, the outside has a voxel value 0, and voxels just inside the surface (surface voxels) can optionally have a value greater than 1 which constitutes a label. This image is converted into a 3D image data set with three attributes at each voxeh The first value (16 bit signed integer) is defined as "infinity" outside the structure and O inside the structure, the second attribute (16 bits) is 0 outside the structure and "infinity" inside the structure, and the third attribute (8 bits) is zero both inside and outside the structure, but contains a surface label value for each surface voxel. A 3D chamfer kernel is passed over the image (see Appendix A). The first attribute thus becomes the distance from the outside of the structure's surface and the second attribute becomes the distance from the inside of the surface. The third attribute contains the influence zone which defines the surface voxel (or more likely patch of surface voxels) which the current voxel is nearest to. The first two attributes are then combined such that the image has two attributes per voxel: 9 a distance from the surface (positive inside and negative outside). 9 an influence zone. This algorithm takes approximately 1 minute to run for a typical 256x256x32 image, on a Sun SparcStation 10. The associated (but not necessarily equivalent) structure in the image to be registered is represented as a non-connected list of points. These points may represent a surface, or points along the centre-line of a linear structure such as a blood vessel, or true "point like" structureS. The distance transform is modified using anatomical knowledge about the relationship between the structures being registered. Knowledge has been acquired from two sources. The first is text-book knowledge about the expected relationships between anatomical structures. The second is derived from registered training datasets. The training data is used to determine the distribution of points on a structure in one modality with respect to the surface of an associated (but not necessarily equivalent) structure in a second modality. The influence zones enable different surface regions to be associated with different point distributions. Inner Table of Skull and Outer Surface of Brain. The inner surface of the skull and

the outer surface of the brain are two associated but non-equivalent surfaces. Knowledge about the expected distance of these structures from each other is used to modify the distance transform. The folding of the cortex, and the presence of cerebro-spinal fluid, vascular and membranous structures between the brain and the skull leads to

186

points on the brain surface lying at a distribution of distances inside the skull. Registered MR and CT images from one patient were used to measure this distribution of distances. In our initial work, this distribution was used to modify the distance transform to produce a cost image as follows: the position of the peak of the distance distribution was subtracted from the distance image and the gradient of the external distances, representing overlap of brain on bone, was doubled to reflect the asymmetry of this distribution.

Surface of Brain and Angiographic Data. The same representation of knowledge can be applied when combining 3D centre-line representations of blood vessels derived from bi-plane digital angiography with MR images. In this case, the anatomical knowledge is that the larger arteries (>O.2mm in diameter) are constrained by the pia mater membrane to lie on the surface of the brain. Where the brain surface folds, the blood vessels follow the resulting fissures. This algorithm was tested using vessel centreline coordinates derived from MR images, so that the registration transformation was known a priori. 2.3 Training. These two algorithms, minimisation of feature space dispersion as measured by VIR and modified chamfer matching provide a means of measuring the "goodness of registration". Before using these algorithms for real registration tasks, it is necessary to determine which intensity ranges lead to the formulation of the best objective function computed from the VIR, and which anatomical structures to use, and how to represent numerically the anatomical knowledge in the objective function for the modified chamfer matching. We have devised techniques which make use of accurately registered training images. We have carried out some initial training for the applications outlined above. For the purposes of training we used an example registered MR and CT dataset of the head. The training dataset was registered using 12 points identified interactively. Our computer simulation of point registration accuracy would indicate that the registration error between the two datasets varies between 1 and 2mm throughout the imaged volume [27].

Training in Minimisation of Feature Space Dispersion. The intensity ranges appropriate for minimisation of VIR were initially selected from the feature space images obtained by plotting MR intensity against CT intensity. Intensity ranges which exhibited strong clustering in feature space for correctly registered images, but were highly dispersed for mis-registered images were chosen. These intensity ranges were subdivided into 'bins'. The utility of each bin as a metric of the registration accuracy was investigated by calculating the standard error on the mean of CT/MR intensity ratios (the objective function) for known mis-registrations separately for each of the six degrees of freedom. Training the Modified Chamfer Matching for MR and CT Registration. The inner table of the skull was interactively segmented using ANALYZE "autotrace" (an interactive intensity threshold based region growing tool) [21] from a 28 slice CT dataset of the brain (including the skull base). A 3D distance transform was generated and modified as described above. Approximately 300 points distributed over the brain surface were manually located from the registered MR dataset of the same patient. The effect of mis-registration was assessed by applying known registration errors separately for

187

each of the 6 degrees of freedom. The use of the distance transform alone and the modification incorporating anatomical knowledge, as described above, were compared. Training the Modified Chamfer Matching for MR and Angiographic Data. The images used for this evaluation were derived from a fixed cadaver brain, with vessels perfused with barium gel, and imaged with MR. The barium gel gave the perfused vessels high contrast. The vessels were tracked in 3D from the MR images so that the correct registration transformation was known a priori. The resulting representation of the vessels, however, is identical to the 3D centreline vessel segment representation generated by our software "SARA" (a System for Angiographic Reconstruction and Analysis) from bi-plane digital X-ray angiograms [22]. 2.40ptimisation. Both algorithms presented in this paper provide a means, given an estimate of the registration transformation, of evaluating an objective function which provides an indication of how good that registration transformation is. As has been widely reported [1,19] both correlation and surface fitting would be expected to be multimodal functions, ie: there are many local minima in addition to the correct solution (the global minimum) in the six dimensional optimisation space. We use the combination of multiple resolutions (which blur out many of the local minima) and stochastic optimisation algorithms (which tend to be more robust at finding the global minimum than conventional down hill methods) to overcome this problem. The optimisation proceeds first at a low resolution with a large search space, then at progressively higher resolutions and smaller search spaces. At each resolution, a genetic algorithm [23] or multi-start stochastic simplex [24] is used to determine the global minimum. These algorithms are described in appendices B and C.

3.0 Results. 3.1 Training Data for Minimisation of VIR for MR and CT Images of the Head. Figure 1 shows the feature space images for MR and CT data of the head. CT intensity is plotted along the horizontal axis, and the MR intensity along the vertical axis. Note in particular the strong association between MR and CT voxel intensities in the region corresponding to bone. This structure, which is visible in the left image where the datasets are registered, is seen to disperse with mis-registration caused by translation in the x direction (laterally) of 5mm and 16 mm respectively.

Fig. 1. Feature space image constructed by plotting MR voxel intensity (vertical axis) against CT voxel intensity (horizontal axis) for images which are registered (left), and mis-registered by 5mm or 16 mm. Appropriate CT bins corresponding to air, brain and bone were selected by inspection of these images. The use of these bins was examined on a pre-registered training

188

data set. Figure 2 shows the change in the objective function computed from the VIR as a function of translation in the x (lateral) and z (cranio-caudal) directions. The minimum cost for translation in the x direction for the selected bone and brain bins corresponds, to the solution obtained using our point-based algorithm. The poor performance of the brain and bone bins in the z translation direction is due to the limited volume of the head imaged in this patient. The upper portion of the skull is absent from the CT images, so there is nothing to prevent the optimisation algorithm in effect shifting the brain out of the skull at no increase in the VIR. The air sinuses of the skull base, however, are sufficient to provide a constraint on the z translation provided the starting estimates are good, so the mean across all the bins is a minimum when the registration corresponds to the point based solution. Figure 3 demonstrates that the mean i

w

-

i

i

|

-

= moan

:

: Done

2'0

'

Ir. = -

= prain = Done

A

v

'

'

2'0

. . . .

'

6

'

x translation (ram) z translation (ram) Fig. 2. Change in variance of intensity ratio with mis-registration translations in the x and z direction, for intensity bins corresponding to air, brain, bone, and to their mean.

40

of the air, bone and brain bins constrains the registration problem for all 6 degrees of freedom.

3.2 Results for Minimisation of Feature Space Dispersion. MR and CT datasets from three patients were registered automatically using this algorithm without any pre-segmentation. The first comprised the un-registered images from the same patient dataset as used for training. The remaining two were registered using the results of the training on the first patient's datasets. These datasets were initially out of registration by up to 15mm translation and 18 degrees rotation. Figure 4 shows example slices from the registered and combined images. The images were combined by thresholding the bone from the CT images and overlaying that on the soft tissue from the registered and transformed MR [7]. The registered images were all visually indistinguishable from the solutions found using our point landmark based algorithm, which we estimate to have an accuracy of within 2mm [27].

189

1.1

o~ 0.9

=----, v translate

~ L.II~ I /#1 I

,,---* ~,~r~ii,~l~te x x rotate =~'--~_~ rotate rotate

'7

0"-4(

"-~

~ ' w t , L%/-/i I

I

-20

=

I

=

0

I

_

LJ~

=

20

40

mis-registration (mm or deg.)

Fig. 3. Graph of feature space dispersion (measured as VIR for the mean of air, soft tissue and bone bins) for mis-registration by translation or rotation.

Fig. 4. Combined images with soft tissue (s) from MR and bone (b) from CT for patients with a glioma (x), anterior fossa meningioma (y) and transtemporal meningioma (z). 3.3 Training data for chamfer matching of MR and CT images of the head Figure 5 shows the distribution of distances of points on the brain surface (from MR) from the inner table of the skull (from the registered CT). In this example, the minimum cost was set at a distance from the skull corresponding to the peak of the distance distribution (at -0.7mm), and the asymmetrical shape of the distance distribution was incorporated into the cost image simply by doubling the distance gradient in the direction towards the bone of the skull. Both the conventional distance transform and the cost image were evaluated using registered training data. Figure 6 demonstrates that using the conventional distance image, the minimum cost solutions for translation in the z direction (cranio-caudal) and the rotation about the x axis (a line passing through both auditory meati) are different from those found using the point landmark based registration. Figure 7 shows the result obtained using the modified chamfer matching. In this case~ the minimum cost

190 350 300 250

v

Inside brain

2O0

"6 E

-r r

inside skull bone.

150 100 5O 0

-

-14

-16

-12

-10 Distance

-8

-6

-4

-2

0

2

f r o m i n n e r s u r f a c e of skull / m m

Fig. 5. Distribution of distances of points on the brain surface with respect to the inner surface of the skull.measured on registered MR and CT images for one patient. !

I

60

--.t

T 62(

: x translation

y translation : translation 9 x rotation ~ y rotation T z rotation ,

i

-10

,

~

0 translation / rotation

,

i

10

,

20

Fig. 6. Conventional chamfer matching of MR and CT images. Change in cost (square of chamfer distance) resulting from translation or rotations in six degrees of freedom. solution is within a voxel of the solution found by the point landmark based algorithm, for all degrees of freedom.

3.4 Evaluation of MR and CT Chamfer Matching Using Patient Data. For two patient studies evaluated so far (one of which was the same study as that used to train the algorithm), both the genetic algorithm and stochastic simplex optimisation algorithms converged to a solution within 2mm translation and 2 degrees rotation of the solution found using point landmark registration.

191

120

:

"

9----- y translation : : 2 translation 9

~ ~ = ~,l

"%~t~

- x rotation

,,-.---~ y rot.ap.on .- - ~. rotation ,

1220

,..///f--* k-YA

J/~/

~

/J//

I

~ F : I

-10

i

I

0 translation / rotation

=

I

i

10

20

Fig. % Chamfer matching of M R and CT images incorporating anatomical knowledge.

Change in cost resulting from translation or rotations in six degrees of freedom. 3.5 Training Data for Chamfer Matching of MR and Vessels in the Head. On the left of figure 8 is an example slice from a coronal MR dataset from a cadaver brain. Adjacent to it are the distance transform of the same coronal slice, and the distance transform modified to incorporate the anatomical knowledge about blood vessels running along the surface of the brain.

Fig. 8. An example slice from a coronal MR dataset of a cadaver brain (left), its distance transform (centre), and modified distance transform (right). Registration using the modified distance transform in figure 8 was evaluated using blood vessel data derived from the same MR images. Two vessel segments were used; a branch of the middle cerebral artery passing through the sylvian fissure, and the pericallosal branch of the anterior cerebral artery. Figure 9 demonstrates that the best registration transformation for these two vessels lies within a 2ram translation and a 4 degrees rotation of the known solution.

192 1000

C

H --'i =:

1

y : z -'- x i y = z |

-20

trans!ate translate rotate rotate

~@l~jt--./" - ~ :It" 7"

rotate I

-10

i

I

i

I

+

0 10 translation (mm) / rotation (deg.)

20

Fig. 9. Modified chamfer matching of brain surface and vessels. The minimum cost solution for all degrees of freedom is within 2mm or 4 degrees of the known solution.

4.0 Conclusions and Discussion Our experience with image registration using an interactive point landmark algorithm has demonstrated that there is significant clinical demand for accurately registered images, especially registered MR and CT images for planning neuro and cranial base surgery [ 13]. A limitation of this and other existing registration algorithms is that they require considerable user interaction, and the results can be very dependant on the skill and motivation of the user. The algorithms presented in this paper provide a means of automating the registration process for many such applications. Anatomical knowledge and knowledge of imager characteristics, combined with the use of registered training images provide a means of customising the algorithms for particular clinical applications. We have shown how this can be done with MR, CT and angiographic images. We are in the process of applying the algorithm to other registration applications (eg: MP-dCT/PET registration in the pelvis and neck, and MP-dPET registration in the brain and heart). The different registration algorithms place different demands on pre-segmentation of the image data. For the point landmark based algorithm, accurate identification of between 8 and 12 equivalent 3D points in each set of images to be registered is essential, and this process is very difficult to automate. The uncertainty in point location can be represented by a weighting in the least square algorithm, but this does not provide much flexibility. For surface fitting by chamfer matching, well defined surface structures need to be identified in MR and CT images. Although we have done this segmentation interactively for the test data presented in this paper, automatic segmentation algorithms have been proposed for delineating these structures [eg 25]. The segmentation task is simpler than that required for conventional chamfer matching because it is not necessary to delineate equivalent structures in the modalities being registered, and because the uncertainty in the surface delineation can be coded into the modified distance transforms. The work presented here has used only a single influence zone. The incorporation of multiple influence zones into the distance transform would allow for the use of multiple associated features with different point or surface distributions.

193

The segmentation requirements for minimisation of feature space dispersion will depend on the types of images being registered, and the measure of dispersion which is being used. For CT and MR registration using variance of intensity ratios as the measure of feature space dispersion, the algorithm only requires some tissue classification by intensity ranges in the reference modality. For CT data, determining the intensity ranges once training has been performed is straightforward, and can be performed automatically, because the image intensity is closely related to the tissue type. For other modalities, some processing of the image histogram may provide the same information. The use of the training images and robust optimisation algorithms removes the requirement that some structures be manually delineated and excluded from the processing. Our algorithm only measures dispersion in one dimension of the feature space. One consequence of this is that translation in the cranio-caudal direction is poorly constrained in images which do not include the upper portion of the skull: the MR brain can slide out of the CT skull akin to "lifting an egg from an egg cup" without increasing the calculated variance of intensity ratios. Two dimensional measures of feature space dispersion might overcome this limitation by assigning high cost to solutions where MR bone (signal void) overlies CT brain as well as to solutions where CT bone overlies MR brain. This limitation does not appear to be an obstacle to automatic registration of MR and CT images where the majority of the brain cavity is imaged. We have successfully registered 3 patients of this type. The registration of MR images and 3D reconstructions of vascular networks in the brain remains an unsolved problem both because of the difficulty in reconstructing the complete cerebral circulation from bi-plane angiograms, and because of the absence of equivalent structures in the two modalities that can be used for registration. We have demonstrated that, given a brain surface, and a 3D reconstruction of a small number of major cerebral artery segments, the modified chamfer matching algorithm can determine the registration transformation relating these datasets. The blood vessel segments used for this work were derived from MR images so that the correct transformation was known a priori. It is possible to reconstruct many of the major vessels of the cerebral circulation by manual tracking of the vessels in bi-plane angiograms provided the x-ray projection geometry is known [22], but it is not currently feasible to reconstruct a significant proportion of the cerebral circulation [26]. If, however, a small number of vessel segments were manually reconstructed from the angiograms, these could be used for registration of those vessels to MR coordinates. For all non-linear optimisation problems, an algorithm which successfully avoids local minima in optimisation space is essential if the results are to be reliable. The genetic algorithm appears to be a rapid and robust method for finding an approximately correct solution (somewhere near the bottom of the global minimum, as opposed to a distant local minimum), provided that the population size is sufficiently large. The genetic algorithm, however, appears to be computationally expensive as a means of finding the very bottom of the global minimum. The stochastic simplex appears to perform better in this respect if the starting estimates are sufficiently good, so we are implementing a hybrid algorithm which uses the genetic algorithm for avoiding local minima, but uses the stochastic simplex to obtain the final solution. This may speed up the time taken to perform the registration compared with using the genetic algorithm alone. An additional way of speeding up the optimisation process would be to provide good starting estimates of the required registration transformation (eg: by means of interactive identification of a small number of equivalent points in the modalities being registered) thus constraining the search space. As currently imple-

194

mented, and depending on the precise parameters used, both algorithms typically took about one hour to run on a Sun SparcStation 10. The work presented here demonstrates that the use of anatomical knowledge, and knowledge of imager characteristics, combined with the use of accurately registered images for training purposes, makes automatic image registration an achievable goal. For one application (MR and CT registration in the head) we have achieved automatic registration on three patients.

5.0 Acknowledgements. We are grateful to the UK Science and Engineering Research Council (Project SMIRC) and The Leverhulme Trust for funding much of this work. We are grateful to Roger Woods from the PET centre at UCLA, the staff at the MRC Cyclotron Unit at Hammersmith Hospital, London, and Prof. Chris Taylor and his colleagues at the University of Manchester for useful discussions about their related work.

Appendix A. Distance Transforms Using a 3D Chamfer Filter. A distance transform converts a binary image into an image in which each voxel is labelled with its distance from the nearest voxel with the value 1. Images are discrete, and therefore it is not straightforward to calculate euclidean distances. The chamfer filter is frequently used to calculate distances which are a close approximation to the Euclidean distance. The chamfer filter involves two passes over the image; one in the forward direction, and one in the reverse direction. In three dimensions, where voxels are cubic, the kernel is a 3 x 3 x 3 cube: Forward: C

BC

B A B

B A B C

A

BC

0

B*

*

*

*

*

*

*

*

*

*

*

*

Backward: 9

*

*

*

*

9

*

*

*

0A

B

C B C B A B

9

*

*

B A B

C B C

For Euclidean distances, A, B and C should be in the ratio 1:sqrt(2): sqrt(3). Integers with approximately this ratio are used. This can be modified to work with non-cubic voxels by replacing the elements A, B and C with Ax,Ay,A z, Bxy,Bxz,Byz and Cxyz. For an image where Dx=Dy and Dz=R*Dx, Ax=Ay=l, Az=R, Bxy=Sqrt(2.0), Bxz=Byz=sqrt(l+R*R), Cxyz=sqrt(2.0+R*R). These need to be approximated by integers. To reduce rounding errors, the we use 16 as our minimum integer. The distance value is determined by replacing the current voxel attribute with the minimum sum of the voxels under the kernel and the kernel mask values [17].

Appendix B. The Genetic Algorithm. The genetic algorithm is a stochastic optimisation algorithm which represents possible solutions to the problem it is solving as individuals whose position in the n dimensional search space is represented by a bit string normally called a chromosome. The chromosome comprises a concatenation of genes, each of which is a (normally binary) representation of one of the n search space degrees of freedom parameters. The search space is defined for each degree of freedom, and an initial estimate of the solution is provided. The starting population of possible solutions includes the starting estimate and other random individuals generated within the search space. Some objective function specified for the optimisation problem is used to assign each member of the population with a fitness value. The optimum solution is found by reproduction of the individuals in the population. The probability of each member of the population mating with another

195

member of the population is proportional to that individual's fitness. Reproduction can be asexual (a new individual is generated with the same genetic material as one of its parents) or sexual (in which ease the offspring's chromosome is derived from the genetic material of both its parents). The probability of sexual reproduction is normally greater than 50%. In either ease there is a small probability that a cosmic ray will cause a mutation, thus introducing new genetic material into the population. Some criteria based either on number of generations, or on the fitness of the population can then be used to stop the algorithm. In our implementation, we are searching six degrees of freedom each represented by 6 bits. This algorithm is applied to the data at multiple scales. The solution from the lowest resolution scale is used as the starting estimate for the next resolution.

Appendix C. The Stochastic Simplex Optimisation Algorithm. A simplex in n dimensional space is a shape with n+l vertices, and is thus the simplest shape that requires that many dimensions to exist. For the purposes of n dimensional optimisation, the simplex can be thought of as a creature with n+l feet which tries to crawl into the global minimum. One foot is the starting estimate, and the others are randomly generated within the search space. The objective function is evaluated for each foot, and the creature decides to move its worst foot to a better position. Because the simplex is the simplest creature that can exist in n dimensions, all of the remaining feet lie in a n-1 dimensional plane. The creature first tries to reflect its worst foot across this plane. If the new position it lands on is better than the one it came from, it tries going further in that direction. Alternatively, it tries to place its foot in an intermediate position. If none of these moves gives it a better solution, it tries moving all of its feet closer to the best one. It then once again selects its worst foot, and has another attempt. The simplex stops when all its feet are within a predefined range of each other. A single simplex is likely to find the nearest local minimum in the search space We modify the traditional simplex by using a randomly generated population of simplexes to start with, and restarting the best simplexes together with some new random ones around the previous solution. Provided the search space is reasonably smooth, a population size of 10, and five re-starts with 30% carry over of previous solutions was used in this work.

6.0 References. 1. C.A. Pelizzari, G.T.Y. Chen, D.R. Spelbring, R.R. Weichselbaum, C-T. Chen: Accurate three dimensional registration of CT, PET and/or MR images of the brain. Journal of Computer Assisted Tomography 13, 20-26 (1989) 2. L.R. Schad, R. Boesecke, W. Schlegel, et al: Three dimensional image correlation of CT, MR and PET studies in radiotherapy planning of brain tumours. Journal of Computer Assisted Tomography 11,948-954 (1987) 3. D.J. Hawkes, D.L.G. Hill, E.D. Lehmann, G.P. Robinson, M.N. Maisey, A.C.F. Colchester: Preliminary work on the interpretation of SPECT images with the aid of registered MRI images and an MR derived neuro-anatomical atlas. In: K.H. Hoehne, H. Fuchs, S.M. Pizer (eds.): 3D Imaging in Medicine: Algorithms, Systems, Applications. Heidelburg: SpringerVerlag 1990, pp. 241-252 4. U. Pietrzyk, K. Herholz, W-D. Heiss: Three dimensional alignment of functional and morphological tomograms. Journal of Computer Assisted Tomography 14, 51-59 (1990) 5. A.C. Evans, S. Marrett, J. Torrescorzo, S. Ku, L. Collins: MRI-PET Correlation in Three Dimensions Using a Volume-of-Interest (VOI) Arias. Journal of Cerebral Blood Flow and Metabolism 11, A69-A78 (1991) 6. C.J. Henri, D.L. Collins, T.M. Peters: Multimodality image integration for stereotactic surgical planning. Medical Physics 18, 167-177 (1991) 7. D.L.G. Hill, D.J. Hawkes, J.E. Crossman, et al: Registration of MR and CT images for skull base surgery using point-like anatomical features. British Journal of Radiology 64, 10301035 (1991) 8. J.S. Fleming: A technique for using CT images in attenuation correction and quantification in SPECT. Nuclear Medicine Communications 10, 83-97 (1989)

196

9. I.L. Kaplan, L.C. Swayne: Composite SPECT-CT images: technique and potential applications in chest and abdominal imaging. American Journal of Roentgenology 152, 865-866 (1989) 10. D.J. Hawkes, L. Robinson, J.E. Crossman, et al: Registration and Display of the Combined Bone Scan and Radiograph in the Diagnosis and Management of Wrist Injuries. European Journal of Nuclear Medicine 18, 752-756 (1991) 11. T.L. Faber, R.W. McColl, R.M. Opperman, J.R. Corbett, R.M. Peshock: Spatial and Temporal Registration of Cardiac SPECT and MR Images: Methods and Evaluation. Radiology 179, 857-861 (1991) 12. A. Gueziec, N. Ayache: Smoothing and matching of 3-D space curves. SPIE 1808, 259-273 (1992) 13. D.L.G. Hill, S.E.M. Green, J.E. Crossman, et al: Visualisation of multi-modal images for the planning of skull base surgery. SHE 1808, 564-573 (1992) 14. EA. Van den Elsen, J.B. Maintz, E.D. Pol, M.A. Viergever: Image fusion using geometrical features. SPIE 1808, 172-186 (1992) 15. H. Jiang, R.A. Robb, K.S. Holton: New approach to 3-D registration of multimodality medical images by surface matching. SPIE 1808, 196-213 (1992) 16. G. Borgefors: Hierarchial Chamfer Matching: A Parametric Edge Matching Algorithm. IEEE transactions in pattern analysis and machine intelligence 10, 849 (1988) 17. G. Borgefors: Distance Transformations in Arbitrary Dimensions. Computer Vision, Graphics and Image Processing 27, 321-345 (1984) 18. D.L.G. Hill, D.J. Hawkes: The use of anatomical knowledge to register 3D blood vessel data derived from DSA with MR images. Proc SPIE 1445, 348-357 (1991) 19. R.E Woods, S.R. Cherry, J.C. Mazziotta: A rapid automated algorithm for accurately aligning and reslicing PET images. Journal of Computer Assisted Tomography (1992) 20. R.E Woods: Personal Communication. (1992) 21. R.A. Robb, M.C. Stacey, C. McEwan: A networked workstation approach to multi-dimensional biomedical image analysis. In: K. De Graaf, M.A. Viergever (eds.): Information processing in medical imaging. New York: Plenum 1987, pp. 349-374 22. D.J. Hawkes, A.C.E Colchester, C.R. Mol: The accurate 3D reconstruction of the geometric configuration of vascular trees from X-ray recordings. In: R. Guzzardi (ed.): Physics and engineering of medical imaging. Holland: Martinus Nijhoff 1987, pp. 250-256 23. D.E. Goldberg: Genetic algorithms in search optimisation and machine learning. Mass. USA:Addison Wesley 1989 24. W.H. Press, B.E Flannery, S.A. Teukolsky, W.T. Verrerling: Numerical Recipes in C - The art of scientific computing. Cambridge:Cambridge University Press 1988 25. G. Gerig, J. Martin, R. Kikinis, O. Kuebler, M. Shenton, EA. Jolesz: Automating segmentation of dual-echo MR head data. In: A.C.E Colchester, D.J. Hawkes (eds.): Information Processing in Medical Imaging. Heidelberg: Springer-Verlag 1991, pp. 175-187 26. C. Smets, G. Verbeeck, P. Suetens, A. Oosterlinck: A knowledge based system for the delineation of blood vessels on subtraction angiograms. Patt Recog Lett 8, 113-121 (1988) 27. D.L.G. Hill, D.J. Hawkes, Z. Hussain, S.E.M. Green, C.E Ruff, G.P. Robinson: Accurate combination of CT and MR data of the head: Validation and applications in surgical and therapy planning. In: C. Roux, G.T. Herman, R. Collerec (eds.): 3D advanced image processing in medicine - Rennes '92. : IEEE-EMBS 1992, pp. 79-84