127
Hybrid rigid and non-rigid registration algorithm for alignment of intra-subject thoracic and abdominal images Z T Lu, Q J Feng, W Yang and W F Chen* School of Biomedical Engineering, Southern Medical University, Guangzhou 510515, China
Abstract: A hybrid rigid and non-rigid registration algorithm has been presented to register thoracic and abdominal CT images of the same subject scanned at different times. The bony structures are first segmented from two different time CT images, respectively. Then, the segmented bony structures in the two respective images are registered based on their boundary points using a soft correspondence matching algorithm, with a rigid transformation constraint on each bony structure. With estimated correspondences in bony structures, the dense deformations in the entire images are interpolated by a thin plate spline (TPS) interpolation technique. To improve the alignment of soft tissues in the images as well, a normalised mutual information based B-spline registration algorithm is used to iteratively refine the registration of soft tissues, and at the same time keep the rigid transformation for each bony structure. This registration refinement procedure is repeated until the algorithm converges. The proposed hybrid registration algorithm has been applied to the clinical data with very encouraging results as measured by two clinical radiologists. Keywords:
1
hybrid image registration, rigid registration, non-rigid registration
INTRODUCTION
Images have been widely used for guiding therapy, and for surgical, radio-surgical and radiotherapeutic planning.1–3 In these applications, the precise registration between preoperative and intra-operative scans is important for increasing the treatment efficacy as well as for minimising damages to the normal structures. Recently, with the development of thoracic and abdominal imaging, there is an increased need of the thoracic and abdominal image registration methods for the comparison of images of the same subject scanned at different times, i.e. before and after pharmacological treatment or surgical intervention. However, the thoracic and abdominal image registration is a very difficult problem due to The MS was accepted for publication on 3 August 2010. * Corresponding author: Wufan Chen, School of Biomedical Engineering, Southern Medical University, Guangzhou 510515, China; email:
[email protected]
IMAG 31 # RPS 2011
DOI: 10.1179/136821910X12863757400169
the rigid motions of bony structures, along with the complicated non-rigid motions of heart, thorax and other soft tissues.4–6 The deformable registration techniques incorporating rigid structures have been recently investigated. Little et al. incorporated independent rigid objects in a modified thin plate spline (TPS) non-rigid registration.7 Tanner et al. proposed a solution to locally couple the control points of a B-spline free-form deformation field to make the transformations rigid within the specified image region of interest.8 On the other hand, Rohlfing et al. proposed a penalty term to impose local tissue incompressibility and volume preservation over the image without need for segmentation. This is achieved by constraining the local Jacobian determinant to be close to unity everywhere in the image.9 We propose a particular rigid and non-rigid registration for alignment of thoracic and abdominal images at different times. Our method first The Imaging Science Journal Vol 59
128
Z T LU ET AL.
segments bony structures from the neighbouring soft tissues, and restrains them to deform rigidly during registration. Then, the corresponding bony structures in the two images are registered based on their boundary points by using a soft correspondence matching algorithm.10 Finally, the registration for the soft tissues is iteratively refined by means of B-spline free form deformations (FFD).8,9 This FFD-based registration technique is modified to allow rigid transformations for bony structures, while non-rigid elastic transformations for soft tissues, thus achieving overall accurate registration for the entire images. The remainder of this paper is organised as follows. In Section 2, we describe our complete method for segmentation of bony structures, and for rigid and non-rigid registration of thoracic and abdominal images. In Section 3, experimental results are reported. Finally, Section 4 closes the paper with some conclusions and directions for future research.
2
METHODS
Given two CT images V1 and V2 scanned at two different times, the goal of our image registration is to find the optimal transformation that maps the information contained in one image into its anatomical correspondence in the other. In our application, the transformations to be estimated in the entire image domain include not only non-rigid transformations, but also rigid transformations corresponding to the rigid deformation of bony structures in the images. To achieve it, we first segment bony structures from two CT images V1 and V2, separately. Then, the boundaries of these segmented bony structures in the two images are registered using a soft correspondence matching algorithm.10 The transformation for each bony structure is constrained to be rigid transformation after soft correspondence detection. Afterwards, the correspondences in the other regions of two images are elastically interpolated by a TPS interpolation technique.7 To further align the regions including soft tissues, the deformations are iteratively refined in the entire images by using a normalised mutual information based B-spline FFD algorithm. In each iterative refinement step, the deformation in every bony structure is kept as its respective rigid transformation. By using this deformation strategy, The Imaging Science Journal Vol 59
we are able to estimate rigid transformations for bony structures and non-rigid transformations for soft tissues, thereby achieving overall accurate registration for entire images. 2.1
Local rigid transform
Our registration method requires segmentation of bony structures from both CT images under registration. We perform the required segmentation by employing an automatic segmentation technique11 that is able to adaptively vary the threshold according to the mean and standard deviation of intensities in the local neighbourhood. Fourier Descriptors was used as a metric to characterise the bone contour shapes. It was suspected that the shapes of neighbouring contours would vary gradually, and that this coherence could be used to validate the segmentation. To remove noise in the segmentations, we also use morphological erosion, dilation, and three-dimensional region growing techniques to extraction bony structures. The boundaries of these segmented bony structures are used to estimate the initial deformations between two images under registration by using a soft correspondence matching algorithm.10 The estimated transformation for each separate bony structure is required to be rigid after soft correspondence detection; thus, for N bony structures, there are N separate rigid transformations estimated. The dense correspondences, or dense deformations in the entire images, are interpolated by using a TPS interpolation technique,7 as briefly described next. Let {(xi,yi,zi)}i50, 1, …, n denote n landmarks in the moving image which will be transformed to new sites {(x’i ,y’i ,z’i )}i50, 1, …, n in the fixed image by pre-determined rigid body transformation. The displacements of these points are: D(xi,yi,zi)5 {(x’i {xi y’i {yi z’i {zi )}i50, 1, …, n. We can form a TPS interpolation as follows n X D(x,y,z)~a0 za1 xza2 yza3 zz bi Ui (r) (1) i~1
where U(r)5r2log (r2), r25(x2xi)2z(y2yi)2z(z2zi)2, and r is the distance between the point being calculated and a landmark point under consideration. This form has favourable properties for image registration. However, different functions could be used, as the one described in Ref. 7. We can calculate the TPS coefficients IMAG 31 # RPS 2011
HYBRID RIGID AND NON-RIGID REGISTRATION ALGORITHM
129
where K is a n6n matrix, P is a n64 matrix, PT is the transpose of matrix P and DI is the displacements of ith points 1 0 0 U(r12 ) U(r1n ) C B B U(r21 ) 0 U(r2n ) C C B C K~B B .. .. .. C; C B . . P . A @ U(rn1 ) U(rn2 ) 0 1 0 1 x 1 y1 z 1 C B B 1 x 2 y2 z 2 C C B C P~B B .. .. .. C .. B. . . C . A @ 1 xn y n z n Once the TPS coefficients are found, we can calculate the deformation at every point x5(x,y,z) in the image n X f (x)~xza0 za1 xza2 yza3 zz bi Ui (r) (3) i~1
1
Examples of registering images with two rigid-motion objects (grey-white circles): (a) the original image with regular grids. Grids are provided to demonstrate the deformations estimated in the middle and right panels; (b) the deformations estimated using piecewise rigid transformations; (c) the deformations estimated using our proposed method; (d) a close-up of a portion of (b); (e) a close-up of a portion of (c)
0 B B B B B B B B B B B B B B @
b1 b2 .. . bn a0 a1 a2
1 C C C C C C C K C~ C PT C C C C C A
a3
IMAG 31 # RPS 2011
0
D1
1
B C B D2 C B C B .. C B C {1 B . C B C P B Dn C B C 0 B 0 C B C B C B 0 C B C @ 0 A 0
(2)
By using TPS to estimate the dense deformations and constraining the rigid transformation for each bony structure, we can obtain reasonable deformations for the entire volumes. Figure 1 demonstrates the advantage of our proposed method in registering images with two rigid-motion objects, i.e. grey-white circles. In order to see clearly, Fig. 1d and e offers a close-up of a portion of the results. Our method can estimate the desired rigid transformation for each object, and simultaneously it can estimate smooth deformations near the circles and in the background. While using a classical piece-wise rigid transformation method,7 it can accurately estimate the rigid transformations for rigid-motion objects, but the deformations are not smooth from the boundaries of objects to the background, and the grad is broken in the background, indicated by red arrows. 2.2
FFD
Although the TPS technique can estimate dense deformations in the entire volumes and bring the soft tissues roughly aligned, it is still required to refine the registration for soft tissues, since they have their own non-rigid motions, which can be independent of the motions of bony structures estimated. Accordingly, we use a B-spline FFD algorithm to iteratively refine the registration based on the initial deformations estimated by bony structures, and at the same time keep the rigid transformation for each bony structure. The Imaging Science Journal Vol 59
130
Z T LU ET AL.
FFD is a parametric model which provides a flexible nonlinear transformation to register images or structures. This model has been successfully applied to different medical imaging applications, such as brain registration or cardiac segmentation,6 pre- and post-contrast magnetic resonance mammogram registration,9 and whole-body PET-CT images.12 The basic idea of FFD is to deform an object by manipulating an underlying mesh of control points. The resulting deformation controls the shape of the 3D object and produces a smooth and C2 continuous transformation. Let w denote a nx6ny6nz mesh of control points wi,j,k with uniform spacing d. Then, at any position x5(x,y,z), the deformation is computed from the positions of the surrounding 46464 neighbourhood of control points 3 X 3 X 3 X T(x)~xz Bl (u)Bm (v)Bn (w) wizl,jzm,kzn (4)
under registration;13 at the same time, the deformation on each bony structure is constrained to be rigid, respectively. By assuming that all points in the same soft structure have similar deformations, we group points with similar displacements into various regions, and then require the control points in each grouped region to have similar displacements. By repeating this registration refinement strategy, we can eventually obtain accurate registration not only for rigid-motion bony structures, but also for non-rigid soft tissues in the volumes. 2.3
Our proposed rigid and non-rigid registration algorithm for alignment of two different time thoracic and abdominal images of the same subject can be summarised as follows:
N
l~0 m~0 n~0
where i, j and k denote the indices of the control point cell containing x5(x,y,z), and u, v and w are the relative positions of x, y and z, respectively, inside that cell in the three dimensions. Bl represents the lth basis function of the B-spline, i.e.
N
B0 (x)~({x3 z3x2 {3xz1)=6 B1 (x)~(3x3 {6x2 z4)=6 B2 (x)~({3x3 z3x2 z3xz1)=6
(5)
N
B3 (x)~x3 =6 The parameters of B-spline-based transformation are the displacement of the control points. Since B-spline is locally controlled, it is computationally efficient even for a large number of control points. In particular, the basis functions of B-spline have a limited support, i.e. changing control point affects the transformation only in the local neighbourhood of that control point. This also saves the computational time. The tradeoff between deformation flexibility and computational complexity is mainly an empirical choice, which is determined by the accuracy required. In our experiment with thoracic and abdominal datasets, we have used a 46464 neighbourhoods of control points. The initial displacements of the control points can be directly obtained from the TPS interpolation results. Then, the refinement of registration is iteratively conducted based on the maximisation of normalised mutual information between two volumes The Imaging Science Journal Vol 59
Summary of our proposed method
N
N
3
first, we segment bony structures from two volumes. Also, bony structures in each volume are separated by using morphological operations and region growing second, we register these segmented bony structures in two volumes, based on their boundary points, using a soft correspondence detection algorithm. The transformation for each bony structure is made to be rigid after soft correspondence detection third, we interpolate the dense correspondences/ deformations in the entire images using TPS. After the interpolation, we also constrain the transformation for each bony structure to be rigid afterwards, we use the interpolated deformations as initial deformations to guide the B-spline-based registration for refining the registration for soft tissues. The resulted transformation in each bony structure is again made to be rigid we repeat the above B-spline-based deformation refinement algorithm, until it converges. Thus, eventually we can register bony structures rigidly and soft tissues non-rigidly, for the thoracic and abdominal images of the same subject scanned at two different times.
RESULTS
Several experimental results on clinical data are presented in this section to demonstrate the performance of the proposed rigid and non-rigid registration IMAG 31 # RPS 2011
HYBRID RIGID AND NON-RIGID REGISTRATION ALGORITHM
2
Example of the original CT data before registration. Left: the first time image used as fixed volume in our experiment; right: the second time image used as moving volume
algorithm for thoracic and abdominal images. In particular, three sets of CT data scanned at different times were acquired using a Philips Brilliance 64-Slice CT scanner. The size of each CT data is 51265126397, and the voxel size is 0.6860.6860.50 mm. A set of CT data without registration is shown in Fig. 2. Figure 3 shows the coronal view (first row) and sagittal view (second row) of CT data after local rigid registration. This result demonstrates that the local rigid transformations, estimated for bony structures, allow for accurate registration of bony structures, but not for the adjacent or distant soft tissues, including liver and the stomach. This is clearly indicated by red lines, which are placed as landmarks in the figure. To improve the registration for soft tissues, a Bspline FFD algorithm, based on normalised mutual information, is used to refine the deformation in the entire images, and at the same time the transformations in the segmented bony structures are kept to be
3
131
rigid during the whole registration refinement procedure. To find the parameters on control points of Bspline-represented deformations, we need an efficient and robust optimisation algorithm. We used an optimisation algorithm similar to that described in Ref. 9, i.e. applying iteratively a gradient descent technique to all control points simultaneously. Note that the gradient can be estimated very efficiently due to the local effect of the control points, which improves the efficiency of optimisation algorithms. Moreover, to further improve robustness and speed of our algorithm, we employed a multi-resolution approach. Starting with coarse control point grid spacing, then we successively refined using a B-spline subdivision algorithm during the registration. To assess the quality of final registration in the patient datasets, we asked two clinical radiologists to place 30 corresponding landmarks in the liver, spine and kidney. By using our registration algorithm, we
Alignment results using rigid transformations estimated for bony structures and the dense deformations interpolated by TPS technique. Coronal and sagittal views are shown in the first and the second rows, respectively. It can be observed that the appropriate registration is achieved for bony structures, but not for the adjacent or distance soft tissues. Left: fixed images; right: moving images after local rigid registration and TPS interpolation. The red lines are used as landmarks for visual inspection of alignment results
IMAG 31 # RPS 2011
The Imaging Science Journal Vol 59
132
Z T LU ET AL.
4
Final registration results after non-rigid registration refinement using the constrained free-form deformations. Both coronal (first row) and sagittal (second row) views are provided. Left: the first time image used as a fixed image; right: the registered version of the second time image, used as a moving image. Red lines are provided as landmarks for visual inspection of registration results
can bring the corresponding landmarks from different time images into the same space; therefore, we can evaluate the errors in all aligned landmarks. The error is ei5|d(xi,x’i )2de(xi,x’i )|, where d(xi,x’i ) and de(xi,x’i ) are the ground truth and estimated distance from the landmark xi to its corresponding x’i , respectively. To measure the quality of the registration, we measure the mean error me and standard deviation st 1 XNb me~ e (6) i~1 i Nb X 1=2 1 Nb 2 (e {me) (7) st~ i~1 i Nb where Nb is the number of landmarks. The mean error for these landmarks is 1.5 mm, and the standard deviation is 0.6 mm. Also, by inspecting the final alignment results in Fig. 4, we can observe that the soft tissues (including liver and stomach) were aligned very well after performing the constrained FFD proposed in this paper. In particular, the results obtained in the liver are very encouraging; even there exist large deformations in the respective regions.
4
CONCLUSION
We have proposed a particular registration algorithm that permits rigid alignments for bony structures and non-rigid elastic alignments for soft tissues, to achieve accurate registration for the thoracic and abdominal images acquired from same subject at The Imaging Science Journal Vol 59
different times. Our approach consists of two steps. The first step is on rigid registration of bony structures in the images. To achieve it, the bony structures are first segmented from the images, and then they are registered according to their boundary points using a soft correspondence detection algorithm. After detecting correspondences, a rigid transformation is estimated for each bony structure in the images, and then the dense correspondences/ deformations are interpolated in the entire images using a TPS interpolation technique. The second step is to refine the registration in the entire images, particularly on the registration of soft tissues, using a constrained B-spline based FFD. The dense deformations estimated in the first step are used to initialise the displacements for the control points in the Bspline-based transformations, which increases not only the registration accuracy but also the registration speed, i.e. 40% deduction of computational cost compared to the case with no initial deformations provided. In each iterative deformation refinement procedure using B-spline-based registration, the transformation for each bony structure is constrained to be rigid, although the actual rigid transformation parameters are allowed to be adjusted slightly. By using this proposed registration algorithm, we achieve the reasonable registration results for thoracic and abdominal images. In the future, we will perform more evaluations on each step of registration algorithm. In particular, the performance of bony structure segmentation is significant for the accuracy of subsequent registration; thus, we may need to refine this step in future, i.e. using the current IMAG 31 # RPS 2011
HYBRID RIGID AND NON-RIGID REGISTRATION ALGORITHM
segmentation and registration results as initialisation for refining the registration of bony structures using their original intensity images. In addition, we will test the effects of changing the number of control points of FFD, since the number of control points affects not only the registration accuracy but also the speed of registration algorithm. Finally, we will test and evaluate our proposed registration algorithm on more other datasets or applications. ACKNOWLEDGEMENTS This work was supported by a grant from National Natural Science Foundation of China (No. 31000450), the Key Program of National Natural Science Foundation of China (No. 30730036) and the Major State Basic Research Development Program of China (973 Program, No. 2010CB732505). The authors also gratefully acknowledge the helpful comments and suggestions of the reviewers, which have improved the presentation.
6
7
8
9
10
REFERENCES 1 Brown, L. A survey of image registration techniques. ACM Comput. Surv., 1992, 24, 325–376. 2 Maintz, J. and Viergever, M. A survey of medical image registration. Med. Image Anal., 1998, 2, 1–36. 3 Zitova, B. and Flusser, J. Image registration methods: a survey. Image Vision Comput., 2003, 21, 977–1000. 4 Lester, H. and Arridge, S. A survey of hierarchical nonlinear medical image registration. Patt. Recogn., 1999, 32, 129–149. 5 Camara, O. ‘Non-linear registration of thoracic and abdominal CT and 18-FDG whole-body emission PET
IMAG 31 # RPS 2011
11
12
13
133
images: methodological study and application in clinical routine’, PhD dissertation, Ecole Nationale Supe´rieure des Te´le´communications, Paris, France, 2003. Frangi, A. F., Rueckert, D., Schnabel, J. A. and Niessen, W. J. Automatic construction of multipleobject three-dimensional statistical shape models: application to cardiac modeling. IEEE Trans. Med. Imag., 2002, 21, 1151–1165. Little, J. A., Hill, D. L. G. and Hawkes, D. J. Deformations incorporating rigid structures. Comput. Vision Image Understand., 1997, 66, 223–232. Tanner, C., Schnabel, J. A., Chung, D., Clarkson, M. J., Rueckert, D., Hill, D. L. G. and Hawkes, D. J. Volume and shape preservation of enhancing lesions when applying non-rigid registration to a time series of contrast enhancing MR breast images. Lect. Notes Comput. Sci., 2000, 1935, 327–337. Rohlfing, T., Maurer, Jr, C. R., Bluemke, D. and Jacobs, M. Volume-preserving nonrigid registration of MR breast images using free-form deformation with an incompressibility constraint. IEEE Trans. Med. Imag., 2003, 22, 730–741. Rangarajan, A., Chui, H., Mjolsness, E., Pappu, S., Davachi, L., Goldman-Rakic, P. and Duncan, J. A robust point matching algorithm for autoradiograph alignment. Med. Image Anal., 1997, 4, 379–398. Wang, L. I., Greenspan, M. and Ellis, R. Validation of bone segmentation and improved 3-D registration using contour coherency in CT data. IEEE Trans. Med. Imag., 2006, 25, 324–334. Maes, F., Collignon, A., Vandermeulen, D., Marchal, G. and Suetens, P. Multimodality image registration by maximization of mutual information. IEEE Trans. Med. Imag., 1997, 16, 187–198. Studholme, C., Hill, D. L. G. and Hawkes, D. J. An overlap invariant entropy measure of 3D medical image alignment. Patt. Recogn., 1999, 32, 71–86.
The Imaging Science Journal Vol 59
