INSTITUTE OF PHYSICS PUBLISHING
PHYSICS IN MEDICINE AND BIOLOGY
Phys. Med. Biol. 49 (2004) 3067–3087
PII: S0031-9155(04)75850-9
Fast free-form deformable registration via calculus of variations Weiguo Lu1, Ming-Li Chen2, Gustavo H Olivera1,2, Kenneth J Ruchala1 and Thomas R Mackie1,2 1 2
TomoTherapy Inc., 1240 Deming Way, Madison, WI 53717, USA University of Wisconsin-Madison, 1300 University Avenue Madison, WI 53705, USA
E-mail:
[email protected]
Received 5 February 2004 Published 28 June 2004 Online at stacks.iop.org/PMB/49/3067 doi:10.1088/0031-9155/49/14/003
Abstract In this paper, we present a fully automatic, fast and accurate deformable registration technique. This technique deals with free-form deformation. It minimizes an energy functional that combines both similarity and smoothness measures. By using calculus of variations, the minimization problem was represented as a set of nonlinear elliptic partial differential equations (PDEs). A Gauss–Seidel finite difference scheme is used to iteratively solve the PDE. The registration is refined by a multi-resolution approach. The whole process is fully automatic. It takes less than 3 min to register two three-dimensional (3D) image sets of size 256 × 256 × 61 using a single 933 MHz personal computer. Extensive experiments are presented. These experiments include simulations, phantom studies and clinical image studies. Experimental results show that our model and algorithm are suited for registration of temporal images of a deformable body. The registration of inspiration and expiration phases of the lung images shows that the method is able to deal with large deformations. When applied to the daily CT images of a prostate patient, the results show that registration based on iterative refinement of displacement field is appropriate to describe the local deformations in the prostate and the rectum. Similarity measures improved significantly after the registration. The target application of this paper is for radiotherapy treatment planning and evaluation that incorporates internal organ deformation throughout the course of radiation therapy. The registration method could also be equally applied in diagnostic radiology.
0031-9155/04/143067+21$30.00 © 2004 IOP Publishing Ltd Printed in the UK
3067
3068
W Lu et al
1. Introduction Medical image registration has been the subject of extensive study in the literature. Recent reviews have appeared in several journal publications (Brown 1992, van den Elsen et al 1993, Maintz and Viergever 1998, Hill et al 2001, Zitova and Flusser 2003) and books (Hajnal et al 2001, Gee et al 2003). Image registration computes a transformation that relates the points in one image to their corresponding points in the other. The transformation can be global or local, rigid or deformable (Lu 2001). The rigid-body model simplifies the registration process, but rigid-body transforms have limited applicability because many organs deform substantially. Rigid-body is a good model for individual bone motion. For example, by far the most important part of the body registered in this way is the cranium and its contents— the brain. However, the extension of the rigid-body model to extra-cranial targets is more difficult, because the assumption of a rigid-body is not valid within the abdomen, pelvis, thorax or extremities. For other regions of the body in the vicinity of bone, the rigid-body model may be used, but may cause large error. Internal structures do enlarge and contract due to normal physiological functions, the region of interest volume does not necessarily maintain a fixed relationship to bony structures. For most organs in the body, many more degrees of freedom (DOF) are needed to describe the different deformations with adequate accuracy. There have been many approaches to deformable registration. As for the similarity measures (Rogelj and Kovacic 2001) considered, most techniques could be categorized as point-based and voxel-based methods. Point-based techniques minimize the distance between features such as points, curves or surfaces of corresponding anatomical structures. They require the identification and matching of these features on both reference and test images. This process usually needs certain amount of human interactions. After point matching, the remaining procedure of registration is only interpolation or approximation. Voxel-based methods use similarity measures such as sum of squared distance, cross correlation or mutual information (Maes et al 1997, Studholme et al 1999, Pluim et al 2003) between images. They have the advantage that they do not require any feature extraction process. As for the spatial transformations, the deformations can be grouped as polynomial transformation (Woods et al 1998), spline and thin-plate spline (Bookstein 1989) transformation and free-form deformation (FFD) (Sederberg and Parry 1986). The DOF of polynomial transformation is defined by its order. Polynomial transformations cannot accommodate local shape changes but only model global shape changes. Therefore, their ability to recover anatomical shape variability is quite limited. In addition, the degrees of higher order polynomials tend to introduce artefacts such as oscillations. Spline and thin-plate spline transformation requires certain number of control points. The DOF of spline and thinplate spline transformation is proportional to the number of control points used. Each control point has global influence on the transformation. In many cases, this global influence of control points is undesirable since it becomes difficult to model local deformations. In addition, the computation complexity makes using large number of control points prohibitive. Usually, an FFD is the best choice for many situations for its ability to model local deformation, its unlimited DOF, and its computation efficiency. The FFD must be constrained/regularized because of ill-posedness of the problem (the existence of many possible solutions). Physical models, such as linear elastic models (Burr 1981, Bajcsy and Kovacic 1989, Lu et al 2001, Lu et al 2001), viscoelastic models (Miller et al 1993), viscous fluid models (Christensen et al 1996, Wang and Staib 1998) and biomechanical model (Hagemann et al 1999, Kyriacou and Davatzikos 1999, Kyriacou et al 1999) are widely used to enforce topological properties of the deformation and constrain the
Deformable registration
3069
enormous solution spaces. Generally, the whole image is modelled as a physical body, and the similarity measures between two images act as external forces that ‘stretch’ the body. These external forces are counterbalanced by physical constraints. The ultimate goal is the determination of a state with minimum energy whose resulting deformation defines the registration. The problems associated with finding the minimum energy state or equilibrium usually involve iterative numerical methods. Physical model based methods usually have heavy computation demands, and even require massive parallel computers (Christensen et al 1996). FFD can also be modelled as optical flow (Horn and Schunck 1981) or ‘demons’ (Thirion 1995, Thirion 1998). In these cases, regularization is applied by filtering/smoothing the deformation field (Thirion 1998, Guimond et al 2001). Conventionally, in 3D radiation therapy, a planning image is assumed to be a valid representation of the patient throughout the course of radiotherapy. Translation and perhaps even rotation set-up verification can be used to perform ‘rigid-body’ realignments of the patient based on transmission planar images. Set-up verification (or rigid-body registration) is not adequate for modern radiotherapy applications. Internal organ deformation throughout the course of radiation therapy is problematic for accurately evaluating the cumulative dose delivered (Yan et al 1999, Lu et al 2000, Olivera et al 2003). Deformable registration is a technique to account for internal organ deformation. Deformable registration is also the basis to build four-dimensional (4D) model for radiation therapy planning and evaluation (Brock et al 2003, Zhang et al 2004). To apply deformable registration routinely in radiation therapy, a desirable method should be accurate, fast and fully automatic. In addition, it should have enough number of degrees of freedom to cope with large deformations. The application nature excludes the point-based methods that require manually selecting a large number of landmarks and physical model based methods that require prior knowledge of the material properties and heavy computation demands. In this paper, we present a new, fast and accurate voxel similarity based free-form deformable registration method. The deformable registration problem is modelled as a functional minimization problem. Rather than using a complicated physics model, we use the smoothness of displacement field as a constraint. Therefore, no prior knowledge about the physical properties of the patient is required. The smoothness constraints are somehow similar as Thirion’s ‘demons’ method (Thirion 1995, 1998, Ibanez et al 2003). In Thirion’s method, the smoothness constraints are explicit in the algorithm since the displacement fields are convolved with a Gaussian filter after each iteration. In our approach, this constraint is implicitly built into the objective functional, therefore, the optimization process itself addresses the trade-off between the similarity measures and smooth constraints. We use calculus of variations to represent the optimization problem as a set of nonlinear elliptic partial differential equations (PDEs). The PDEs are iteratively solved using a Gauss–Seidel finite difference scheme. A multi-resolution approach is used to refine the registration. The method and our implementation are general and can be used to register any dimension data, such as 1D signals, 2D images or 3D volume data. The whole process is fully automatic and efficient. The registration of two 3D volumes is computed within minutes using a single personal computer. Theoretically, the method itself does not eliminate local minima. But in practice, local minima are rarely met due to the combination of multi-resolution technique and Gause–Seidel updating scheme. With the combination of all these techniques, images with large deformations can also be accurately registered. The target application of this paper is radiotherapy planning and evaluation that incorporates internal organ deformation. Multiple CT scans of the same patient are required. But the method itself could also be equally applicable to diagnostic radiology, both CT and MR in situations where patients are scanned on multiple occasions.
3070
W Lu et al
2. Theory 2.1. Free-form deformation When the relative positions of particles in a continuous body are altered, we say that the body is strained and the change in the relative position of points is a deformation. To describe the deformation of the body, we need to know the position of any point in the body with respect to a reference configuration. We use a Eulerian reference frame. The Eulerian description of material deformations specifies the time evolution of particle positions and velocities as observed at fixed points. At the reference state, the location of material point P is x = (x 1 , x 2 , x 3 ). We use x as a label for that point. As time goes on, the point moves. Its location has history: x = x (x, t) refers to the same reference frame. Let us define a displacement mapping from a domain A(reference) to domain B(test): x (x, t) = x + u(x, t). (2.1) u:A→B Equation (2.1) means that a point originating at x will move to point x + u(x, t) at time t. We call (2.1) a registration map. The vectors u(x, t) are the displacement vectors. The objective of deformable registration is to find out these displacement vectors. The deformation can be local or global, small or large, elastic or inelastic. The most general deformation is ‘free-form deformation’; that is, each voxel can move freely without any explicit constrains. In free-form deformation, the degrees of freedom are 3N, where N is the total number of voxels. 2.2. Minimization of the energy functional In this paper, we describe the free-form deformable registration problem as finding the displacement fields uˆ that minimize the energy functional ε(u): uˆ = arg min ε(u) (2.2) u
where
ε(u) = x∈R 3
3 3 2 R 2 (x, u) + λ vj i dx.
(2.3)
i=1 j =1
Here R(x, u) = B(x + u) − A(x) is the residual. A is the reference data and B is the test ∂ui data, and vj i ≡ ∂x j . Here we assume that two images are similar in intensity level so that intensity residual is appropriate to describe their misalignment. This variational formulation follows a standard principle of making the result smooth when there are no data. In particular, we see that when the residual square R 2 is large, the first term dominates, and is minimized by setting R = 0, that is, two images match each other in voxel intensity. On the other hand, when R 2 is small, the energy is dominated by summation of the squares of the partial derivatives of the displacement vector, yielding a slowly varying field. Equation (2.2) is a nonlinear minimization problem with thousands to millions of variables. It is prohibitive to solve it with the conventional optimization algorithm on a personal computer within reasonable time. 2.3. Calculus of variations By using the calculus of variations (Keener 1988, Xu 2000), it can be shown that the displacement field can be found by solving the following Euler–Lagarange equations, ∂R(x, u) =0 (2.4) λ∇ 2 u − R(x, u) ∂u
Deformable registration
3071
3
2
where ∇ 2 is the Laplacian operator with respect to x; ∇ 2 = i=1 ∂x∂ i 2 and λ is a constant weight. Equation (2.4) provides further intuition behind the displacement formulation. We note that the region where two images have matched intensity (R = 0), the second term in equation (2.4) is zero. Therefore, within such a region, displacement vector u is determined by Laplacian equation, and the resulting displacement vector is interpolated from other unmatched region. This results in a smoothly varying displacement field. As in equation (2.4), the deformable registration problem becomes the problem of solving the nonlinear elliptic partial differential equations (PDE). Further expanding ∂R(∂xu,u) , we have ∂B(x + u) ∂R(x, u) = ∂u ∂u
∂z ∂B(z)
=
∂z z=x+u ∂ u
∂B(z)
= ∂z z=x+u = (∇B)(x + u) = g (x + u )
(2.5)
where g (x) = ∇B(x). So we have λ∇ 2 u − [B(x + u) − A(x)]g (x + u) = 0.
(2.6)
3. Numerical implementation 3.1. Finite difference scheme Equation (2.6) is a system of nonlinear elliptic PDE. We use a finite difference scheme to solve it. To set up the finite difference scheme, the problem needs to be discretized. Let indices i, j, k correspond to x, y and z coordinates, (i = 1, 2, . . . , Lx , j = 1, 2, . . . , Ly , k = 1, 2, . . . , Lz ). Let indices n correspond to dimensions (n = 1, 2, 3). Let the combined indices (i, j, k) be organized as in a certain sequence (for example, a 3D volume is organized as a 1D array), and let the indices for that parametric sequence be m (m = 1, 2, . . . , N ), N is the total number of voxels. Discretizing the left-hand side of equation (2.6), and let it be Lm,n , we have Lm,n = λ∇ 2 (um,n ) − [B ∗ m − Am ]g ∗ m,n Tri-linear interpolation is used to calculate transformed tested imageB g ∗ m,n ,
(3.1) ∗
m
and its gradient
B ∗ m = B(xm + um )
(3.2)
g ∗ m,n = gn (xm + um )
(3.3)
and ∇ 2 (um,n ) =
ui+1,j,k,n + ui−1,j,k,n + ui,j +1,k,n + ui,j −1,k,n + ui,j,k+1,n + ui,j,k−1,n − ui,j,k,n 6 (3.4)
3072
W Lu et al
which is the difference between the displacement of 6 neighbours about a point and the point itself. Note that in equation (3.4), the index m on the left-hand side is decomposed as indexes i, j, k on the right-hand side. One step of Newton iteration was used to update um,n , that is, unew m,n = uold m,n −
Lold m,n ∂Lold m,n ∂um,n
where ∂Lm,n ∂(∇ 2 (um,n )) =λ − ∂um,n ∂um,n
(3.5)
∂(B ∗ m ) ∗ ∂(g ∗ m,n ) ∗ g m,n + (B m − Am ) . ∂um,n ∂um,n
(3.6)
From equation (3.3), we have ∂(∇ 2 (um,n )) = −1 ∂um,n
(3.7)
and from equation (2.5), we have ∂B ∗ m = g ∗ m,n . ∂um,n
(3.8)
We discard the second derivative term ∂g ∗ m,n /∂um,n , then we have ∂Lm,n = −(λ + (g ∗ m,n )2 ). ∂um,n
(3.9)
Therefore, our update scheme is unew m,n = uold m,n +
Lold m,n 2 . λ + g ∗old m,n
(3.10)
Gauss–Seidel method (Press et al 1992) is used to calculate Lold m,n and g ∗old m,n , that is, we use the updated version of um,n as soon as they become available. In summary, the update scheme for each iteration is as follows: For n = 1, 2, 3 For m = 1, 2, . . . , N Calculate ∇ 2 (um,n ) according to (3.4) Calculate B ∗ m and g ∗ m,n using tri-linear interpolation (3.2) and (3.3) Calculate Lm,n according to (3.1) Update um,n according to (3.10) Endfor Endfor. 3.2. Multi-resolution strategy Multi-resolution technique has emerged as a standard technique for deformable registration (Bajcsy and Kovacic 1989, Lester and Arridge 1999, Hellier et al 2001, Mattes et al 2003). Multi-resolution technique is a systematic way for structuring local information into global. The major motivation for using a multi-resolution technique is to alleviate the deficiencies of the one-resolution model (Bajcsy and Kovacic 1989), e.g., the limitation of small deformations, the limitation to propagate the boundary deformation further into the inner region and the computational complexity.
Deformable registration
3073
Figure 1. Flowchart for multi-resolution registration. A and B are reference and test images, respectively. Two image pyramids are constructed before the registration (left-most and right-most column). U are displacement fields. The algorithm accepts an initial displacement, which could be from any initial guess or previous registration. The registration is from top (coarsest) to bottom (finest). Each level registration is used as the initial guess for the latter level. The reduce operation means down-sampling of the images. The expand operation means up-sampling and scaling of the displacement field.
In multi-resolution registration, a coarse level registration is obtained first, which is later improved with the finer resolution calculation. For the multi-resolution algorithm (figure 1), an image pyramid is constructed. The bottom of the pyramid is the original image. Each level of pyramid is the down-sampled version (reduce operation) of its lower level. The down-sample factor is 2 when the downsampled dimension is not less than 32 and 1 otherwise. As for the optimization iterations, each level uses twice the number of iterations than that of its lower level. For example, suppose the original image set has dimension 512 × 256 × 68 and bottom level uses 8 iterations, then the pyramid has 5 levels, each level has dimensions, from bottom to top, (512 × 256 × 68), (256 × 128 × 34), (128 × 64 × 34), (64 × 32 × 34) and (32 × 32 × 34) respectively. The
3074
W Lu et al
number of iterations for each level is 8, 16, 32, 64 and 128, respectively. Registration starts on the top level (with coarsest resolution). The resulting displacement field u from the coarse level, after it is appropriately expanded (expand operation), is used as the initial displacement field for the finer level registration. 4. Evaluation There are generally two criteria to evaluate a deformable registration algorithm: (a) The difference between the calculated displacement field with that of the gold standard. (b) The similarity measures between the deformed test image and the reference image. The first criterion requires a gold standard displacement field, which only exists on the simulation study or well-designed phantom study. The second criterion could apply on any image data sets, but it cannot tell how well the registration matches the real deformation. As for the similarity measures, we use (normalized) cross correlation (CC) and mutual information (MI). The CC is defined as ¯ ¯ x) − B) (A(x) − A)(B( CC = (4.1) 1 . ¯ 2 (B(x) − B) ¯ 2 2 (A(x) − A) Here, A¯ and B¯ are the mean intensity of the reference and the test image, respectively. If the reference and the target image are identical (perfect registration), CC is unity. The MI is defined as (Studholme et al 1999) pAB (i, j ) MI = − pAB (i, j ) log (4.2) pA (i)pB (j ) where i, j are the intensity level of the reference image and the target image, respectively. p(·) is the probability distribution. The larger the MI, the better the registration. 5. Experiments and results The presented algorithm could be applied to any dimension (e.g., a 1D signal, a 2D slice, and a 3D volume data) image. The algorithm is implemented in C++ programming language. The same code is used to register any dimension image data. There are only two parameters needed to run the code. One is the Laplacian weight λ, the other is the number of iterations for the finest resolution. Fortunately, as we have extensively tested, the algorithm is not very sensitive to these two parameters, and a rough guess is good enough. In the following tests, we always use λ = 0.1, and use 16 iterations for the finest level. 5.1. Synthetic data 1D registration. Figure 2 illustrates how the algorithm works for 1D signal. Here we have reference signal (red solid line) and test signal (green dashed line). The objective of deformable registration is to deform the test signal so that it approaches the reference signal. The blue dashed line shows the deformed test signal after each iteration. The bottom panel shows the growth of displacement field after each iteration. 2D registration. In this experiment, we use a 2D image, which is CT slice (512 × 512), as the reference image. This reference image is deformed using a known formula, which we call harmonic deformation. The harmonic deformation is defined as x (x, y) = (1 + b cos mθ )x
y (x, y) = (1 + b cos mθ )y.
(5.1)
Deformable registration
3075
Figure 2. Illustration of 1D signal registration. The registration objective is to deform the test signal to approach the reference signal. The top panel shows the reference (red solid line), test (green dasheded line) and the deformed test (blue dashed line) signal. Ten iterations are illustrated. The bottom panel shows the growth of the displacement field after each iteration.
Figure 3. Synthetically deformed test images and their differences from the reference image. The left three columns show the test images. The right three columns show the difference images. All these test images are deformed from the same one slice CT (512 × 512) according to a harmonic deformation formula. From the top row to the bottom row, the harmonic parameter was m = 2, 4, 8 respectively. From left to right column, the harmonic parameter b = 0.1, 0.2, 0.3, respectively.
Here θ = tan−1 xy . Two parameters m, b are used in the harmonic deformation, where m specifies the complexity of deformation, and bspecifies the magnitude of deformation. Figure 3 shows the deformed test images with different harmonic parameters. All these images are deformed from the same reference image. From top row to bottom row, the harmonic parameter m = 2, 4, 8 respectively. From left to right column, the harmonic parameter b = 0.1, 0.2, 0.3, respectively.
3076
W Lu et al
Figure 4. The registered version of the synthetic test images as illustrated in figure 3. The right three columns show the difference images after registration. The registrations use the technique presented in this paper.
We registered the test images to the reference image using the algorithm developed in this paper. A multi-resolution technique as discussed in section 3.2 is used. The Laplacian weight was set to beλ = 0.1, and 16 iterations were used for the finest level. For the comparison purpose, we also registered the same date using Thirion’s ‘demons’ method (Thirion 1995, 1998, Ibanez et al 2003). Same multi-resolution strategies were used. As required by ‘demons’ method for the regularization purpose, a standard deviation (σ = 1.0) of the Gaussian smoothing kernel was applied to the demons’ deformation field after each iteration. For registering 2D images of size 512 × 512 using proposed multi-resolution technique, the registration time is approximately 4 s when using our method and about 7 s when using ‘demons’ method, both were run on the same personal computer with clock speed of 933 MHz. Figure 4 illustrates the registered version of figure 3 using our method. Figure 5 illustrates the registered version of figure 3 using ‘demons’ method. Figure 6 shows the similarity measures (CC) between the reference image and the test images before and after registration. The left panel is the CC before registration. The middle is the CC after registration using our method. The right panel shows the CC after registration using ‘demons’ method. For small and simple deformation (m = 2, b 1), both our method and ‘demons’ method do a good job, the deformed images have been registered back to their reference counterpart. But for the large deformations (m 4, b > 1), our method is superior to the ‘demons’ method. There is large registration error in ‘demons’ method. While our method does a great job, the similarity measures after registration approaches unity even for large and complex deformations (m = 8, b = 0.3). 5.2. Phantom data To set-up a gold standard for testing the deformable registration algorithm, we designed a gel–balloon phantom (Lu 2001). In this phantom, a balloon was surrounded by the gel, and can be inflated and deflated with insertion or removal of heavy oil. The volume of balloon is controlled by the amount of oil injected, which can be read from an attached syringe. In total 320 plastic beads were implanted around the balloon in a regular cube grid (7 × 7 × 7,
Deformable registration
3077
Figure 5. Similar to figure 4 but using ‘demons’ method for registration.
Figure 6. Similarity measures (CC) between reference image and test images. The left panel shows CC before registration. The middle panel shows CC after registrations based on the technique presented in this paper. The right panel shows CC after registrations based on the ‘demons’ method.
3078
W Lu et al
except the locations taken up by the balloon). The diameter of each bead is about 3.5 mm. The distances between beads were about 10 mm in each direction. The beads were assumed to move in the same way as the surrounding gel since they are moulded into the gel. By adjusting the amount of oil injected into the balloon, five deformation stages were approached. We use the medial deformation stage as the reference (R) and the other four stages are categorized as either inflation (I) or the deflation (D) stages. Thus, as in the increasing order of balloon size, the five deformation stages are D2, D1, R, I1 and I2 (1 and 2 stands for two levels of deformation). CT images of the gel–balloon phantom were taken in order to study the movement and deformation of the gel as a function of the inflation state of the balloon. A Somatom HiQ CT scanner (Siemens Medical Systems, Inc., New Jersey, USA) at the University of Wisconsin Hospital and Clinics was used. The scan protocol was 140 KVp, 95 mAs, with voxel size 0.85 × 0.85 × 2 mm3. Based on the CT image, the positions of the beads can be measured, either manually or automatically. They can also be calculated by the deformable registration algorithm. The evaluation of the algorithm includes comparing these two results: measurement and calculation. Since the deformation is quite small (of the order of 1 mm) in this experiment, the measurement must be in sub-voxel resolution. We use an automatic algorithm (Lu 2001) to determine the positions of the beads. Figure 7 shows the overlapped orthogonal (transverse, sagittal and coronal) view of the reference stage and that of inflated stage. The bottom (red) is the reference stage, while the top (green) is the inflated stage. The inflation of the balloon pushed the beads out, which is demonstrated in all transversal, coronal and sagittal views. Figure 8 illustrates the registered version of figure 7. It shows that both balloon and beads are registered to their reference position, and no significant difference between the reference and inflated stages is visible. Figures 9 and 10 are similar to figures 7 and 8, except now a deflation stage is registered with the reference stage. The cumulative histograms of beads displacement are illustrated in figure 11. The y-axis of histogram shows the percentage of beads that has displacement smaller than certain value. The x-axis shows the amount of displacement. The left panel shows the histogram before registration, the right panel shows the histogram after registration. From the histogram plots, we found that the displacement of beads between the inflation (deflation) stages and the reference stage could be 3–4 mm. But after the registration, all displacements are less than 1 mm. This means that for small deformation, the registration error is less than 1 mm. 5.3. Clinical data ‘Fractionation’ (ICRU 1993, 1999) is a common technique in external beam radiotherapy; that is, a prescribed dose is delivered through multiple (such as 30–40) fractions, usually one fraction per day. The delivery time for each fraction lasts several minutes. We test our algorithm on both ‘intra-fraction’ images and ‘inter-fraction’ images. ‘Intra-fraction’ images stand for images that reveal ‘intra-fraction’ deformation, examples are CT images of different breathing phases. ‘Inter-fraction’ images stand for images that reveal daily deformation, such as daily CT images of the same patient during whole radiotherapy treatment courses. 5.3.1. Intra-fraction image registration. In this experiment, we use the CT data from a lung patient with IRB approval and de-identification. The CT scans were taken at University of Wisconsin-Madison Hospital and Clinics using a GE LightSpeed CT scanner with four row detectors (GE Medical System, Waukesha, WI). CT images at the exhalation phase and deep
Deformable registration
3079
Figure 7. Overlapped orthogonal views of the reference stage (red) and one of the inflated stage (green) before registration. The left panel shows the transverse view, and the right panels show coronal (lower) and sagittal (upper) view, respectively.
Figure 8. The registered version of figure 7. The red component is the reference stage and the green component is the registered test stage.
inspiration phase were obtained by breath-holding technique. The scan at deep inspiration phase took about 20 s with 0.8 s per gantry rotation. The exhalation phase was scanned with 0.4 s per gantry rotation and finished within 10 s. Both scans are helical with pitch = 1.5. The voxel size is 0.9 × 0.9 × 2.5 mm3.
3080
W Lu et al
Figure 9. Overlapped orthogonal views of the reference stages (red component) and one of the deflated stages (green component) before registration. The left panel shows the transverse view, and the right panels show coronal (lower) and sagittal (upper) view, respectively.
Figure 10. The registered version of figure 9. Red component is the reference stage and green component is the registered test stage.
CT images of two respiration stages, namely, inspiration and deep exhalation are used for the registration test. The inhale image is used as the reference image. The exhale stage is used as the test image. Figure 12 shows the overlapped orthogonal view of the inhale and exhale of CT image before deformable registration. The red component is the inhale image and the green component is the exhale image. We can see that there is large lung deformation from
Deformable registration
3081
Figure 11. Histograms of absolute beads displacement before (left) and after (right) the registration.
Figure 12. Overlapped orthogonal views of the intra-fraction CT images. The red component is the inhale image, which is used as reference image. The green component is the exhale image, which is used as test image.
3082
W Lu et al
Figure 13. Overlap of inhale and the exhale image after registration.
inhale to exhale stages. In addition, the exhale data are reconstructed using an incomplete field of view. Figure 13 illustrates the overlapped orthogonal CT views of the inhale and exhale phases after deformable registration. The large deformation of lung and other parts of the body is registered. Significant misregistration only occurs at the edge portion of image, which is due to an incomplete test data set. This experiment shows that our method is robust for large deformations. 5.3.2. Inter-fraction image registration. In this experiment, we retrospectively study the inter-fractionCT data from three prostate patients. These data were taken on alternate days of external beam radiation therapy course in William Beaumont Hosiptal (courtesy Dr Di Yan) in 1997. The numbers of CT data set for these patients are 18, 15 and 16, respectively. For each patient, the first CT image set is treatment planning CT and all others are images on treatment days (fraction CT). We want to register the fraction images with the planning image. The planning CT is used as the reference image and the fraction CTs are used as the target image. All images are down-sampled to size of 256 × 256 × 61, with voxel resolution 1.9 × 1.9 × 3 mm3. The whole registration time between two images is about 3 min when run on a single processor 933 MHz Pentium III personal computer. Figure 14 illustrates the overlapped orthogonal views of the reference image (bottom layer, red) and one of the rigid-body registered test images (top layer, green) of the first patient. The rigid-body registration is based on bony structures. It is still obvious that there is large local deformation between the reference images and the test images, most notably in the prostate and rectum region. The rectum is full in the reference image, while it is gas-filled in the test image. The prostate was deformed as well. Figure 15 illustrates the deformable registered version of figure 14. It shows that the local deformation is largely recognized and registered. There is hardly any significant difference between the reference image and thereak registered ¯ image. Figure 16 shows the enlarged view of two transverse slices from figures 14 and 15. The top is before deformable registration. The bottom is after deformable registration.
Deformable registration
3083
Figure 14. Overlapped orthogonal view of the CT images of the same prostate patient acquired during different days. The red component is the planning CT image, which is used as the reference image. The green component is the CT image acquired during the treatment day, which is used as the test image.
Figure 15. Overlap orthogonal views of planning and fraction image after registration.
Figure 17 compares similarity measures of cross correlation (CC) and mutual information (MI) after rigid-body registration and after deformable registration on three prostate patient
3084
W Lu et al
Figure 16. Enlarged view of two transverse slices as shown in figures 14 and 15. The top is before registration. The bottom is after registration.
data sets. The first row is CC and the second row is MI. The ‘squares’ represent similarity measures after the rigid-body registration but prior to the deformable registration. The ‘circles’ represent the similarity measures after the deformable registration. Prior to the deformable registration, the fraction images and the planning image have low similarity measures, which imply that there are large misalignments between these images. The similarity measures after deformable registration are significantly larger than that after rigid-body registration. It means that deformable registration is an indispensable step to account for local deformation.
6. Discussions and future work As any voxel intensity similarity (sum of squared distance) based methods, the algorithm developed in this paper relies on the assumption that voxel representing the same homologous point on an object has the same intensity on both the reference and the test images. This assumption is reasonable for some applications in radiotherapy treatment planning and evaluation, where deformable registration of same modality images (daily CT data and 4D CT data) is required. This assumption prohibits this technique to be applied directly to multimodality registration. But this limitation can somehow be bypassed by certain pre-processing steps. For example, the voxel intensity of both reference and test images can be mapped to physical density before the registration. The intensity mapping can also be achieved by some adaptive intensity correction technique (Guimond et al 2001). Based on these pre-processing
Deformable registration
3085
Figure 17. Similarity measures between planning CT (reference image) and fraction CT (test images) before (square) and after (circle) deformable registration for three patient images. The top row shows cross correlation, the bottom row shows mutual information. The numbers of CT set for each patient are 18, 15 and 16, respectively.
steps, this method could be potentially extended to multi-modality image registration, such as registration of KV-CT versus MV-CT, CT versus MRI, etc. On the other hand, using the mutual information (Likar and Pernus 2001) rather than the residual difference in the first part of equation (2.3) could be a more general solution. All these extensions need further investigations. One problem common to any regularization-based method is how to choose the regularization parameter, the Laplacian weight λ as in our algorithm. Though through multi-resolution technique, the dependence on λ is much alleviated in our algorithm. A rather conserved choice of λ is enough for most applications as in radiation therapy planning and evaluation. Adaptively selecting λ for different iterations and/or tissues is worth a further investigation for the applications that require more accurate control of local deformation. Another interesting topic of deformable registration is how to map points that are missing in the test image to that structure in the reference image. The driving force of our updating scheme (equations (3.10) and (3.1)) is λ∇ 2 (um,n ) − [B ∗ m − Am ]g ∗ m,n ; that is, if an object is missing in the test image, then g ∗ ≈ 0 at that region, therefore, the displacement of such region is solely determined by the Laplacian term. The resulting displacement field of such region is slowly varying in accordance with its neighbour regions. This feature follows the standard principle of making the result smooth when there are no data. This topic will be further studied in the future work.
3086
W Lu et al
7. Conclusions A fully automatic, fast and accurate deformable registration algorithm is presented. We tested the algorithm extensively using simulation data, phantom data, as well as the clinical data. Experimental results show that our model and algorithm are suitable for temporal deformable registration of same modality images. Simulation and phantom studies show that registration error of our method is sub-voxel. When applied to the intra-fraction CT images of a lung patient and inter-fraction image of a prostate patient. The simulation study and the lung patient study show that our method is suitable to account for large deformations. The prostate results show that registration based on local free-form deformation with smoothness constrains is appropriate to describe the local deformations in the prostate and the rectum. Similarity measures improved significantly after the registration. Acknowledgments The authors would like to thank Dr Di Yan of Beaumont Hospital to provide the prostate patient CT data, Tiezhi Zhang of University of Wisconsin-Madison to provide the lung patient CT data, Dr Harry Keller and Stewart Becker of University of Wisconsin-Madison to help on the phantom experiments. References Bajcsy R and Kovacic S 1989 Multiresolution elastic matching Comput. Vis. Graph. Image Process. 46 1–21 Bookstein F 1989 Principal warps: Thin-plate spline and the decomposition of deformations IEEE Trans. Pattern Anal. Mach. Intell. 11 567–85 Brock K, Balter J, Dawson L, Kessler M and Meyer C 2003 Automated generation of a four-dimensional model of the liver using warping and mutual information Med. Phys. 30 1128–33 Brown L G 1992 A survey of Image registration techniques ACM Comput. Surv. 24 325–76 Burr D J 1981 A dynamic model for image registration Comput. Graph. Imaging Process. 15 102–12 Christensen G E, Rabbitt R D and Miller M I 1996 Deformable templates using large deformation kinematics IEEE Trans. Image Process. 5 1435–47 Gee J C, Antoine Maintz J B and Vannier M W 2003 Biomedical Image Registration: Revised Papers. Second International Workshop, WBIR 2003 (Lecture Notes in Computer Science) (Philadelphia, PA: Springer) Guimond A, Roche A, Ayache N and Meunier J 2001 Three-dimensional multimodal brain warping using the Demons algorithm and adaptive intensity corrections IEEE Trans. Med. Imaging 20 58–69 Hagemann A, Rohr K, Stiehl H S, Speteger U and Gilsbach J M 1999 Biomechanical modeling of the human head for physically based, nonrigid image registration IEEE Trans. Med. Imaging 18 875–84 Hajnal J V, Hawkes D L G and Hill D J (eds) 2001 Medical Image Registration (Boca Raton, FL: CRC Press) Hellier P, Barillot C, Memin E and Perez P 2001 Hierarchical estimation of a dense deformation field for 3D robust registration IEEE Trans. Med. Imaging 20 388–402 Hill D L G, Batchelor P G, Holden M and Hawkes D J 2001 Medical image registration Phys. Med. Biol. 46 1–45 Horn B and Schunck B 1981 Determining optical flow Artif. Intell. 17 185–203 Ibanez L, Schroeder W, Ng L and Cates J 2003 The ITK Software Guide: The Insight Segmentation and Registration Toolkit (New York: Kitware Inc.) ICRU 1993 Prescribing, recording, and reporting photon beam therapy ICRU Report 50 ICRU 1999 Prescribing, recording, and reporting photon beam therapy Supplement to ICRU Report 50 Keener J P 1988 Principles of Applied Mathematics: Transformation and Approximation (Reading, MA: AddisonWesley) Kyriacou S K and Davatzikos C 1999 A 3D model of quasi-static brain mechanics: application to tumor-bearing image registration 1st Joint BMES/EMBS Conference p 1161 Kyriacou S K, Davatzikos C, Zinreich S J and Bryan R N 1999 Nonlinear elastic registration of brain images with tumor pathology using a biomechanical model [MRI] IEEE Trans. Med. Imaging 18 580–92 Lester H and Arridge S R 1999 A survey of hierarchical nonlinear medical registration Pattern Recognition 32 129–49
Deformable registration
3087
Likar B and Pernus F 2001 A hierarchical approach to elastic registration based on mutual information Image Vis. Comput. 19 33–44 Lu W 2001 Motion detection and correction for image guided radiation therapy PhD Thesis University of WisconsinMadison Lu W, Chen M and Mackie T R 2001 Deformable image registration: I. Theory Biological & Physical Basis of IMRT & TomoTherapy ed B R Paliwal, D E Herbert, J K Fowler and M P Mehta (Madison WI: Medical Physics Publishing) pp 322–31 Lu W, Keller H and Mackie T R 2001 Deformable Image Registration: II. Experiments Biological & Physical Basis of IMRT & Tomotherapy ed B R Paliwal, D E Herbert, J K Fowler and M P Mehta (Madison, WI: Medical Physics Publishing) pp 332–42 Lu W, Mackie T R, Keller H, Ruchala K J and Olivera G H 2000 A generalization of adaptive radiotherapy and the registration of deformable dose distribution ICCR 2000 pp 521–3 Maes F, Dollignon A, Vandermeulen D, Marchal G and Suetens P 1997 Multimodality image registration by maximization of mutual information IEEE Trans. Med. Imaging 16 187–98 Maintz J B A and Viergever M A 1998 A survey of medical imaging registration Med. Image Anal. 2 1–36 Mattes D, Haynor D, Vesselle H, Lewellen T and Eubank W 2003 PET-CT image registration in the chest using free-form deformations IEEE Trans. Med. Imaging 22 20–8 Miller M I, Christensen G E, Amit Y and Grenander U 1993 Mathematical textbook of deformable neuroanatomaies Proc. Natl. Acad. Sci. USA 90 11944–8 Olivera G H, Ruchala K, Lu W, Kapatoes J, Reckwerdt P, Jeraj R and Mackie R 2003 Evaluation of patient setup and plan optimization strategies based on deformable dose registration Int. J. Radiat. Oncol. Biol. Phys. 57 S188–9 Pluim J P W, Maintz J B A and Viergever M A 2003 Mutual-information-based registration of medical images: a survey IEEE Trans. Med. Imaging 22 986–1004 Press W H, Teukolsky S A, Vetterling W T and Flannery B P 1992 Numerical Recipes in C (Cambridge: Cambridge University Press) Rogelj P and Kovacic S 2001 Similarity measures for non-rigid registration Proc. SPIE 4322 569–78 Sederberg T and Parry S 1986 Free form deformation of solid geometry models SIGGRAPH 20 151–60 Studholme C, Hill D L G and Hawkes D J 1999 An overlap invariant entropy measure of 3D medical image alignment Pattern Recognition 32 71–86 Thirion J P 1995 Fast non-rigid matching of 3d medical images (MRCAS’95: Medical Robotics and Computer Aided Surgery Baltimore) pp 47–54 Thirion J P 1998 Image matching as a diffusion process: an analogy with Maxwell’s demons Med. Image Anal. 2 243–60 van den Elsen P A, Pol E J D and Viergever M A 1993 Medical image matching-a review with classification IEEE Eng. Med. Biol. Mag. 1 26–39 Wang Y and Staib L H 1998 Integrated approaches to non-rigid registration in medical images IEEE WACV ’98 pp 102–8 Woods R, Grafton S, Watson J, Sicotte N and Mazziotta J 1998 Automated image registration: II. Intersubject validation of linear and nonlinear models J. Comput. Assist. Tomogr. 22 153–65 Xu C 2000 Deformable models with application to human cerebral cortex reconstruction in magnetic resonance images PhD Thesis John Hopkins University Yan D, Jaffray D A and Wong J W 1999 A model to accumulate fractionated dose in a deforming organ Int. J. Radiat. Oncol. Biol. Phys. 44 665–75 Zhang T, Jeraj R, Keller H, Lu W, Olivera G H, McNutt T R, Mackie T R and Paliwal B 2004 Treatment plan optimization incorporating respiratory motion Med. Phys. 31 1576–86 Zitova B and Flusser J 2003 Image registration methods: a survey Image Vis. Comput. 21 977–1000
