veloped in [5, 7, 6, 17, 13, 12, 9] where the registration process is viewed as finding .... In order to get a better understanding of this template propagation method, one ... Frequency domain representation of two families of reg- ularization filters ...
FLOWS OF DIFFEOMORPHISMS FOR MULTIMODAL IMAGE REGISTRATION C. Chefd’hotel, G. Hermosillo, O. Faugeras INRIA Sophia-Antipolis, France ABSTRACT We present a theoretical and computational framework for nonrigid multimodal registration. We proceed by minimization of statistical similarity criteria (global and local) in a variational framework, and use the corresponding gradients to drive a flow of diffeomorphisms allowing large deformations. This flow is introduced through a new template propagation method, by composition of small displacements. Regularization is performed using fast filtering techniques. This approach yields robust matching algorithms offering a good computational efficiency. We apply this method to compensate distortions between EPI images (fMRI) and anatomical MRI volumes. 1. INTRODUCTION Under different acquisition modalities, image intensities can no longer be compared by their difference. In this case, registration methods1 are usually built upon statistical similarity measures, such as the cross-correlation, the correlation ratio [14], or the mutual information [18]. The idea is to realign two images by maximizing these criteria over low-dimensional parametric sets of transformations. In two recent papers [2, 8] we proposed to extend this approach to an infinite-dimensional, variational framework. We computed the first variation of global and local versions of the previous criteria, and defined PDE-based matching method using a linear elastic regularization. The main drawback of this approach was a limited capture range and a slow convergence, partially overcome using multiresolution strategies. To solve these problems, we propose here to recover directly large deformations using a flow of diffeomorphisms2 driven by the gradient of statistical similarity measures. We borrow some of the ideas developed in [5, 7, 6, 17, 13, 12, 9] where the registration process is viewed as finding a geodesic or building a flow in a suitable infinite-dimensional group of smooth, invertible transformations. We proceed as follows. First, we use simple heuristics to design a new template propagation method that allows large deformations by composition of small displacements. This method combines the gradients of statistical similarity criteria with a regularization step performed by low-pass filtering. Then, we show that this algorithm forms a consistent approximation of a continuous flow of diffeomorphisms. The well-posedness of this flow follows easily under mild assumptions on the regularization filter. The flexibility offered by the template propagation method, combined with fast recursive filtering techniques, yields a robust and This research was partially supported by EC grant Mapawamo (QLG3CT-2000-30161) and INRIA ARCs IRMf. 1 For surveys on registration techniques and their applications to medical imaging, we refer to [11], [16] and references therein. 2 A bijective map is a diffeomorphism if and 1 are continuously differentiable.
computationally efficient numerical implementation. We present synthetic and real experiments on functional and anatomical MRI datasets. In the following, we model the reference and template images as two functions f and g defined on an open bounded subset
Rn . We consider that a diffeomorphism acts on the template g by composition to form a new image g .
Æ
2. STATISTICAL SIMILARITY CRITERIA We first recall the results presented in [2, 8] concerning the first variation of various statistical similarity criteria. We note X and Y two random variables modeling the intensities of f and g . The cross-correlation (CC), the correlation ratio (CR) and the mutual information (MI) of X and Y are defined from the corresponding joint density denoted p . p is estimated from f and g using a non-parametric Parzen model
Æ
Æ
p (i; j ) =
1 Z G (f (x) i; (g Æ )(x) j ) dx; j j
where G is a familly of smooth density kernels (usually Gaussian windows of variance ). By extension, one can also build a spacedependent Gaussian-weighted version of this estimator, such that
G (f (x) i; (g Æ )(x) j ) G (x x0 ) dx R ; p (i; j; x0 ) =
G (x x0 ) dx R
where controls the neighborhood size. Note that the latter model provides a simple way to relax the stationarity assumption underlying the use of a single random variable for each image. In particular, new similarity measures are obtained by integration over of the local expression of CC, CR and MI. Now, let S ( ) be the expression of a similarity criterion in terms of p . Using the previous density estimates, its gradient S ( ) (for a standard L2 -metric) has a generic form given by
r
rS ( )(x) = L(f (x); g( (x)); x) rg( (x)); where L(i; j; x) is a local intensity comparison function which depends on the joint density p . Note that with global estimates of the joint density, L does not explicitely depend on x. We refer to [2, 8] for the detailed computations and the expression of L in the different cases (CC, CR and MI, both local and global). In practice, gradient evaluations reduce to several Gaussian smoothings of discrete histograms, a procedure which is efficiently approximated using recursive filters [4].
1
3.1. Heuristics The natural approach to maximize a similiarity criterion is to build a sequence of transformations that follows its gradient direction. Since belongs to a function space, such a procedure requires an additional regularization (by adding a penalty term, or directly to S ( )). Starting from applying a regularization operator 0 = id, the first step of a gradient algorithm would yield a new transformation
1 = 0 + "R(rS ( 0 )) = id + "R(rS (id)): Instead of building the sequence k by applying this procedure iteratively, we propose to immediately propagate the template g into g1 = g (id + " ( S (id))), and consider the next iteration as a first step in the resolution of a new matching problem between f and g1 starting at = id. If gk is the propagated template after k iterations of this new procedure, we define k the transformation that satisfies k; gk = g k . The previous argument reduces to a simple algorithm:
Æ
L(f; g Æ k )r(g Æ k ) (displacement field) R(vk ) (regularization ) k Æ (id + "k vk ) (update)
(1)
Since the regularization is applied locally to each vector field vk instead of the whole transformation k , large deformations are not penalized. Moreover, if k is a diffeomorphism and vk is sufficiently smooth, one can always find "k such that k+1 is a diffeomorphism. 3.2. Continuous formulation In order to get a better understanding of this template propagation method, one can study its behavior as the time step goes to zero. We define a step forward operator " ( ) = (id + " v ) corresponding to the update part of the previous iterative procedure. Assuming is differentiable, one can easily check by derivation that " satisfies
K
K
(
K0 ( )
K" ( ) " "=0
Æ
= ; = D v;
and thus defines a consistent, first order accurate, step forward operator for the approximation of a continuous flow given by
= D v: t
K
(2)
Note that " is an alternative to the explicit forward Euler operator " ( ) = + "D v (We refer to [3] for a complete survey on approximation techniques for evolution equations). In this context, the iterative procedure (1) can be regarded as an approximation of a continuous matching flow defined by
K
with
0.9
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0 −6
s=2 s = 10 s = 40
0.1
−4
−2
0 ξ
2
4
6
0 −6
−4
−2
0 ξ
2
4
6
Fig. 1. Frequency domain representation of two families of regularization filters (providing Hs or C1 regularity): (a) Sobolev (s = 1; 0 = 1 = 1; i2 = 0), Gaussian (s = 2; i = 1=(i!2i )), (b) Sobolev for s = 2; 10; 40.
Rr
8
vk vk k+1
0.9
0.1
R r
Æ
1 Sobolev Gaussian
3. FLOWS OF DIFFEOMORPHISMS
= D v; (0; ) = id; t v = R(L(f; (g Æ ))r(g Æ )):
By construction, one expects the solution to this inital value problem to be a one-parameter familly of diffeomorphisms. Previous theoretical developments on similar equations [6, 17] have shown
that this assumption is satisfied if the time-dependent vector field v is sufficientlyR smooth. , for > 0 and a suitable Indeed, if 0 v (t; ) 2B dt < norm , there exists a unique familly of diffeomorphisms B solution of the equation t = v ; (0; ) = id, on [0; ℄. From this result, we can use the chain rule to show that , the familly of diffeomorphisms such that t [0; ℄; (t; (t; x)) = x, is solution of equation (2). From [6; 17℄, we also know that the norm of the Sobolev space s ( ), the set of n-dimensional vector valued functions whose components have square integrable derivatives up to the order s, is a suitable choice for B , provided s > n=2 + 1.
jj jj
jj
jj
1
Æ
8 2
H
jj jj
3.3. Regularization operators
R
Ideally, one should choose a regularization operator that not only provides the suitable s regularity for v (to guarantee the well-posedness of the matching problem), but that is also compatible with some prior knowledge on the smoothness of the underlying deformation field. Operators based on elasticity theory have been proposed in [5] and implemented by convolution with linear filters in [1]. However, this model, borrowed from fluid mechanics, is not always relevant and lacks flexibility for many applications. Here, we propose to set (v ) = ? v (? denotes the convolution with each component of v ) where is a generic linear filter whose Fourier transform is
H
R
F ()() =
" X
i0
i j j2i
#
s=2
:
For instance, if the gradient of the statistical similarity criterion is square integrable, one can simply take s > n=2+1, 0 = 1 = 1, and i2 = 0, in order to obtain an admissible regularized vector field v . In fact, the corresponding filter is directly related to the metric of the Sobolev space s (see [15]). For s = 2 and i = 2i =(i!2i ), corresponds to a Gaussian window of parameter > 0. In this case, the components of the resulting vector field are infinitely differentiable (C1 ) and it always satisfies the regularity requirements. The one-dimensional frequency responses of these filters are presented in Fig 1. Both types of filters have been implemented, using the FFT and a fast recursive approximation [4]. The Gaussian-based regularization seems the most effective in practice. In particular, the role of its parameter is very intuitive to recover local or global deformations, depending on the application.
H
4. EXPERIMENTS We developed a generic C++ implementation of the previous equations including the different similarity criteria. The following results were obtained with a local cross-correlation criterion and a Gaussian based regularization. The local cross-correlation was chosen due to its rather low computational cost and its very good efficiency compared to the best global criteria. The computation times provided below are given for a Pentium III 935 Mhz processor. The first synthetic experiment (Fig. 2) simply shows the ability of our method to extract complex intensity dependencies while handling large deformations. The image size is 256x256 and the computation time of 2 minutes 50 seconds. is represented as a displacement field u such that = id + u. Magnetic resonance imaging offers various acquisition modes that usually yield images with different intensity profiles. In this context, we consider the problem of mapping an EPI volume (used in functional MRI) into the corresponding anatomical reference frame. Note that EPI images often present strong geometric distortions due to magnetic field inhomogeneities in the phase-encoding gradient direction. We first apply our algorithm to monkey brain images (Fig. 3). Starting from an affine initialization, the algorithm recovers a distortion which is essentially located in the upper part of the brain (probably due to nonlinearities induced by the additional antenna used during this acquisition). The computation time was 4 minutes 40 seconds. The algorithm robustness is also tested by applying large artificial deformations to the EPI volume (Fig. 4). On human brain images (Fig. 5 and 6) we obtain deformations of an amplitude up to 5 voxels, mostly in the phase-encoding direction (vertical axis of the transverse view). This solution seems consistent with previous results obtained by Kybic, Th´evenaz et al. [10] on the same dataset. In this case, the computation time was 15 minutes.
(a)
(b)
(c)
(d)
Fig. 3. Monkey brain data (courtesy of Pr. G.A. Orban, D. Fize and W. Vanduffel, Lab. of Neuro- and Psychophysiology, K.U. Leuven). First row: orthogonal slices of the initial T1-w (a) and EPI (b) volumes. Second row: contours of the EPI volume are superimposed on the T1-w acquisition, (c) initial configuration (affine intialization), (d) after realignment. Volume size 81x87x81 voxels.
5. CONCLUSION The main contribution of this paper is the consistent template propagation technique described with equation (1). Our approach combines the robustness of statistical similarity measures with the flexibility of diffeomorphic matching techniques. Preliminary experiments show the efficiency of this approach. Its validation on larger datasets and other modalities is an ongoing work. 6. REFERENCES
(a)
(b)
[1] M. Bro-Nielsen and C. Gramkow, “Fast fluid registration of medical images,” in Visualization in Biomedical Computing, K. H¨ohne and R. Kikinis, Eds., pp. 267–276. Springer, 1996. [2] C. Chefd’hotel, G. Hermosillo, and O. Faugeras, “A variational approach to multi-modal image matching,” in IEEE Workshop on Variational and Level Set Methods, Vancouver, Canada., July 2001, pp. 21–28, IEEE Computer Society. [3] A. Chorin, T. Hugues, M. McCracken, and J. Marsden, “Product formulas and numerical algorithms,” Communications on Pure and Applied Mathematics, vol. 31, pp. 205–256, 1978. [4] R. Deriche, “Fast algorithms for low-level vision,” IEEE Transactions on PAMI, vol. 12, no. 1, pp. 78–87, 1988.
(c)
(d)
Fig. 2. Synthetic example: (a) reference image, (b) template, (c) template after realignment, (d) recovered displacement field.
[5] G. Christensen, R. Rabbitt, and M. Miller, “Deformable template using large deformation kinematics,” IEEE Transactions on Image Processing, vol. 5, no. 10, pp. 1437–1447, 1996. [6] P. Dupuis, U. Grenander, and M. Miller, “Variational problems on flows of diffeomorphisms for image matching,” Quarterly of Applied Mathematics, vol. 56, pp. 587-600, 1998.
(a) (a)
(b) 5
(b)
0
−5
(c)
(c)
(d)
Fig. 4. Consistency of the solution for two large artificial deformations: (a) and (c) initial configurations, (b) and (d) after realignment.
(d)
(e)
Fig. 6. Contours of the anatomical image superimposed on the EPI volume: (a) before and (b) after registration. Fig. (c), (d) and (e): x,y and z components of the final deformation field at grid points corresponding to the previous transverse view (Fig. 5).
[7] U. Grenander and M. Miller, “Computational anatomy: an emerging discipline,” Quarterly of Applied Mathematics, vol. 56, no. 4, pp. 617–694, 1998. [8] G. Hermosillo and O. Faugeras, “Dense image matching with global and local statistical criteria: a variational approach,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Hawaii, 2001. [9] A. Hirani, J. Marsden, and J. Arvo, “Averaged template matching equations,” in Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR’2001), Sophia-Antipolis, France, vol. 2134 of Lecture Notes in Computer Science, Springer. [10] J. Kybic, P. Th´evenaz, A. Nirkko, and M. Unser, “Unwarping of unidirectionally distorted EPI images,” IEEE Transactions on Medical Imaging, vol. 19, no. 2, pp. 80–93, 2000. [11] J. Maintz and M. Viergever, “A survey of medical image registration,” Medical Image Analysis, vol. 2, no. 1, pp. 1–36, 1998.
(a)
(b)
[12] M. Miller and L. Younes, “Group actions, homeomorphisms, and matching : A general framework,” International Journal of Computer Vision, vol. 41, no. 1-2, pp. 61–84, 2001. [13] D. Mumford, “Pattern theory and vision,” Lecture notes, Centre Emile Borel, Paris, 1998. [14] A. Roche, G. Malandain, X. Pennec, and N. Ayache, “Multimodal image registration by maximization of the correlation ratio,” Technical report 3378, INRIA, August 1998. [15] M. Taylor, Partial Differential Equations, Springer, 1996.
(c) Fig. 5. Human brain data (courtesy of Jan Kybic, data provided by Arto Nirkko, Inselspital Bern): (a) anatomical image, (b) EPI image, (c) EPI image after registration (see Fig. 6 for image details). Transverse view, central slice, volume size of 256x256x30 voxels (the EPI volume is resampled).
[16] A. Toga and P. Thompson, “The role of image registration in brain mapping,” Image and Vision Computing, vol. 19, no. 1-2, pp. 3–24, 2001. [17] A. Trouv´e, “Diffeomorphisms groups and pattern matching in image analysis,” International Journal of Computer Vision, vol. 28, no. 3, pp. 213–21, 1998. [18] W. Wells III, P. Viola, H. Atsumi, S. Nakajima, and R. Kikinis, “Multi-modal volume registration by maximization of mutual information,” Medical Image Analysis, vol. 1, no. 1, pp. 35–51, 1996.
