Proceedings of the 20th Annual International Conference of the ZEEE Engineering in Medicine and Biology Society, Vol. 20, No 2,1998
Automatic Retinal Registration using Global Optimization Techniques N. Mouravliansky, G. K. Matsopoulos, K. Delibasis and K. S. Nikita Department of Electrical and Computer Engineering, National Technical University of Athens, E-mail:
[email protected]. Only the translation component of motion was determined neglecting any rotation effects. Furthermore, an interactive method was employed to register early and late frames of FA to assess macular oedema, based on the gradient of fluorescein intensity [5]. A non-real-time registration on monochromatic photographic and FA images by non-optimized simple template matching or Hausdorff-distance-based algorithms was developed considering only translation and scaling factors [6]. The log-polar transformation of the spatial frequency spectrum of FA images was employed to achieve automatic registration, using translation, rotation and uniform scaling [7]. The use of this transformation removes the effects of translation and converts rotation and uniform scaling into independent shifts in orthogonal directions. However, the requirement for non-uniform scale transformation models (e.g. affine, bilinear or projective) is often encountered when registering retinal images.
Abstract - In this paper, a new automatic scheme to register retinal images is presented and is currently tested in a clinical environment. The scheme considers the suitability and efficiency of different image transformation models and function optimization techniques, following an initial preprocessing stage. Three different transformation models -Affine, Bilinear and Projective- as well as two global optimization techniques, - Simulated Annealing and Genetic Algorithms- are investigated and compared in terms of accuracy and efficiency. The registration of 26 pairs of Fluoroscein Angiography (FA) and Indocyanine Green Chorioangiography (ICG) images with the corresponding Red Free (RF) retinal images, showed the superiority of combining Genetic Algorithms with the Affine and Bilinear transformation models.
Key words: retinal registration, traivformation models, simulated annealing, genetic algorithms
AND PREPROCESSING OF RETINAL IMAGES 11. ACQUISITION
I. INTRODUCTION Ophthalmologists commonly compare Red-Free (RF) retinal images, which is a reference image taken without intravenous injection of a dye while illuminating the retina with a green light, with the corresponding FA or ICG image of the patient, acquired at different times. This is a very difficult task due to the misalignment of the retinal images caused by the geometry during the acquisition at different times and to possible progression of various diseases. The relative study of retinal images enhances the information on the reference RF images by superimposing useful information contained in FA or ICG retinal images and it is considered an important step towards a carefully directed laser treatment; a process that is commonly used in the clinical practice. Human-interactive registration is considered as a standard method, widely used in the clinical practice. Automatic retinal registration techniques have been developed to overcome failures due to human interaction. An automatic registration of sequential FA frames was firstly reported by enhancing edges and vessel crossings and cross-correlating small windows around these features to determine optimal matching [l]. Another technique was based on identifying vessels using an adaptive thresholding technique and then applying a feature based sequential detection method to find the best match between the vessels [2]. Sequence of retinal images was also registered locally by finding corresponding triangles formed by vessel bifurcation points [3]. Registration experiments were carried out on low quality photographs of the ocular fundus, obtained with a contrast medium, implementing a phase correlation algorithm with and without a weighting function to meet the FFT hardware requirements for real time processing [4]. 0-7803-5164-9/98/$10.00 0 1998 IEEE
i) Acquisition of retinal images Retinal images were acquired using the IMAGEnet 1024 system, which is a fully functional digital imaging system for acquisition, analysis, storage and retrieval of retinal images. Digital FA and ICG images of size 1024 x 1024 were directly obtained using a CCD camera mounted on a Topcon TRX-SOX fundus camera. The use of this motorized common camera allows image acquisition at a rate of one image every 0.8 seconds. The acquired retinal images are also driven from the CCD camera to a Silicon Graphics (SGI) workstation, where the developed automatic registration scheme is currently running. The automatic and the manual registration were tested on 512 x 5 12 retinal images to reduce the execution time.
ii) Preprocessing of retinal images In the case of retinal images, the objects of interest are the retinal vessels: arteries and veins. It is found that the registration algorithm operates more efficiently if the vessels are segmented, using at least a crude segmentation. Therefore, image preprocessing is required prior to registration. A simple thresholding is not sufficient, since retinal vessels and background structures are of comparable intensity. The preprocessing involves two main processes with the addition of an inversion procedure applied only to the RF images in order to emulate the appearance of FA and ICG images: a) vessel enhancement and border suppression using a mathematical morphology operation according to the following equation:
567
zenhunccd = lorfgfnul
- (zor,ginu/ OB)
)
where 0 denotes the erosion operation and B is the 7 x 7 flat structuring element. The resulting image presents significantly suppressed background while enhances vessel visibility. It is also noted that the morphological operation suppresses pixels at the retinal image border caused by the aperture of the fundus camera. b) vessel detection by a matched filter, which is convoluted with the images to be registered [SI The kernel is of size n x n, with the vessels modeled as line segments with a length of L pixels is defined as follows:
K’(x, Y ) = K ( x , v)- m (2) 1 where m = - x K ( x , y )is the mean value of the kernel. A Kernel The modified kernel can detect long vessel segments in a specific direction. Assuming an angular resolution of 30 degrees and by rotating the kernel 5 times with this step, the algorithm performs the convolution 6 times and detects vessels of arbitrary orientation. Fig. 1 shows the segmented retinal vessels of FA retinal images.
a
transformed image, without preserving the parallelism. It strongly resembles the bilinear and is completely defined by 9 independent parameters. ii) The objectivefunction: Measure of Match In the case of the retinal images, after the preprocessing, two binary images of the retinal vessels are obtained, ZA and I,. The problem is formulated as determining a transformation that, when applied to the second image, a best match with the first is achieved. Ideally all the nonzero pixels of the transformed image should be mapped on non-zero pixels of the first image. The problem can be mathematically formulated as the maximization of the following objective function:
where MOM denotes the Measure Of Match (objective function) and Tx and Ty are the transformations for the x and y coordinates and n is the number of non-zero pixels of IB. Assuming that the non-zero pixels of the preprocessed images have a value of 255, then the absolute maximum (optimal) value of the above quantity equals 255. In general, the maximization achieved is significantly lower. The reason is not the optimization method’s inefficiency, but the fact that the two images are in most of the cases not identical and may also contain noisy pixels. In this case, there is no transformation that can match every non-zero pixel Of IB to non zero pixels of ZA. iii) Global optimization techniques The most attractive solution for search methods using nontrivial transformations is based on global optimization techniques. In this work we consider the use of two global optimization techniques, Simulated Annealing (SA) [91 and Genetic Algorithms (GAS) [lo]. The transformation models employed in the case of 2-D retinal images, introduce 6 , 8 and 9 independent parameters (corresponding to the affine, bilinear and projective transformation models, respectively), defined over a wide range of values to achieve robustness. This fact, combined with the presence of multiple local extrema of the objective function, necessitates the use of global optimization techniques.
b
Figure 1. Coarse segmentation of the retinal vessels: (a) original FA image and (b) its final segmentation.
111. REGISTRATION OF RETINAL IMAGES The process of image registration can be formulated as a problem of optimizing a function that quantifies the match between the original and the transformed image. In the case of registering retinal images from different modalities (RF with FA or ICG), the need for an accurate registration with less human interaction, the absence of clear reference anatomical regions and the low quality of the retinal images, suggest the use of robust global algorithms. More specifically, we are concentrated on the determination of the translation and rotation displacements as well as on uniform and non-uniform scaling deformation occurred during retinal image acquisition from different modalities.
a) Simulated Annealing (SA): presents an optimization technique that can process cost functions with arbitrary degrees of non-linearities and arbitrary boundary is conditions. A function of a system’s state, fi(?), minimized using a process called Annealing. The system searches for optimal values of the function
c j Transformation models
The AfJine Transformation: can be decomposed into a linear transformation and a simple translation and is completely defined by 6 independent parameters. The Bilinear Transformation: is the simplest polynomial transformation, which maps straight lines from the original image to curves and is completely defined by 8 independent parameters, The Projective Transformation: maps any straight line in the original image onto a straight line in the 568
f(?), while adding to the function a noise component, whose magnitude is a descending function of time. For every minimization problem, a quantity T, known as “Temperature”, is defined as a descending function of time k and the function T(k) is called Annealing Schedule. The Annealing Schedule implemented in this work is the Adaptive Exponential Simulated Annealing (AESA) where:
T ( k ) = T, exp(k(c - 1))
at the “Ophthalmological Diagnostic, Therapeutic and Research Center” in Athens.
(4)
Genetic Algorithms (GAS): The method starts by creating a population of random solutions of the optimization problem. A solution to the problem usually consists of the values of the independent parameters of the function to be optimized (objective function). These values are often converted to binary and concatenated to a single string, called individual. In several cases it has been shown that this conversion does not offer substantial advantages and real encoding is used instead. This process is formulated in pseudocode as follows: initialize theJirst generation of n individuals randomly while (termination-condition is false) calculate the objectivefunction of the n individuals select N/2 pairs of individuals apply crossover and mutation operator to produce offspring replace the current generation by the n offspring
I AESA vs GAS: A comparative study of the three transformation models (affine, bilinear and projective) in conjunction with the two global optimization techniques (SA and GAS) is performed on 26 retinal image pairs. In order to obtain comparable results, the following assumptions are made: a) same total number of function evaluations (in our case 20,000) and b) same allowed range for the independent parameters of the transformations. It becomes evident that the GAS achieved better optimization than the SA for 23 retinal image pairs using the affine and bilinear transformation models. In Fig. 2(a), the two optimization methods, SA and GAS, using the affine transformation, and the three transformation models using the GAS (Fig. 2(b)), are compared by plotting the values of the objective function (measure of match) against the number of function evaluations, for a retinal pair. The measure of match is averaged for 10 different program executions. It becomes evident that the GAS performed better than the SA with significantly higher convergence rate.
Figure 3: (a) preprocessed RF image (white colored vessels); (b) preprocessed FL image (gray colored vessels); (c) registration performed by the best individual of the first generation of the GAS; (d) final result of the registration (produced by the best individual of all generations).
The automatic registration scheme is based on the bilinear transformation, in conjunction with GAS and applied on 512 x 512 retinal images. The parameter ranges of the transformation model are kept as they were defined during the experimental process of the scheme and the total number of function evaluations has been reduced to 10,000. Fig. 3 shows the result of the automatic registration scheme on RF and FL retinal images. The automatic registration scheme is also applied on a pathologic pair of RF and FL images of a male patient suffering from Harada’s Syndrome. Fig. 4(a) shows the registered FL image of the patient with three boundaries traced by the expert: one corresponding to the fovea (best fitting ellipse) and the other two to the areas of the abnormality (free drawing).
slmilanty Measure
0
Number of Evaluation (x 10000)
05
1
IS
2
Numberof E~aluation(X 10000)
a b Figure 2: Performance of the automatic registration technique for a retinal image pair: (a) GAS (continuous line) vs SA (dotted line); (b) Performance of the three transformation models (affine (dotted line) vs bilinear (continuous line) vs projective (dashed line)) with GAS.
IV. CLINICAL RESULTS The proposed automatic retinal registration scheme is currently running on an Indigo’ SGI workstation, installed
Figure 4: (a) a registered pathological FL image with three boundaries traced by an expert; (b) superposition of the boundaries on the corresponding RF image.
These boundaries are easily identified only in the FL image and are of extreme importance in directing the laser treatment procedure. Since the laser treatment planning is
569
based on the RF image of the retina, these boundaries are traced by the ophthalmologist on the registered FL image and siinultaneously placed on the corresponding RF image, displayed besides, according to the implementation of the scheme (Fig. 4(b)). The automatic retinal registration scheme is compared to the manual registration for all the 26 retinal pairs. The disadvantages of this human-interactive approach include inaccuracy in the placement of special reference points, inconsistency of the registration results and increased interaction time between the expert and the system. Fig. 5 shows randomly selected areas of registered FA images superimposed on the corresponding RF images, for a pair, using the automatic, in Fig. 5(a), and the manual, in Fig. 5(b), registration techniques. It cm be observed that the manual registration failed to preserve the continuity of the vessels along the borders of the superimposed regions, whereas the proposed automatic approach achieved registration of all vessels.
registration algorithm seems to perform local than global registration, whereas when the landmarks are distributed over the whole image, the manual registration seems to improve. This confirms the inconsistency characteristic of the method due to human intervention. V. CONCLUSIONS An automatic retinal image registration scheme was presented in this paper. Results were presented in the form of the best-achieved registration, both numerically and visually, for a sufficient number of data, applied on three different retinal images. The experimental results showed the superiority of Genetic Algorithms as a global optimization process in conjunction with the Affine or Bilinear transformation models. The automatic registration scheme was implemented on SGI workstation and installed on a clinical setting, requiring minimum human interaction. In terms of execution time, the manual registration method requires a minimum of 3.5 min for each retinal image pair whereas the automatic registration requires an overall execution time of 4,5 min, for 5 12 x 5 12 retinal images.
(b)
(a)
Figure 5: Comparison between the automatic and the manual registration of RF and FA retinal images: (a) automatic and (b) manual registration. A quantitative analysis between automatic and manual retinal registration methods has also been performed. Three independent experts, using the IMAGEnet system, performed manual registration of all thie 26 retinal pairs, and the average value of objective function (Eq. 3) for the three trials and for all pairs, was evalluated. The results were then compared to the average objective function values obtained by the automatic registration scheme (10 independent executions) and are displayed in Fig. 6, against all the retinal image pairs.
04 : : : : : : : : : r
O
I
O
F
(
l
l
; r
~
: : : : : O
w
C
dmb& M k k ;
y
z
I
Figure 6: Comparison between the automatic andl the manual registration for 26 retinal image pairs.
The values of the objective function of d l manual trials are significantly lower than the one achieved by the automatic registration scheme for all retinal image pairs (except for the Pair 9). This confirms the advantage of the proposed automatic registration scheme in terms of accuracy. It was also noticed that the values of the objective function obtained by the three manual registration trials differ substantially for all retinal image pairs. When the landmarks are placed close to each other, the manual 570
REFERENCES P. Nagin, B. Scwartz, and G. Reynolds, “Measurement of fluoroscein angiograms of the optic disc and retina using computerized image analysis”, Ophthalmology, V O ~ .92, pp.547-552, 1985. E Peli, R. Augliere, and G. Timberlake, “Feature-based registration of retinal images”, IEEE Trans. Med. Imaging, vo1.6, pp. 272-278, 1987. R. Jagoe, J. Amold, C. Blauth, P. Smith, K. Taylor, and R. Wootton, “Retinal vessel circulation patterns visualized from a sequence of computer-aligned angiograms”, Invest Ophthalmology Vis. Sci., vol. 34, pp. 2881-2887, 1993. E. de Castro, G. Christini, A. Martelli, C. Morandi, and M. Vascotto, “Compensation of random eye motion in television opthalmoscopy: preliminary results’”, IEEE Trans. Med. Im., vol. 6, no. 1, pp. 74-81, 1987. R. Philips, T. Spencer, P. Ross, P. Sharp, and J. Forrester, “Quantification of diabetic maculopathy by digital imaging of the fundus”, Eye, vol. 5, pp. 130137,1991. J. W. Berger, M. E. Leventon, N. Hata, W. Wells 111, and R. Kikinis, “Design considerations for a computervision-enabled ophthalmic augmented reality environment”, In CVRMED/MRCAS, Grenoble, France, 1997. A. Cideciyan, “Registration of ocular fundus images”, IEEE Engng. in Med. Biol., vol. 14, pp. 52-58, 1995. S. Chaudhuri, S. Chatterjee, N. Katz, M. Nelson, and M. Goldbaum, “Detection of blood vessels in retinal images using two-dimensional matched filters”, IEEE Trans. Med. Im., vol. 8, no. 3, pp. 263-269, 1989. E. Aarts, and Van Laardhoven, “Simulated Annealing: theory and practice”, John Wiley and Sons, New York, 1987. [ 1 O]D. Goldberg, “Genetic Algorithms in optimization, search and machine learning”, Addison-Wesley, 1989.
