Multimodal image registration for potential diagnosis and monitoring of ...

2016 1st International Conference on Biomedical Engineering (IBIOMED), Yogyakarta, Indonesia

Multimodal Image Registration for Potential Diagnosis and Monitoring of Morphoea using a Hybrid NGC Method L. I. Izhar, I. Elamvazuthi

T. Stathaki

Dept. Of Electrical and Electronic Engineering Universiti Teknologi PETRONAS Bdr. Seri Iskandar, Malaysia [email protected]

Dept. of Electrical and Electronic Engineering Imperial College London London, United Kingdom

K. Howell

Z. Omar

Institute of Immunity and Transplantation University College London London, United Kingdom

Dept. of Electronic and Computer Engineering

Abstract— In medicine, thermal imaging diagnostic tool is increasingly employed thanks to its low cost, non-harmful and non-invasive nature. However, a thermogram can be quite ambiguous due to its low spatial resolution. This ambiguity can be reduced by incorporating information or data from different imaging sensors. This paper is an extension of a modified Normalized Gradient Correlation (NGC) employed in the initial phase of this study for multimodal image registration to assist in diagnosis and monitoring of linear morphoea. The proposed method is an improved, hybrid version of the modified NGC that incorporates an iterative based normalized cross-correlation coefficient (NCC) method for retrieval of translational differences based on the spatial domain in the initial method. The hybrid NGC method is found to reduce misregistration due to inaccurate retrieval of translational differences suffered by the initial NGC method in this multimodal image registration by up to 77.4% for over-detection error. Keywords— Thermogram, image registration, normalized gradient cross-correlation, morphoea, Blaschko’s lines

I. INTRODUCTION Integration between multimodal images is increasingly employed in diverse applications especially in medicine, computer vision, remote sensing and military to name a few [7, 11, 17]. By integrating important features between multimodal images to complement one image to another, further analysis can be carried out in a more accurate and efficient manner [6, 8, 14]. Thus, image registration is a very important first step towards a successful integration as it transforms the different sets of data into one coordinate system [10, 11]. Image registration is the key element in all image analysis tasks where the integration of various data sources is the final outcome. Fourier transform based methods [5, 13, 15],

Universiti Teknologi Malaysia Johor, Malaysia correlation matrix based methods [1], and feature based methods [3, 8] are some of the most applied methods in image registration. With a Fast Fourier Transform (FFT) algorithm, more and more methods based on Fourier Transform have been used. This is also contributed by the resilience of the FFT to random noise as well as to uneven illumination unlike the correlation matrix based methods [17]. The Fourier transform based methods are also more considered in image registration than the others as it is able to solve for the best correlation between frequency domain features. Information provided by the Blaschko’s lines has been the main focus of this work [18] (see Fig. 1(c)) for not only to improve the diagnosis of morphoea (a fibrosing disorder of the skin and underlying tissues), but also for monitoring of its treatment efficacy. Weiber and Harper [16] have shown that linear morphoea lesions follow the Blaschko’s lines based on visual correlation performed on 65 patients. As morphoea in children can result in irreversible structural deformities, hence it is of paramount important to detect the disease in its early stage to avoid cosmetic and functional complications. Hence, this paper discusses a hybrid version of the modified normalized gradient cross-correlation method in [18] (hybrid NGC) proposed for initial registration between face thermograms and a face sketch (Fig.1) towards integration of the Blashko’s lines and face thermograms. This integration of information not only able to reduce the ambiguity of a thermal image but also able to assist in the current diagnosis and monitoring of morphoea as the lines represent lesion patterns of linear morphoea.

978-1-5090-4142-8/16/$31.00 ©2016 IEEE


cases, the collar shirt were mistaken as the jawline hence the misregistration.

(a) (b) (c) Fig. 1: Multimodal registration between (a) a face thermogram, and a face sketch (b) without the Blaschko’s lines and (c) with the Blaschko’s lines (reproduced from [16]). (a)

II. METHODOLOGY A. Pre-Processing Registration between the two different image modalities such as between a thermogram and an optical image or a drawing, suffers from the problem of having non-direct relationship in their pixel intensities. Hence, in this work, edge detection using Sobel [5, 11, 18] is performed onto thermograms prior to image registration as it can efficiently detect discontinuities in low noise images [12]. By using the salient features of the thermograms, problems related to the low-pass nature of a thermogram can also be solved. Similarly for the face sketch, its negated image will be used as the edge map. Next, edge thickening by dilation with a 3 by 3 square structuring element is carried out in order to spread the information conveyed by each edge map and facilitate the registration process. The distribution of the thickened lines is further spread by a Gaussian smoothing filter [12] to increase the potential for pixel overlapping between the edge maps of thermograms and the face sketch. Now, smoother, more distributed and spreaded edge maps as shown in Fig. 2 are obtained. Next, Tukey window which is a cosine tapered window [2] is applied onto the edge maps to sufficiently suppress error due to leakage upon FFT processing.

(d)

(c)

Fig. 3 Examples of good registration results achieved; (a) and (b), and misregistration results achieved; (c) and (d), by the modified NGC method [18].

In order to solve this problem, an approach based on incorporating an iterative normalized cross-correlation coefficient (NCC) technique performed in the spatial domain to replace the final registration part of the NGC method whereby translational differences are solved based on the NGC formula [18] in the spatial Fourier domain, was employed. It is found that the performance and robustness of the NGC method especially in the retrieval of translational differences is greatly improved (see Fig. 4). Hence, the NGC method that incorporates the NCC technique to recover translational differences in the spatial domain known as the hybrid NGC is investigated in this work. A fast normalized cross-correlation (NCC) algorithm developed by Lewis [9] is employed using image gradients to give estimation of translational differences once the scaling and/or angle differences from the target image are removed by the modified NGC method explained earlier. In this algorithm, the NCC is defined as ,

(a)

(b)

∑ , ∑ ,

̅ ,

, ̅ ,

,

̅

, ∑ ,

̅

,

(1)

(b)

Fig. 2: Pre-processing result, edge maps of; (a) thermogram and (b) face sketch.

B. Hybrid Normalized Gradient Cross-Correlation (Hybrid NGC) In the initial part of this study as discussed in [18], the NGC method, which is a modified version of a phase correlation (PC) method [1, 4, 11, 13] has been employed with some modifications to register between images of different modalities and subjects (see Figs. 3 (a) and (b)). In this multimodal image registration that involves different subjects, it is found that the NGC method [18] suffers from misregistration mostly due to inaccurate translational differences being recovered for some cases of thermograms as shown in Fig.3 (c) and Fig.3 (d). The misregistration for the two thermograms clearly show that the translational differences recovered were not accurate. In both

where , is the reference image and t is the target image, ̅ , denotes the mean value of , in the region under the t, while ̅ denotes the mean value of the target image, t. Replacing both the images with their gradient magnitudes , , (1) takes the form ,

∑ , ∑ ,

, ,

,

, ,

,

∑ ,

,

(2)

In this work, the magnitude of gradients for both the reference and the rescaled and/or rotated target image are used as it gives pixel locations strong edge responses and suppress the contribution of uniform areas which do not provide any reference points for motion estimation. The translational


differences (u,v) is found by taking the maximum value of , . Using a fast algorithm such as designed by Lewis [9] for the NCC, makes the method becomes more attractive to be used for image matching, feature detection and opens up other new applications. This new approach that combines the NGC method performed in the Fourier domain for the retrieval of scaling factor and/or rotational angle with the NCC algorithm performed in the spatial domain for the retrieval of translational differences, known as the hybrid NGC, is proposed in this work. III. COMPARISON ANALYSIS BETWEEN THE NGC AND THE HYBRID NGC METHOD

This section presents comparison analysis between the hybrid NGC method and NGC method [18] in the multimodal image registration. The face thermograms are acquired from 11 normal subjects at 7 time-points during a day for each subject using VarioCAM hr head 700 by InfraTec with a thermal sensitivity of up to 30mK or 0.03°C. The pixel resolution of the face thermograms and the face sketch are 386×290 and 348×306 respectively. Figure 4 shows examples of improvements in the registrations for two subjects where the NGC method has failed in recovering for the correct translations.

(a)

(b)

Overall, based on the error measures, the hybrid NGC method outperforms the NGC method by achieving less underdetection error (by up to ~9.2%) and over-detection error (by up to ~77.4%) in the multimodal image registration between face thermogram and the face sketch. Table 1 compares between the two methods in terms of their mean errors. In [18], we have compared between the NGC method and the PC method in solving this multimodal registration problem. It has been shown that the NGC method outperforms the PC method especially in recovering the scale difference between the multimodal images. The good performance of the NGC method in this aspect (by up to 1.77 of scaling factor achieved) has contributed to the good performance of the hybrid NGC method as removal of scale difference from target image by the NGC method is very crucial in ensuring an accurate retrieval of translational differences by the NCC method in the hybrid NGC method. By using the same registration parameters, the Blascho’s lines can be incorporated with the face thermogram as can be seen in Fig. 5(b) to improve the current diagnosis and monitoring of morphoea. TABLE I.

MISREGISTRATION ANALYSIS BETWEEN THE HBRID NGC METHOD AND THE NGC METHOD

Error Measures

Mean

Std.

NGC

Hybrid NGC

NGC

Hybrid NGC

eu

0.61

0.54

0.1000

0.0891

eo

0.43

0.12

0.1265

0.1120

(c)

(d) (e) (f) Fig. 4: (a) and (d) are misregistered images obtained by the NGC method. (b) and (e), and (c) and (f) are improved registration results achieved by the hybrid NGC method represented by edge maps and intensity images respectively.

For this analysis purpose, error measures based on underdetection error, eu, and over-detection error, eo are computed; edge map of registered face sketch S(x,y) is compared against manually outlined edge maps of its reference image which in this case is thermogram, R(x,y). To compute eu and eo, the followings are performed based on 8-pixel neighbourhood [18]. 1. For every edge pixel (EP) in S(x,y), with coordinates (i,j), if there is an edge pixel with the same coordinates in R(x,y) or any of its neighbouring 8 positions, count it with a counter A; Ʃ , , , , or else, count it with a counter B; Ʃ , , ∩ , . Then, the under-detection error, eu , is computed as B/(A+B). 2. For every edge pixel in , , with coordinates , , if there is an edge pixel with the same coordinates in , or any of its neighbouring 8 positions, count it with a counter C; Ʃ , , , , or else, count it with a counter D; D; Ʃ , , ∩ , . Then, the over-detection measure, , is computed as D/(C+D).

(a)

(b)

Fig. 5: (a) The face sketch with the Blaschko’s lines (b) The Blascho’s lines incorporated with the thermogram.

IV. DISCUSSIONS AND CONCLUSIONS By performing the NGC in the so-called hybrid approach, it is shown that the registration errors can be reduced significantly especially for the over-detection error by up to 77.4% with a mean value of 0.12. This is the result of reducing inaccuracies in retrieval of translational differences achieved by incorporating the NCC technique in the modified NGC [18]. However, to achieve further improvement in the registration between multimodal images, there is a need for the registration to be approached by a non-rigid transformation based method or a local registration method as this involves images of different nature. On the other hand, this multimodal registration problem may also be approached by using an atlas image to replace the face sketch. This atlas image can be created by combining optical images of all the subjects involved in the study using a group-wise based registration method.


REFERENCES [1]

[2] [3] [4] [5]

[6] [7] [8]

[9]

[10] [11] [12] [13]

[14]

[15]

[16] [17] [18]

D. I. Barnea and H. F. Harvey, “A class of algorithms for fast digital image registration”, IEEE Transactions on Computers, C-21(2):179-186, 1972. P. Bloomfield, Fourier Analysis of Time Series: An Introduction, WileyInterscience, New York, 6:69, 2000. L.G. Brown, “A survey of image registration techniques”, ACM, Computing Surveys, 24:326-376. D. Casasent and D. Psaltis, “Position, rotation, and scale invariant optical correlation”, Applied Optics, 15:1795-1799, 1976. C. E. Costa and M. Petrou, “Automatic registration of ceramic tiles for the purpose of fault detection”, Machine Vision and Applications, 11: 225-230, 1999. A. Goshtaby, “2-D and 3-D image registration for medical, remote sensing, and industrial applications”, Wiley Press, 2005. J. V. Hajnal, D. L. G. Hill and D. J. Hawkes. Medical Image Registration, The Biomedical Engineering Series, CRC Press, 2001. A. Jarc, J. Pers, P. Rogelj, M. Perse and S. Kovacic, “Texture features for affine registration of thermal (FLIR) and visible images”, Computer Vision Winter Workshop, Graz Technical University, St. Lambrecht, Austria, February 6-8, 2007. J. P. Lewis, "Fast normalized cross-correlation," Industrial Light & Magic. An extended version of J. P. Lewis, “Fast template matching,” Vis. Inf., pp. 120–123, 1995. L. I. Izhar, M. Petrou. Thermal Images in Medicine. Advances in Imaging and Electron Physics, 171(2): 41-114, 2012. M. Petrou, “Image registration: an overview”, Advances in Imaging and Electron Physics, 130:243-291, 2004. M. Petrou and C. Petrou, Image Processing: the fundamentals, 2nd Edition, John Wiley, 2010. B. S. Reddy and B. N. Chatterji, “An FFT-based technique for translation, rotation and scale-invariant image registration”, IEEE Transactions on Image Processing, 5(8): 1266-1271, 1996. G. Schaefer, R. Tait, K. Howell, A. Hopgood, P. Woo and J. Harper, “Automated overlay of infrared and visual medical images”, User Centered Design for Medical Visualization, 8:174- 183, 2008. G. Tzimiroupoulos, V. Argyriou, S. Zafeiriou and T. Stathaki, “Robust FFT-based scale-invariant image registration with image gradients”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(10):1809-1908, 2010. L.Weibel and J.I. Harper, “Linear morphoea follows Blaschko’s lines”, British Journal of Dermatology, 159:175-181, 2008. B. Zitova and J. Flusser, “Image registration methods: a survey”, Image and Vision Computing, 21:977-100, 2003. L. I. Izhar, T. Stathaki, K. Howell, “Global based Thermal Image Registration for Diagnosis of Morphoea”, Paper accepted for International Conference for Innovation in Biomedical Engineering and Life Sciences (ICIBEL 2015), 7-8 November 2015.