A novel multi-focus image fusion algorithm based on ...

A novel multi-focus image fusion algorithm based on feature extraction and wavelets Rodrigo Navaa , Boris Escalante-Ram´ırezb and Gabriel Crist´obalc a Posgrado

en Ciencia e Ingenier´ıa de la Computaci´on, Universidad Nacional Aut´onoma de M´exico, M´exico b Dep. de Procesamiento de Se˜ nales, Facultad de Ingenier´ıa, Universidad Nacional Aut´onoma de M´exico, M´exico c Instituto de Optica ´ “Daza de Vald´es” (CSIC), Serrano 121, Madrid, 28006, Spain ABSTRACT Focusing cameras is an important problem in computer vision and microscopy. Due to the limited depth of field of optical lenses in CCD devices, there are sensors which cannot generate images of all objects with equal sharpness. Therefore, several images of the same scene have different focused parts. One way to overcome this problem is to take different in-focus parts and combine them into a single composite image which contains the entire focused scene. In this paper we present a multi-focus image fusion algorithm based on feature extraction and wavelets. Classical wavelet synthesis is known to produce Gibbs phenomenon around discontinuities. The approach of wavelet on the interval transform is suitable to orthogonal wavelets and does not exhibit edge effects. Since Canny filter’s operator is a Gaussian derivative, a well known model of early vision, we used it to get salient edges and to build a decision map who determines which information to take and at what place. Finally, quality of fused images is assessed using both traditional and perception-based quality metrics. Quantitative and qualitative analysis of the results demonstrate higher performance of the algorithm compared to traditional methods. Keywords: image fusion, wavelet transform, feature extraction, perceptual quality metrics

1. INTRODUCTION At present time, image fusion is widely recognized as an important tool for improving performance in imagebased applications such as remote sensing, machine vision, medical imaging, optical microscopy and so on. Image fusion allows merging images from multiple sensors or even multiple images from the same sensor. Its goal is to integrate complementary and redundant information to provide a composite image which could be used to better understanding of the entire scene. Image fusion can be defined as the process by which two or more images or some of their features are combined together to form a single image more suitable for human and machine perception or further task such as segmentation, feature extraction and object recognition.1 In this paper, we considered multi-focus images, i.e, situations where two or more images that depict the same scene will not be in-focus everywhere(if one object is in-focus, another one will be out of focus). This occurs because there are sensors which cannot generate images of all objects at various distances with equal sharpness. Another example of multi-focus images is found in optical microscopy. Limited depth of field is a common problem with conventional light microscopy. Since the specimen can be sectioned by moving the object along the optical axis, portions of the objects’ surface outside the optical plane appear defocused in the acquired image.2 The advantages of multi-focus images can be fully exploited by merging the sharply focused regions into one image that will be in-focus everywhere.3, 4 Further author information: (Send correspondence to R.N.) R.N.: E-mail: [email protected] B.E.: E-mail: [email protected], Telephone (52)55-56223213, Fax: (52)55-56223579 G.C.: E-mail: [email protected], Telephone (34)91-5616800(x2313), Fax: (34)91-5645557 Optical and Digital Image Processing, edited by Peter Schelkens, Touradj Ebrahimi, Gabriel Cristóbal, Frédéric Truchetet, Proc. of SPIE Vol. 7000, 700028, (2008) · 0277-786X/08/$18 · doi: 10.1117/12.781403

Proc. of SPIE Vol. 7000 700028-1

In the previous literature, many techniques for multi-focus image fusion have been suggested. An overview on this issue can be found in.5, 6 Some methods such as wavelet-based algorithms and the Hermite transform have shown better results for visual interpretation.7, 8 Wavelet methods are widely employed, classical approach uses quadrature mirror filter banks (QMF) and Fourier transform (FT), but exhibits edge effects and artifacts due to aliasing in the fused image.9, 10 In this work, we used a orthonormal wavelets on the interval construction in order to avoid some of disadvantages of the classical approach.11 The existing multi-focus image fusion methods can be classified into several groups. A common classification is to distinguish three different levels according to the stage at which the fusion takes place: pixel, feature and decision level.12 • Pixel level fusion generates a fused image in which information content associated with each pixel is determined from a set of pixels in source images. Fusion at this level can be performed either in spatial or in frequency domain. However, pixel level fusion may conduct to contrast reduction.13 • Feature level fusion requires the extraction of salient features which are depending on their environment such as pixel intensities, edges or textures. This similar features from the input images are fused. This fusion level can be used as a means of creating additional composite features. The fused image can also be used for classification or detection.14 • Decision level is a higher level of fusion. Input images are processed individually for information extraction. The obtained information is then combined applying decision rules to reinforce common interpretation.15 An important preprocessing step in image fusion is image registration. Corresponding pixel positions in the source images must refers to the same location. In this paper, we assumed that source images are already registered. In the next section a new feature level fusion algorithm is proposed. Firstly, multiresolution wavelet is presented and then the detailed algorithm is given. Section 3 presents some experimental results. Finally, conclusions are given in section 4.

2. MULTI-FOCUS IMAGE FUSION First, we give a brief description of the fusion methodology based on the multiresolution analysis (MRA). An image often contains physically salient features at many different scales or resolutions. There is strong evidence that the human visual system (HVS) processes information in a MRA fashion and it is especially sensitive to contrast changes.4 At the beginning of analysis all scales or resolutions have the same importance. MRA scheme decomposes input images into several coefficients each of which captures information present at a given scale. One major advantage of MRA is that spatial as well as frequency domain localization of an image is obtained simultaneously. The basic idea in MRA is to decompose input images at first by applying wavelet transform. Then the fusion operation on the transformed images is performed and finally the fused image is reconstructed by inverse transform that restores intensity values and spatial resolution. See Fig. 1. In addition to this, wavelet transform of an image provides a multiple scale pyramid decomposition. There are four coefficients after each decomposition. These are Low-Low, Low-High, High-Low and High-High coefficients. The next decomposition stage operates only on Low-Low coefficient. See Fig. 1(a). General theory of MRA assumes that signals or images have infinite length. If the MRA analysis is applied to images with finite support, undesirable edge effects will occur. However, filtering operations require values of the images outside its supported range. Images can be extrapolated by means of periodic, symmetric or polynomial based methods, which allow filters to be applied on the images ∈ L2 (R). Also, special filters can be designed to replace original filters at the image borders.11 These filters are adapted to interval boundaries and do not require values outside interval. In this work we have implemented a polynomial method for orthogonal wavelets that appears in.16

Proc. of SPIE Vol. 7000 700028-2

(a) Analysis stage

(b) Synthesis stage Figure 1. Block diagram for MRA scheme.

2.1 Fusion Algorithm The proposed algorithm is shown in Fig. 2. We only show a bi-modal case but can be easily extended to a multi-modal case. The algorithm consists of the following steps: 1. Decomposition of the input images using wavelet on the interval transform in one or more levels. According to wavelet theory, the bigger decomposition level is, the smaller low frequency information contains. Justification of the number of decomposition levels appears in.17 2. Extraction of salient features in each wavelet coefficient using the Canny’s filter. The edge detection process serves to simplify the analysis of images by drastically reducing the amount of data to be processed, while at the same time preserving useful structural information about object boundaries. We chose this filter because is based on Gaussian derivatives (GD) (see Fig. 3). Since receptive fields of simple cells in the primate visual cortex were well fit in the space and time domains by the GD model, filters based on GD are good operators for the detection of relevant image patterns at different spatial scales.18

i III.p A

SII i I]g E

Figure 2. Fusion algorithm scheme

Proc. of SPIE Vol. 7000 700028-3

3. Build a decision map using Eq. 1. ⎧ k a ⎪ ⎪ ⎨ A akB   yFk = max akA , akB ⎪ ⎪ ⎩ CICB

k k if akA ∈ EA and akB ∈ / EB k k k k if aB ∈ EB and aA ∈ / EA k k k k if aA ∈ EA and aB ∈ EB otherwise.

(1)

k k and EB are found edges where akA and akB are intensity values of decomposed input images at level k, EA k of image A and B at level k respectively. yF is intensity value of fused image in wavelet domain. CICB are context independent constant behavior operators.19 CICB is composed of operators which have the same behavior whatever the values of the information to combine and which are computed without any contextual or external information, i. e., ∨ (a, b) ∈ image F (a, b) ≥ max(a, b)

4. Inverse Wavelet on the Interval transform of fused image yFk .

Figure 3. Canny operator in 3D

3. EXPERIMENTAL RESULTS We have performed various experiments using different orthogonal wavelets in order to compare the proposed algorithm with traditional fusion methods using a toolbox developed in Java language called FUSION LAB and described in.20 Fig. 4 shows a snapshot of the evaluation tool. We have separated our experiments in two classes: • Multi-focus image fusion with ground truth. Images with a reference image (distortion free). • Multi-focus image fusion without ground truth. Images without a reference image.21

—

Figure 4. Fusion Lab developed with Java 1.5 and Eclipse IDE

Proc. of SPIE Vol. 7000 700028-4

3.1 Evaluation of multi-focus image fusion with ground truth In experiment #1, we considered gray scale images of size 512×512 (see Fig. 5). The ground truth image where both the apple and the napkin container are in-focus is shown in Fig. 5(a), Fig. 5(b) has the napkin container blurred and the apple in-focus and Fig. 5(c) the other way around.

(a) Ground truth

(b) Input image A

(c) Input image B

Figure 5. Input images for apple and napkin container experiment.

In scheme 1 we fused input images using Haar wavelet with two levels of decomposition∗ . Larger values of low-low and detail coefficients were selected for reconstructing the fused image (MAX-MAX). In this scheme we did not perform any feature extraction. In scheme 2 we fused input images with the same rule of previous scheme (MAX-MAX) and we performed feature extraction with parameters: filter = Prewitt, σ = 0.6, low-threshold = -100 and high-threshold = -10. Fig. 6(a) shows the decision map obtained from image A and Fig. 6(b) shows the decision map obtained from image B for scheme 2. Fused image result of scheme 1 is shown in Fig. 7(a) and fused image result of scheme 2 is shown in Fig. 7(b). We compared both fused images with ground truth image in order to assess our proposed algorithm.22 Table 1 shows quantitative results.

(a) Decision map for image A

(b) Decision map for image B

Figure 6. Decision Maps for proposed algorithm ∗

Haar coefficients a0 =

√1 , 2

a1 =

1 √ 2

Proc. of SPIE Vol. 7000 700028-5

(a) Fused image for scheme 1

(b) Fused image for scheme 2

Figure 7. Results for apple and napkin container experiment Table 1. Results for apple and napkin container experiment. Fourth column shows results using signal to noise ratio of Wigner-Ville distribution SNRW .23 In the fifth column we used a perceptual quality metric based on structural similarity (SSIM)24

scheme #1 #2

PSNR [dB] 26.466 45.301

MSE

SNRW

SSIM

−4

7.600

0.952

−5

22.552

0.996

2.256e 2.950e

In experiment #2, we used human white blood cell images from confocal microscope (see Fig. 8). A confocal microscope uses point illumination and a pinhole in an optically conjugate plane in front of the detector to eliminate out of focus information, only the light within the focal plane can be detected. Fig. 8(a), is the ground truth image, Fig. 8(b) and Fig. 8(c) are blurred versions of it.

(a) Ground truth

(b) Input image A

(c) Input image B

Figure 8. Human white blood cells images from confocal microscope for experiment #2.

In scheme 3 we fused input images using DB2 wavelet with one level of decomposition† . Average values of low-low and detail coefficients were selected for reconstructing the image one (AV-AV). In this scheme we did not perform any feature extraction. In scheme 4 we fused input images with the same rule of previous scheme (AV-AV) and we performed feature extraction. †

DB2 coefficients a0 =

√ 1+√ 3 , 4 2

a1 =

√ 3+√ 3 , 4 2

a2 =

√ 3−√ 3 4 2

and a4 =

√ 1−√ 3 4 2

Proc. of SPIE Vol. 7000 700028-6

Fig. 9(a) shows the decision map obtained from image A and Fig. 9(b) shows the decision map obtained from image B for scheme 4. Fused image of scheme 3 is shown in Fig. 7(a) and Fig. 10(b) shows fused image of scheme 4. The quantitative results appear in Table 2.

(a) Decision map of image A

(b) Decision map of image B

Figure 9. Decision maps for scheme 4

(a) Fused image for scheme 3

(b) Fused image for scheme 4

Figure 10. Fused images of human white blood cells experiment. Table 2. Results for human white blood cells experiment.

scheme

PSNR [dB]

MSE −4

SNRW

SSIM

#3

30.086

9.8117e

7.696

0.965

#4

44.391

3.638e−5

16.431

0.995

3.2 Evaluation of multi-focus image fusion without ground truth Among the quality metrics, the mean square error (MSE) and signal to noise ratio (SNR) are widely employed because they are easy to calculate and usually they have low computational cost, but these objective metrics require a reference image together with the processed image in order to evaluate the visibility of artifacts. This imposes obvious limitations on those applications where such metric cannot be used. Non-reference metrics are much more difficult to define, since the metrics are not relative to the original image by providing an absolute value associated to a given image. Here, we used a non-reference quality metric based on information that appears in.21

Proc. of SPIE Vol. 7000 700028-7

For experiment #3 we used images by courtesy of Dr. Shutao Li (see Fig. 11). Input image A Fig. 11(a). has letters in-focus and drink blurred. Fig. 11(b) the other way around. In scheme 5 we fused input images using SYM4 wavelet with one levels of decomposition‡ . We use (MAXMAX) rule for merging coefficients. In this scheme we did not perform any feature extraction. In scheme 6 we fused input images with the same rule of previous scheme (MAX-MAX) and we perform feature extraction with parameters: filter = Prewitt, σ = 0.7, low-threshold = 15 and high-threshold = 25. IS Magnth. Quaty

'I. •s. o •1U 1I LW

IT Mfl

(a) Input image A

(b) Input image B

Figure 11. Input images for experiment #3.

Fused image result of scheme 5 is shown in Fig. 12(a) and fused image result of scheme 6 is shown in Fig. 12(b). The resulting performance metric for both schemes are given in Table 3. S Mig&fisr Quaity

.11 IJ

•1• •1l

o

IIM11 LIN

15T

(a) Fused image for scheme 5

(b) Fused image for scheme 6

Figure 12. Results for drink and letters experiment. Table 3. Results of the fusion performance for experiment #3. Mutual Information uses α = 0.8

Fusion

I q (F, A)

I q (F, B)

MFq AB

N MFq AB

#1

2.277

2.181

4.459

0.320

#2

2.363

2.155

4.519

0.324

In the last experiment #4 we fused three artificial images taken in our laboratory with a digital Reflex camera. Input images are shown in Fig. 14(a). These input images have distinct focal planes of the same scene. ‡

SYM4 coefficients a0 = −0.1294, a1 = 0.2241, a2 = 0.8365 and a3 = 0.4830

Proc. of SPIE Vol. 7000 700028-8

In scheme 7 we fused images using DB2 wavelet with one level of decomposition and MAX-MAX criteria for merging coefficients. In this scheme we did not perform any feature extraction. In scheme 8 we fused images whit the same criteria of scheme 7 but we performed feature extraction with parameters: filter = Sobel, σ = 0.4, low-threshold = 0 and high-threshold = 25.

1

TAmPOn

lLF*i

(a) Input image A

(b) Input image B

(c) Input image C

Figure 13. Input images for experiment #4.

Fused image result of scheme 7 is shown in Fig. 14(a) and fused image result of scheme 8 is shown in Fig. 14(b).

TAMPOI1

!•,, TAmPOn

ts lb — flROEAUX

j!I

CTV LENS

(a) Fused image of scheme 7

(b) Fused image of scheme 8

Figure 14. Fused images for experiment #4.

4. CONCLUSIONS In this paper we proposed a feature level fusion algorithm that incorporates a Gaussian derivative model of early vision and we developed an evaluation tool called Fusion Lab. We evaluated the performance of the proposed algorithm. Based on results showed in Tables 1 to 3 and based on visual perception we can conclude that the feature level fusion method proposed performs better than tradition fusion methods. A possible explanation is due to the fact, that the feature level method preserves the information from the edges.

ACKNOWLEDGMENTS This work has been sponsored by the following grants: UNAM PAPIIT IN106608, TEC2007-67025/TCM; TEC2006-28009-E and PI040765.

Proc. of SPIE Vol. 7000 700028-9

REFERENCES 1. G. Pajares and J. M. de la Cruz, “A wavelet-based image fusion tutorial,” Pattern Recognition 37(9), pp. 1855–1872, 2004. 2. B. Forster, D. Van de Ville, J. Berent, D. Sage, and M. Unser, “Complex wavelets for extended depth-of-field: A new method for the fusion of multichannel microscopy images,” Microscopy Research and Technique 65(1), pp. 33–42, 2004. 3. W. B. Seales and S. Dutta, “Everywhere-in-focus image fusion using controllable cameras,” Proc. SPIE 2905, pp. 227–234, 1996. 4. I. De and B. Chanda, “A simple and efficient algorithm for multifocus image fusion using morphological wavelets,” Signal Processing 86(5), pp. 924–936, 2006. 5. Z. Wang, D. Ziou, C. Armenakis, D. Li, and Q. Li, “A comparative analysis of image fusion methods,” IEEE Transactions on Geoscience and Remote Sensing 43(6), pp. 1391–1402, 2005. 6. A. G. Valdecasas, D. Marshall, M. Becerra, and J. J. Terrero, “On the extended depth of focus algorithms for bright field microscopy,” Micron 32(6), pp. 559–569, 2001. 7. S. Mallat, “A theory for multi-resolution signal: The wavelet representation,” IEEE Transactions on Pattern Analysis and Machine Intelligence 11(10), pp. 2003–2012, 1989. 8. B. Escalante-Ram´ırez and A. L´ opez-Caloca, “Image fusion with the Hermite transform,” International Conference on Image Processing 2, pp. 145–152, 2003. 9. Y. Nievergelt, Wavelets Made Easy, Birkh¨ auser, 1999. 10. M. Shensa, “The discrete wavelet transform: Wedding the a´ trous and mallat algorithms,” IEEE Transactions on Signal Processing 40(10), pp. 2464–2482, 1992. 11. A. Cohen, I. Duabechies, and P. Vial, “Wavelets on the interval and fast wavelet transforms,” Applied and Computational Harmonic Analysis 1, pp. 54–81, 1993. 12. C. Pohl and J. L. Van Genderen, “Multisensor image fusion in remote sensing: concepts, methods and applications,” International Journal on Remote Sensing 19(5), pp. 823–854, 1998. 13. S. Li, J. T. Kwok, and Y. Wang, “Combination of images with diverse focuses using the spatial frequency,” Information Fusion 2, pp. 169–176, 2001. 14. S. Kor and U. Tiwary, “Feature level fusion of multimodal medical images in lifting wavelet transform domain,” IEEE International Conference of the Engineering in Medicine and Biology Security 1, pp. 1479– 1482, 2004. 15. A. H. Gunatilaka and B. A. Baertlein, “Feature-level and decision-level fusion of non coincidently sampled sensors for land mine detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence 23(6), pp. 577–589, 2001. 16. W. S. Lee and A. A. Kassim, “Signal and image approximation using interval wavelet transform,” IEEE Transactions on Image Processing 16(1), pp. 46–56, 2007. 17. Y. Dongmei and Z. Zhongming, “Wavelet decomposition applied to image fusion,” International Conference on Info-tech and Info-net 1, pp. 291–294, 2001. 18. J. Canny, “A computational approach to edge detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence 8(6), pp. 679–698, 1986. 19. I. Bloch, “Information combination operators for data fusion: A comparative review with classification,” IEEE Transactions on Systems, Man and Cybernetics 26(1), pp. 52–67, 1996. 20. R. Nava, “Desarrollo de un m´etodo de fusi´ on de im´ agenes basado en energ´ıa utilizando la transformada wavelet discreta,” Master’s thesis, National Autonomous University of Mexico, 2007. 21. R. Nava, G.Crist´ obal, and B. Escalante-Ram´ırez, “Nonreference image fusion evaluation procedure based on mutual information and a generalized entropy measure,” Proc. SPIE 6592, p. 659215, 2007. 22. R. Nava, A. Gallego, and G. Crist´ obal, “Una nueva herramienta para la evaluaci´ on de la calidad perceptible en im´agenes,” Congreso Anual de la Sociedad Espa˜ nola de Ingenier´ıa Biom´edica, Navarra , pp. 77–88, 2006. 23. A. Beghdadi and R. Iordache, “Image quality assessment using the joint spatial/spatial-frequency representation,” EURASIP Journal on Applied Signal Processing 1, pp. 40–47, 2006. 24. Z. Wang and A. C. Bovik, “A universal image quality index,” IEEE Signal Processing Letters 9(3), pp. 81–84, 2002.

Proc. of SPIE Vol. 7000 700028-10