Multimodal Image Fusion Algorithm Using Dual-Tree Complex ...

Multimodal Image Fusion Algorithm Using Dual-Tree Complex Wavelet Transform and Particle Swarm Optimization Junli Tao, Shutao Li, and Bin Yang College of Electrical and Information Engineering, Hunan University, Changsha, China [email protected]

Abstract. In this paper, a multimodal image fusion algorithm based on multiresolution transform and particle swarm optimization (PSO) is proposed. Firstly, the source images are decomposed into low-frequency coefficients and high-frequency coefficients by the dual-tree complex wavelet transform (DTCWT). Then, the high-frequency coefficients are fused by the maximum selection fusion rule. The low-frequency coefficients are fused by weighted average method based on regions, and the weights are estimated by the PSO to gain optimal fused images. Finally, the fused image is reconstructed by the inverse DTCWT. The experiments demonstrate that the proposed image fusion method can illustrate better performance than the methods based on the DTCWT, the support value transform (SVT), and the nonsubsampled contourlet transform (NSCT). Keywords: Image fusion; Dual-tree complex wavelet transform; Image segmentation; Particle swarm optimization.

1 Introduction With many multimodal sensors have been developed, processing all massive image data directly is ineffective. Image fusion technique offers a solution to this problem. Image fusion is to merge the useful information of the input images into a new composite one. It enables a human or a computer vision system to analyze a single image only instead of all input images simultaneously. In the last decade, many software and techniques have been developed to resolve the image fusion problem. In all those methods, the fusion schemes based on multiresolution transform have attracted a considerable amount of research attention. Some popular transforms include discrete wavelet (DWT) [1], stationary wavelet (SWT) [2], dual-tree complex wavelet (DTCWT) [3], curvelet (CVT) [4], contourlet (CT) [5] and nonsubsampled contourlet transform (NSCT)[6]. In addition, Zheng et al. in [7] proposed an image fusion method based on the support value transform, which used the support value to represent the salient features of image. D.-S. Huang et al. (Eds.): ICIC 2010, CCIS 93, pp. 296–303, 2010. © Springer-Verlag Berlin Heidelberg 2010

Multimodal Image Fusion Algorithm Using DTCWT and PSO

297

In this paper, a DTCWT based fusion scheme with PSO is proposed to produce the optimal fused result adaptively. The source images are firstly decomposed into low-frequency and high-frequency coefficients by DTCWT. Then, the high-frequency coefficients are fused by the maximum selection fusion rule. For the fusion of lowfrequency coefficients, which contain some significant information, the fusion rule will directly affect the natural appearance of the fused results. The commonly used weighted average fusion rule which average the low-frequency coefficients entirely is likely to reduce the contrast of the fused image in some degree. Thus, in this paper, to preserve the region features and highlight different parts adaptively, we fuse the lowfrequency coefficients with a region based weighted average fusion rule. The weights are optimized with PSO. The fused image is reconstructed with fused low-frequency and high-frequency coefficients. The rest of this paper is organized as follows. In Section 2, overview of the DTCWT based image fusion scheme is given. Section 3 provides the procedure of the proposed method followed by some experiments in Section 4 and conclusions in Section 5.

2 Overview of the DTCWT Based Image Fusion Scheme 2.1 Framework of Image Fusion Scheme Based on DTCWT Without loss of generality, we illustrate the fusion scheme with two input images I1 and I2 which are assumed to have been registered. The proposed fusion scheme is shown in Fig.1. Firstly, each of the registered input images is transformed to lowfrequency coefficients A and series of high-frequency coefficients D by applying DTCWT, defined as:

( A, D ) = DTCWT ( I ) .

(1)

Then, in general, the low-frequency coefficients A1, A2 and high-frequency coefficientsD1, D2 are handled separately to give fused version, defined as: AF = φ A ( A1 , A2 ) DF = φD ( D1 , D2 )

.

(2)

where φ A , φD are fusion rules for low-frequency and high-frequency coefficients, and AF, DF indicate the fused low-frequency and high-frequency coefficients. Finally, the fused image F is obtained by performing the inverse transform (IDTCWT) on AF and DF, defined as: F = IDTCWT ( AF , DF ) .

(3)

298

J. Tao, S. Li, and B. Yang

Fig. 1. The fusion scheme based on DTCWT

2.2 Fusion Rules

For fusion of the high-frequency coefficients, a general and effective image fusion rule is the maximum selection fusion rule [8], formulated as: ⎧D , DF = ⎨ 1 ⎩ D2 ,

abs ( D1 ) ≥ abs ( D2 ) otherwise

.

(4)

It is vital to appropriately fuse the low-frequency coefficients, since it preserves global features which help ensure a natural appearance of the fused image. The lowfrequency coefficients are usually combined by pixel based weighted average fusion rule (PBAVE) to give the fused coefficients, defined as: AF = w1 × A1 + w2 × A2 ,

(5)

where weights w1,w2 take values between 1 and 0, and w1 + w2 = 1 . When they are separately set to be 1 and 0, Eq.(5) becomes to the selection fusion rule. Usually the weights take equal value 0.5. It works well with images from the same modality, but when used to fuse multi-modal images of different dynamic ranges, PBAVE will significantly alter the intensity range of the images and reduces contrast in the fused image. There are still some region-based (RB) fusion rules. These rules work by calculating measures of the importance of a region as priority and selecting the corresponding region with higher priority. The region features in the source images are preserved, but the optimal results could not always be obtained by simply selecting coefficients from one input.

3 Proposed Fusion Method with DTCWT and PSO This section proposes a multimodal image fusion method to obtain optimal fused image, with DTCWT and PSO. After decomposing input images into low-frequency and high-frequency coefficients as shown in Section 2, the fusion rule, shown in Fig.2, is proposed and used to fuse low-frequency coefficients. A1, A2 are low-frequency coefficients of two inputs, as shown in Fig.1. In order to highlight region features of the source images, A1 and A2 is merged based on region. The segmentation map is generated from a false color image produced by the combination of A1 and A2. In addition, the corresponding regions are fused by weighted average fusion rule. The values of weights are adaptively estimated by PSO with Piella’s index to be the fitness function. The fused image F is


299

reconstructed by the fused low-frequency coefficients AF and the fused highfrequency coefficients DF, obtained by performing the maximum selection fusion rule defined in Eq.(4).

Fig. 2. The proposed low-frequency fusion rule

3.1 Image Segmentation

To obtain a better segmentation, we segment false color image instead of gray image. The false color image is generated by assigning A1, A2 and their average (A1+A2)/2 into RGB channels separately. Next the false color image is segmented with the method described in [9], where Timothee et al. presented a spectral segmentation with multiscale graph decomposition called multiscale Ncuts. { R1 , R2,…, RK } indicates the segmentation map, where Rk denotes a region. 3.2 Proposed Fusion Rule with PSO

With the segmentation map, low-frequency coefficients are fused region by region with weighted average fusion rule to highlight different parts of the image, defined as:

AF , Rk = wk × A1, Rk + (1 − wk ) × A2, Rk

k = 1,2,..., K ,

(6)

where AF , Rk denotes the fused coefficients of Rk corresponding to A1, Rk and A2, Rk . Weights, w1 , w2 ,..., wK , for each region take values between 0 and 1. In order to adaptively find an optimal contrast setting wo1 , wo 2 ,..., woK , a populationbased optimization algorithm PSO is employed. In PSO [10] system, a particle represents a candidate solution to the optimization problem. Here a single particle represents weights of K regions. That is, each particle xi is constructed as: xi = ( wi ,1 ,..., wi , k ,..., wi , K ) ,

(7)

where wi,k refers to the kth region’s weight of the ith particle. Fitness function depending on the optimization problem is used to measure the performance of each particle. In order to obtain optimal local contrast in fused results, Piella’s index Qw [11] is adopted to be the fitness function. It estimates how well the salient information (contrast) from the inputs is presented in the fused image. The formulation of Qw is defined as:

Qw ( I1 , I 2 , F ) =

∑ c(n)(λ (n)Q ( I , F | n) + (1 − λ (n))Q ( I

n∈N

0

1

0

2

, F | n)) ,

(8)

300


where Q0 =

and λ (n) =

4σ xy x y

,

(9)

s ( I1 | n) . s ( I1 | n ) + s ( I 2 | n )

(10)

2

2

( x + y )(σ x2 + σ 2y )

In Eq. (9), similarity Q0 between images x and y is defined with local image statistics (variance σ x , σ y , covariance σ xy and mean x, y ) for all blocks (n) across the images. In

Eq. (8), Q0 ( I1 , F | n) is Q0 that is defined between input image I1 and fused image F for local window n. The Qw uses saliency weight λ (n) of local information quality to estimate Q0 as defined in Eq.(8). The local saliency weight λ (n) is calculated with s ( I1 | n) and s ( I 2 | n) as defined in Eq. (10), where s ( I1 | n) is some saliency of image I1 in window n, e.g entropy, sharpness and contrast. Here contrast is used as saliency to estimate optimal local contrast in the fused results. The overall saliency of a window is defined as c(n) = max( s( I1 | n), s ( I 2 | n)) . According to the definition of the fitness function in Eq.(8), the larger the value of Qw is, the more salient information (contrast) contained in the input images has been transferred into the fused image without introducing distortions. With optimization of Qw, the optimal local contrast in fused image is obtained adaptively. The proposed image fusion approach is summarized below: 1) Decompose input images I1, I2 into low-frequency coefficients A1, A2, and high-frequency coefficients D1, D2, using DTCWT. 2) Produce a single set of high-frequency coefficients DF with the maximum selection fusion rule defined in Eq. (4); 3) Generate a segmentation map by segmenting false color image using the method described in Subsection 3.1. 4) Search optimal weights: * Initiate the particles xi and fuse the low-frequency coefficients to a fused version AF’ with Eq. (6); * Search optimal contrast setting wo1 , wo 2 ,..., woK using PSO with Piella’s index Qw of fused result F’ reconstructed by AF’ and DF to be fitness function; * Stop searching until reach the maximum iteration times. 5) Fuse low-frequency coefficients using Eq. (6) with optimized weights wo1 , wo 2 ,..., woK . 6) Perform the inverse transformation on the fused low-frequency coefficients AF and high-frequency coefficients DF, to obtain the fused image F.

4 Experimental Results In our experiments, the ‘Dune’, ‘Tree’ and ‘UNcamp’ datasets are employed to test the performance of the proposed image fusion approach. Source images are


301

Fig. 3. Fusion of IR and VS images from ‘UN Camp’ sequence (frame 1818 to 1822 left to right): seven rows from top down show IR images in the first row, VS images in the second row, false color image in the third row, segmentation map with sixteen regions in the fourth row, fused images with proposed method RBPSO in the fifth row, fused results with SVT in the sixth row and results with NSCT in the last row

302


decomposed into three levels in the proposed method and other DTCWT based fusion methods performed in this section. We set one hundred to be maximum iteration times and thirty particles for PSO. The Piella’s Index is calculated with blocks size to be 8 × 8 . For performance comparison, three other fusion rules based on DTCWT are employed as well. The high-frequency coefficients are all fused by the maximum selection fusion rule. The low-frequency coefficients are merged by PBAVE with w1=w2=0.5, region-based selection fusion with average energy to be priority (RBENRG). Image fusion scheme based on support value transform (SVT) [7] and nonsubsampled contourlet transform (NSCT) [6] are also performed here. The level of decomposition is set to be 2,3,3,4 for NSCT and 4 for SVT. Fig. 3 shows the fused results of ‘UNcamp’ sequence (frame 1818 to 1822). The first four rows are source images, false color images and segmentation map with sixteen regions respectively. The fused images produced by proposed fusion method are depicted in the fifth row. The fused images of the methods described in literature [6] and [7] are shown in the sixth and seventh row respectively. It is obvious that fused results using proposed method contains most scene information in visible image and the man included in the scene is clearer than the other. Table 1. Average Fusion Performance with Piella’s Index for the three Datasets of this Experimental Section

‘Dune’ ‘Tree’ ‘Uncamp’

PBAVE 0.908 0.8046 0.6706

RBENRG 0.8984 0.7848 0.7236

SVT 0.9118 0.8153 0.7054

NSCT 0.9155 0.8207 0.7198

RBPSO 0.9156 0.8418 0.7694

5 Conclusions This paper describes a multimodal image fusion algorithm based on DTCWT and PSO to obtain optimal fused images. For fusion of the multimodal images of different intensity range, it is vital to adjust the different intensity range between the input images and highlight different parts of the image. In the context of DTCWT based image fusion scheme, low-frequency coefficients which contain the intensity information should be fused with an appropriate fusion rule. The commonly used fusion rules such as PBAVE and RBENRG usually reduce contrast in fused image and cannot adaptively give optimal fused results. This paper proposes a novel fusion scheme with PSO to automatically find optimal contrast setting to obtain an optimal fused image. Experimental results demonstrate that the proposed fusion method outperforms some well-known methods both visually and quantitatively. Acknowledgment. This paper is supported by the National Natural Science Foundation of China (No. 60871096 and 60835004), the Ph.D. Programs Foundation of Ministry of Education of China (No.200805320006), the Key Project of Chinese Ministry of Education (2009-120), and the Open Projects Program of National Laboratory of Pattern Recognition, China.


303

References 1. Mallat, S.: A Wavelet Tour of Signal Processing, The Sparse Way, 3rd edn. Academic Press, San Diego (2009) 2. Li, T.J., Wang, Y.Y.: Biological Image Fusion Using a SWT Based Variable-Weights Selection Scheme. In: Proc. of the 3rd International Conference on Bioinformatics and Biomedical Engineering, pp. 1–4 (2009) 3. Ray, L.A., Adhami, R.R.: Dual Tree Discrete Wavelet Transform with Application to Image Fusion. In: Proc. of the 38th Southeastern Symposium on System Theory, pp. 430–433 (2006) 4. Mahyari, A.G., Yazdi, M.: A Novel Image Fusion Method Using Curvelet Transform Based on Linear Dependency Test. In: Proc. of International Conference on Digital Image Processing, pp. 351–354 (2009) 5. Yang, L., Guo, B.L., Ni, W.: Multimodality Medical Image Fusion Based on Multiscale Geometric Analysis of Contourlet Transform. Neuro computing 72, 203–211 (2008) 6. Zhang, Q., Guo, B.L.: Multi-Focus Image Fusion Using the Nonsubsampled Contourlet Transform. Signal Process 89, 1334–1346 (2009) 7. Zheng, S., Shi, W., Liu, J., Zhu, G., Tian, J.: Multisource Image Fusion Method Using Support Value Transform. IEEE Trans. Image Process 16, 1831–1839 (2007) 8. Lewis, J.J., O’Callaghan, R.J., Nikolov, S.G., Bull, D.R., Canagarajah, N.: Pixel- and Region-Based Image Fusion with Complex Wavelets. Information Fusion 8, 119–130 (2007) 9. Timothee, C., Florence, B., Jianbo, S.: Spectral Segmentation with Multiscale Graph Decomposition. In: Proc. of Computer Vision Pattern Recognition, pp. 1124–1131 (2005) 10. Lai, C.C., Wu, C.H., Tsai, M.C.: Feature Selection Using Particle Swarm Optimization with Application in Spam Filtering. International Journal of Innovative Computing Information and Control 5, 423–432 (2009) 11. Piella, G., Heijmans, H.: A New Quality Metric for Image Fusion. In: Proc. of International Conference on Image Processing (2003)