JOURNAL OF NETWORKS, VOL. 9, NO. 2, FEBRUARY 2014
501
An Image Fusion Method Based on NSCT and Dual-channel PCNN Model Nianyi Wang 1, 2 1. School of Mathematics and Computer Science Institute, Northwest University for Nationalities, Lanzhou 730000, China 2. School of Information Science and Engineering, Lanzhou University, Lanzhou 730000, China Email:
[email protected]
Yide Ma School of Information Science and Engineering, Lanzhou University, Lanzhou 730000, China Email:
[email protected]
Weilan Wang School of Mathematics and Computer Science Institute, Northwest University for Nationalities, Lanzhou 730000, China Email:
[email protected]
Shijie Zhou School of Biomedical Engineering, Dalhousie University, Halifax, Nova Scotia, Canada Email:
[email protected]
Abstract—NSCT is one of useful multiscale geometric analysis tools, which takes full advantage of geometric regularity of image intrinsic structures. The dual-channel PCNN is a simplified PCNN model, which can process multiple images by a single PCNN. This saves time in the process of image fusion and cuts down computational complexity. In this paper, we present a new image fusion scheme based on NSCT and dual-channel PCNN. Firstly, the fusion rules of subband coefficients of NSCT are discussed. For the fusion rule of low frequency coefficients, the maximum selection rule (MSR) is used. Then, for the fusion rule of high frequency coefficients, spatial frequency (SF) of each high frequency subband is considered as the gradient features of images to motivate dual-channel PCNN networks and generate pulse of neurons. At last, fused image is obtained by using the inverse NSCT transform. In order to show that the proposed method can deal with image fusion, we used two pairs of images as our experimental subjects. The proposed method is compared with other five methods. The performance of various methods is mathematically evaluated by using four image quality evaluation criteria. Experimental comparisons conducted on different fusion methods prove the effectiveness of the proposed fusion method. Index Terms—Image Fusion; Nonsubsampled Contourlet Transform (NSCT); Pulse Coupled Neural Network (PCNN); Dual-Channel PCNN; Spatial Frequency
I.
INTRODUCTION
Image fusion plays an important role in image analysis and image application. Image fusion is an information processing system by which we can obtain more image information at the same time and integrated them into one
© 2014 ACADEMY PUBLISHER doi:10.4304/jnw.9.2.501-506
single image; finally we can get a new presentation and explanation for an image. Fused image benefits the image analysis in many fields, such as in remote sensing, intelligent robot, machine vision, clinical medicine and molecular biology. In most cases, image fusion algorithm is divided into two categories: image fusion algorithm based on spatial domain and image fusion algorithm based on transform domain. There are many kinds of image fusion methods in transform domain. Among them, those methods that based on multiscale decomposition (MSD) of source images become more important tools in recent years. Contourlet is one of MSD methods. It was proposed by M.N.Do and M.vetteri. It provides different and flexible number of directions at each scale and can capture intrinsic geometrical structure [1], [2], but contourlet lacks shift-invariance property and causes pseudo-Gibbs phenomena around singularities since it needs downsampling and upsampling operations [3]. In order to get rid of the frequency aliasing of the contourlet and enhance directional selectivity and shift-invariance, Cunha, Zhou, and M.N.Do proposed nonsubsampled contourlet transform (NSCT) [3]. Thus, in this paper NSCT is used as the MSD method. PCNN is a biologically inspired neural network. It has been proven that PCNN is very suitable for image processing such as image segmentation, image enhancement and image fusion. As to image fusion field, researchers have developed some image fusion algorithms based on PCNN [4], [5]; however, all the image fusion methods using traditional PCNN have one common trait: one traditional PCNN cannot finish the
502
JOURNAL OF NETWORKS, VOL. 9, NO. 2, FEBRUARY 2014
whole process of image fusion effectively. Because the traditional PCNN model is too complex and has too many parameters; it needs long time calculation in most cases, especially for a real-time system. So researchers proposed kind of methods to simplify the traditional PCNN model and combine it with other method together to process image. Literature [6] proposed a fusion algorithm based on NSCT and PCNN. After source images are decomposed by nonsubsampled contourlet transform, low frequency subband coefficients and bandpass subband coefficients are obtained. And then a simplified PCNN is applied to determine bandpass subband coefficients. Finally, fused image is obtained after inverse nonsubsampled contourlet transform. This algorithm proves that the use of nonsubsampled contour let transform and PCNN for image fusion is effective and feasible. Beside this, other researchers also proposed image fusion methods that combining NSCT and PCNN together. However, the defects of this kind of methods are: firstly, the PCNN model is too complex and has too many parameters; secondly, each pixel of the corresponding fusion coefficient only reflects the information of one source image; however the effects of another image are not considered [7]. In order to resolve this problem and improve the performance of image fusion methods based on traditional PCNN model and decrease computation complexity, a series of modified and simplified PCNN models have been proposed. Literature [8] proposed a dual-channel PCNN model and applied it into medical image fusion, and was proved effective. In this paper, we propose an image fusion method by using the shift-invariance, multi-scale and multi-directional properties of NSCT along with dual-channel PCNN model in such a way that can capture more details of source images, and finally result in fused images with high contrast. II.
RELATED WORK
A. NSCT Contour let is proposed by M.N.Do to obtain a sparse expansion for smooth contours, which overcomes the limitation of wavelet in representing contours by using square-shaped brush strokes and many fine "dots". Contour let employs Laplacian pyramid (LP) for multi-scale decomposition, and the directional filter bank (DFB) for directional decomposition. The number of direction decomposition at each level can be different, which is much more flexible than the three directions in wavelet. Unfortunately, in the original contour let, down samplers and up samplers are presented in both LP and DFB. Thus, it is not shift-invariant and causes pseudo-Gibbs phenomena around singularities. NSCT eliminates the down samplers and up samplers during the decomposition and reconstruction of image. Figure 1 shows the decomposition framework of NSCT. Nonsubsampled pyramid filter bank (NSPFB) and nonsubsampled DFB (NSDFB) are used in NSCT. The NSPFB is achieved by using two-channel nonsubsampled © 2014 ACADEMY PUBLISHER
2-D filter banks. The NSDFB is achieved by switching off downsamplers and upsamplers in each two-channel filter bank in DFB tree structure and upsampling filters accordingly [1], [3]. The NSCT not only retains the features of contourlet, but also has the properties of shift-invariance. When it is introduced into image fusion, sizes of different subbands are identical, which makes it easy to find the relationship among different subbands. This is beneficial for designing fusion rules. The common NSCT-based image fusion approach consists of the following steps: Firstly, perform NSCT on source images to obtain lowpass subband coefficients and bandpass directional subband coefficients at each scale and each direction. Secondly, apply some fusion rules to select NSCT coefficients of the fused image. Finally, employ inverse NSCT to the selected coefficients and obtain the fused image [9]. B. Dual-channel PCNN In traditional PCNN model, neuron receives input signals from feeding input Fi,j and linking input Li,j through the receptive field. Si,j is input stimulus such as the normalized gray level of image pixels in (i, j) position. Ui,j is the internal activity of neuron, and Ti,j is the dynamic threshold. Yi,j stands for the pulse output of neuron and it gets either the binary value 0 or 1. Behind these variables, a series of parameters need to be determined including threshold parameters, the decay constants, the weighted factor and the connection coefficients. This limited the use of PCNN. In order to correct this defect, the dual-channel PCNN model was proposed [8], as shown in Figure 2. Like the original PCNN neuron, each neuron also consists of three parts: receptive domain, modulation domain and pulse generator domain. The role of the receptive domain is to receive two kinds of inputs. One is from the external stimulus and another is from the surrounding neurons. Modulation domain is the place where all data is fused. The role of the pulse generator domain is to generate output pulse. Compared with traditional PCNN model, dual-channel PCNN model has simpler network architecture [8]. III.
PROPOSED IMAGE FUSION METHOD
The notations used in this section are as follows: A, B, R represent two source images and final fused image, respectively. C= (A, B, R). LFSC indicates the low frequency subband (LFS) of image C. HFSCg,h indicates the high frequency subband (HFS) of image C at scale g and direction h. (i, j) denotes spatial location, thus LFSC (i, j), HFSCg,h (i, j) denote coefficients located at (i, j) of low frequency and high frequency subband, respectively. In NSCT, images are decomposed into low frequency and high frequency subbands. The former determines gradation of light and the later relates with detail structure. They should be fused separately [9]. The dual-channel PCNN model we used in this paper is mathematically described as follows:
Hi1, j [n] = M (Yi , j [n 1]) + Si1, j
(1)
JOURNAL OF NETWORKS, VOL. 9, NO. 2, FEBRUARY 2014
503
ω2
（π,π）
Lowpass Subband coefficients Bandpass directional Subband coefficients
NSPFB
ω1
NSDFB Image
NSPFB
Bandpass directional Subband coefficients
NSDFB
（-π, -π） （a）
（b）
Figure 1. Nonsubsampled contourlet transform: (a) decomposition framework and (b) idealized frequency partition Surrounding neurons
S1ij
· · · ·
M
Σ
H1ij
β1
1
×
Σ
σ Eij Threshold
× W
Σ
H2ij
S2ij
VT
1 0
Yij
Uij
β2
receptive domain
Tij
Σ
×
exp(T )
Σ
1 pulse generator domain
modulation domain
Figure 2. Dual-channel PCNN model
Hi2, j [n] = W (Yi , j [n 1]) + Si2, j
(2)
Ui , j (n) = (1 1 Hi1, j [n])(1 2 Hi2, j [n])+
(3)
1， U i , j [n] Ti , j [n 1] Yi , j [n] 0, otherwise
(4)
Ti , j [n] exp(T )Ti , j [n 1] VT Yi , j [n]
(5)
The dual-channel PCNN has two external input channels. Let H1 and H2 stand for two symmetrical channels of current neurons，where M(·) and W(·) are feed functions, which shows the influence of surrounding neurons on the current neuron. Yi,j explains the internal state of the neuron, where β1 and β2 are the weighting coefficients of H1 and H2, respectively. The value of βk indicates the importance of the kth channel. The σ is the level factor to adjust the average level of internal activity. Other parameters Si,j, Ui,j, Ti,j and Yi,j are the same as parameters in the traditional PCNN model. Using dual-channel PCNN, multiple-source images are processed by single PCNN. Multiple external stimuli work in a neuron at the same time, which makes multiple-images processed in a parallel way. This saves time in the process of image fusion and cuts down computational complexity, as well. A. The Fusion Rule of Low Frequency Coefficients The coefficients in coarsest scale subband represent approximation component of source image. In most fusion applications, maximum selection rule (MSR) [10] was adopted to choose low frequency coefficients. According to this fusion rule, we select the low frequency
© 2014 ACADEMY PUBLISHER
coefficients of LFSR from LFSA or LFSB by using following formula: A A B LFS i, j , LFS i, j LFS i, j LFS i, j B LFS i, j , otherwise R
(6)
B. The Fusion Rule of High Frequency Coefficients The coefficients of HFS of source images are fused using dual-channel PCNN. Spatial frequency (SF) is considered as the gradient features of images to motivate dual-channel PCNN networks. SF proposed by Eskicioglu et al. is calculated by row and column frequency [11]. It reflects the whole activity level of an image which means the larger the SF the higher the image resolution. The SF is defined as:
SFi ,gj, h ( I ig, ,jh I ig,1,h j )2 ( I ig, ,jh I ig, ,jh1 )2
(7)
i, j
where SFg,h, and Ig,h, denote the SF and the coefficients of i,j i,j the pixel that located at (i,j) on scale g and direction h, respectively. SF in each high frequency subbands of source images A and B is set as feeding input of the dual-channel PCNN, and is inputted to each channel of PCNN respectively to motivate neurons and generate pulse of neurons by using formulas (1) ~ (5). Remember, each neuron in the whole process only ignites once. When the total number of ignition neurons is less than the total number of neurons in the dual-channel PCNN, then the process continues till every neuron ignites once. After that, all received coefficients signals are mixed together in the internal activity matrix U. We normalize internal activity matrix U and get the
504
JOURNAL OF NETWORKS, VOL. 9, NO. 2, FEBRUARY 2014
high frequency coefficients from different scales and different directions. We can express as follow:
HFSRg, h i, j U g ,h i, j
(8)
C. Detailed Fusion Algorithm Source images to be fused must be registered. The steps of the proposed image fusion algorithm are described briefly as follows: 1). Decompose source images A and B by NSCT to get each low frequency subbands coefficients LFSA(i,j), LFSB(i,j) and each high frequency subbands coefficients HFS Ag,h (i,j), HFS Bg,h (i,j) of each image. Note: the decomposition parameter of NSCT is provided in Section Ⅳ. 2). Select low frequency coefficients by using formula (6), and get LFSR(i,j) for final fused image. 3). Calculate spatial frequency as described in formula (7) by using overlapping window on coefficients of HFS, and get SFg,h, i,j for each high frequency coefficients. 4). Input SF of each HFS of images A and B into each channel of dual-channel PCNN to motivate the neural networks and generate pulse of neurons with formula (1) ~ (5). 5). Parameter Cn is used to record the number of the firing neuron in the current iteration. Each neuron only fires one time in the whole process. 6). Let Sum = Sum´+ Cn, where Sum denotes the total number of the fired neurons after the current iteration. Sum´denotes the total number of the fired neurons before the current iteration. 7). Let Num denotes the total number of all neurons in the network. If Sum < Num, turn to step 6); otherwise, continue to execute next step. 8). Normalize the internal state U. The normalized U is the high frequency coefficients of different scales and different directions, as formula (8) describes. 9). Apply inverse NSCT on the fused LFS and HFS coefficients to get the final fused image. IV.
image. EOL is one of the useful indices to describe clarity of image. QAB/F is proposed by C.S.Xydeas et al. as an objective image fusion performance measure. Figure 3 shows two groups of source images: satellite image group (images (a) and (b) in Figure 3), brick image group (images (c) and (d) in Figure 3). Images (a) and (b) are focused at the central area and peripheral area, respectively. Images (c) and (d) are focused on the bar brick and flate brick, respectively. In order to prove the validity of the proposed fusion technique, several experiments are conducted. Five other methods are adopted to compare with our proposed one (M6), which are Averaging method (M1), discrete wavelet transform (DWT) with DBSS (2, 2) (M2), Laplacian pyramid (M3), morphological pyramid (M4), PCA method (M5). Parameters of these methods are set by: pyramid level = 4, selection rules: high-pass = select max, lowpass = average [17].
Figure 3. Two groups of source images: satellite image group and brick image group.
RESULTS AND DISCUSSION
Our experiments are conducted by MATLAB R2007b with Intel Core (TM) 2 Duo T7500 2.2GHz. The decomposition parameter of NSCT is set as: levels= [1, 2, 4], pyramid filter and directional filter are set as ‘pyrexc’ and ‘yk’, respectively. For dual-channel PCNN, the parameters are set as: β1=0.5, β2=0.5, σ=1, αT=0.05, VT=1000, Convolution core K = [0.1091, 0.1409, 0.1091; 0.1409, 0, 0.1409; 0.1091, 0.1409, 0.1091]; M (·) = W (·) = Y [n-1] ⊗ K, where ⊗ denotes convolution operation. Quantitative assessment of different image fusion algorithms are compared using evaluation criteria Mutual information (MI) [12], Standard deviation (SD) [13], Energy of laplacian (EOL) [14] and QAB/F [15], which have been proved to be effective to a great degree [16]. MI can be used to measure amount of information transferred from source images to the final fused image. The fusion performance would be better and better with MI increasing. SD indicates deviation degree between grey values of pixels and the average one of the fused
© 2014 ACADEMY PUBLISHER
Figure 4. Fusion results of different methods conducted on satellite images. Image (a): Averaging method (M1); (b): discrete wavelet transform (DWT) with DBSS (2, 2) (M2); (c): Laplacian pyramid (M3); (d): morphological pyramid (M4); (e): PCA method (M5); (f): proposed method (M6)
Figure 4 and Figure 5 show fusion results by using above mentioned 6 methods conducted on two image groups. In each figure, Mi (i=1, 2… 6) indicates different fusion methods. By carefully inspect the fused images obtained by 6 fusion methods in Figure 4 and Figure 5, we can find that our proposed method possesses a
JOURNAL OF NETWORKS, VOL. 9, NO. 2, FEBRUARY 2014
satisfied visual effect compared to other 5 methods. In Figure 4 and Figure 5, M1 (Averaging method) as one of the easiest methods, eliminated too much image details and blurred the fused images, M2 (discrete wavelet transform (DWT) with DBSS (2, 2)) produced noise around edges, margins and lines. M4 (morphological pyramid) suffered from the problem of blocking effect. M5 (PCA method) blurred dot information around outlines and contours. As far as these six methods are concerned, the results of both M3 (Laplacian pyramid) and M6 (our proposed one) obtained better visual effects. Therefore from the subjective visual evaluation we can see that the proposed fusion method is effective.
Figure 5. Fusion results of different methods conducted on brick images. Image (a): Averaging method (M1); (b): discrete wavelet transform (DWT) with DBSS (2, 2) (M2); (c): Laplacian pyramid (M3); (d): morphological pyramid (M4); (e): PCA method (M5); (f): proposed method (M6) TABLE I.
COMPARISON OF SIX FUSION METHOD OF SATELLITE IMAGE GROUP
M1 M2 M3 M4 M5 M6 TABLE II.
MI 3.684 3.190 3.551 3.060 3.705 3.737
SD 8.311 8.248 8.275 8.202 8.300 8.271
EOL 0.519 1.000 1.008 1.653 0.816 1.042
QABF 0.235 0.273 0.294 0.244 0.230 0.291
COMPARISON OF SIX FUSION METHOD OF BRICK IMAGE GROUP
M1 M2 M3 M4 M5 M6
MI 1.383 0.773 0.962 1.071 1.427 1.468
SD 10.488 11.261 11.404 10.948 10.468 11.404
EOL 0.473 2.352 2.369 1.869 0.532 2.372
QABF 0.288 0.354 0.388 0.392 0.298 0.381
Table 1 and Table 2 report objective evaluations of the above-mentioned 6 methods. Experimental data shows that the differences of SD values of 6 fusion methods are slight and tiny. Thus we pay more attention to the other three evaluation indices. In the satellite image group, the value of MI of our proposed M6 is the best. In the brick image group, the values of MI and EOL of M6 are the best. Though in the satellite image group the EOL and QABF values of M6 are not the highest, both of them still outperformed the other 4 methods. In the brick image group, the value of QABF of M6 gets the third high value. The quantitative objective evaluation and comparison we
© 2014 ACADEMY PUBLISHER
505
discussed above verified that the proposed method is an effective fusion method for image. V.
CONCLUSIONS
NSCT is one of useful multiscale geometric analysis tools, which take full advantage of geometric regularity of image intrinsic structures. The dual-channel PCNN is a simplified PCNN model, which can process multiple images by a single PCNN. This saves time in the process of image fusion and cuts down computational complexity. In this paper, we present a new image fusion scheme based on NSCT and dual-channel PCNN. The flexible multi-resolution, anisotropy, and directional expansion characteristics of NSCT are associated with pulse synchronization and light computation features of dual-channel PCNN. In order to show that the proposed method can deal with image fusion, we used two pairs of images as our experimental subjects. At the same time, our method is compared with other five methods (Averaging method, discrete wavelet transform, Laplacian pyramid, etc.), the performance of various methods is evaluated by using Mutual information, Standard deviation, Energy of laplacian and QAB/F. Experimental comparisons conducted on different fusion methods prove the effectiveness of the proposed fusion method. ACKNOWLEDGEMENT The authors would like to thank the anonymous reviewers and editors for their invaluable suggestions. This work was jointly supported by the National Natural Science Foundation of China (No. 61162021, No.61375029, No. 61175012), Innovative Team Subsidize of Northwest University for Nationalities, and Fundamental Research Funds for the Central Universities under NO.lzujbky-2013-38. REFERENCES [1] Do, M. N., and Vetterli, M. "The contourlet transform: an efficient directional multiresolution image representation." IEEE Transactions on Image Processing, 14(12) pp. 2091-2106, 2005. [2] Do, M. N., and Vetterli, M. "Framing pyramids." IEEE Transactions on Signal Processing, 51(9) pp. 2329-2342, 2003. [3] Da Cunha, A. L., Zhou, J., and Do, M. N. "The nonsubsampled contourlet transform: theory, design, and applications." IEEE Transactions on Image Processing, 15(10) pp. 3089-3101, 2006. [4] Broussard R P, Rogers S K, Oxley M E, et al. "Physiologically motivated image fusion for object detection using a pulse coupled neural network." IEEE Transactions on Neural Networks, 10(3) pp. 554-563, 1999. [5] Li W, Zhu X F. "A new image fusion algorithm based on wavelet packet analysis and PCNN." Machine Learning and Cybernetics. Proceedings of 2005 International Conference on. IEEE, 9 pp. 5297-5301, 2005. [6] L Meili, L Yanjun, W Hongmei. "Multi-focus image fusion method based on contourlet transform." Comput. Eng. Appl, 45 (10) pp. 20-22, 2009.
506
[7] Baohua Z, Xiaoqi L, and Weitao J. "A multi-focus image fusion algorithm based on an improved dual-channel PCNN in NSCT domain." Optik-International Journal for Light and Electron Optics, 2013. [8] Wang, Z., and Ma, Y. "Medical image fusion using m-PCNN." Information Fusion, 9(2) pp. 176-185, 2008. [9] Wang N Y, Wang W L, Guo X R. "A New Image Fusion Method Based on Improved PCNN and Multiscale Decomposition." Advanced Materials Research, 834 pp. 1011-1015, 2014. [10] Li, H., Manjunath, B. S., and Mitra, S. K. "Multisensor image fusion using the wavelet transform." Graphical models and image processing, 57(3) pp. 235-245, 1995. [11] Eskicioglu, A. M., and Fisher, P. S. "Image quality measures and their performance." IEEE Transactions on Communications, 43(12) pp. 2959-2965, 1995. [12] Qu, G., Zhang, D., and Yan, P. "Information measure for performance of image fusion." Electronics letters, Vol. 38, No. 7, pp. 313-315, 2002. [13] Chang, Dah-Chung, and Wen-Rong Wu. "Image contrast enhancement based on a histogram transformation of local standard deviation." Medical Imaging, IEEE Transactions on, Vol. 17, No. 4, pp. 518-531, 1998. [14] Aslantas, V., and R. Kurban. "A comparison of criterion functions for fusion of multi-focus noisy images." Optics Communications, Vol. 282, No. 16, pp. 3231-3242, 2009. [15] Xydeas, C. S., and Petrovic, V. "Objective image fusion performance measure." Electronics Letters, Vol. 36, No. 4, pp. 308-309, 2000. [16] Bhatnagar, G., Jonathan Wu, Q. M., and Liu, Z. "Human visual system inspired multi-modal medical image fusion framework." Expert Systems with Applications, 2012. [17] Wang, Z., Ma, Y., Cheng, F., and Yang, L. "Review of pulse-coupled neural networks." Image and Vision Computing, 28(1) pp. 5-13, 2010.
© 2014 ACADEMY PUBLISHER
JOURNAL OF NETWORKS, VOL. 9, NO. 2, FEBRUARY 2014
Nianyi Wang received the B.S. degree in computer science from Lanzhou University, Gansu, China in 2002, and received the M.S. degree in computer application technology from Lanzhou University in 2007. He is currently a lecturer in the School of Mathematics and Computer Science Institute, Northwest University for Nationalities. He is currently pursuing the Ph.D. degree in radio physics at Lanzhou University. His current research interests include artificial neural networks, image processing, and pattern recognition. Yide Ma received the B.S. and M.S. degrees in radio technology from Chengdu University of Engineering Science and Technology, Sichuan, China, in 1984 and 1988, respectively. He received the Ph.D. degree from the Department of Life Science, Lanzhou University, Gansu, China, in 2001. He is currently a Professor in the School of Information Science and Engineering, Lanzhou University. He has published more than 50 papers in major journals and international conferences and several textbooks, including Principle and Application of Pulse Coupled Neural Network (Beijing: Science Press, 2006), and Principle and Application of Microcomputer (Beijing: Science Press, 2006). His current research interests include artificial neural networks, digital image processing, pattern recognition, digital signal processing, and computer vision. Weilan Wang is currently a professor in the School of Mathematics and Computer Science Institute, Northwest University for Nationalities. Her current research interests include image processing, pattern recognition, Tibetan information processing, digital protection of Thangka image. Shijie Zhou is currently pursuing the Ph.D. degree in the School of Biomedical Engineering，Dalhousie University. His current research interests include artificial neural networks, image processing, and Biomedical Engineering.
