Image Fusion Using Computational Intelligence: A ... - Semantic Scholar

Image Fusion Using Computational Intelligence: A Survey Humayun Irshad

Muhammad Kamran

Department of Computer Sciences National University of Computer & Emerging Science Islamabad, Pakistan [email protected]

Department of Computer Science National University of Computer & Emerging Science Islamabad, Pakistan [email protected]

Abdul Basit Siddiqui

Ayyaz Hussain

Department of Computer Sciences National University of Computer & Emerging Science Islamabad, Pakistan [email protected]

Department of Computer Sciences National University of Computer & Emerging Science Islamabad, Pakistan [email protected]

Abstract— Image fusion is a process to combine information from different images of the identical scene to make an image that has more information. Main focus of this survey is on image fusion techniques using Computational Intelligence. Image fusion techniques have agreed upon few standards, whereas most of the schemes rely upon multi-scale decompositions. It contains many steps and complex decision models that may be difficult to implement in real-time systems. They are also susceptible to artifacts and noise enhancement because they treat source images as equally likely contributors to the fused result. This research survey will focus on the study of existing and new image fusion techniques. Keywords: Image fusion, image enhancement, discrete wavelet transform, fuzzy logic, multiresolution image fusion.

I.

INTRODUCTION

The term fusion is generally defined as an approach to extract the information which is acquired from several domains. In image fusion (IF), the objective is to combine corresponding multi-sensors, multi-view or multi-temporal information in order to obtain a good quality image which can lead the observers to some better decision. The meanings and the measurement of the quality of the fused image vary application to application. Image fusion has got immense importance in many application areas. In the fields of remote sensing and astronomy, to obtain high spatial and spectral resolutions, multi-sensor image fusion is performed where the sensors have high spatial and spectral resolutions respectively. In medical imaging, we find many fusion applications which simultaneously evaluate CT, MRI, and/or PET images. A number of applications using multi-sensor fusion of visible and infrared images have appeared in military, security, and surveillance areas. In case of multi-focus or multi-view image fusion techniques, the images of the same scene taken by the same sensor are fused to achieve an image in which all the objects are in focus. These fusion techniques usually generate an image with higher resolution than the resolution provided by the sensors. In multi-temporal image fusion applications, the images of the same scene are recorded at

different times to keep track of different changes in the scene or to get a less degraded image of the scene. The list of applications mentioned above demonstrates the variety of problems we face when fusing images. It is impossible to devise a universal method valid to all image fusion tasks. Every scheme should take into account not only the fusion reason and the characteristics of individual sensors, but also particular imaging conditions, imaging geometry, noise corruption, required accuracy and application-dependent data properties [9]. In this survey, we explore different techniques based on computational intelligence like fuzzy logic. We also find out the limitations in these techniques and propose a hybrid technique. The subsequent sections of this survey are organized as follows. Section II describes the background of IF. Section III reviews the multi-resolution image fusion based on fuzzy region feature. Section IV reviews the pixel & feature level multi-resolution image fusion based on fuzzy logic. Section V reviews the algorithm for multi-focus image fusion using morphological wavelet. Section VI describes a framework of region-based dynamic image fusion. Section VII lists the limitations of these techniques. In Section VIII, proposed model is described. II.

BACKGROUND

The most important prerequisite of image fusion is image registration. In this survey, it is assumed that the source images are already registered. Some generic requirements could be imposed on the fusion algorithms are: (a) the fusion process should preserve all relevant information of the input imagery in the composite image i.e., pattern conservation, (b) the fusion techniques should not introduce any artifacts or inconsistencies which would distract the human observer or following processing stages, (c) the fusion process should be shift and rotational invariant i.e., the fusion result should not depend on the location or orientation of an object the input imagery, and (d) irrelevant features and noise should be suppressed to a maximum extent [1]. The fusion process is performed at three different levels including pixel, feature and decision level. In pixel level image fusion techniques, the fusion takes place directly on

the pixel intensities of the source images. For example, average or maximum of the corresponding pixels of the source images is calculated and it is taken as the corresponding pixel value of the fused image. In this kind of image fusion, the pixels are least processed. In feather level image fusion, first the features are extracted from the source images and then on the basis of some selection criteria, the important features are selected. Based on the selected features, the fusion process is performed. These features are extracted through segmentation of the source images. Decision level image fusion involves the detection and classification of objects in the source images and then the information is fed into the fusion algorithm. In the literature, we also find the hybrid techniques which combine two different levels of fusion such as combination of pixel and feature level fusion. Fig. 1 describes the different levels of image fusion. The image fusion techniques fall in two categories. In the first category, images are fused in spatial domain. In spatial domain, corresponding pixel values of the source images are operated using simple mathematical operators such as max, mean etc. and the fused image is generated. In this kind of fusion, some undesired effects such as blur are also introduced in the fused image which can mislead to the observer. To overcome these problems, in the second category, the source images are first transformed into another domain. Multi-scale transforms such as wavelets, Laplacian pyramids, Morphological pyramids and gradient pyramids have been proposed in the second category of the image fusion techniques. Discrete wavelet transform provide directional information in decomposition levels and contain unique information at different resolutions. The information flow diagram of wavelet based image fusion algorithm is shown in fig. 2. In wavelet based image fusion scheme, the source images I1(x, y) and I2(x, y), are decomposed into approximation and detailed coefficients at required level using discrete wavelet transform (DWT). The approximation and detailed coefficients of both images are combined using fusion rule. The fused image If(x, y) is obtained by taking the inverse discrete wavelet transform (IDWT).

Figure 2.

III.

Information flow diagram in image fusion scheme

MULTIRESOLUTION IMAGE FUSION BASED ON FUZZY REGION FEATURE

This scheme is a novel region based image fusion scheme using multi-resolution analysis. They choose an object image from the source images to perform segmentation and to define region attribute. The low frequency band of the object image is segmented into important regions, sub-important regions and background regions by using K-Means clustering algorithm according to the pixel level distribution [6]. Each feature of the regions is used to determine the region’s degree of membership, and then fed into fusion system [4]. Fig. 3 is the framework of the multi-resolution image fusion based on fuzzy region feature algorithm. 1. Select one image as object image from all input images. 2. Perform multi-resolution decomposition of all images. 3. Segment the object image into important, sub-important and background regions by using K-Means clustering algorithm and also find regions features. 4. Each feature of regions is used to determine the region’s degree of membership in fuzzy space. The function of membership i region belongs to j is: μi,j

exp

MEi Ej Lmax Lmin 2

(1)

where Lmax and Lmin are the highest gray level and lowest gray level, E1= Lmin, E2= (Lmax - Lmin)/2, E3 = Lmax ; MEi is the mean of pixel gray level within region i; µi,1, µi,2, and µi,3 are the values of membership in important, sub-important and background region. 5. Fussification rules on following: i. MEi=E1 indicates that the region i is background, fusion result F1 is corresponding region of image B; ii. MEi=E2 indicates that the region i is sub-important, fusion result F2 is obtained by single pixel based fusion; iii. MEi=E3 indicates that the region i is important, the fusion result F3 is the corresponding region of image A. 6. Based on every region feature, the membership of each pixels is defined as µi,1, µi,2, and µi,3. Defuzzification process is performed using the membership: ∑

,

∑

Figure 1. Image Fusion level

2

   

 

     2  

where F is the multi-resolution representation of the fused image. The final fused image is obtained by performing inverse discrete wavelet frame transform.

Figure 3. Framework of the Multi-resolution Image Fusion Based on Fuzzy Region Feature algorithm

IV.

Figure 4. Flowchart of the Pixel & Feature Level Multiresolution Image Fusion based on Fuzzy Logic

PIXEL & FEATURE LEVEL MULTIRESOLUTION IMAGE FUSION BASED ON FUZZY LOGIC

The objective behind fusing multi-resolution images is to create a single image with more information and better interpretability. In algorithm, images are first segmented into regions using fuzzy clustering and are then fed into a fusion system, based on fuzzy “if-then” rules. Fuzzy clustering is more flexible than traditional strict clustering and hence allowing more robustness as compared to other segmentation techniques (e.g. K-means algorithm) [2]. Fuzzy c-means is a technique for clustering which allows one data item to belong to more than one cluster and every data item has a certain membership in every cluster. In fuzzy c-mean clustering algorithm, data items are assigned membership values and cluster centers are determined. This iterative process continues to calculate the new membership values and cluster centers according to the distance between entities and centers. The general framework of this algorithm is shown in fig. 4. 1. This process comes to stop when a maximum number of iteration is reached or an objective function reaches a required threshold value [7]. Apply DWFT to two registered source images. 2. Use Fuzzy c-mean clustering algorithm to segment the approximations into three regions, important region, sub-important region and background region. 3. Feature Level Fusion:- Segmented approximations are fed into a fusion system based on fuzzy “if-then” rules, to get fused approximations [3]. 4. Pixel Level Fusion:- The details are fused by absolute maximum coefficient selection method. 5. Apply morphological filtering [8] which use “fill” and “clean” operators to sweep isolated points. 6. With fused Approximations and fused details get fused wavelet frame coefficients map. Take IDWFT and get fused image.

V.

MULTI-FOCUS IMAGE FUSION USING MORPHOLOGICAL WAVELETS

There are two problems with linear wavelets. The first problem is that during the decomposition, the range of original data is not preserved. Secondly, linear wavelets such as Haar wavelets perform low-pass filtering. As a result, the edges in the fused image become smooth which decrease the contrast. To overcome these problems, the nonlinear behavior of discrete wavelet transformed is introduced using the morphological operators. De and Chanda introduced a new nonlinear morphological wavelet transform which preserves the range in the scaled images and involves integer arithmetic only [10]. Suppose we have a signal X Є V0, where V0 is multiresolution signal decomposition scheme. So X is a function from domain D to G where D is a subset of Z2 and G is the set of gray-values, where Z2 is a discrete two dimensional space of finite set of gray values. Let A Є Z2 which is a structuring element. Then the morphological operators, dilation δA(X) and erosion εA(X) of X by A are defined as δ

,

,

 Є  ,   

max                      ,  Є 

1,

1 , 3)

ε

,

,

 Є  ,   

min                      ,  Є 

1,

1 , (4)

It proposes a non-separable two-dimensional uncoupled morphological wavelet decomposition scheme, which will be used for image-fusion algorithm. Unique analysis operators (ψ↓, ω↓) are used at all levels. A summarized form of algorithm is given under: 1.

Analysis Step: Apply the analysis operators’ ψ↑ and ω↑, 1,2, … , and k times recursively, on image   , get    ,   ,   , … ,   , where  is the scaled

image at level k and   , 1,2, … , are the details at levels 1,2, … , , respectively. 2. Fusion Step: Compare , , , … ,   , where     are given by , max |  , |, |  , |, … , |  , | , max | , |, | , |, … , | , | and  respectively. 3. Synthesis step: Reconstruct the fused image  at level , 1, … ,0, by applying the synthesis operators,   and following by addition, respectively, on r, c   ω Y   r, c . i.e. X   ψ    X This method has some computational advantages. The technique guarantees lossless data processing due to integer computations. Secondly, the memory space required during decomposition remains fixed irrespective of the frequency of analysis operators applied. Thirdly, simple arithmetic operations like addition, subtraction and comparison are the only operations used in this method. Fourthly, due to the non-linear nature, important geometric details such as edges are nicely-preserved at lower level of resolutions. Finally, the method is very fast due to its simplicity. VI.

3. 4.

5.

REGION-BASED DYNAMIC IMAGE FUSION

It will be more appropriate if the useful features of images in the fusion process are selected. These features are used in dynamic image fusion then the disadvantage is that the important moving information of image sequences will not be utilized sufficiently [11]. A framework of regionbased dynamic image fusion is proposed, as shown in fig. 5. The multi-resolution analysis is used for every preprocessed frame, and the source images are decomposed by means of steerable non-separable wavelet transform. At the same time, the technique of target detection is applied to separate the target from background in the source image sequences and obtain a set of candidate regions. Subsequently, the decomposition coefficients are obtained by using different fusion rules in different regions. The fusion image sequences are constructed by performing an inverse transform on the fused decomposition coefficients in the end. For each candidate region, a confidence measure [11] is defined. If the brightness ratio between a candidate region and its neighborhood goes higher, the value of the confidence will go higher at the same time and close to 1, otherwise it will be close to 0. Candidate regions with high confidence are selected as target regions. The new method is described as follows: 1. 2.

Figure 5. Framework of the region-based dynamic image fusion

Let i=1. Use the method of image segmentation to process the ith frame of the source image sequences. Note the centroid of the target region in the ith frame. Use the intensity-based information of the target regions as target pattern. If the ith frame is the last frame in the image sequence, stop; else go to Step 5.

6.

Let i=i+1. Do the pattern matching in the ith frame according to the target pattern and the region which the centroid is located in the (i−1)th frame. If the target region of the ith frame is obtained, go to Step 3; else go to Step 2.

and Region-based fusion rule: The frames are denoted as . The target regions detected in   and  are represented by   , ,…,   and   , ,…,   respectively, where M and N are the number of the regions in IA and IB respectively. The union target region can be calculated by      . There are three region aggregates in the current frame: single target region set  , and background region set superposed target region set B. In the single target region, the fusion rule is: ,

, ,

 

,     ,    

, ,

   

(6)

  are the decomposition coefficients of where   , is the fused coefficient. and respectively, and For the closed region t in the superposed target region is used to denote the similarity between set      , the region t in and  . ∑ ,    

 ∑ ,           ,

 

,

·  

 

,

 ∑ ,    

 

 

,

(8)

Let     0, 1  be the close threshold, which is usually set as 0.85. If the similarity M(t) is smaller than , the fusion rule in superposed target region     is defined: ,

 

, ,

,       ,    

   

 ,

(10)

If     , the fusion rule is calculated by the method of weighted means ,  ·  ,    ·  , ,               (11)  ·  ,    ·  , ,            

where and are maximal and minimal weighted coefficients, respectively. In the background region, the basic MS (maximum selection) rule is applied for fusion. This method has better capabilities of target recognition and preserves clear background information; it can be used effectively to enhance the inspectors’ target awareness capability. VII. LIMITATION In first image fusion scheme, it is based on fuzzification of region feature and discrete wavelet frame for merging multiple sensor images. The scheme preserves the image contrast and obtains better region similarity. It blurs the edges due to k-mean cluster algorithm which doesn’t generate good segments. In second image fusion scheme, it improves quality of fused image c-means fuzzy clustering algorithm. This algorithm has drawbacks that it is computationally expensive due to DWFT which not only required are more computations but also is linear wavelets in which range of the original data is not preserved. Since linear wavelet perform low-pass filtering and hence the edges are smooth out. Due to this reason the contrast in the image is decreased. In third image fusion scheme, nonseparable, non linear wavelets (morphological operators) are used for multi-resolution decomposition; it preserves the contrast in images. In fourth image fusion scheme, a framework for region based dynamic image fusion is used. VIII. PROPOSED MODEL Those techniques, discussed in previous section, either do not preserve the edge information in the fused image or they perform the blurring of the edges. It is very important to preserve the edge information in the fused image in order to get better contrast. We have proposed a model to preserve the edges in the fused image. The images are fused in two different situations: when the reference image is available and when it is not available (blind image fusion). 1. When the reference image is available, first we find the edges from reference image and fused image denoted by Re and Fe using some filter such as laplacian or sobel. By superimposing Re and Fe, the information of edges can be incorporated in the resultant fused image. 2. When the reference image is not available, first we measure the blur in the source multi-focus images by using some blur metric such as one given under [13] and some threshold. Using the blur metric, we find the nonblur areas and find out the edges in these areas denoted by S1e and S2e. Combine the edge images S1e and S2e using some criteria and generate a resultant combined edge image FSe. We find the edges from the fused image denoted by Fe. By superimposing FSe and Fe, the information of the edges can be incorporated in the resultant fused image. Blur measure = Sum of all edges width / no. of edges

IX.

CONCLUSION

In this survey, the uses of DWT, PCA, Morphological operators and fuzzy logic in fusion of images are studied. Above mentioned image fusion techniques illustrates the diversity of problems we face when we fused images. It is impossible to design a universal method applicable to all image fusion tasks. Every technique should take into account not the fusion purpose and the characteristics of individual sensors, but also particular image conditions, image geometry, noise corruption, required accuracy, and application-dependent data properties. In all above methods, no one has used edge preservation. So we need a method that also preserves image details during fusion. X.

REFERENCES

[1]

V.P.S Naidu, J.R. Raol, “Pixel-level image fusion using Wavelets and Principal Component Analysis,” Defence Science Journal, Vol. 58, No. 3 May 2008, pp. 338-352 © 2008, DESIDOC.

[2]

B. N. Kayani, A. M. Mirza, A. Bangish, H. Iftikhar, “Pixel & Feature Level Multiresolution Image Fusion Based on Fuzzy Logic,” logic in T. Sobh (ed.), Innovations and Advanced Techniques in Computer and information Sciences and Engineering, 129-132 © 2007 Springer.

[3]

L. Zhao., B. Xu., W. Tang., Z. Chen., “A pixel-level Multisensory Image Fusion Algorithm based on fuzzy logic,” in L. Wang and Y. Jin (Eds.): FSKD 2005, LNAI 3613, pp. 717-720, 2005. © SpringerVerlag Berlin Heidelberg 2005.

[4]

L. Gang., J. Zhong-liang., S. Shao-yuan., “Multiresolution image fusion scheme based on fuzzy region feature,” in Journal of Zhejiang University SCIENCE A 2006 7(2):117-122.

[5]

Y. Wang, Z. Chen, “Performance Evaluation of Several Fusion Approaches for CCD/SAR Images,” Chinese Geographical Science, 13 (2003) 91-96.

[6]

http://mathworld.wolfram.com/K-MeansClusteringAlgorithm.html.

[7]

S.R. Jang, C.T. Sun, E. Mizutani, “Neuro-Fuzzy and Soft Computing: A Computational Approach to Learning and Machine Intelligence,” Prentice Hall Inc, USA, 1997.

[8]

Y. Zheng, E. A. Essock, B. C. Hansen, “An Advanced Image Fusion Algorithm Based on Wavelet Transform – Incorporation with PCA and morphological Processing,” in Proceedings of the SPIE, vol 5298, pp. 177- 187, 2004.

[9]

J. Flusser, F. Sroubek, B. Zitov, “Image Fusion: Principles, Methods, and Applications, Tutorial EUSIPCO 2007,” Institute of Information Theory and Automation Academy of Sciences of the Czech Republic, 182 08 P.

[10] I. De, B. Chanda, “A simple and efficient algorithm for multifocus image fusion using morphological wavelets,” Signal Processing 86 (2006) 924–936. [11] W. Zhong-hua, Q. Zheng, L. Yu, “A framework of region-based dynamic image fusion,” Journal of Zhejiang University Science A 2007 8(1):56-62. [12] J. Heijmans, J. Goutsias, “Multiresolution signal decomposition schemes,” Part 2: morphological wavelets, IEEE Trans. Image Process. 9 (November 2000) 1897–1913. [13] S. Md. Mansoor Roomi, T. Pandy Maheswari, V. Abhai Kumar, “A Detail Preserving Filter for Impulse Noise Detection and Removal”, ICGST-GVIP. Volume 7, Issue 3, November 2007.