Research Article
Multi-sensor image fusion based on regional characteristics
International Journal of Distributed Sensor Networks 2017, Vol. 13(11) Ó The Author(s) 2017 DOI: 10.1177/1550147717741105 journals.sagepub.com/home/ijdsn
Fanjie Meng1, Ruixia Shi1, Dalong Shan1, Yang Song2, Wangpeng He1 and Weidong Cai2
Abstract Multi-sensor data fusion method has been widely investigated in recent years. This article presents a novel fusion algorithm based on regional characteristics for combining infrared and visible light images in order to achieve an image with clear objects and high-resolution scene. First, infrared objects are extracted by region growing and guided filter. Second, the whole scene is divided into the objects region, the smooth region, and the texture region according to different regional characteristics. Third, the non-subsampled contourlet transform is used on infrared and visible images. Then, different fusion rules are applied to different regions, respectively. Finally, the fused image is constructed by the inverse non-subsampled contourlet transform with all coefficients. Experimental results demonstrate that the proposed objects extraction algorithm and the fusion algorithm have good performance in objective and subjective assessments. Keywords Image fusion, objects extraction, non-subsampled contourlet transform, multi-sensor data processing, image processing
Date received: 30 August 2017; accepted: 15 October 2017 Handling Editor: Zhi-Bo Yang
Introduction Multi-sensor data-based techniques have the advantages of providing sufficient information and thus have drawn much attention in recent years. For example, in the fields of structural health monitoring (SHM), vibration signal analysis has been proved as an effective tool for condition monitoring and fault diagnosis.1–3 However, in some cases, single-sensor data-based SHM cannot provide enough information to accurately detect the operational condition of complex mechanical equipment. Therefore, it is of great importance to develop effective multi-sensor data-based signal processing techniques. Image fusion is a technique that extracts available information from two images or more to merge into an image showing more information about the scene. Infrared (IR) images are acquired mainly based on the spatial distribution of IR radiation of objects while visible (ViS) light images are obtained based on the light reflection of objects. At night or in
the harsh climate condition, IR images can represent objects better than ViS images, but the resolution of IR images is relatively low and the visual effect is fuzzy.4 Therefore, in order to obtain an image with clear objects and high-resolution scene, IR and ViS images should be combined to fuse. So far, the fusion of IR and ViS images has been widely applied in many fields such as concealed weapon detection5 and helicopter navigation aid.6
1
School of Aerospace Science and Technology, Xidian University, Xi’an, P.R. China 2 School of Information Technologies, The University of Sydney, Sydney, NSW, Australia Corresponding author: Wangpeng He, School of Aerospace Science and Technology, Xidian University, Xi’an 710071, P.R. China. Email:
[email protected]
Creative Commons CC-BY: This article is distributed under the terms of the Creative Commons Attribution 4.0 License (http://www.creativecommons.org/licenses/by/4.0/) which permits any use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access pages (http://www.uk.sagepub.com/aboutus/ openaccess.htm).
2
International Journal of Distributed Sensor Networks
In recent years, many fusion methods have been developed. Most of these methods can be divided into two categories which are spatial domain fusion and transform domain fusion. Fusion methods in spatial domain are directly performed in pixel gray level7 or color space,8 such as linear weight method, principal component analysis (PCA) method, and pseudo color image fusion method. For methods in transform domain, source images are converted into a transformed representation, then different fusion rules are applied to different scales and resolutions, and finally the inverse transform is used to obtain the fused image. Traditional transform tools include the pyramid transform, wavelet transform, curvelet transform, and the non-subsampled contourlet transform (NSCT).9–11 However, a good fusion method not only relies on the transform but also depends on the fusion of coefficients in the transform domain. Traditional methods that adopt a single fusion rule have the disadvantage of dimming objects in the fused image.12–14 In this article, a fusion method for IR and ViS images based on regional characteristics is proposed. Our work differs from existing works in three aspects: 1.
2.
3.
Objects are extracted from the IR image with methods of region growing and guided filter. Each object in the IR image is taken into account, and adaptive filtering is realized by improving the size of local window of guided filter. The proposed objects extraction method not only guarantees the integrity of objects but also avoids the introduction of the background. Aiming at obtaining a scene with higher resolution and making full use of complementary information from IR and ViS images, the gradient map of the ViS image is computed and the gradient map is clustered. By combining the segmentation of the gradient map and the object image, the whole scene can be segmented effectively. Image fusion is performed on the enhanced IR image and the region division map, which makes the fused image have clear objects and highresolution scene.
The rest of this article is organized as follows: Recent fusion methods are introduced in section ‘‘Related works.’’ The fusion framework is given in section ‘‘The system overview.’’ The extraction of the objects region is described in section ‘‘Extraction of IR objects.’’ The segmentation of the scene is presented in section ‘‘Division of the scene.’’ The fusion rule is given in section ‘‘Fusion algorithm based on regional characteristics of the image.’’ Experimental results are performed in section ‘‘Experimental results and analysis’’
and conclusions ‘‘Conclusion.’’
are
summarized
in
section
Related works Up to date, most image fusion algorithms are performed in transform domain, and the tool of the NSCT is the most popular. The NSCT can implement multiscale and multi-directional decomposition, respectively, based on a non-subsampled pyramid (NSP) and nonsubsampled directional filter banks. The structure of the NSCT is similar to that of contour transform, but compared with contour transform, the NSCT retains multi-resolution and multi-direction information and provides translation invariance as well as elimination of the Gibbs phenomenon. Furthermore, it overcomes the limitation of the direction in traditional wavelet transform and variation in translation in contour transform and can effectively reduce the impacts of misregistration in fusion results. Therefore, the NSCT is more suitable for image fusion. However, most NSCTbased image fusion algorithms mainly apply fusion rules of the independent pixel gray value and the independent window, which may influence the quality of the fused image. In particular, two main issues can result from such a fusion approach: 1. 2.
The contrast between objects and background is low. Visual effect of the fused image is poor, and there is a loss of detailed information.
To solve the above problems, an increasing number of algorithms based on region division and object extraction are proposed. These algorithms apply different fusion rules to different regions of source images according to characteristics of different regions, which is of great help to improve the quality of the fused image. Sun et al.15 combine region growing and the non-subsampled shearlet transform and propose the fusion method based on object extraction and adaptive pulse coupled neural network. This method applies the regional information to the fusion process, but it fails to concern the objects region for region division. Han et al.16 feed the co-occurrence of hot spots and motion into a markov random field (MRF) model to generate the saliency map for the IR image, then source images are decomposed by wavelet transform, and low coefficients are fused by the method of weighted average and the choice of weighted coefficient is determined by the saliency map. This method depicts the salient foreground object clearly and the speed of fusion has been improved greatly; however, the edge of objects in the fused image is poor. Zhao et al.17 identify regions-ofinterest based on region saliency detection in both
Meng et al. imagery frames, and then segment the images into target and background regions. Different fusion rules are adopted respectively in target and background regions. Although the quantitative fusion result is improved, the visual effect is still not very well. Yang et al.18 propose a region fusion method based on watershed. The method improves the objective evaluation index of the fused image, but the segmentation method merely considers the global characteristics of source images without considering the local ones. Luo et al.19 propose a fusion method based on structure similarity. The region map is generated from the structural similarity (SSIM) index between source images and the fused image. However, the region map is influenced by fusion rules of the initial fused image. Wang and Fu20 and Xu et al.21 have proposed novel algorithms using region growing and gray theory to extract IR object. However, these approaches consider only objects region while without characteristics of the scene. Wang and Du22 proposed an algorithm based on image segmentation. After object segmentation, different rules are applied to objects region and background region. However, the disadvantage is that the robustness of the segmentation approach is not high, and the segmented object would include some background region.
3
Extraction of IR objects Location of the objects region The significance of image fusion of IR and ViS images is to use the complementary information of both images to get a fused image that expresses objects and the whole scene accurately, which not only include clear objects but also the higher resolution scene. Therefore, the precise localization of IR objects is particularly important. Current methods of IR objects extraction are based on gray theory,20 OTSU,23 Renyi entropy,24 and so on. These methods can extract small objects, but always include information from the background and cannot extract larger objects completely. To solve these problems, we combine region growing and guided filter to extract objects. The detail of the proposed method is as follows. To improve the contrast of objects and the background, and get the clear edge of objects, the IR image is enhanced 255 k expð10 A(i, j)=255 5Þ + 1 u = mean(A) t
A1(i, j) =
k=
The system overview Figure 1 depicts the proposed system architecture with its main functional units and data flows. The functions of the key modules are as follows: 1.
2.
3.
Extraction of IR objects: IR objects are extracted from the IR image based on approaches of region growing and guided filter. Segmentation of the whole scene: first, the gradient map of the ViS image is calculated by Sobel operator. Second, the ViS image is divided into the smooth region and the texture region using K-means clustering on the gradient map. By combining the map of background division of the ViS image and the object image, the division map of the scene is obtained. Fusion of images: first, region mapping is conducted on the enhanced IR image, the ViS image, and the IR image according to the region division map. Second, high-frequency and lowfrequency direction subbands are obtained by the NSCT from source images. Third, according to the different characteristics of regions, different fusion rules are used to fuse the highfrequency and low-frequency subbands of source images, respectively. Finally, the fused image is obtained by the inverse NSCT transform.
ð1Þ
255=u 1 expðð10 (u=255) 5ÞÞ
where A(i, j) represents the gray value of the pixel localized at (i, j), A1(i, j) is the enhanced gray value, and t is the constant coefficient. The above process will enhance pixels whose gray values are greater than u and weaken pixels whose gray values are less than u; therefore, this process can greatly improve the contrast of objects and the background of the IR image. In order to avoid the enhancement of the background, it is shown by experiments that the most appropriate value of t is set between 1 and 2. We introduce the region growing method to extract objects from the IR image which examines neighboring pixels of initial seed points and determines whether the pixel neighbors should be added to the region. Step 1. Since objects in the IR image have larger thermal emissivity, the brightness of objects is always higher than the background. Considering multiobjects of the IR image and in order not to lose objects, select pixels with relatively higher gray value as seed points. The selection of seeds is as follows Z = fA(i, j)jA(i, j).M AV =8g
ð2Þ
where M is the max gray value of the IR image A, AV is the average gray value of the IR image A, and Z is the set of seed points of the IR image A.
4
International Journal of Distributed Sensor Networks
Figure 1. Block diagram of the proposed method.
Step 2. According to seed points of the IR image A, pixels in the corresponding position of the enhanced IR image A1 are regarded as the set of seed points Z1 of A1. The rule of the growth is as follows: 1. Record the max gray value of pixels in A1 as T, T = max(A1). 2. Take T as the threshold and apply it into the following growth formula absðA1(i, j) Z1(i, j)Þ\
T 4
ð3Þ
where A1(i, j) represents the gray value of pixel (i, j) in the enhanced IR image A1, Z1(i, j) represents the gray
value of pixel in the set of Z1; if A1(i, j) satisfies equation (3), pixel (i, j) in the enhanced IR image A1 will be incorporated into the grown region.
3. Traverse each seed point of Z1 and repeat Step (2), the initial object map IO as shows in Figure 2(a3)–(d3) can be achieved. The selection process of seed points may lead to the introduction of local region of the background, and therefore interference region needs to be eliminated. It is believed that relatively large regions are always objects, while relatively small ones are always
Meng et al.
5
Figure 2. Extraction of IR objects: (a1)–(d1) source images; (a2)–(d2) the enhanced images of (a1)–(d1); (a3)–(d3) the initial object images IO; (a4)–(d4) the images IO1; (a5), (b5), (c5)–(b7), (d5)–(d8) the windows of the guided filter; (a6), (b6), (c8)–(c10), (d9)– (d12) the images filtered by (a5), (b5), (c5)–(b7), (d5)–(d8); (a7), (b7), (c11), (d13) the final object images IO2.
6
International Journal of Distributed Sensor Networks interference regions. Small regions which satisfied the following formula will be eliminated IO1 = fIO(m)\IOM =3g,
m 2 f1, 2, . . . , Lg
ð4Þ
where IOM is the largest region in the initial object map IO, L is the number of regions in the initial object map IO, and IO(m) is the mth region the initial object map. As shown in Figure 2(a4)–(d4), IO1 is the approximate location of objects. To achieve complete objects, IO1 is needed for further processing.
Acquisition of the local region of objects Guide filter25 is an edge aware filter which filters the input image by taking into account contents of the guidance image. Guided filter is a local linear model between the guidance image I and the output image Q. The guided filter is used to obtain the local region of objects in this article. The definition is as follows Qi = ak Ii + bk , 8i 2 vk P 8 1 Ii pi uk pk bvc < i2vk ak = d2k + e : bk = pk ak uk
n
n
n
n
n
J 2 f1, 2, . . . , Nn g ð8Þ
To obtain a filtered image including complete object, calculate the mean value of all RnJ and set nJ =5. rn = max (RnJ ) R Third, according to the filtering formula (8), set vk = rn 3 rn , n 2 f1, 2, . . . , L1), and let I = A1 as the guidance image and pn which is the object image separated from IO1 as the input image, then the output images qn including local objects region are achieved, as shown in Figure 2(a6), (b6), (c8)–(c10), and (d9)– (d12).
Extraction of objects
ð6Þ
The method of region growing is used in qn , n 2 f1, 2, . . . , L1g. First, pixels with the maximum gray value in qn are selected as seeds
"
n
1=2 RnJ = (iJ Oni )2 + (jJ Onj )2 ,
ð5Þ
where (ak , bk ) is a constant coefficient in vk and vk is the size of a local window centered at pixel k. jvj is the number of pixels in vk and pk is the mean of P in vk . uk and d2k are the mean and variance of I in vk , respectively. The enhanced IR image A1 retains good edge characteristics of IR objects, and therefore A1 is selected as the guidance image and IO1 as the input image. The output image which includes more complete objects and maintains the edge features of A1 much better is achieved. However, the determination of the size of the local window is a problem since there may be multiple objects in the input image and the size of each object is different. If the value of vk is larger, objects will be mixed with the background; on the contrary, if the value of vk is smaller, complete objects cannot be obtained. To obtain an output image with clean and complete objects, the allocation of the appropriate value of vk to each object is important. The selection of the size of vk is as given below. First, separate the region of each object in the object image IO1 and use a single image to represent each object, and then the centroid of each object image On is calculated # 1 X 1 X On = (Oni , Onj ) = in , jn , Nn (i , j )2N Nn (i , j )2N
image, and (in , jn ) is the coordinates of the pixel in the object of the nth image. Second, calculate the distance between each pixel in the object and its centroid
ð7Þ
n 2 f1, 2, . . . , L1g
where L1 is the number of objects in the object image IO1, Nn is the number of pixels in the object of the nth
Tn = maxfqn (i, j)g
ð9Þ
Then, the growth criterion is chosen as follows absðqn (i, j) Tn Þ\Tn =2,
n 2 f1, 2, . . . , L1g
ð10Þ
where qn (i, j) is the gray value of the pixel located in (i, j) of the image. Finally, all the object images are integrated to achieve the final object image IO2, as shown in Figure 2(a7), (b7), (c11), and (d13).
Division of the scene The ViS image has an advantage of high spatial resolution, thus the ViS image can be divided into different regions according to different regional characteristics of the scene. First, the gradient map of the ViS image is calculated by the Sobel operator. The Sobel operator horizontal gradient is hx = ½1, 0, 1; 2, 0, 2; 1, 0, 1
The vertical gradient is hy = hx0 . Then the ViS image is filtered by hx and hy, respectively Gx = filter(hx, B), Gy = filter(hy, B) G = abs(Gx) + abs(Gy)
ð11Þ
where Gx and Gy are the horizontal gradient and the vertical gradient of the ViS image B, respectively. G is
Meng et al.
7
Figure 3. Region division results of images: (a1)–(k1) ViS images, (a2)–(k2) IR images, (a3)–(k3) IR object images, (a4)–(k4) region division results of ViS images, and (a5)–(k5) final region division images.
the gradient map of ViS image B, in which the value of each pixel represents the gradient of each pixel in the ViS image. The greater the value represents the richer detailed information the current location has. Then, the K-means method26 is used to cluster image G. According to the average gradient of each cluster, the ViS image is divided into the smooth region and the texture region. Joining the object image and the scene divided image, an image which is divided into the objects region, the smooth region, and the texture region is achieved. The experiment results are shown in Figure 3(a5)–(k5), in which the black region represents
the texture region, the white region represents the objects region, and the gray region represents the smooth region.
Fusion algorithm based on regional characteristics of the image Fusion of the objects region To enhance the contrast of objects and the background, the objects region of the enhanced IR image A1 and the objects region of the ViS image B are used to fuse. First, the local energies of low-frequency coefficients are calculated
8
International Journal of Distributed Sensor Networks X
ET (i, j) =
LT (i + x, j + y)2 ,
T 2 (oA1, oB) ð12Þ
ETL (i, j) =
where W is the size of the window and LT (i, j) is the low-pass subband coefficient of point (i, j). Fusion rules of low-frequency coefficient of the objects region are as follows LoF (i, j) = aLoA1 (i, j) + bLoB (i, j)
ð13Þ
where a = EoA1 (i, j)=(EoA1 (i, j) + EoB (i, j)), b = 1 a, LoF (i, j) is the coefficient of fused objects region, and LoA1 (i, j) and LoB (i, j) are the low coefficients of oA1 and oB, respectively. High-frequency coefficient used the rule of higher absolute value HoF (i, j) =
HoA1 (i, j) if absðHoA1 (i, j)Þ.absðHoB (i, j)Þ ð14Þ HoB (i, j) if absðHoA1 (i, j)Þ absðHoB (i, j)Þ
where HoA1=oB (i, j) is the high-frequency coefficient of pixel (i, j).
Fusion of the smooth region Since the information in the smooth region is not rich enough and the brightness information are the dominant regional characteristics, the following fusion rules are used LsF (i, j) = HsF (i, j) =
LsA (i, j) if EsA (i, j) EsB (i, j) LsB (i, j) if EsA (i, j)\EsB (i, j)
ð15Þ
HsA (i, j) if absðHsA (i, j)Þ absðHsB (i, j)Þ HsB (i, j) if absðHsA (i, j)Þ\absðHsB (i, j)Þ ð16Þ
where Lst (i, j), Hst (i, j), t 2 (A, B) are low-frequency and high-frequency coefficients of source images, respectively. ESA (i, j) and ESB (i, j) are the local energy of point (i, j) of SA and SB, respectively.
Fusion of the texture region The texture region contains a great deal of edge, texture, and direction information. In order to extract more texture information and high-frequency information, local gradient energy of image (EOG) and local correlation coefficient are used to fuse low-frequency coefficients. EOG is defined as follows EOG(i, j) =
EOGT L (i + p, j + q)2 ,
T 2 (tA, tB)
p, q2w
x, y2W
X
XX
ðf (i + 1, j) f (i, j)Þ2
i2m j2n
+ ðf (i, j + 1) f (i, j)Þ2
ð17Þ
ð18Þ E3T (i, j) = 2 3 ETL (i 1, j 1) ETL (i 1, j) ETL (i 1, j + 1) 6 7 ETL (i, j) ETL (i, j + 1) 5, 4 ETL (i, j 1) ETL (i + 1, j 1) ETL (i + 1, j) ETL (i + 1, j + 1) T 2 (tA, tB) ð19Þ CR1(i, j) = corr2(E3tA(i, j) , E3tB(i, j) )
ð20Þ
where CR1(i, j) denotes the correlation coefficient of local texture between tA(i, j) and tB(i, j); the greater the value, the more similar they are. Set h = (1 CR1(i, j))=2 + 0:5, when CR1(i, j) h, indicating that the texture information of tA is quite different from that of tB; in order to make full use of the complementary information of tA and tB, the following fusion rules are applied LtF (i, j) =
EtAL (i, j) LtA (i, j) EtAL (i, j) + EtBL (i, j) EtBL (i, j) + LtB (i, j) EtAL (i, j) + EtBL (i, j)
ð21Þ
When CR1(i, j).h, indicating that the texture information of tA is similar to tB ItF (i, j) =
LtA (i, j), EtAL (i, j).EtAL (i, j) LtB (i, j), EtAL (i, j) EtBL (i, j)
ð22Þ
where ETL (i, j) is the local EOG of the texture region, E3T (i, j) is the 3 3 3 neighborhood of the matrix of ETL at point (i, j), CR1(i, j) is the correlation coefficient between E3tA (i, j) and E3tB (i, j), LtA (i, j) and LtB (i, j) are the low coefficients of the texture region of tA and tB, respectively, and LtF (i, j) is the coefficient of the fused texture region. The sharpness27 of point and local correlation coefficient are used to fuse high-frequency coefficients. Sharpness of point is the total weighted deviations between the point and its eight neighborhoods. It is a statistic of the variations’ degree of gray around each point in the image; the higher the value, the clearer the image. The fusion rules are as follows
Meng et al.
9
X ∂2 f psT (i, j) = ∂l2 = j2 f (i, j) f (i 1, j) f (i + 1, j)j + j2 f (i, j) f (i, j 1) f (i, j + 1)j 1 + pffiffiffi j4 f (i, j) f (i 1, j 1) f (i + 1, j + 1) f (i 1, j + 1) f (i + 1, j 1)j 2 PS3T (i, j) = 2 psT (i 1, j 1) 6 4 psT (i, j 1) psT (i + 1, j 1)
psT (i 1, j)
psT (i 1, j + 1)
3
7 psT (i, j) psT (i, j + 1) 5, psT (i + 1, j) psT (i + 1, j + 1)
T 2 (tA, tB) ð24Þ CR2(i, j) = corr2(PS3tA(i, j) , PS3tB(i, j) )
ð25Þ
Set h = (1 CR2(i, j))=2 + 0:5, when CR2(i, j) h HtF (i, j) =
pstA (i, j) HtA (i, j) pstA (i, j) + pstB (i, j) pstB (i, j) + HtB (i, j) pstA (i, j) + pstB (i, j)
ð26Þ
When CR2(i, j).h HtF (i, j) =
HtA (i, j), pstA (i, j).pstB (i, j) HtB (i, j), pstA (i, j) pstB (i, j)
ð27Þ
where psT (i, j) is the sharpness of point (i, j) of the texture region, PS3T (i, j) is the 3 3 3 neighborhood of the matrix of psT at point (i, j), CR2(i, j) is the correlation coefficient between ps3tA (i, j) and ps3tB (i, j), HtA (i, j) and HtB (i, j) are the high coefficients of the texture region of tA and tB, respectively, and HtF (i, j) is the coefficient of the fused texture region.
Experimental results and analysis In this article, our experiment platform consisted of Core i5-3470 CPU at 3.20 GHz, with 8.0 GB memory, using MS-Windows 7, on which we ran MathWorks MATLAB R2013a software. Our experiments can be divided into two parts. First, we test our objects extraction algorithm. Second, the fusion algorithm is assessed.
Evaluation of the objects extraction algorithm Our objects extraction algorithm is tested with 11 pairs of IR images. The ground truth data for each image are manually labeled with Adobe Photoshop CS3. We also compared it with two recent algorithms, which are based on gray theory20 and Renyi entropy.24 The algorithm based on gray theory takes the IR image as gray system which includes a part of the known information and part of the unknown information, and then the
ð23Þ
theory of gray correlation is applied to detect and extract the target of IR image. The algorithm based on Renyi entropy through finding out a threshold makes the sum of entropy of the segmented target image and background image reach maximum to extract object. As shown in Figure 4, although the algorithm based on Renyi entropy can extract the object, it has not considered the introduction of interference regions. The algorithm based on gray theory can extract smaller object of IR image, and its result is quiet impressive. However, when the object is large and the scene is sophisticated, this algorithm cannot extract the object completely. On the contrary, the proposed algorithm can not only extract object completely but also avoid the introduction of the background information. To evaluate the effectiveness of the proposed algorithm accurately and comprehensively, we also adopt objective evaluation criteria of mean precision and recall rate as well as the comprehensive evaluation index Fmeasure to evaluate the proposed algorithm. The precision, recall, and F-measure27–29 are given as follows precision =
TP TP ; recall = TP + FP TP + FN
Fmeasure =
(1 + b) 3 precision 3 recall b 3 precision + recall
ð28Þ ð29Þ
where TP is true positive, FP refers to false positive, and FN means false negative. Experimental results are shown in Table 1 and Figure 5.
Evaluation of the fusion algorithm To validate the fusion effect, the proposed fusion algorithm is compared with several recently developed methods. The experimental results shown in Figure 6(a1)–(a4), (b1)–(b4), (c1)–(c4), (d1)–(d4), (e1)– (e4), (f1)–(f4) are fusion methods based on NSCT in recent years. It can be seen that the contrast between objects and the background is low and objects are blur. (a5)–(f5) are the results of applying the fusion rules of the objects region to the smooth region and the texture region. (a6)–(f6) are the results of applying fusion rules of the smooth region to the smooth region and the texture region. As can be seen from (a5)–(f5) and (a6)– (f6), a relatively complete objects region is achieved and the contrast of objects and background is enhanced, but the resolution of background is low. (a7)–(b7) are fusion results of our method, which
10
International Journal of Distributed Sensor Networks
Figure 4. (a) The source IR image, (b) the ground truth image of (a), (c) the extraction result based on Renyi entropy, (d) the extraction result based on gray theory, and (e) the extraction result of our proposed method.
Table 1. Objective evaluation of different methods of object extraction.
Gray theory Renyi entropy Our method
Precision
Recall
F-measure
0.3315 0.6712 0.9300
0.5249 0.9407 0.9508
0.4064 0.7838 0.9043
Meng et al.
11
Figure 5. Precision, recall, and F-measure for different algorithms of object extraction.
Table 2. Objective evaluation of different fusion methods of the first image.
Bhatnagar et al.12 Chen et al.13 Cao et al.14 Ge et al.30 Smooth Object Proposed
E
SF
AVG
EI
MI(V,F)
6.2625 6.8654 6.8323 6.8327 6.8368 6.8424 6.8772
13.8036 18.2587 17.9922 18.0406 17.6351 18.4917 18.7389
4.6042 6.9885 6.8626 6.8781 6.7949 7.0331 7.1306
47.0901 73.4108 71.9108 72.0914 71.0500 73.5343 74.6827
1.1676 2.4030 2.2234 2.2255 2.5033 2.1720 2.6179
E: entropy; SF: spatial resolution; AVG: average gradient; EI: edge intensity; MI(V,F): mutual information between the ViS image and the fused image.
Table 3. Objective evaluation of different fusion methods of the second image.
Dong et al.10 Kong et al.11 Bhatnagar et al.12 Hou and Zhang28 Smooth Object Proposed
E
SF
AVG
EI
MI(V,F)
6.0163 6.9340 6.9652 6.9651 6.9375 6.9699 6.9930
9.9073 16.0474 16.0827 16.1072 17.3072 16.0197 17. 5079
2.9681 5.2626 5.3235 5.3348 5.4830 5.3205 5.5172
30.3375 52.9714 53.6326 53.7318 55.1405 53.5215 55.5198
2.1179 3.6191 3.8452 3.8374 4.2921 3.8554 4.37701
E: entropy; SF: spatial resolution; AVG: average gradient; EI: edge intensity; MI(V,F): mutual information between the ViS image and the fused image.
include objects of higher contrast and the scene of higher resolution. In terms of visual effects, the fused image of the proposed method contains more details and obtains more salient information from source images, thus it can be seen that the proposed method has a better performance than other methods. In order to further evaluate the performances of our algorithm, Tables 2–7 and Figure 7 show the comparison
of different methods based on NSCT measured by objective criteria. In our experiments, objective evaluation criteria of entropy (E), spatial resolution (SF), average gradient (AVG), edge intensity (EI), and mutual information between the ViS image and the fused image (MI(V,F)) are selected to evaluate the proposed algorithm. It further proves the superiority of the proposed algorithm.
12
International Journal of Distributed Sensor Networks
Table 4. Objective evaluation of different fusion methods of the third image.
Dong et al.10 Kong et al.11 Bhatnagar et al.12 Hou and Zhang28 Smooth Object Proposed
E
SF
AVG
EI
MI(V,F)
6.6890 6.5671 6.6044 6.7562 6.7124 6.7185 6.7744
11.6666 11.7206 11.3636 11.5389 17.7196 16.8829 17.8542
4.3933 4.5264 4.1985 4.3766 6.0231 5.8735 6.0846
42.6594 44.3499 40.3014 42.6020 59.9959 58.6755 60.8256
1.3932 3.6100 3.4072 3.3953 4.2553 3.3864 4.0750
E: entropy; SF: spatial resolution; AVG: average gradient; EI: edge intensity; MI(V,F): mutual information between the ViS image and the fused image.
Table 5. Objective evaluation of different fusion methods of the fourth image.
Dong et al.10 Kong et al.11 Bhatnagar et al.12 Hou and Zhang28 Smooth Object Proposed
E
SF
AVG
EI
MI(V,F)
6.7629 6.8431 6.8942 6.8427 6.8863 6.9841 7.0264
9.6550 10.0243 10.4369 9.9982 10.5456 10.9444 10.9841
3.2241 4.1838 4.4110 4.1779 4.3481 4.5604 4.5758
32.9688 41.4499 44.1855 41.3665 42.9141 45.4240 45.5464
1.0893 1.5943 2.0201 1.5966 1.6652 1.5664 1.9324
E: entropy; SF: spatial resolution; AVG: average gradient; EI: edge intensity; MI(V,F): mutual information between the ViS image and the fused image.
Table 6. Objective evaluation of different fusion methods of the fifth image.
Dong et al.10 Kong et al.11 Bhatnagar et al.12 Hou and Zhang28 Smooth Object Proposed
E
SF
AVG
EI
MI(V,F)
6.2272 6.5161 6.6557 6.5144 6.5323 6.6686 6.6733
10.0691 11.0762 11.2280 11.0174 11.5658 11.7995 11.8704
4.2637 4.8795 5.0294 4.8680 5.1219 5.2399 5.2722
42.9642 48.3829 50.2894 48.1853 50.6205 51.8830 52.4385
0.8813 1.2589 1.4005 1.2603 1.4037 1.2170 1.4976
E: entropy; SF: spatial resolution; AVG: average gradient; EI: edge intensity; MI(V,F): mutual information between the ViS image and the fused image.
Table 7. Objective evaluation of different fusion methods of the sixth image.
Dong et al.10 Kong et al.11 Bhatnagar et al.12 Hou and Zhang28 Smooth Object Proposed
E
SF
AVG
EI
MI(V,F)
6.9507 7.3688 7.0645 7.3685 7.1258 7.1225 7.3731
8.2222 13.1059 13.2954 13.0930 13.5257 13.3822 13.6547
3.5206 4.8392 4.8988 4.8303 5.1114 5.0423 5.1547
35.9645 48.0746 48.9511 47.9689 51.3916 50.3368 51.7167
1.3406 3.1915 4.6221 3.1910 3.4215 4.0050 4.2553
E: entropy; SF: spatial resolution; AVG: average gradient; EI: edge intensity; MI(V,F): mutual information between the ViS image and the fused image.
Conclusion This article proposes a multi-sensor fusion method based on regional characteristics of the image. This is
inspired by the fusion method based on object extraction. The major difference between traditional algorithms and our algorithm is that our algorithm takes the multi-objects and the introduction of the scene
Meng et al.
Figure 6. Fusion results of different fusion methods: (a1)–(f1) fused images of Bhatnagar et al.,12 (a2)–(f2) fused images of Chen et al.,13 (a3)–(f3) fused images of Achanta et al.,29 (a4)–(f4) fused images of Cao et al.,14 (a5)–(f5) fused images based on the fusion rule of the objects region, (a6)–(f6) fused images based on the fusion rule of the smooth region, and (a7)–(f7) fused images of the proposed method.
13
14
International Journal of Distributed Sensor Networks
Figure 7. Objective evaluation of our proposed algorithm and other algorithms.
information into account. In order to enhance the contrast of the objects region and the background and get the scene with higher resolution, we fused the objects region of enhanced IR image and ViS image. The scene is divided into different regions according to different regional characteristics, and then different fusion rules are applied to different regions. This method not only retains the IR objects but also obtains the spatial domain information of source images effectively. Through the objective and subjective analyses, the experimental results show that the proposed method is superior to other methods. Declaration of conflicting interests The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Natural Science Foundation of China under grant 61305040 and the Fundamental Research Funds for the Central Universities under grant JB161305.
References 1. Yang Z, Radzienski M, Kudela P, et al. Damage detection in beam-like composite structures via Chebyshev pseudo spectral modal curvature. Compos Struct 2017; 168: 1–12. 2. Xiang J, Zhong Y, Gao H, et al. Rolling element bearing fault detection using PPCA and spectral kurtosis. Measurement 2015; 75: 180–191. 3. He W, Ding Y, Zi Y, et al. Repetitive transients extraction algorithm for detecting bearing faults. Mech Syst Signal Pr 2017; 84: 227–244.
Meng et al. 4. Fan X and Su S. An enhanced infrared and visible image fusion method based on wavelet transform. In: Proceedings of the 5th international conference on intelligent humanmachine systems and cybernetics, Hangzhou, China, 26–27 August 2013, vol. 49, pp.453–456. New York: IEEE. 5. Hussein N, Hu F and He F. Multisensor of thermal and visual images to detect concealed weapon using harmony search image fusion approach. Pattern Recogn Lett 2017; 94: 219–227. 6. Amolins K, Zhang Y and Dare P. Wavelet based image fusion techniques—an introduction, review and comparison. ISPRS J Photogramm 2007; 62(4): 249–263. 7. Prakash O, Kumar A and Khare A. Pixel-level image fusion scheme based on steerable pyramid wavelet transform using absolute maximum selection fusion rule. In: Proceedings of the 2014 international conference on issues and challenges in intelligent computing techniques (ICICT), Ghaziabad, India, 7–8 February 2014, pp.765– 770. New York: IEEE. 8. Yang S, Liu W, Deng C, et al. Color fusion method for low-light-level and infrared images in night vision. In: Proceedings of the 2012 5th international congress on IEEE image and signal processing (CISP), Chongqing, China, 16–18 October 2012, pp.534–537. New York: IEEE. 9. Singh R and Khare A. Fusion of multimodal medical images using Daubechies complex wavelet transform—a multiresolution approach. Inform Fusion 2014; 19: 49–60. 10. Dong L, Yang Q, Wu H, et al. High quality multispectral and panchromatic image fusion technologies based on Curvelet transform. Neurocomputing 2015; 159: 268–274. 11. Kong W, Lei Y, Lei Y, et al. Image fusion technique based on non-subsampled contourlet transform and adaptive unit-fast-linking pulse-coupled neural network. IET Image Process 2011; 5(2): 113–121. 12. Bhatnagar G, Wu Q and Liu Z. Directive contrast based multimodal medical image fusion in NSCT domain. IEEE T Multimedia 2013; 15(5): 1014–1024. 13. Chen Z, Zhang C and Wang P. High-quality fusion for visible and infrared images based on the double NSCT. In: International congress on image and signal processing, 2015, pp.223–227. New York: IEEE. 14. Cao L, Jin L, Tao H, et al. Multi-focus image fusion based on spatial frequency in discrete cosine transform domain. IEEE Signal Proc Let 2015; 22(6): 220–224. 15. Sun W, Hu S, Liu S, et al. Infrared and visible image fusion based on object extraction and adaptive pulse coupled neural network via non-subsampled Shearlet transform. In: Proceedings of the 12th international conference on signal processing, Hangzhou, China, 19–23 October 2014, pp.946–951. New York: IEEE. 16. Han J, Pauwels E and Zeeuw P. Fast saliency-aware multi-modality image fusion. Neurocomputing 2013; 111(6): 70–80. 17. Zhao B, Li Z, Liu M, et al. Infrared and visible imagery fusion based on region saliency detection for 24-hour-surveillance systems. In: Proceedings of the 2013 IEEE international conference on robotics and biomimetics,
15
18.
19.
20.
21.
22.
23.
24.
25. 26.
27.
28.
29.
30.
Shenzhen, China, 12–14 December 2013, pp.1083–1088. New York: IEEE. Yang Y, Dai M and Zhou L. Fusion of infrared and visible images based on NSUDCT. Infrared Laser Eng 2014; 43(3): 961–966. Luo X, Zhang Z and Wu X. Image fusion using region segmentation and sigmoid function. In: Proceedings of the 2014 22nd international conference on pattern recognition (ICPR) Stockholm, 24–28 August 2014, pp.1049–1054. New York: IEEE. Wang C and Fu Y. Fusion of infrared and visible light images based on the grey theory to object extraction. In: Proceedings of the 2015 12th IEEE international conference on electronic measurement & instruments, Qingdao, China, 16–18 July 2015, vol. 3, pp.1203–1207. New York: IEEE. Xu Q, Cheng Y and Zhao H. An infrared and visible image fusion method based on improved region growing algorithm. Electron Optic Contr 2014; 21(8): 54–57. Wang J and Du S. Study on the image segmentation of field crops based on the fusion of infrared and visiblelight images. In: Proceedings of the 2010 symposium on photonics and optoelectronic (SOPO), Chengdu, China, 19–21 June 2010, pp.1–4. New York: IEEE. Chen Q, Zhao L, Lu J, et al. Modified two-dimensional OTSU image segmentation algorithm and fast realisation. IET Image Process 2012; 6: 426–433. Fan S and Yang S. Infrared electric image segmentation using fuzzy Renyi entropy and chaos differential evolution algorithm. In: Proceedings of the 2011 international conference on future computer sciences and application (ICFCSA), Hong Kong, China, 18–19 June 2011, pp.220–223. New York: IEEE. He K, Sun J and Tang X. Guided image filtering. IEEE T Pattern Anal 2013; 35(6): 1397–1409. Wang J and Su X. An improved K-means clustering algorithm. In: Proceedings of the 2011 IEEE 3rd international conference on communication software and networks (ICCSN), Xi’an, China, 27–29 May 2011, pp.44–46. New York: IEEE. Hassen R, Wang Z and Salama M. Image sharpness assessment based on local phase coherence. IEEE T Image Process 2013; 22(7): 2798–2810. Hou X and Zhang L. Saliency detection: a spectral residual approach. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2007 (CVPR 2007), Minneapolis, MN, 17–22 June 2007, pp.1–8. New York: IEEE. Achanta R, Hemami S, Estrada F, et al. Frequency-tuned salient region detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2009 (CVPR 2009), Miami, FL, 20–25 June 2009, pp.1597– 1604. New York: IEEE. Ge W, Ji P and Zhao T. Infrared image and visual image fusion algorithm based on NSCT and improved regional cross entropy. In: Proceedings of the 2015 27th Chinese control and decision conference (CCDC), Qingdao, China, 23–25 May 2015, pp.3645–3649. New York: IEEE.
