Guided Image Filtering-Based Pan-Sharpening Method

International Journal of

Geo-Information Article

Guided Image Filtering-Based Pan-Sharpening Method: A Case Study of GaoFen-2 Imagery Yalan Zheng 1 , Qinling Dai 2,3 , Zhigang Tu 4 and Leiguang Wang 1, * 1 2 3 4

*

Faculty of Forestry, Southwest Forestry University, Kunming 650224, China; [email protected] School of Printing and Packaging, Wuhan University, Wuhan 430079, China; [email protected] Faculty of Design, Southwest Forestry University, Kunming 650224, China School of Electrical and Electronic, Nanyang Technological University, Singapore 637553, Singapore; [email protected] Correspondence: [email protected]; Tel.: +86-150-8702-0570

Received: 13 September 2017; Accepted: 7 December 2017; Published: 12 December 2017

Abstract: GaoFen-2 (GF-2) is a civilian optical satellite self-developed by China equipped with both multispectral and panchromatic sensors, and is the first satellite in China with a resolution below 1 m. Because the pan-sharpening methods on GF-2 imagery have not been a focus of previous works, we propose a novel pan-sharpening method based on guided image filtering and compare the performance to state-of-the-art methods on GF-2 images. Guided image filtering was introduced to decompose and transfer the details and structures from the original panchromatic and multispectral bands. Thereafter, an adaptive model that considers the local spectral relationship was designed to properly inject spatial information back into the original spectral bands. Four pairs of GF-2 images acquired from urban, water body, cropland, and forest areas were selected for the experiments. Both quantitative and visual inspections were used for the assessment. The experimental results demonstrated that for GF-2 imagery acquired over different scenes, the proposed approach consistently achieves high spectral fidelity and enhances spatial details, thereby benefitting the potential classification procedures. Keywords: remote sensing; image fusion; guided filtering; GF-2 imagery

1. Introduction Currently, numerous digital aerial cameras and optical earth observation satellites such as QuickBird, WorldView-3, and GaoFen-2 (GF-2) exist that can simultaneously obtain multispectral (MS) and panchromatic (Pan) images [1]. Due to physical constraints, high-resolution Pan images lack the spectral information of MS images, while MS images often have a lower spatial resolution. To synergistically utilize these images for various applications, such as detailed land cover classification, change detection, and so on, it has become increasingly important to integrate the strengths of both types [2,3]. The GF-2 satellite was launched in August 2014. It is a civilian optical remote sensing satellite developed by China and the first satellite in China with a resolution below 1 m. This satellite is equipped with both a panchromatic sensor and multispectral sensor that can be used simultaneously. The GF-2 can achieve a spatial resolution of 0.8 m with a swath of 48 km in panchromatic mode; in contrast, the satellite acquires images with a resolution of 3.2 m in 4 spectral bands in multispectral mode. Furthermore, it is also characterized by high radiation accuracy, high positioning accuracy, and fast attitude maneuverability, among other features. With its low cost and availability, this satellite can benefit many possible applications in China, such as detailed land cover/use classification, change detection, and landscape design. As a recently launched optical satellite, exploring effective sharpening approaches to expand the application scope of the images is important. ISPRS Int. J. Geo-Inf. 2017, 6, 404; doi:10.3390/ijgi6120404

www.mdpi.com/journal/ijgi

ISPRS Int. J. Geo-Inf. 2017, 6, 404

2 of 22

Many pan-sharpening methods have been proposed to achieve high-spatial and high-spectral resolutions. These methods can be roughly classified into three categories: ratio enhancement (RE) methods, multiresolution analysis (MRA) methods, and component substitution (CS) methods [4]. In general, RE methods [5,6] use image division to compute a synthetic ratio; then, the pan-sharpening result is obtained by multiplying an MS image by the ratio. The MRA methods [7] utilize some multi-scale analysis tools, such as Laplacian pyramids or wavelet transform, to divide the spatial information of each image into many channels and then insert the high-frequency channels of the Pan image into the corresponding MS channels, before restoring them to produce a fused image. CS methods [8] first project the MS image into a vector space; then, one structural component of the MS bands is replaced by a Pan image, before applying an inverse transformation. The CS methods can be summarized into four steps [9,10], including: (a) resampling the MS image to the scale of the Pan image; (b) computing the intensity component (e.g., acquired by weighted summation of the MS image); (c) matching the histograms of the Pan image to the intensity component; and (d) injecting the extracted details according to a set of weight coefficients. Some studies [11] also indicate that the MRA methods can be formulated in the same way as the CS methods, but the main difference lies in the method used to compute the intensity component. CS methods are more practical and popular because of their fast calculation speeds and convenient implementation. Representative cases of CS methods include principal component analysis (PCA), Gram-Schmidt transformation (GS), Intensity-Hue-Saturation (IHS), and the University of New Brunswick (UNB) method [12], among others. These typical methods are widely used and can retain the spatial details of original Pan images well. However, spectral distortion will occasionally occur in pan-sharpened images [13]. Yun Zhang [12] attributes this distortion to the inefficiency of classical techniques on new sensors. Xie et al. [14] indicate that neglecting the spectral consistency term results in fused images that are not strictly spectrally consistent. A local adaptive method, i.e., an adaptive GS method (GSA), is proposed in Ref. [10] that can preserve the spectral features without diminishing the spatial quality. Xie et al. [14] reveal the implicit statistical assumptions of the CS methods from a Bayesian data fusion framework, and demonstrate that all pixel values in different vectors are independent and identically distributed; considering this assumption in a local sliding window is always a better solution to spectral distortion. Furthermore, these popular methods are also employed to fuse data of different resolution ratios. For example, Fryskowska et al. [15] analyze the multispectral image integration abilities of Landsat 8 with data from the high spatial resolution panchromatic EROS B satellite. The authors test six algorithms (Brovey, Multiplicative, PCA, IHS, Ehler, and HPF) and the experimental results show that the Brovey and Multiplicative algorithms can achieve better visual effectiveness. Santurri et al. [16] compare the pansharpened results of different methods on SPOT-HRV panchromatic and Landsat-TM multispectral images. The IHS-based and GSA methods are reported in the literature as the more effective techniques, whereas, other traditional methods barely achieve satisfactory results. Recent developments in pan-sharpening approaches have also included a fast pan-sharpening method based on nearest-neighbor diffusion (NND) [17] and deep-learning based algorithms [18,19]. The NND method assumes that each spectral value in the fused image is a linear mixture of the spectra of its direct adjacent superpixels, and it takes each pixel spectrum as the smallest unit of operation. The structure of a deep-learning network includes multiple artificial neural networks with hidden layers. Such models have excellent feature learning abilities [18], and they have recently been introduced for use in image fusion. For instance, Liu et al. [19] propose a multi-focus image fusion method that utilizes a deep convolutional neural network trained by both high-quality image patches and their blurred versions to encode the mapping. Many experiments have shown that both the NND and deep learning approaches can achieve a strong fusion performance; however, the fused results produced by the NND method may result in spectral distortion in some specific scenes of very high-resolution images. Meanwhile, methods based on deep learning require large amounts of training data to achieve acceptable performance, and their complex model structures often make

ISPRS Int. J. Geo-Inf. 2017, 6, 404

3 of 22

explaining the results difficult. In this context, increasing numbers of emerging satellite images provide the motivation for developing new methods to counteract these limitations. In recent years, applications of edge-preserving filtering, such as bilateral filtering [20], mean shift [21,22], and guided image filtering [23], have attracted a great deal of attention in the image processing community. Among these, guided image filtering, proposed by He et al. [23] in 2010, is quite popular due to its low computational cost and excellent edge-preserving properties. Guided image filtering has been widely used for combining features from two different source images, such as image matting/feathering [24], HDR compression [25], flash/no-flash de-noising [26], haze removal [27], and so on. By transferring the main boundaries of the guidance image to the filtered image, the original image can be smoothed; meanwhile, the gradient information of the guidance image can also be retained. Guided image filtering provides an interesting way to fuse the features of multi-source data sets. However, the application of guided filtering to remote sensing image pan-sharpening tasks remains to be considered. Li et al. [28] developed an image fusion method with guided filtering that has been tested on multi-focus or multi-exposure images of nature scenes. In this context, a novel pan-sharpening method based on guided image filtering for fusing GF-2 images is proposed. In detail, the spectrum coverage of the Pan and MS bands is considered, and a simulated low-resolution Pan band is simulated through a linear regression model. During the filtering process, the resampled MS image is taken as the guiding image for the simulated Pan band. Next, the spatial information is obtained by subtracting the filter output from the original Pan image. Finally, the pan-sharpened image is synthesized by adaptively injecting the spatial details into each band of the resampled MS image. The remaining sections of this paper are organized as follows. Section 2 reviews guided image filtering. Section 3 presents the proposed image pan-sharpening method. The experimental settings are introduced in Section 4, and Section 5 provides the experimental results and discussion. Finally, Section 6 provides conclusions. 2. Guided Image Filtering Pan-sharpening can be considered as a process that combines the strengths of both panchromatic and multispectral images. Correspondingly, guided filtering [23] combines the characteristics of an original image and a guidance image. When the original and guidance images are properly selected, it is feasible to integrate the characteristics of both images. In this section, we first briefly review the guided image filtering algorithm; then, we analyze the properties of the guided image filtering with different parameter settings. 2.1. Guided Image Filtering The guided image filter [23] assumes that the filtering output follows a local linear model between filter output Q and guidance image I in a local window ωk centered at pixel k. Qi = ak Ii + bk , ∀i ∈ ωk ,

(1)

where ak and bk are linear coefficients considered to be constant in a small square image window ωk with a radius of (2r + 1) × (2r + 1). The local linear model guarantees ∇ Q = a∇ I, that is, that filter output Q has an edge only if the guidance image I has an edge. Here, the coefficients ak and bk are computed by minimizing the following cost function: E ( a k , bk ) =

∑

i ∈ ωk

[( ak Ii + bk − pi )2 + εak 2 ],

(2)

ISPRS Int. J. Geo-Inf. 2017, 6, 404

4 of 22

where p is the filter input, and ε is a regularization parameter assigned by users that prevents ak from becoming too large. The linear coefficients are directly resolved by the linear ridge regression [29] as follows: 1 |ω |

ak =

∑i∈ωk Ii pi −µk pk σk2 +ε

bk = p k − a k µ k pk = |ω1 | ∑ pi

,

(3)

i ∈ ωk

where µk and σk2 are the mean and variance of I in ωk , |ω | is the number of pixels in ωk , and pk is the mean of p in ωk . Because all windows that contain i will involve pixel i, different windows will have different values of Qi . One effective method to resolve this problem is to average all the possible values of Qi to obtain a filtered output image Q. Therefore, after calculating ( ak , bk ) for all windows ωk in the image, the filter result is computed by: Qi =

1 |ω |

∑ ( ak Ii + bk )

k:i ∈ωk

(4)

= ai Ii + bi , where ai =

1 a | ω | ∑ k ∈ ωi k

and bi =

1 b . | ω | ∑ k ∈ ωi k

2.2. Influence of Parameters Two important parameters, the radius r of the local windows and the regularization parameter ε, determine the filtering performance. Some guided filtering results using various parameters are shown in Figure 1. Figure 1a is one band of the MS image, defined as the input image, and Figure 1c is the corresponding Pan image, which serves as the guidance image; Figure 1b,d show the filtering results of the guided filter with different parameter settings. The edge-preserving performance is emphasized by the local rectangles. As shown, with different extents, the structures in the guidance image are preserved in the filtering outputs. More specifically, in Figure 1b as the filter size r increases, the edge information gradually becomes visible. This means that an increasing amount of structural information is transferred from the guidance image to the filtering results. Meanwhile, as the degree of smoothing ε increases, only slight changes occur in the filtered outputs (Figure 1d). However, as more structures of the guidance image (the Pan band) are transferred to the filtering outputs, the spectral information inherited from the original input image (MS) is reduced. In contrast, the greater the value of ε, the more the details of the input image are smoothed. Therefore, to consider the trade-off between the spectral and spatial qualities, empirically, the values of r and ε cannot be too large.

ISPRS Int. J. Geo-Inf. 2017, 6, 404

5 of 22

ISPRS Int. J. Geo-Inf. 2017, 6, 404

5 of 22

(a)

(b)

(c)

(d)

Figure 1. (a) The input image (one band of multispectral image); (c) The guidance image (the

Figure 1. (a) The input image (one band of multispectral image); (c) The guidance image2 (the panchromatic band); (b) The outputs: from left to right, the regularization parameter ε is set to 0.1 , panchromatic band); (b) The outputs: from left to right, the regularization parameter ε is set to and the radius r is set to 2, 4, 6, and 8; (d) The outputs: from left to right, the radius r is set to 4, 2 0.1 , and the regularization radius r is setparameter to 2, 4, 6, εand 8; (d) The outputs: from tothe right, r is set to 4, 2, 0.2 2, 0.42, and 2. As is set to 0.1 0.8left filterthe sizeradius and the r increases, 2 , 0.22 , 0.42 , and 0.82 . As the filter size r increases, the and the parameter set to 0.1 the regularization edge information graduallyε is becomes visible. As the degree of smoothing ε increases, the edge changes information gradually becomes visible. As the degree of smoothing ε increases, the changes in in filtering outputs are slight. filtering outputs are slight. 3. Proposed Algorithm for GaoFen-2 (GF-2) Datasets

3. Proposed Algorithm GaoFen-2 (GF-2) Datasets In this section, wefor formulate the problem and subsequently introduce the proposed pan-sharpening method on guided Then,the we verify the effectiveness of the proposed method. the proposed In this based section, we filtering. formulate problem and subsequently introduce pan-sharpening method based on guided filtering. Then, we verify the effectiveness of the 3.1. Problem Formulation and Notations proposed method. The goal of the proposed algorithm is to obtain new MS images, which simultaneously possess

both high spectral and high spatial properties. In the following section, we use M and P to denote 3.1. Problem Formulation and Notations the original MS and Pan images, respectively. After resampling all the MS bands into the same spatial

The theband, proposed to obtain new MS images, which y) size goal as theofPan P( x , yalgorithm ) is the Panisband pixel value of the position ( x , y)simultaneously , Mi and Mi ( x ,possess both high spectral and high spatial properties. In the following section, we use M and P to denote the with i ∈ {1, 2, 3, 4} are the i-th MS band and the pixel value of the i-th MS band at the position ( x , y) original MS and Pan images, respectively. After resampling all the MS bands into the same spatial respectively, and the final pan-sharpened output is denoted as F . size as the Pan band, P( x, y) is the Pan band pixel value of the position ( x, y), Mi and Mi ( x, y) with i ∈ {3.2. 1, 2,Guided 3, 4} are the i-th band and the pixel value of the i-th MS band at the position ( x, y) Filtering BasedMS Pan-Sharpening respectively, and the final pan-sharpened output is denoted as F.

In this section, a pan-sharpening method based on guided image filtering is proposed. The flow chart the method shown in Figure 2 can be described as follows. 3.2. GuidedofFiltering Based Pan-Sharpening

(i) The original multispectral image is registered and resampled to be the same size as the original In this section, a pan-sharpening method based on guided image filtering is proposed. The flow Pan image P . chart of the method shown in Figure 2 can be described as follows. (ii) By minimizing the residual sum of squares (Equation (5)), the weights wi (with i = 1, 2, 3, 4 )

(i) (ii)

The can original multispectral be easily estimated. image is registered and resampled to be the same size as the original Pan image P. By minimizing the residual sum of squares (Equation (5)), the weights wi (with i = 1, 2, 3, 4) can be easily estimated. !2 RSS(wi ) =

∑∑ x

y

4

P( x, y) − ∑ wi Mi ( x, y) i =1

,

(5)

ISPRS Int. J. Geo-Inf. 2017, 6, 404

6 of 22

ISPRS Int. J. Geo-Inf. 2017, 6, 404

6 of 22

2

4   RSS( wi ) =  wi Mi ( x,P ey )can (5) (6).  P( x, y ) −  image  , be obtained with Equation Thereafter, a synthetic low-resolution panchromatic x y  i =1 

4 Thereafter, a synthetic low-resolution panchromatic image P can be obtained with Equation (6). e

P=

∑ w i Mi ,

i =1

4

P =  wi Mi ,

(6)

(6)

i =1

where Pe is the simulated low-resolution panchromatic image and wi is the weight for the i-th where P i (isx,the panchromatic image and wi is the weight for the i-th band band M y),simulated which is low-resolution constant for the given band. Mi ( x , yM ) ,i which constant thetaken givenas band. (iii) Each (with is i= 1, 2, 3,for 4) is the guidance image to guide the filtering process of the e low-resolution Pan image P,4 and the filter Mi0image (withto i= 1, 2,the 3, 4)filtering is obtained asoffollows: Each Mi (with i = 1, 2, 3, ) is taken as theoutput guidance guide process the (iii) low-resolution Pan image P , and the filteroutput Mi′ (with i = 1, 2, 3, 4 ) is obtained as follows:

Mi0 = GF Mi , Pe , i = 1, 2, 3, 4,

(

)

Mi′ = GF Mi , P , i = 1, 2, 3, 4 ,

(7)

(7)

where GF (u, v) denotes the process of guided filtering, and u and v represent the guidance and

denotes the process of guided filtering, and u and v represent the guidance and where GF ( u , v )respectively. input images,

images, respectively. (iv) input The pan-sharpening result Fi is obtained by extracting the spatial information from the Pan image Fi is obtained (iv) The pan-sharpening by M extracting the spatial information from the Pan and injecting it into theresult resampled MS image i according to the weight αi ( x, y ). This process image and injecting it into the resampled MS image can be formulated as shown in Equations (8) and (9):Mi according to the weight αi ( x, y) . This process can be formulated as shown in Equations (8) and (9)::


Fi ( x, y) = P( x, y) − Mi0 ( x, y) × αi ( x, y) + Mi ( x, y), i ∈ {1, 2, 3, 4}, Fi ( x, y ) = ( P( x, y ) − M i′( x, y ) ) × αi ( x, y ) + M i ( x, y ), i ∈ {1, 2 , 3, 4} ,

αi ( x, y) = s αi ( x, y) =

1

1

, i ∈ {1, 2, 3, 4},

, i ∈ {1, 2 2, 3, 4} ,

− P) ( p, q)) i ( p, ( p,q ) −qP)( p,q ) ∑ ( M( M 2

((p,q )∈w p , q )∈w ( x,y)

(8)

i

(8) (9)

(9)

( x ,y )

where F image, P Pan image, image, M Mi((xx, the resampled resampled MS , yy) ) isis the is the the fusion fusion image, is the the original original Pan where Fii (( xx,, yy)) is P (( xx,, yy)) is i 0

image, M ( x, y) is the filtering output, α ( x, y) is the weight corresponding to i-th MS band at a MS image, i M i′ ( x , y ) is the filtering output, i αi ( x, y) is the weight corresponding to i-th MS band at

position ( x, y), w( x,y) denotes a local square window centered at ( x, y), ( p, q) expresses a pixel in

w( x , y ) denotes square window centered pixel of athe position y ) , ( p ,and q ) expresses ( x , y ) , window ( x ,image local square w a ,local i is the band number of theatMS the total anumber ( x,y)

w(Obviously, in the local square window , i is thethe band number the MS image the total of bands in the MS image is 4. greater theofdistance is, theand smaller the number weight should x,y)

be; otherwise, weight should be the large. bands in the MSthe image is 4. Obviously, greater the distance is, the smaller the weight should be; otherwise, the weight should be large.

Figure 2. The flow chart of the proposed method. The numbers in parentheses reference the related

Figure 2. The flow chart of the proposed method. The numbers in parentheses reference the equations. related equations.

In the proposed algorithm, the guided filtering involves both the resampled spectral band Mi (as the guidance image) and the simulated Pan band Pe (as the input band); therefore, the output band e This process results in less spectral distortion when Mi0 preserves the structures of both Mi and P. extracting the spatial details from the Pan band.

ISPRS Int. J. Geo-Inf. 2017, 6, 404 ISPRS Int. J. Geo-Inf. 2017, 6, 404

7 of 22 7 of 22

In the proposed algorithm, the guided filtering involves both the resampled spectral band Mi

the proposed algorithm, the guided filtering involves both the resampled spectral band Mi 7 of 22 ISPRS (as Int. the J.In Geo-Inf. 2017, image) 6, 404 guidance and the simulated Pan band P (as the input band); therefore, the output

(as the guidance image) and the simulated Pan band P (as the input band); therefore, the output band M i′ preserves the structures of both Mi and P . This process results in less spectral distortion band M i′ preserves the structures of both Mi and P . This process results in less spectral distortion when extractingthe thealgorithm spatial details from the the Pan extracted band. Furthermore, modulates spatial details with a position-dependent when extracting the spatial detailsmodulates from the Pan band. the algorithm extracted(9), spatial detailspixel withlocated a position-dependent ratio αi ( x,Furthermore, y). More specifically, as shown inthe Equation for each at position ( x, y) Furthermore, thespecifically, algorithm modulates the extracted spatial details with a position-dependent α ( x, y ) ratio . More as shown in Equation (9), for each pixel located at position centered at ai window of size (2R + 1) × (2R + 1), the Euclidean distance between each Mi (( xx,, yy)) and ratio αi ( x, y) . More specifically, as shown in Equation (9), for each pixel located at position ( x , y) ( x ,the y) and Euclidean distance as between each Miof at a window of size R + 1) × ( 2 Rof+ 1the P( x, ycentered ) is calculated. Then, the (2 reciprocal distance is defined an indicator amount ) , the centered at a window of size (2 R + 1) × ( 2 R + 1) , the Euclidean distance between each Mi ( x , y) and of spatial details that should be injected into a specific MS band. A small distance indicates P( x , y) is calculated. Then, the reciprocal of the distance is defined as an indicator of the amountaofsmall P ( x , y) is calculated. Then, the reciprocalpixels of the distance isthe defined as an indicator of in thewhich amount of the spectrum of should the corresponding between MS bands; case, spatialdifference details that be injected into a specific MS band. A and smallPan distance indicates a small spatial details that should be injected intopixels a specific MSis, band. Aand small distance indicates a be; small weight should be large. However, the larger the distance the weight should thus, a spectrum difference of the corresponding between the MSsmaller Panthe bands; in which case, the spectrum difference of the corresponding pixels between the MS and Pan bands; in which case, the weight should be large. However, the larger the distance is, the smaller the weight should be; thus, a weak combination is assumed. In this way, the spectral distortion is further reduced. weight should be large. However, the larger thespectral distancedistortion is, the smaller the weight should be; thus, a weak combination is assumed. In this way, the is further reduced. There are three important parameters in the proposed algorithm: the radius r of local windows, weakThere combination is assumed. In this way, theproposed spectral distortion is further reduced. of local windows, the are three important infilter, the radius r windows the regularization parameter ε inparameters the guided and thealgorithm: radius Rthe of the local for calculating of local windows, the There are three important parameters in the proposed algorithm: the radius r regularization parameter ε in the guided filter, and the radius R of the local windows for calculating the weights. A detailed discussion concerning parameter selections is local provided in Section 5.1. guided filter, and the selections radius R isofprovided the for calculating regularization ε in theconcerning the weights. A parameter detailed discussion parameter inwindows Section 5.1. the weights. A detailed discussion concerning parameter selections is provided in Section 5.1.

3.3. Effectiveness of the Proposed Method

3.3. Effectiveness of the Proposed Method

3.3. Effectiveness of the Proposed The effectiveness of the fusedMethod results depends primarily on the detail injection models, that is, the The effectiveness of the fused results depends primarily on the detail injection models, that is, injection weights. A demonstration detail and spectral preservation with different The effectiveness ofAthe fused of results depends on the detail injection models, that is, the injection weights. demonstration of enhancement detail primarily enhancement and spectral preservation with the injection weights. A demonstration of detail enhancement and spectral preservation with weights is provided in Figures 3 and 4. different weights is provided in Figures 3 and 4. different weights is provided in Figures 3 and 4.

Figure 3. 1-D illustrations of different detailinjection injection models: models: (a) model, detail = SP=×SP αi; × αi ; Figure 3. 1-D illustrations of different detail (a)The Theproposed proposed model, detail Figure 3. 1-D illustrations of different detail =injection models: (a)× The proposed model, detail = SP × αi; (b) Equal proportion injection model, detail SP (spatial detail) 1; (c) Generic GS based model [30]. (b) Equal proportion injection model, detail = SP (spatial detail) × 1; (c) Generic GS based model [30]. (b) Equal proportion injection model, detail = SP (spatial detail) × 1; (c) Generic GS based model [30].

(a) (a)

(b) (b)

(c) (c) Figure 4. Cont.

(d) (d)

ISPRS Int. J. Geo-Inf. 2017, 6, 404

8 of 22

ISPRS Int. J. Geo-Inf. 2017, 6, 404

8 of 22

(e) Figure 4. Example of the fused results with different injection weights: (a) The original MS

Figure 4. Example of the fused results with different injection weights: (a) The original MS (multispectral) image; (b) The proposed model, detail = SP × αi; (c) Equal proportion injection model, (multispectral) image; (b) The proposed model, detail = SP × αi ; (c) Equal proportion injection model, detail = SP × 1; (d) Generic GS (Gram-Schmidt transformation) based model [30]; (e) Local patches detailfrom = SP (d) Generic GS (Gram-Schmidt transformation) based model [30]; (e) Local patches (b)×to1;(d). from (b) to (d).

A 1-D example of detail enhancement with different models is shown in Figure 3. The 1-D input spectral signal (blue) is theenhancement spectral curve with of partdifferent of the features in is the first resampled A 1-D example of detail models shown in Figuremultispectral 3. The 1-D input M . The product of the SP (spatial detail) and weight is the injected detail layer (red); among band 1 spectral signal (blue) is the spectral curve of part of the features in the first resampled multispectral

SP expresses between the Pan band and the output.detail The injection weights bandthese, M1 . The productthe ofdifference the SP (spatial detail) and weight isfiltered the injected layer (red); among α ( x, y ) (Equation (9)), the equal proportion injection model, and the GS-based model include i these, SP expresses the difference between the Pan band and the filtered output. The injection[30] weights (theαicovariance between thethe Pan andproportion first resampled multispectral band). The enhanced signal include ( x, y) (Equation (9)), equal injection model, and the GS-based model [30] (the (green) is the combination of the input signal and the detail layer. covariance between the Pan and first resampled multispectral band). The enhanced signal (green) is Figure 3aofshows that the result obtained fromlayer. the proposed method with weight αi preserves the combination the input signal and the detail the gradient information well, and the spatial details are obviously is Figure 3a shows that the result obtained from the proposedsimultaneously method withenhanced. weight αThis i preserves α is calculated pixel-by-pixel, and the spatial details are not lost during because the weight i the gradient information well, and the spatial details are obviously simultaneously enhanced. This is processing. However, spatial details are injected in equal in theare model; shown in Figure because the weight αi is calculated pixel-by-pixel, and theproportion spatial details not as lost during processing. 3b, the enhanced signal appears to have an abnormal protrusion. In Figure 3c, the result based on However, spatial details are injected in equal proportion in the model; as shown in Figurethe 3b, the GS model has the same trend as the input signal, but the curve is smoother. Due to this global model, enhanced signal appears to have an abnormal protrusion. In Figure 3c, the result based on the GS some detail may be lost in the fused image. model has the same trend as the input signal, but the curve is smoother. Due to this global model, Figure 4 shows a comparison of output images with different injection weights as mentioned someabove. detailDuring may beprocessing, lost in thethe fused image. radius and regularization parameter in the guided filter were set to 3 Figure 4 shows a comparison of injection weights αi mentioned patchesimages indicatewith that different the proposed method, using as as the and 10−8, respectively. The zoomed-inoutput above. During processing, the radius and regularization parameter in the guided filter were set to injection weight, achieves better spectral preservation and detail enhancement than do other models. 3 and 10−8 , respectively. The zoomed-in patches indicate that the proposed method, using αi as the injection weight, achieves better Settings spectral preservation and detail enhancement than do other models. 4. Datasets and Experimental In this the data Settings sets are introduced first; then, some state-of-the-art fusion methods are 4. Datasets andsection, Experimental used for comparisons, and several evaluation metrics are briefly described.

In this section, the data sets are introduced first; then, some state-of-the-art fusion methods are Datasets and several evaluation metrics are briefly described. used4.1. for GF-2 comparisons, The characteristics of the employed datasets are shown in Table 1. The MS image consists of four

4.1. GF-2 Datasets bands, including blue, green, red, and near infrared (NIR), and the spectral range of the MS bands

are covered by of thedatasets Pan band. pairsinofTable GF-21.images acquired urban, Theexactly characteristics of the the range employed areFour shown The MS imageover consists of four water body, cropland, and forest areas were employed to evaluate the effectiveness of the proposed bands, including blue, green, red, and near infrared (NIR), and the spectral range of the MS bands are method. The characteristics of each image are displayed in Table 1 (the original MS images were exactly covered by the range of the Pan band. Four pairs of GF-2 images acquired over urban, water resampled to be the same size as the Pan images).

body, cropland, and forest areas were employed to evaluate the effectiveness of the proposed method. The characteristics of each image are displayed in Table 1 (the original MS images were resampled to be the same size as the Pan images).

ISPRS Int. J. Geo-Inf. 2017, 6, 404

9 of 22

Table 1. Characteristics of the employed GaoFen-2 (GF-2) datasets. Spatial resolution

MS: 3.2 m Pan: 0.8 m

Spectral range

Blue: 450–520 nm Green: 520–590 nm Red: 630–690 nm NIR: 770–890 nm Pan: 450–900 nm

Image locations

Guangzhou

Land cover types

Urban, rural, water body, cropland, forest, concrete buildings, etc. 1

2

Image size 3

4

MS: 250 × 250 Pan: 1000 × 1000 MS: 1250 × 1250 Pan: 5000 × 5000 MS: 1250 × 1250 Pan: 5000 × 5000 MS: 1250 × 1250 Pan: 5000 × 5000

4.2. Methods Considered for Comparison Five state-of-the-art pan-sharpening methods, including Gram-Schmidt transformation [30] and NND pan-sharpening [17] from the ENVI software, the University of New Brunswick method [14] from the PCI Geomatica software, the adaptive GS (GSA) method [12], and the GD method [31] were selected for comparisons. These approaches are either integrated into commercial software or are newly developed, and have all been shown to be efficient for fusing remote sensing images. (1)

(2)

Gram-Schmidt Transformation (GS) [30]: The general GS method uses the Blue, Green, Red and NIR bands of an MS image to simulate a low-resolution Pan band according to corresponding predefined weights. Thereafter, the GS transformation is applied to the synthetic Pan and low-resolution MS images using the first band from the former. Finally, the high-resolution Pan image replaces the first band of the GS transformed bands, and the inverse GS transformation is employed to produce a fused MS image. This method has been integrated into the ENVI 5.3 software. The average of the low-resolution multispectral bands is calculated as the low-resolution Pan band. Adaptive GS method [12]. The GSA method has the same processing procedures as the general GS method except that the GS method uses equal weight coefficients for each MS band to obtain an intensity image, whereas the GSA method employs the weight coefficients derived using a regression between the MS and degraded low-resolution Pan images. Both methods enjoy the injection gains given by Equation (10): gi =

(3)

cov( Mi , IL ) , i = 1, · · · , N, var( IL )

(10)

where cov( Mi , IL ) is the covariance of the i-th MS band and the low-resolution Pan image, var( IL ) denotes the variance of the low-resolution Pan image, and N is the total number of bands in the MS image. Nearest-neighbor Diffusion-based Pan-Sharpening (NND) [17]: The NND method first downsamples the high-resolution Pan image to match the size of the MS image. Then, it calculates the spectral band contribution vector using linear regression and obtains the difference factors from the neighboring super pixels of each pixel in the original Pan image. Finally, this method applies a linear mixture model to acquire a fused image. Two important external parameters, an intensity smoothness factor and a spatial smoothness factor, are set based on the

ISPRS Int. J. Geo-Inf. 2017, 6, 404

(4)

(5)

10 of 22

intended application. In this study, the default values of the parameters were utilized across all the experiments. University of New Brunswick method (UNB) [14]: The UNB pan-sharpening method first equalizes the histogram of the MS image and Pan image. Then, the spectral bands of the MS image, that are covered by the Pan band, are employed to produce a new synthetic image using the least squares technique. Finally, all the equalized bands of the MS image are fused with the synthesized image to obtain a high-resolution multispectral image. This method is currently integrated into the PCI Geomatica software. In this study, the method was executed using the default parameters. GD method: Zhao et al. [31] proposed a fusion method based on a guided image filter that takes the resampled MS image as the guidance image and the original Pan image as the input image to implement the filtering process. Next, it obtains the filtered image with the related spatial information. Finally, the spatial details of the original Pan image are extracted and injected into each MS band according to the weight wi defined by Equation (11) [12] to obtain the fused image. Equation (11) calculates the optimal coefficient, which indicates the amount of spatial detail that should be injected into the corresponding MS band: wi =

cov( P, MSi ) , var( P)

(11)

where cov( P, MSi ) is the covariance of P and the i-th MS band, and var( P) denotes the variance of the original Pan image. It is worth mentioning that the GD method employs a global detail-injection model in which the weight for a given band is fixed and calculated according to Equation (11). In contrast, the proposed pan-sharpening method employs a local-based injection model, which is adaptively determined according to a local statistic defined by Equation (9). 4.3. Evaluation Methods In this paper, both visual interpretations and quantitative evaluation methods were employed to verify the effectiveness of the proposed method. In general, the quality evaluation approaches consist of two types, a reduced-resolution and a full-resolution assessment. In the first approach, some evaluation metrics, such as the relative dimensionless global error in synthesis (ERGAS) [32], the Spectral Angle Mapper (SAM) [33], and Q4 [34], require a reference image for comparison. In many previous experiments, the original Pan and MS images are degraded by four; then, the degraded images were pan-sharpened so that the original images could be used as references for comparison [35]. However, the scale invariance assumption does not always hold in practice, and the accuracy is fundamentally influenced by the way that the resolution degradation is performed [13]. Therefore, to avoid potential bias introduced from the degradation, we directly utilized the resampled MS and original Pan images as the reference images for quantitative assessments of the different pan-sharpening results. Using the resampled MS images as reference images, four widely used metrics were selected for quantitative assessment from spatial and spectral qualities, including Entropy, the correlation coefficient (CC) [36], the universal image quality index (UIQI) [37], and the relative dimensionless global error in synthesis (ERGAS) [32]. Furthermore, the pan-sharpened images were also classified based only on their spectral features; therefore, the classification accuracy was employed as a metric to indirectly verify the effectiveness of the proposed method.

ISPRS Int. J. Geo-Inf. 2017, 6, 404

(1)

11 of 22

Entropy is used to measure the spatial information contained in a fused image. The higher the Entropy is, the richer the spatial information possessed by the fused image is. Entropy is expressed as follows: 255

Entropy = − ∑ F (i ) log2 F (i )

(12)

0

(2)

where F (i ) is the probability of pixel value i in the image. CC [36] measures the correlation between the MS image and fused image. The value of CC ranges from 0 to 1. A higher correlation value indicates a better correspondence between the MS image and fused image, the ideal correlation coefficient value is 1. CC is defined as follows: m n    ∑ ∑ M (i, j) − M F (i, j) − F

i =1 j =1

CC = v u " #2   " #2  u m n  m n  u t ∑ ∑ F (i, j) − F ∑ ∑ M(i, j) − M   i =1 j =1   i =1 j =1

(3)

where M (i, j) is the pixel value of original MS image, F (i, j) is the pixel value of the fused image, and M and F respectively denote the mean values of original and fused images. UIQI [37] models any distortion as a combination of three different factors: loss of correlation, luminance distortion, and contrast distortion. It is suitable for most image evaluations, and the best value is 1. UIQI is given by: U IQI =

(4)

(13)

σxy 2σx σy 2xy × × 2 2 2 2 σx σy σ x + σy ( x ) + (y)

(14)

where x and y are the mean values of the fused and original images, and σx and σy are the standard deviation of the fused and original images, respectively. ERGAS [32] evaluates the overall spectral distortion of the pan-sharpened image. The lower the ERGAS value is, the better the spectrum quality of the fused image is. The best ERGAS value is 0. The definition of ERGAS is as follows: v u K dP u RMSE2 (i ) ∆ t1 ERGAS = 100 (15) ∑ d MS K i=1 MEAN 2 (i ) where d P /d MS is the ratio between pixel sizes of the Pan and MS images, K is the number of bands, and MEAN (i ) is the mean of the i-th band, whereas RMSE(i ) is the root-mean-square error between the i-th band of the reference image and the i-th band of the pansharpened image.

5. Results and Discussion In this section, the influence of the three parameters is discussed first. Successively, four groups of experimental results and some image quality assessment are presented and discussed. Finally, the computational complexity of the proposed method is reported. 5.1. Analysis of the Influence of Parameters 5.1.1. Parameter Influences in the Guided Filter As mentioned in Section 2.2, the parameters r and ε affect the filtering size and smoothing degree of the guided filter, respectively. To obtain the optimal parameter settings, an image of size 500 × 500 pixels was employed to conduct a parameter analysis, and two metrics, Entropy and SAM, were used as measures. Entropy is related to the spatial quality, while SAM [33] quantifies the spectral distortion

ISPRS Int. J. Geo-Inf. 2017, 6, 404

12 of 22

by computing the angle between the corresponding pixels of the pan-sharpened and reference images. Figures 5 and 6 show the influences of these two parameters on the pan-sharpening performance. ISPRS Int. J. Geo-Inf. 2017, 6, 404 12 of 22

7.31 7.28 7.28 7.25 7.25 7.22 7.22 7.19 7.19 7.16 7.16

1 1

Entropy Entropy

7.33 7.33 7.32 7.32 7.31 7.31 7.3 7.3 7.29 7.29 7.28 7.28 7.27 7.27 7.26 7.26

1 1

7 7 6 6 5 5 4 4 3 3 2 2 1 1

1 3 5 7 3 5 7 9 11 13 r 1 3 5 r 7 3 ε=10e-3 5 7 ε=10e-6 9 11 13 ε=10e-3r r ε=10e-3 ε=10e-3 ε=10e-6 Figure 5. Analysis of the influence of the parameter r . Figure 5. Analysis of the influence of the parameter r. Figure 5. Analysis of the influence of the parameter r .

Spectral Angle Mapper Spectral Angle Mapper

Entropy Entropy

7.34 7.34 7.31

12 of 22

Spectral Angle Mapper Spectral Angle Mapper

ISPRS Int. J. Geo-Inf. 2017, 6, 404

9 11 9 11 ε=10e-6 ε=10e-6

13 13

3.05 3.05 2.9 2.9 2.75 2.75 2.6 2.6 2.45 2.45 2.3 2.3 2.15 2.15 2

10e-1 10e-2 10e-3 10e-4 10e-5 10e-6 1 10e-1 10e-2 10e-3 10e-4 10e-5 10e-6 2 10e-1 10e-2ɛ 10e-3 10e-4 10e-5 10e-6 1 10e-1 10e-2ɛ 10e-3 10e-4 10e-5 10e-6 r=2 ɛ r=4 r=2 ɛ r=4 r=2 r=4 r=2 r=4 Figure 6. Analysis of the influence of the parameter ε . Figure 6. Analysis of the influence of the parameter ε .

Figure 6. Analysis of the influence of the parameter ε. In these experiments, the window size of the weight was fixed to 7 × 7 , and 7 groups of both r the window size of the weight wasanalyzed, fixed to 7 ×ε 7 was , andfixed 7 groups both valuesexperiments, were evaluated. When the influence of r was to 10−3ofand 10−6r , and In ε these −3 In these experiments, the window size of the weight was fixed to 7 × 7, and 7 groups of both r wereto evaluated. Whenthe theinfluence influenceofofε r was wasanalyzed. analyzed, ε was fixed to 10 and 10−6 , and ε rvalues was fixed 2 and 4 when while − 3 − 6 was fixed 2 and 4see when the influence ofof εr was was analyzed. r were and ε while values evaluated. the influence analyzed, ε was to 10valueand a larger thefixed Entropy and10 , From Figure 5 to we canWhen that when ε is fixed, r reduces is εfixed, athe larger the Entropy value and From 5 we4However, can seethe that when ε less r reduces than changes in the Entropy and SAM values theFigure SAM when r is whileincrease r was fixed to 2 value. and when influence of was3,analyzed. is less than 3, the changes in the Entropy and SAM values increase the SAM value. However, when r is greater than 3, the Entropy value gradually decreases. Therefore, are not obvious; and when r From Figure 5 we can see that when ε is fixed, a larger r reduces the Entropy value and increase than the the Entropy value gradually decreases. Therefore, are not obvious; and when r is greater should not beSAM too large or too considering the trade-off between the than two metrics, valuein ofthe r Entropy the SAM value. However, when r is less 3,3,the changes and values are not considering the trade-off between the two metrics, the value of r should not be too large or too small. obvious; and when r is greater than 3, the Entropy value gradually decreases. Therefore, considering small. In Figure 6, when r is fixed, as the ε value decreases, the Entropy value becomes larger, while the trade-off between the twoismetrics, the value of decreases, r should not be too large or too small. fixed, as the Entropy becomes larger, while In Figure when r to ε valuean greater thanvalue 10−3 has a negligible effect the SAM value6,continues decrease. However, ε valuethe In Figure 6, when r is fixed, as the ε value decreases, the Entropy value becomes larger, while the −3 10 has value a negligible effect the the SAM value continues decrease. an than ε value 10−4 greater causes than the Entropy to decrease on Entropy and SAMtovalues, an However, ε value less −3 has a negligible effect on −4 SAM slowly, value continues to decrease. However, an ε value greater than 10 less than 10 causesthe thevalue Entropy decrease on the Entropy SAM value values, an ε value also to should not whereas and the SAM decreases continuously. Therefore, of εvalue the Entropy and SAM values, an ε value less than 10−4 causes the Entropy value decrease slowly, also slowly, whereas SAMConsequently, value decreases continuously. Therefore, the ofvalues ε to be too large or toothe small. in the following experiments, wevalue set the ofshould r andnot ε and be be 3too large too small. Consequently, in the following experiments, we set values of r not ε too whereas the SAM value decreases continuously. Therefore, the value of the ε also should , respectively. to and 10−8or −8, respectively. to 3 and 10 large or too small. Consequently, in the following experiments, we set the values of r and ε to 3 and The Influence of the Window Radius R for Calculating Weights 10−8 , 5.1.2. respectively. 5.1.2. The Influence of the Window Radius R for Calculating Weights The window radius R of the weight during the fusion process is another important parameter. 5.1.2. Here, The The Influence the Window Radius for Calculating R εofwere the weight during the RWeights fixed R at 3 and 10−8fusion , and process was is setanother to 1, 3, important 5, 7, and 9 parameter. to analyze thewindow valuesof ofradius r and −8, and R was set to 1, 3, 5, 7, and 9 to analyze and were fixed at 3 and 10 Here, the values of r ε its influence the pan-sharpening performance. quality index Q4 [34] wasimportant employed parameter. as the The windowon radius R of the weight during the Afusion process is another its influence on the pan-sharpening performance. A quality index Q4 [34] was evaluation employed index, as the measure of influence. Q4 was averaged over the whole image to produce a global − 8 Here, the values of r and ε were fixed at 3 and 10 , and R was set to 1, 3, 5, 7, and 9 to analyze its measure of influence. Q4 was averaged over the whole image to produce a global evaluation lower and all calculations were based on N × N blocks. The value of Q4 ranges from 0 to 1, and index, influence the pan-sharpening performance. A quality index of Q4Q4 [34] was from employed thelower measure N × N blocks. value ranges 0 to may 1,as and and on allreflect calculations were of based on distortion values the amount spectral in theThe fused product, while high values indicate of influence. Q4 is was averaged over the whole to produce global evaluation and all values reflect the amount spectral distortion in the fused product, while indicate N values was setmay toindex, 32. that a result closer to theofreference image. In image the experiments, theavalue ofhigh calculations were based on N × N blocks. The value of Q4 ranges from 0 to 1, and lower values reflect N was set to 32. that aAs result is closer to the reference image. In the experiments, the value of shown in Figure 7, the R increases, the value of Q4 tends to rise, thus the fusion result had R increases, the value of Q4 tends to rise, thus the fusion result had As shown in Figure 7, the the amount of spectral distortion in the fused product, while high values may indicate that a result better spectral preservation. However, in some of the detailed pictures of the edge of the building is preservation. in some of detailed closerbetter to thespectral reference image. InHowever, the experiments, thethe value of Npictures was setoftothe 32.edge of the building

ISPRS Int. J. Geo-Inf. 2017, 6, 404

13 of 22

As shown in Figure 7, the R increases, the value of Q4 tends to rise, thus the fusion result had better preservation. However, in some of the detailed pictures of the edge of 13 the building ISPRSspectral Int. J. Geo-Inf. 2017, 6, 404 of 22 ISPRS Int. Geo-Inf. 2017, 6, 404 13 of 22 displayed in J.Figure 8a, the greater the radius is, the more the spatial details are blurred. Meanwhile, Figurecurves 8a, theof greater radius is, the more details areare blurred. Meanwhile, thedisplayed spectral in profile thesethe detailed pictures in the thespatial same position shown in Figure 8b. As in Figure 8a, the greater the radius the more the position spatial details are blurred. Meanwhile, thedisplayed spectral profile curves of these pictures in the are shown in Figure 8b. As shown, the smaller the R value, thedetailed sharper theis, edges of same the corresponding curves become. Therefore, the spectral profile curves of these detailed pictures in the same position are shown in Figure 8b. As shown, the smaller the R value, the sharper the edges of the corresponding curves become. to achieve a better overall effect, the radius R of the weight should not be too small or too large. value,effect, the sharper the Redges ofweight the corresponding become. shown, to theachieve smaller the Roverall of the should not becurves too small or Therefore, a better the radius Considering this trade-off, R wasoverall consistently setradius to 3 in R theofexperiments, that is, the window size for the weight should not be too small Therefore, to achieve a better effect, the the or too large. Considering this trade-off, R was consistently set to 3 in the experiments, that is, weight was 7 ×this 7. trade-off, consistently set to 3 in the experiments, that is, the toocalculation large. Considering . window size for weight calculation was 7R× 7was window size for weight calculation was 7 × 7 . 0.9 0.9

Value of Q4 Value of Q4

0.8 0.8

0.7 0.7

0.6 0.6

0.5 0.5

0.4 0.4

1

3 1

3

R

5

R

7 5

9 7

9

Figure 7. Analysis of the influence of the parameter R . Figure 7. Analysis of the influence of the parameter R. Figure 7. Analysis of the influence of the parameter R .

(a) (a)

(b) (b)

Figure 8. (a) Details of the building edges with different window radii. (b) The corresponding spectral Figure 8. (a)ofDetails of the building profile specific in (a). edges with different window radii. (b) The corresponding spectral Figure 8.curves (a) Details of theparts building edges with different window radii. (b) The corresponding spectral profile curves of specific parts in (a).

profile curves of specific parts in (a). In the above experiments, we found that, according to different metrics (i.e., Entropy, SAM, and the above experiments, that, according to differentwere metrics (i.e., Entropy, 7 × 7 and . Q4), theIn optimal window sizes for we thefound filtering and weight calculations consistently set toSAM, 7 × 7 . and Q4), window forfound the window filtering and were set to SAM, In thethe above experiments, that, according tocalculations different metrics (i.e., Entropy, Therefore, itoptimal is reasonable tosizes setwe the two sizesweight as the same, as both areconsistently related to modeling Therefore, it window is reasonable tofor set the window as same,that as both areconsistently related tosizes modeling thethe connections between the Pan and MStwo bands. Therefore, wethe assume the two window areto 7 × 7. Q4), optimal sizes the filtering andsizes weight calculations were set the connections between the Pan and MS bands. Therefore, we assume that the two window sizes are the same. Therefore, it is reasonable to set the two window sizes as the same, as both are related to modeling the same.

ISPRS Int. J. Geo-Inf. 2017, 6, 404

14 of 22

the connections between the Pan and MS bands. Therefore, we assume that the two window sizes are the same. ISPRS Int.ofJ. Different Geo-Inf. 2017, Pan-Sharpening 6, 404 5.2. Comparison Approaches

14 of 22

5.2. subsection, Comparison of Different Pan-Sharpening Approaches In this the proposed pan-sharpening method was compared with some other state-of-the-art approaches. Detailed information about these methodswith is provided in Section 4.2. In this subsection, the proposed pan-sharpening method was compared some other stateof-the-art approaches. Detailed information about these methods is provided in Section 4.2. As shown As shown in Figures 9–12, local patches with various land cover types were clipped from the fused Figures 9–12, local patches with various land cover types were clipped from the fused results and results andindisplayed in true color using the same stretching mode. Quantitative assessments of these displayed in true color using the same stretching mode. Quantitative assessments of these four sets four sets of test data are shown in Tables 2–5. The best performance of each metric is in bold. of test data are shown in Tables 2–5. The best performance of each metric is in bold

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 9. The fused results and some local patches using different methods on GF-2 images over an

Figure 9. urban The fused results and some local patches using different methods on GF-2 images over area: (a) original Pan (panchromatic) image; (b) original MS image; (c) GS; (d) NND (nearestan urban area: (a) original Pan (panchromatic) image; (b) original MS image; (c) GS; (d) NND (nearest-neighbor diffusion); (e) UNB (University of New Brunswick method); (f) GSA (adaptive GS method); (g) GD; and (h) the proposed method.

ISPRS Int. J. Geo-Inf. 2017, 6, 404

15 of 22

ISPRS Int. J. Geo-Inf. 2017, 6, 404

15 of 22

Table diffusion); 2. Quality(e) evaluation of fused images: urban areas (corresponding to Figure 9). neighbor UNB (University of New Brunswick method); (f) GSA (adaptive GS method); (g) GD; and (h) the proposed method. Method

Entropy

UIQI

CC

ERGAS

Table 2. Quality evaluation of fused images: urban areas (corresponding to Figure 9).

MS GS NND UNB GSA GD Proposed

7.189

6.877 Entropy Method MS 6.863 7.189 GS 6.685 6.877 NND 6.912 6.863 UNB 6.685 6.891 GSA 6.912 7.156 GD 6.891 7.156 Proposed

0.848 UIQI 0.779 0.848 0.824 0.779 0.893 0.824 0.878 0.893 0.959 0.878 0.959

CC 0.878 0.881 0.881 0.888 0.902 0.962

0.878 ERGAS 0.881 22.504 0.881 36.835 0.888 26.102 0.902 21.001 0.962 25.731

22.504 36.835 26.102 21.001 25.731 14.150

14.150

In general, all of the methods yield visually better images than the original. For the urban area In general, all of the methods yield visually better images than the original. For the urban area (Figure 9 and9 and Table 2), the NND method exhibits andthe theERGAS ERGAS value of (Figure Table 2), the NND method exhibitsobvious obviousspectral spectral distortion, distortion, and value the NND method is the largest. From the local detail images, however, the differences between the GS, of the NND method is the largest. From the local detail images, however, the differences between the GSA,GS, UNB, and GD methods are not obvious. Furthermore, Table 2 shows that the proposed method GSA, UNB, and GD methods are not obvious. Furthermore, Table 2 shows that the proposed achieves better spectral and spatialand performance; its values all four are the method achieves better spectral spatial performance; its for values for allmetrics four metrics arebest. the best.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 10. The fused results using different methods on GF-2 images over water bodies: (a) original Pan image; (b) original MS image; (c) GS; (d) NND; (e) UNB; (f) GSA; (g) GD; and (h) the proposed method.

ISPRS Int. J. Geo-Inf. 2017, 6, 404

16 of 22

ISPRS Int. J. Geo-Inf. 2017, 6, 404

16 of 22

Table10.3.The Quality fused methods images: on water Figure 10). Figure fused evaluation results usingof different GF-2bodies images(corresponding over water bodies:to(a) original Pan image; (b) original MS image; (c) GS; (d) NND; (e) UNB; (f) GSA; (g) GD; and (h) the proposed method. Method

Entropy

UIQI

CC

ERGAS

Table 3. Quality of fused images: water bodies (corresponding to Figure 10). MS evaluation4.113

GS NND UNB GSA GD Proposed

3.978 Entropy Method 3.997 4.113 MS GS 3.802 3.978 NND 3.916 3.997 UNB 3.805 3.802 GSA 3.933 3.916 GD 3.805 Proposed 3.933

0.608 UIQI 0.393 0.608 0.606 0.393 0.700 0.606 0.538 0.700 0.726 0.538 0.726

CC 0.633 0.593 0.661 0.757 0.598 0.790

0.633 ERGAS 0.593 31.796 0.661 70.754 0.757 40.418 0.598 39.723 0.790 44.323 39.589

31.796 70.754 40.418 39.723 44.323 39.589

As shown in Figure 10, the GSA, UNB, and GD methods cannot preserve the spectral information As shown in Figure 10, the GSA, UNB, and GD methods cannot preserve the spectral well for water bodies, and the color of the river is paler and not as deep blue as in the original MS information well for water bodies, and the color of the river is paler and not as deep blue as in the image. In Table 3, we can see that of thevalue NNDofmethod the best; however, original MS image. In Table 3, wethe canEntropy see that value the Entropy the NNDismethod is the best; its values on the other metrics are the worst. This demonstrates that the NND method performs well in however, its values on the other metrics are the worst. This demonstrates that the NND method enhancing detail does notdetail perform wellnot on perform spectralwell preservation; this result may have occurred performs wellbut in enhancing but does on spectral preservation; this result may because NND method is more fusing low-resolution images. havethe occurred because the NND suitable method isfor more suitable for fusing low-resolution images.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 11. The fused results using different methods on GF-2 images over cropland: (a) original Pan image; (b) original MS image; (c) GS; (d) NND; (e) UNB; (f) GSA; (g) GD; and (h) the proposed method.

ISPRS Int. J. Geo-Inf. 2017, 6, 404

17 of 22

ISPRS Int. J. Geo-Inf. 2017, 6, 404

17 of 22

Table 4. Quality evaluation of fused images: cropland (corresponding to Figure 11).

Figure 11. The fused results using different methods on GF-2 images over cropland: (a) original Pan image; (b) original MS image; (c) GS; (d) NND; (e) UNB; (f) GSA; (g) GD; and (h) the proposed method.

Method

Entropy

UIQI

CC

ERGAS

MS 6.709 Table 4. Quality evaluation of fused images: cropland (corresponding to Figure 11). GS NND UNB GSA GD Proposed

6.642

Method Entropy 6.765 MS 6.709 6.464 6.642 GS 6.625 6.765 NND 6.658 6.464 UNB GSA 6.563 6.625 GD 6.658 Proposed 6.563

0.855 UIQI 0.661 0.828 0.855 0.908 0.661 0.777 0.828 0.908 0.905 0.777 0.905

CC 0.859 0.766 0.844 0.912 0.841 0.921

0.859 ERGAS 0.766 0.844 14.697 0.912 40.106 0.841 18.575 15.126 0.921

14.697 40.106 18.575 15.126 29.136 16.460

29.136 16.460

As seen from Figure 11, the fused images obtained by the NND and GD methods had serious seenspectral from Figure 11, the fusedthe images by the NND methods had seriousquite problemsAs with performance: colorobtained of the cropland in and theirGD results is obviously problems with spectral performance: the color of the cropland in their results is obviously quiteUIQI, different from that of the original MS image. In addition, in Table 4, the NND and GD methods’ different from that of the original MS image. In addition, in Table 4, the NND and GD methods’ UIQI, CC and ERGAS values were the worst, but the overall spatial information was well preserved from a CC and ERGAS values were the worst, but the overall spatial information was well preserved from qualitative point of view. This is because the GD method employs a global detail-injection model; thus, a qualitative point of view. This is because the GD method employs a global detail-injection model; colorthus, distortion will occur inoccur someinspecific scenes. However, the the proposed method color distortion will some specific scenes. However, proposed methodbased basedon onthe the local injection can be a can good to spectral distortion. The GS, and proposed method local model injection model besolution a good solution to spectral distortion. TheGSA, GS, GSA, and proposed performed than the others. methodbetter performed better than the others.

(a)

(c)

(b)

(d) Figure 12. Cont.

(e)

ISPRS Int. J. Geo-Inf. 2017, 6, 404

18 of 22

ISPRS Int. J. Geo-Inf. 2017, 6, 404

18 of 22

( f)

( g)

(h)

Figure 12. The fused results using different methods on GF-2 images over a forest area: (a) original Pan

Figure 12. The fused results using different methods on GF-2 images over a forest area: (a) original Pan image; (b) original MS image; (c) GS; (d) NND; (e) UNB; (f) GSA; (g) GD; and (h) the proposed method. image; (b) original MS image; (c) GS; (d) NND; (e) UNB; (f) GSA; (g) GD; and (h) the proposed method. Table 5. Quality evaluation of fused images: forest area (corresponding to Figure 12).

Table 5. Quality evaluation of fused images: forest area (corresponding to Figure 12). Method

Method Entropy MS 5.882 Entropy 6.749 GS NND 6.578 5.882 UNB 6.482 6.749 6.588 GSA 6.578 6.667 GD 6.482 6.504 Proposed

UIQI

CC

UIQI 0.663 0.732 0.727 0.688 0.663 0.720 0.727 0.675 0.688 0.856

0.745 0.752 0.792 0.753 0.896

ERGAS

CC 56.697

ERGAS

28.991 56.064 0.732 55.977 0.745 59.855 0.752 34.666

MS GS 56.697 NND 28.991 UNB 56.064 GSA 6.588 0.720 0.792 55.977 GDshown in Figure 6.667 12 and Table 0.6755, from a 0.753 59.855 For the forest area visual analysis, the proposed method Proposed 6.504 0.856 0.896 34.666 achieves the best spectral and spatial information performance compared to the other methods, followed by the GSA, UNB, GS, and GD methods and finally, the NND method. In the quantitative evaluation, the area UIQI shown and CC in values of the the best, and its the ERGAS value was For the forest Figure 12 proposed and Tablemethod 5, fromwere a visual analysis, proposed method second best, which is consistent with the visual analysis results. achieves the best spectral and spatial information performance compared to the other methods, Many pan-sharpened images are employed not only for manual interpretation, but also for followed by the GSA, UNB, GS, and GD methods and finally, the NND method. In the quantitative computer-based interpretation. Therefore, classification accuracy was used as an indirect evaluation evaluation, UIQI the andeffectiveness CC values of the proposed proposedmethod. methodAwere best,method and itsshould ERGAS value methodthe to verify of the betterthe fusion result in was second best, which is consistent with the visual analysis results. fused images with higher interclass variance and, thus, should obtain better classification results. Many pan-sharpened are employed notareonly for manual interpretation, but The also for Therefore, in our work, images the pan-sharpened images classified based on spectral features. computer-based interpretation. Therefore, classification accuracy was used asaccuracy. an indirect evaluation overall accuracy (OA) and Kappa are employed as the measures of classification The higher the OA and Kappa values are, theof better classification effectA is.better fusion method should result method to verify the effectiveness the the proposed method. In detail, a sample image with a 400 × 600 pixels size was employed to conduct the panin fused images with higher interclass variance and, thus, should obtain better classification results. sharpening process using different methods; then, a supervised classification using a Support Vector Therefore, in our work, the pan-sharpened images are classified based on spectral features. The overall Machine (SVM) method [38] was applied to the fused results. Figure 13 shows the classification maps, accuracy (OA) and Kappa are employed as the measures of classification accuracy. The higher the OA among which, Figure 13b,c shows the test sample with 157 blocks, including 10 classes. The number and Kappa values are, the better the classification effect is. of each class in the test sample was 1/10 of the number in the training samples. As can be seen in In detail, sample image with a 400 × 600 pixels wasand employed to conduct the particularly pan-sharpening Figure 13,athe classification results obtained by thesize NND UNB methods are not process using different methods; then,fusion a supervised classification a Support Vector Machine satisfactory. However, the proposed method achieves a more using consistent classification result. (SVM) method [38] was applied to the fused results. Figure 13 shows the classification maps, to Table 6 shows the OA and Kappa precisions achieved by these different methods, corresponding among which, 13b,c shows the test sample with of 157 blocks, including 10 classes. The number of each theFigure classification maps. Although many types misclassifications occurred, the proposed method thesample highestwas accuracy, thatinthe is more effective at in spectral class achieved in the test 1/10 which of the means number theproposed trainingmethod samples. As can be seen Figure 13, preservation than other tested methods. the classification results obtained by the NND and UNB methods are not particularly satisfactory.

However, the proposed fusion method achieves a more consistent classification result. Table 6 shows the OA and Kappa precisions achieved by these different methods, corresponding to the classification maps. Although many types of misclassifications occurred, the proposed method achieved the highest accuracy, which means that the proposed method is more effective at spectral preservation than other tested methods.

ISPRS Int. J. Geo-Inf. 2017, 6, 404

19 of 22

ISPRS Int. J. Geo-Inf. 2017, 6, 404

19 of 22

(a)

(b)

(c)

(d)

(e)

( f)

( g)

(h)

(i)

Figure 13. The classification results of the fused image produced from different methods: (a) the

Figure 13. The classification results of the fused image produced from different methods: (a) the original image displayed in true color; (b) and (c) ground truth data; (d) GS; (e) NND; (f) UNB; (g) original image displayed in true color; (b) and (c) ground truth data; (d) GS; (e) NND; (f) UNB; (g) GSA; GSA; (h) GD; and (i) the proposed method. (h) GD; and (i) the proposed method. Table 6. Classification accuracy achieved from images produced by the different methods.

Table 6. Classification accuracy achieved from images produced by the different methods. Method OA (%) Kappa

Method OA (%) GS Kappa

74.08 0.7029

GS NND 74.08 73.91 NND0.7009 0.7029

73.91 0.7009

UNB GSA GD 72.50 74.77 74.77 UNB 0.7107 GSA 0.6844 0.7103

72.50 0.6844

74.77 0.7107

Proposed 76.04 GD 0.7256

74.77 0.7103

Proposed 76.04 0.7256

ISPRS Int. J. Geo-Inf. 2017, 6, 404

20 of 22

In conclusion, these experimental results are sufficient to demonstrate that the proposed approach both enhances the spatial information and effectively preserves the spectral characteristics of the original MS images with less distortion. In addition, the produced images can improve the classification accuracy, which is important. 5.3. Computational Complexity In this section, the computational time of each pan-sharpening method is described to evaluate the computational efficiency. We employed MATLAB on a laptop with 4 GB of memory and a 2.4 GHz CPU to perform the experiments. For a 500 × 500-pixel image, the proposed method requires 11.28 s, while the GS, NND, UNB, GSA, and GD methods require 1.84 s, 1.67 s, 1.52 s, 1.43 s, and 2.36 s, respectively. Compared to the other algorithms, the proposed approach consumes more time; this may be because the weight calculation during the pan-sharpening process is performed in a pixel-by-pixel fashion, and the loop is not efficient enough. Therefore, the speed of the proposed algorithm can be further improved by using a more efficient computational approach. 6. Conclusions The goal of pan-sharpening methods is to simultaneously increase the spatial resolution of an original multispectral image while retaining its spectral features. In this study, we proposed a novel pan-sharpening method based on guided image filtering, and applied it to GF-2 images. The underlying idea of this approach is to consider the spectral difference of each pixel between a resampled MS image and a corresponding Pan image, and to adaptively inject the details of the Pan image into the MS image to yield high-resolution MS images. The experimental results and quality assessments demonstrated that, for GF-2 imagery acquired over different scenes, the proposed pan-sharpening approach consistently achieves high spectral fidelity and enhances the spatial details, regardless of the image content. Furthermore, it can also improve the classification accuracy, which is an important aspect in applications of GF-2 images. Finally, adaptively selecting the window size for the weight calculations and estimating the parameters of the guided filtering process requires further research. Acknowledgments: The authors would like to acknowledge the support from the National Natural Science Foundation of China (no. 41571372, no. 41771375 and no. 41301470). The authors would also like to thank the editors and the reviewers for their insightful comments and suggestions. Author Contributions: Y.Z. conducted the experiment and prepared the manuscript. Q.D. helped write the manuscript. L.W. proposed the general framework and helped write the manuscript. Z.T. also helped write the manuscript. Conflicts of Interest: The authors declare no potential conflicts of interest.

References 1.

2. 3. 4. 5. 6.

Zhang, Y.; Mishra, R.K. A review and comparison of commercially available pan-sharpening techniques for high resolution satellite image fusion. In Proceedings of the Geoscience and Remote Sensing Symposium, Munich, Germany, 22–27 July 2012; pp. 182–185. Dong, J.; Zhuang, D.; Huang, Y.; Fu, J. Advances in multi-sensor data fusion: Algorithms and applications. Sensors 2009, 9, 7771–7784. [CrossRef] [PubMed] Ehlers, M.; Klonus, S.; Johan Åstrand, P.; Rosso, P. Multi-sensor image fusion for pansharpening in remote sensing. Int. J. Image Data Fusion 2010, 1, 25–45. [CrossRef] Xu, Q.; Li, B.; Zhang, Y.; Ding, L. High-Fidelity Component Substitution Pansharpening by the Fitting of Substitution Data. IEEE Trans. Geosci. Remote Sens. 2014, 52, 7380–7392. [CrossRef] Gillespie, A.R.; Kahle, A.B.; Walker, R.E. Color enhancement of highly correlated images. II. Channel ratio and chromaticity transformation techniques. Remote Sens. Environ. 1987, 22, 343–365. [CrossRef] Zhang, Y. A new merging method and its spectral and spatial effects. Int. J. Remote Sens. 1999, 20, 2003–2014. [CrossRef]

ISPRS Int. J. Geo-Inf. 2017, 6, 404

7. 8. 9. 10. 11. 12. 13. 14. 15.

16.

17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31.

32.

21 of 22

Murga, J.N.D.; Otazu, X.; Fors, O.; Arbiol, R. Multiresolution-based image fusion with additive wavelet decomposition. IEEE Trans. Geosci. Remote Sens. 1999, 37, 1204–1211. [CrossRef] Tu, T.-M.; Huang, P.S.; Hung, C.-L.; Chang, C.-P. A fast intensity–hue–saturation fusion technique with spectral adjustment for IKONOS imagery. IEEE Geosci. Remote Sens. Lett. 2004, 1, 309–312. [CrossRef] Dou, W.; Chen, Y.; Li, X.; Sui, D.Z. A general framework for component substitution image fusion: An implementation using the fast image fusion method. Comput. Geosci. 2007, 33, 219–228. [CrossRef] Aiazzi, B.; Baronti, S.; Selva, M. Improving Component Substitution Pansharpening Through Multivariate Regression of MS +Pan Data. IEEE Trans. Geosci. Remote Sens. 2007, 45, 3230–3239. [CrossRef] Vivone, G.; Alparone, L.; Chanussot, J. A Critical Comparison Among Pansharpening Algorithms. IEEE Trans. Geosci. Remote Sens. 2015, 53, 2565–2585. [CrossRef] Zhang, Y. Understanding image fusion. Photogramm. Eng. Remote Sens. 2004, 70, 657–661. Pohl, C.; Van Genderen, J.L. Review article Multisensor image fusion in remote sensing: Concepts, methods and applications. Int. J. Remote Sens. 1998, 19, 823–854. [CrossRef] Xie, B.; Zhang, H.; Huang, B. Revealing Implicit Assumptions of the Component Substitution Pansharpening Methods. Remote Sens. 2017, 9, 443. [CrossRef] Fryskowska, A.; Wojtkowska, M.; Delis, P.; Grochala, A. Some aspects of satellite imagery integration from eros b and landsat 8. ISPRS–Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2016, XLI-B7, 647–652. [CrossRef] Santurri, L.; Carlà, R.; Fiorucci, F.; Aiazzi, B.; Baronti, S.; Guzzetti, F. Assessment of very high resolution satellite data fusion techniques for landslide recognition. In Proceedings of the ISPRS Centenary Symposium, Vienna, Austria, 5–7 July 2010; Volume 38, pp. 492–497. Sun, W.; Chen, B.; Messinger, D.W. Nearest-neighbor diffusion-based pan-sharpening algorithm for spectral images. Opt. Eng. 2014, 53, 013107. [CrossRef] Hinton, G.E.; Salakhutdinov, R.R. Reducing the dimensionality of data with neural networks. Science 2006, 313, 504–507. [CrossRef] [PubMed] Liu, Y.; Chen, X.; Peng, H.; Wang, Z. Multi-focus image fusion with a deep convolutional neural network. Inf. Fusion 2017, 36, 191–207. [CrossRef] Jiang, W.; Baker, M.L.; Wu, Q.; Bajaj, C.; Chiu, W. Applications of a bilateral denoising filter in biological electron microscopy. J. Struct. Biol. 2003, 144, 114–122. [CrossRef] [PubMed] Fukunaga, K.; Hostetler, L.D. The estimation of the gradient of a density function, with applications in pattern recognition. IEEE Trans. Inf. Theory 1975, 21, 32–40. [CrossRef] Cheng, Y.Z. Mean shift, mode seeking, and clustering. IEEE Trans. Pattern Anal. Mach. Intell. 1995, 17, 790–799. [CrossRef] He, K.; Sun, J.; Tang, X. Guided Image Filtering. IEEE Trans. Pattern Anal. Mach. Intell. 2013, 35, 1397–1409. [CrossRef] [PubMed] Levin, A.; Lischinski, D.; Weiss, Y. A closed form solution to natural image matting. IEEE Trans. Pattern Anal. Mach. Intell. 2006, 30, 228–242. [CrossRef] [PubMed] Durand, F.; Dorsey, J. Fast bilateral filtering for the display of high-dynamic-range images. ACM Trans. Graph. (TOG) 2002, 21, 257–266. [CrossRef] Petschnigg, G.; Szeliski, R.; Agrawala, M.; Cohen, M.; Hoppe, H.; Toyama, K. Digital photography with flash and no-flash image pairs. ACM Trans. Graph. 2004, 23, 664–672. [CrossRef] He, K.; Sun, J.; Tang, X. Single Image Haze Removal Using Dark Channel Prior. IEEE Trans. Pattern Anal. Mach. Intell. 2011, 33, 2341–2353. [CrossRef] [PubMed] Li, S.; Kang, X.; Hu, J. Image Fusion With Guided Filtering. IEEE Trans. Image Process. 2013, 22, 2864–2875. [CrossRef] [PubMed] Draper, N.; Smith, H. Applied Regression Analysis, 2nd ed.; John Wiley: New York, NY, USA, 1981. Laben, C.A.; Brower, B.V. Process for Enhancing the Spatial Resolution of Multispectral Imagery Using Pan-Sharpening. U.S. Patent 6011875 A, 4 January 2000. Zhao, W.; Dai, Q.; Zheng, Y.; Wang, L. A new pansharpen method based on guided image filtering: A case study over Gaofen-2 imagery. In Proceedings of the IGARSS IEEE International Geoscience and Remote Sensing Symposium, Beijing, China, 10–15 July 2016; pp. 3766–3769. Wald, L.; Ranchin, T.; Mangolini, M. Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images. Photogramm. Eng. Remote Sens. 1997, 63, 691–699.

ISPRS Int. J. Geo-Inf. 2017, 6, 404

33. 34. 35. 36.

37. 38.

22 of 22

Klonus, S.; Ehlers, M. Image fusion using the Ehlers spectral characteristics preserving algorithm. GIS Remote Sens. 2007, 44, 93–116. [CrossRef] Bovik, A.; Wang, Z. A universal image quality index. IEEE Signal Process. Lett. 2002, 9, 81–84. [CrossRef] Ranchin, T.; Wald, L. Fusion of high spatial and spectral resolution images: The ARSIS concept and its implementation. Photogramm. Eng. Remote Sens. 2000, 66, 49–61. Yuhas, R.H.; Goetz, A.F.H.; Boardman, J.W. Discrimination among semi-arid landscape endmembers using the Spectral Angle Mapper (SAM) algorithm. In Proceedings of the Summaries 3rd Annual JPL Airborne Geoscience Workshop, Pasadena, CA, USA, 1–5 June 1992; pp. 147–149. Alparone, L.; Baronti, S.; Garzelli, A.; Nencini, F. A global quality measurement of Pan-sharpened multispectral imagery. IEEE Geosci. Remote Sens. Lett. 2004, 1, 313–317. [CrossRef] Boser, B.; Guyon, I.; Vapnik, V. A training algorithm for optimal margin classifiers. In Proceedings of the 5th Annual Workshop on Computational Learning Theory, Pittsburgh, PA, USA, 27–29 July 1992; pp. 144–152. © 2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).