does multispectral / hyperspectral pansharpening ... - IEEE Xplore

DOES MULTISPECTRAL / HYPERSPECTRAL PANSHARPENING IMPROVE THE PERFORMANCE OF ANOMALY DETECTION ? Ying Qu and Hairong Qi EECS Department The University of Tennessee, Knoxville, TN 37996 {yqu3, hqi}@utk.edu ABSTRACT Pansharpening refers to the fusion of a high spatial resolution panchromatic image with high spectral resolution multispectral or hyperspectral images (MSI or HSI) to yield high resolution data in both spectral and spatial domains. It has been widely adopted as a primary preprocessing step for numerous applications. In this paper, we perform a literature survey of various pansharpening algorithms including the most advanced deep learning approaches for both multispectral and hyperspectral images. We further evaluate the effect of the resolution difference on anomaly detection. Synthetic multispectral and hyperspectral images are generated to evaluate the performance of anomaly detection on high resolution images. Eight state-of-the-art MSI and HSI pansharpening methods are compared in this paper. Experimental results show that, performing anomaly detection on high resolution images improves the detection rate, and at the mean time suppresses the false alarm rate. Index Terms— Hyperspectral images, multispectral images, pansharpening, anomaly detection, deep learning 1. INTRODUCTION Multispectral images (MSI) and Hyperspectral images (HSI) can provide important spectral information that benefits remote sensing applications. However, due to the limitations of remote sensors, there is a trade-off between spatial resolution and spectral resolution. Pansharpening refers to the fusion of high spatial resolution panchromatic images (PAN) with high spectral resolution MSI or HSI to generate images with high resolution (HR) in both spectral and spatial domains. Recent technologies are able to acquire PAN with the corresponding MSI or HSI simultaneously using commercial satellites such as Google Earth and Bing Maps, that has made the pansharpening possible. Nowadays, pansharpening plays an increasingly important role for remote sensing, and is becoming one of the essential preliminary processing procedures for many applications such as environmental monitoring, change detection, object recognition, and classification [1]. For these applications, high resolution images in both spatial and spectral domains

978-1-5090-4951-6/17/$31.00 ©2017 IEEE

Bulent Ayhan, Chiman Kwan

Richard Kidd

Applied Research LLC, Rockville, MD

Jet Propulsion Lab Pasadena, CA

are required to generate more promising results [2]. Anomaly detection is one of the remote sensing applications, which aims to identify anomaly pixels inside an image. It can be modeled as an unsupervised binary classification problem between the background class and the anomaly class. In this paper, we evaluate the performance of HR MSI or HSI achieved by pansharpening through the application of anomaly detection and investigate if adopting pansharpening as a preprocessing step will influence the performance of anomaly detection. The detection method we choose for anomaly detection is Subspace-RX (SSRX) [3], which performs RX in the primary subspace. We choose SSRX because it is a popularly used detection method that works effectively especially on small anomalies [4], which has been the common scenario of anomalies appeared in MSI or HSI. The contribution of this work is two-fold. First, several stateof-the-art pansharpening methods for both multispectral and hyperspectral images are surveyed and investigated including the most advanced deep learning based approaches. Second, to the best of our knowledge, we make the first attempt to evaluate the pansharpening technique through anomaly detection and investigate if pansharpening can benefit the task of anomaly detection. The rest of the paper is organized as follows. Sec. 2 reviews the pansharpening methods for both MSI and HSI. Sec. 3 evaluates the state-of-the-art pansharpening algorithm through anomaly detection. Conclusion are drawn in Sec. 4. 2. PANSHARPENING TECHNIQUES 2.1. Pansharpening for Multispectral Images MSI pansharpening can be dated back to the 19th century, and has been well developed through decades of works [5]. Traditional widely used pansharpening algorithms can be roughly classified into two groups, the component substitution (CS) and the multiresolution analysis (MRA) approaches. CS–based approaches [5] mainly project the low resolution (LR) MSI onto a predefined space, which separates the spectral information from spatial information.By substituting the spatial components with histogram-matched PAN and converting back to the MSI space, the resolution of data is effectively improved. Band-dependent spatial detail(BDSD)

6130

IGARSS 2017

[6] extends the general framework of CS-based approaches by defining coefficients and rewriting them into a different format. According to Vivone et al. [2], in the CS family, BDSD yields the most reasonable results, thus we include this method for evaluation. The multiresolution analysis (MRA) approaches achieve HR by injecting spatial details of PAN to an interpolated MSI. The difference between the PAN and the low-pass filtered PAN is used to improve the quality of MSI. Spatial details can be extracted by various transforms such as decimated wavelet transform (DWT) [7], adaptive wavelet transform (ATWT) [8] and Laplacian pyramid (LP) [9]. Two injection schemes are usually used, including the additive (e.g., MTF-GLP-HPM [10]) and multiplicative (e.g., MTF-GLP [11]) approaches, which, according to [2], have yielded decent results. 2.2. Pansharpening for Hyperspectral Images Pansharpening methods have been extended to HSI due to the increasing demand for high resolution hyperspectral images. However, very few methods have been proposed to address this problem [12, 13], leaving great potential in the development of this field. Most traditional MSI pansharpening methods can be extended to HSI, but in this section, we only focus on recent developments. The Bayesian approaches calculate the posterior distribution of the HR HSI given MSI or PAN. The Bayesian framework offers a convenient way to regularize the solution of HR HSI by employing a proper prior distribution [14] [15]. Take the fusion of the observed LR HSI and HR MSI as an example. Assume that the LR HSI YH is acquired by convolving HR HSI X with blur kernel B and downsampled by S. The observed HR PAN YP is acquired by the multiplication of spectral response of MSI R and X. The problem can be formulated as YH = XBS + NH ;

YP = RX + NP ,

min kYH − HUSk2F + kYPAN − RHUk2F H,U

(2)

Different methods vary by choosing different prior distribution p(U). In this paper, three state-of-the-art approaches of this group including Bayesian Na¨ive [16], Bayesian Sparse [15] and HySure [17] are evaluated through their effect on the task of anomaly detection. Another group of approaches addresses the pansharpening based on non-negative matrix factorization (NMF) or unmixing. These approaches are based on the assumptions similar to Eq. 1. However, the difference is that, the methods in this group assume X is a linear combination of H and U. Thus it is intuitive and natural to keep U and H non-negative, which leads to a part-based representation. The algorithms proposed

(3)

The problem is solved separately by optimizing the first term with respect to H, and second term with respect to U. In this way, the HR HSI is generated by X = HU. CNMF [18] is a representative method in this group and will be evaluated. 2.3. Deep Learning-based Pansharpening Deep learning based approaches have shown great potential in many field of studies. There have been three recent attempts to address pansharpening for HSI or MSI. In 2012, Licciardi et al. [20] proposed the first deep learning based fusion method, which reduced the dimension of HSI automatically using autoencoder. In their structure, a sigmoid function is defined as the active function in the first layer to compress the data. Then an induction technique is proposed to fuse the dimension reduced HSI and PAN, based on the rule that we should recover the same LR HSI by downsampling the improved HR HSI. In 2015, a modified sparse tied-weights denoising autoencoder was proposed by Huang et al. [21]. The authors assumed that there exists a mapping function between LR and HR images for both PAN and MSI. During the training process, an LR PAN was generated by the interpolated MSI, then the mapping function was learned by given the LR PAN patches as the input and HR PAN patches as the output. In 2016, a supervised three-layer SRCNN was proposed by [22] to learn the mapping function between the input HR Pan with the interpolated LR MSI, and the output HR MSI. 3. EXPERIMENTAL RESULTS

(1)

where NH and NP are independent noise. Since HSI can be assumed to lie in a low-dimensional subspace with basis H, X = HU, the problem can be converted to p(U|YH , YP ) ∝ p(YH |U)p(YP |U)p(U).

by Yokoya et al. [18] and Lanaras et al. [19] are similar methods in this group with the only difference being the different NMF algorithms they adopted. The fusion is done by jointly unmixing HR PAN and LR HSI into endmembers and abundance vectors by the objective function defined as

In this section, we evaluate the performance of pansharpening algorithms through the application of anomaly detection using synthetic data. Since in real applications, anomalies are usually of small size, both MSI (120 × 120 × 4) and HSI (120 × 120 × 176) data are generated by implanting small dots with different reflections from 0.1 to 1 on the real HR images. Then they are convolved with a 9 × 9 Gaussian blur kernel with a standard deviation of 0.849, and downsampled in both horizontal and vertical directions with a factor of 2. The PAN is generated by averaging the spectral bands of the HR images. Four advanced MSI pansharpening algorithms, BSDS [6], MTF-GLP-HPM [10], MTF-GLP [11], and MTFGLP-CBD [11], are chosen to perform the fusion, which are used as a preprocessing step before anomaly detection. Similarly, HSI images are fused with four advanced HSI pansharp. ening algorithms including Bayesian Na . . ive [16], Bayesian Sparse [15], CNMF [18], and HySure [17]. Three metrics, RMSE, SAM and ERGAS [12], are performed to evaluate the

6131

quality of the HR results with the optimal value 0. A traditional anomaly detection algorithm Subspace-RX (SSRX) [3] is applied on both pansharpened HR and subsampled HR images. For quantitative comparison, detection results are converted into binary images according to different thresholds. Based on these binary images, receiver operating characteristics (ROC) curves are generated by calculating the detection rate versus false alarm rate. 3.1. Anomaly Detection on Multispectral Pansharpening Images Figure 1 shows the MSI pansharpening results and their subsampled results using different algorithms. According to the metrics demonstrated in Table 1, MRA-based methods generally achieve better results as compared to CS-based methods. Figure 2 shows the ROC curve of performing SSRX on HR results and subsampled HR results, respectively. Note that, REF refers to the ROC on original HR MSI, and LR shows the ROC on degraded LR MSI. From the ROC, we observe that the detection rate on HR MSI is higher than that on the degraded data, and the better the HR results, the higher the detection rate can be achieved. For most of the methods, we achieve better ROC on subsampled HR results. Table 1. Pansharpening evaluation metrics. MSI RMSE SAM ERGAS BSDS 0.036 4.01 7.95 MTF-GLP-HPM 0.025 3.55 5.35 MTF-GLP 0.023 3.33 5.06 MTF-GLP-CBD 0.027 4.10 5.78 HSI RMSE SAM ERGAS Bayesian Nave 0.04 3.67 6.93 Bayesian Sparse 0.04 6.8 7.55 CNMF 0.03 4.53 6.26 HySure 0.06 7.55 10.28

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

Fig. 2. Generated ROC curves. LEFT: the ROC on HR MSI. The RIGHT: ROC on subsampled HR MSI 3.2. Anomaly Detection on Hyperspectral Pansharpening Images Pansharpening results and subsampled results for HSI are shown in Table 1 and Fig. 3, from which we observe that . From visualization and metrics shown in Table 1, the recovered HR HSI of Bayesian Sparse [15] and CNMF [18] are more promising and contain more detailed information. By applyingThrough the same anomaly detection algorithm SSRX, the ROC curves are generated in Fig. 4. From the results, we can observe that as long as the super high resolution algorithm performs well, anomaly detection works shows better results on HR HSI than on the degraded LR HSI. Compared to applying SSRX on HR HSI directly,In addition, we are able to achieve better results by applying the detection method on subsampled HR HSI, because subsampling can reduce the noise and distortion of the HR HSI results while keeping adequate amount of details.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

Fig. 3. Hyperspectral pansharpening results using state-ofthe-art algorithms. (a) LR HSI. (b) HR PAN. (d), (f), (h) and (j) are HR HSI generated by Bayesian Nave, Bayesian Sparse, CNMF and HySure, respectively. (c),(e),(g) and (i) are subsampled HR HSI of (d), (f), (h) and (j), respectively. (k) subsampled ground truth. (l) ground truth.

Fig. 1. Multispectral pansharpening results using state-of-theart algorithms. (a) LR MSI. (b) HR PAN. (d), (f), (h) and (j) are HR MSI generated by BDSD, MTF-GLP, MTF-GLPHPM and MTF-GLP-CBD, respectively. (c),(e),(g) and (i) are subsampled HR MSI of (d), (f), (h) and (j), respectively. (k) subsampled ground truth. (l) ground truth.

4. CONCLUSION In this paper, we performed a literature survey of various state-of-the-art multispectral and hyperspectral pansharpening algorithms including the most advanced deep learning approaches. Totally eight advanced pansharpening approaches were studied and evaluated through the task of anomaly detection. Based on the experiments, for both MSI and HSI data,

6132

[10] B Aiazzi, L Alparone, S Baronti, A Garzelli, and M Selva, “An mtf-based spectral distortion minimizing model for pansharpening of very high resolution multispectral images of urban areas,” in Remote Sensing and Data Fusion over Urban Areas, 2003. 2nd GRSS/ISPRS Joint Workshop on, 2003, pp. 90–94.

Fig. 4. Generated ROC curves. LEFT:ROC on HR HSI. RIGHT:ROC on subsampled HR HSI applying anomaly detection on promising high resolution images generated by pansharpening algorithms will increase the detection rate while suppressing the false alarm rate simultaneously. Furthermore, if we conduct the anomaly detection on subsampled high resolution images, the detection rate will increase further, because the subsampling process will reduce the noise and distortion to certain extent without the risk of loosing too much information. 5. REFERENCES [1] Marcus Borengasser, William S Hungate, and Russell Watkins, Hyperspectral remote sensing: principles and applications, 2007. [2] Gemine Vivone, Luciano Alparone, Jocelyn Chanussot, Mauro Dalla Mura, Andrea Garzelli, Giorgio A Licciardi, Rocco Restaino, and Lucien Wald, “A critical comparison among pansharpening algorithms,” IEEE Transactions on Geoscience and Remote Sensing, vol. 53, no. 5, 2015. [3] A. Schaum, “Joint subspace detection of hyperspectral targets,” in Aerospace Conference, 2004. Proceedings. 2004 IEEE, March 2004, vol. 3, p. 1824 Vol.3. [4] Y. Qu, R. Guo, W. Wang, H. Qi, B. Ayhan, C. Kwan, and S. Vance, “Anomaly detection in hyperspectral images through spectral unmixing and low rank decomposition,” in 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), July 2016. [5] Claire Thomas, Thierry Ranchin, Lucien Wald, and Jocelyn Chanussot, “Synthesis of multispectral images to high spatial resolution: A critical review of fusion methods based on remote sensing physics,” IEEE Transactions on Geoscience and Remote Sensing, vol. 46, no. 5, pp. 1301–1312, 2008. [6] Andrea Garzelli, Filippo Nencini, and Luca Capobianco, “Optimal mmse pan sharpening of very high resolution multispectral images,” IEEE Transactions on Geoscience and Remote Sensing, vol. 46, no. 1, pp. 228–236, 2008. [7] Stephane G Mallat, “A theory for multiresolution signal decomposition: the wavelet representation,” IEEE transactions on pattern analysis and machine intelligence, vol. 11, no. 7, 1989. [8] Mark J Shensa, “The discrete wavelet transform: wedding the a trous and mallat algorithms,” IEEE Transactions on signal processing, vol. 40, no. 10, pp. 2464–2482, 1992. [9] Peter Burt and Edward Adelson, “The laplacian pyramid as a compact image code,” IEEE Transactions on communications, vol. 31, no. 4, pp. 532–540, 1983.

[11] B Aiazzi, L Alparone, S Baronti, A Garzelli, and M Selva, “Mtf-tailored multiscale fusion of high-resolution ms and pan imagery,” Photogrammetric Engineering & Remote Sensing, vol. 72, no. 5, pp. 591–596, 2006. [12] Laetitia Loncan, Luis B de Almeida, Jos´e M Bioucas-Dias, Xavier Briottet, Jocelyn Chanussot, Nicolas Dobigeon, Sophie Fabre, Wenzhi Liao, Giorgio A Licciardi, Miguel Simoes, et al., “Hyperspectral pansharpening: a review,” IEEE Geoscience and remote sensing magazine, vol. 3, no. 3, 2015. [13] Jin Zhou, Chiman Kwan, and Bence Budavari, “Hyperspectral image super-resolution: a hybrid color mapping approach,” Journal of Applied Remote Sensing, vol. 10, no. 3, pp. 035024– 035024, 2016. [14] Qi Wei, Nicolas Dobigeon, and Jean-Yves Tourneret, “Bayesian fusion of hyperspectral and multispectral images,” in 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2014. [15] Qi Wei, Jos´e Bioucas-Dias, Nicolas Dobigeon, and Jean-Yves Tourneret, “Hyperspectral and multispectral image fusion based on a sparse representation,” IEEE Transactions on Geoscience and Remote Sensing, vol. 53, no. 7, 2015. [16] Qi Wei, Nicolas Dobigeon, and Jean-Yves Tourneret, “Bayesian fusion of multi-band images,” IEEE Journal of Selected Topics in Signal Processing, vol. 9, no. 6, 2015. [17] Miguel Sim˜oes, Jos´e Bioucas-Dias, Luis B Almeida, and Jocelyn Chanussot, “A convex formulation for hyperspectral image superresolution via subspace-based regularization,” IEEE Transactions on Geoscience and Remote Sensing, vol. 53, no. 6, 2015. [18] Naoto Yokoya, Takehisa Yairi, and Akira Iwasaki, “Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion,” IEEE Transactions on Geoscience and Remote Sensing, vol. 50, no. 2. [19] Charis Lanaras, Emmanuel Baltsavias, and Konrad Schindler, “Hyperspectral super-resolution by coupled spectral unmixing,” in Proceedings of the IEEE International Conference on Computer Vision, 2015. [20] Giorgio Antonino Licciardi, Muhammad Murtaza Khan, Jocelyn Chanussot, Annick Montanvert, Laurent Condat, and Christian Jutten, “Fusion of hyperspectral and panchromatic images using multiresolution analysis and nonlinear pca band reduction,” EURASIP Journal on Advances in Signal processing, vol. 2012, no. 1, pp. 1–17, 2012. [21] Wei Huang, Liang Xiao, Zhihui Wei, Hongyi Liu, and Songze Tang, “A new pan-sharpening method with deep neural networks,” IEEE Geoscience and Remote Sensing Letters, vol. 12, no. 5, pp. 1037–1041, 2015. [22] Giuseppe Masi, Davide Cozzolino, Luisa Verdoliva, and Giuseppe Scarpa, “Pansharpening by convolutional neural networks,” Remote Sensing, vol. 8, no. 7, 2016.

6133