HYPERSPECTRAL CHANGE DETECTION WITH STEREO DISPARITY INFORMATION ENHANCEMENT Ali Can Karaca, Davut Çeşmeci, Alp Ertürk, M. Kemal Güllü, SarpErtürk Kocaeli University Laboratory of Image and Signal Processing (KULIS), Kocaeli University, Turkey {alican.karaca1, alp.erturk, kemalg, sertur}@kocaeli.edu.tr,
[email protected] ABSTRACT A hyperspectral change detection method with stereo depth information enhancement is proposed in this paper. The method operates on the hyperspectral data acquired by the ground-based hyperspectral stereo imaging system. The imaging system combines the properties of panoramic and stereo imaging with the high spectral resolution of hyperspectral cameras, and is of use especially for surveillance applications. Stereo and spectral information provided by the system are fused in the proposed method for enhanced change detection. Experimental results are evaluated on two hyperspectral datasets acquired by the system. Preliminary results show the improved performance of the system and the proposed method.
change detection methods are proposed for hyperspectral imagery. In [9], removing parallax-induced changes by using epipolar geometry is proposed for enhancing hyperspectral change detection. That is the only work the authors came across in literature on improving hyperspectral change detection by using stereo information. In this paper, depth information, obtained by utilizing stereo hyperspectral images, is utilized to improve hyperspectral change detection performance. A novel stereo hyperspectral imaging system is utilized to acquire the data. In section 2, this hyperspectral imaging system is introduced. The proposed method is described in section 3. Experimental results and conclusions are given in section 4 and section 5, respectively. 2. STEREO HYPERSPECTRAL IMAGING SYSTEM
Index Terms— Change detection, disparity, hyperspectral imaging, stereo imaging. 1. INTRODUCTION Hyperspectral imaging systems collect data in visible and infrared regions of electromagnetic spectrum and acquire hundreds of narrow spectral bands, resulting in near-continuous electromagnetic spectrum information for each pixel. The spectral information of each pixel is dependent on the chemical and physical properties of the underlying material, and can provide significant features. These significant features enable improved performance for many applications such as change detection. There are various studies in the literature on hyperspectral change detection. Image differencing and image ratio change detection methods are two of the simplest [1]. Principal Component Analysis (PCA) based change detection is proposed in [2]. A statistical analysis based change detection algorithm called, "chronochrome" is proposed in [3]. "Chronochrome" change detection method assumes linear changes between multi-temporal hyperspectral images. Covariance equalization (CE) method that avoids calculating cross covariance matrix directly, unlike the "chronochrome", is proposed in [4]. In [5], a subspace based hyperspectral change detection approach is proposed. In stereo imaging systems, depth information can be utilized to improve change detection performance. In literature, there are several work which combine change detection and stereo information for improving change detection accuracy. 3Dchange detection in urban areas using multi-temporal satellite images is proposed in [6]. In [7], region based change detection for forest areas using single-channel stereo images and LIDAR data is proposed. 3D building change detection using stereo images and GIS database is proposed in [8]. However, none of these stereo
The novel stereo hyperspectral system, presented in Figure 1, consists of three major parts, namely the rotating stage, the pushbroom VNIR hyperspectral camera couple, which are mounted on the rotating state symmetrically, and the notebook.
Figure 1. Hyperspectral imaging system. Imaging spectrometer provides narrow spectral bands with 2.8 nm spectral resolution. For objective lens, 25 mm F/1.8 lenses are used. To increase frame per second (FPS) rate, 400 × 300 resolution is used in this work. In final configuration, wavelength range is decreased to 500 - 875 nm and to the spectral band to 300 so that the cameras capture with 40.4 FPS. Note that spatial alignments and spectral calibrations of both hyperspectral cameras are performed before hyperspectral image acquisition. Even though the camera pair is mounted on the rotating stage carefully, a small point of view difference is prone to occur between them. Hence, geometrical correction must be performed as preprocessing, in the vertical axis for matching epipolar lines to correspond to the same row. For this purpose, phase correlation [10] is applied to average spectral band images of left and right hyperspectral data. It is worth mentioning that, there is no registration problem between hyperspectral data acquired at different times when the
system is located at the same spot, because of the stability of the system. Owing to precisely aligned datasets, pixel-wise change detection can be accomplished without the need for registration. 3. PROPOSED METHOD In this paper, a novel method is introduced to improve change detection performance by fusing spectral change map and depth map calculated from disparity maps.
applied. The equations of aggregation step are given in (1) and (2). In this part, 𝑝 is any pixel in reference image(𝐼 ) in the 𝑊 matching window which has a size of 𝑊 × 𝑊 . The matching window is asymmetrically weighted depending on the Euclidian distance (𝑑 ) between 𝑝 and center pixel 𝑝 and labeled to same segment (𝑆 ) with center pixel 𝑝 . ℎ (𝑝 , 𝑝 ) = 𝑒𝑥𝑝 −
3.1. Spectral Change Detection Spectral change detection is carried out for hyperspectral data, acquired by the same camera (i.e. left or right) of the hyperspectral stereo imaging system, at different times. Even though the datasets are collected in the same day, with little time difference, there are still shadow effects and some illumination changes. Spectral angle distance (SAD) [11] measurement, due to its robustness against illumination variations, has been used for change detection in this work. SAD is also preferred because it provides improved performance with respect to PCA and Euclidian distance [12] measures while still providing much lower computation time with respect to high performance methods such as CC and CE. Two hyperspectral images, acquired at different times, are compared pixel-by-pixel by the SAD measure and a change detection map, which includes the cost value for each pixel pair, is obtained. 3.2 Segment Weighted Stereo Matching Algorithm After geometrical offset correction of stereo images, segment weighted stereo matching algorithm is performed in 4 steps, namely cost initialization, segmentation, cost aggregation and evaluation of disparity constraints. Census transform is quite popular for stereo applications and chosen in this work due to its high performance under different distortion effects [13, 14]. A multiple-band Census transform based matching cost is proposed in this work for the cost initialization step. The proposed cost considers spatial neighbors of the center pixel in different spectral band images for each hyperspectral image and provides radiometric robustness. Firstly, the number of spectral band is reduced to N by averaging each consecutive 300 / N bands to one band, in a non-overlapping manner, for noise reduction. Then, both image pairs are coded over 𝑊 × 𝑊 size sliding windows for N bands using Census Transform. After this step, each pixel is represented by a binary code which has 𝑊 × N size. Binary code vectors are compared using Hamming distance, and the cost values are generated. For segmentation of hyperspectral images, region growing based on SAD over the full spectrum of pixels are used. Segmentation algorithm has two parameters: the maximum number of segments 𝑆 and spectral angle threshold 𝑇 (in radians). In the algorithm, a random pixel is selected and the spectral angles with its neighbors are calculated using SAD. The region is grown as long as the neighbors’ distances are smaller than T. After the region growing stops, another pixel, which is not already segmented, is randomly selected and the same steps are performed iteratively until the number of segment labels is equal to 𝑆 . After the segmentation of hyperspectral images and cost initializations are completed, segment-weighted cost aggregation is
∑ 𝐴𝐶(𝑝 , 𝑞 ) =
,
𝑑 (𝑝 , 𝑝 ) , 𝛾 , 0
𝑖𝑓 𝑝 ∈ 𝑆 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
ℎ (𝑝 , 𝑝 ) × 1 − 𝑒𝑥𝑝 − ∑
,
ℎ (𝑝 , 𝑝 )
(1)
( , )
(2)
where 𝑝 ∈ 𝑊 and 𝑞 ∈ 𝑊 . Here, C is the multiple band Census transform based cost value, AC is the aggregated cost, 𝑑 is the Euclidean distance between the two pixels, 𝛾 is the cost parameter, 𝛾 is the weighting parameter, 𝑝 is a pixel in the reference image, 𝑞 is a pixel in target image, and 𝑊 is the matching window in the target image. Disparity values of any pixel are determined with Winner-Takes-All procedure. In the final step, to eliminate ambiguous disparity values, smoothness and uniqueness constraints [15] are checked for the disparity maps. For the smoothness constraint, disparity values more than half of all neighbors in 3 × 3 neighborhood is assigned to the center pixel. After that, for the uniqueness constraint, bidirectional matched pixels, which explain consistent disparity values for left and right images, are evaluated and mismatched regions are labeled with zero value. 3.3. Fusion of Depth and Change Information Disparity maps are more reliable for closer objects, and less with distance. Therefore, in this work, disparity values are weighted by a coefficient that emphasizes high disparities and, vice versa. These coefficients are obtained in a Gaussian distribution, the mean of which is the highest disparity value and standard deviation is selected as half of the mean value. In the disparity maps, small areas that have quite different disparity values with their neighborhoods are removed. Then, the disparity maps, obtained for hyperspectral data acquired at different times, are subtracted from each other and negative and positive valued depth maps are obtained. Positive and negative values represent object insertion and deletion from the scene, so two masks are created for each of them. As a last noise reduction step, open and close morphological operations are applied on both of the masks. Separately processing insertion / deletion masks provides better noise reduction, especially towards white noise effects. Finally, these masks are combined by OR operator and used as masking absolute valued depth-map. Before the spectral change map and depth map are fused, cost values are normalized to [0, 1] range. These normalizations are conducted by performing exponential operation [16] as follows: 𝜌(𝐶, 𝜆) = 1 − 𝑒𝑥𝑝(− 𝐶⁄𝜆 )
(3)
where C defines considered cost value, and λ parameter controls the cost peak effects. After arranging cost values, a linear fusion operation is carried out as the weighted summation given in (4). 𝐶 is the resulting “depth-change” cost map. 𝐶
=𝑤
×𝜌 𝐶
,𝜆
+𝑤
× 𝜌(𝐶
,𝜆
)
(4)
4. EXPERIMENTAL RESULTS The proposed method has been applied on two datasets. Each dataset consist of two stereo hyperspectral data pairs, acquired at different times within the same day. The spatial size of hyperspectral images in the first and second datasets are 361 × 773 and 356 × 575, respectively. Hyperspectral images consist of 300 spectral bands in both of the datasets. The scenes contain not only changes of interest (inserted/relocated targets) but also natural effects (movement of leaves due to wind, shadow effects, etc.). It has to be noted that, this work focuses on changes of interest and any detected change that corresponds to changes that occur due to the time of day and sun angle are considered false alarms. In the first dataset, a cardboard box (located close to middle of scene) and a plaster board covered by green cardstock (in front of the bush) are placed as targets on the scene prior to second acquisition (Figure 2 (a),(b)). In the second dataset, two cardboards (smaller one in front of the column of building, bigger one next to the bush) are placed and a moving dog (left side of the scene) accepted as changes of interest between the two acquisitions of the scene (Figure 2 (g),(h)). Ground truths are constructed manually for the two datasets (Figure 2 (c),(i)). In multiple-band Census coding step, the number of spectral bands is reduced to 10 (i.e. N = 10) and census coding block size (WC×WC) is set experimentally as 5 × 5. In segmentation step, maximum number of segments (Smax) is selected as 3000, and clustering threshold (T) as 0.05 rad. In disparity map extraction step, matching window size (WM×WM) is selected as 25 × 25, γp and γc are set as 12 and 83, respectively. According to given parameters, SAD based change cost map (Figure 2 (f),(l)) and disparity maps(Figure 2 (d), (e), (j), (k)) are extracted. In noise reduction step (In section 3.3), open and close morphological operations are applied by using 3 × 3 kernel matrix, in which all elements are equal to 1. λspe and λste parameters are selected as 0.5 and 8, respectively. In this work, depth-change map calculated by weighting spectral cost and depth maps equally (wspe = wste = 0.5). Calculated depth-change maps for two datasets given in Figure 3. Although spectral change detection methods are good at catching changes that occurred by spectral variations, they suffer towards to changes such as zoom in/out movements of target. On the contrary, depth maps only sensible to distance variations between target and imaging system. Consequently, the change detection performance is enhanced by using fusion of spectral change map and depth map. According to the changes occurred in the scene, the cost weights (wspe , wste) can be easily adjusted for best results. Receiver-operating-characteristics (ROC) curves, obtained over the two datasets, used to compare the performance of standard SAD based change detection and the proposed method, are provided in Figure 3 (c).The stability of the system ensures that there is no registration problem and that false alarms are low. Also,
there is little time difference between collected datasets, which result in less diurnal variations. For these reasons, both of the methods perform well. However, it can be observed that the proposed method enhances the change detection accuracy of standard SAD based change detection approach. The proposed depth-change method results in lower false positive and higher true positive ratios. 5. CONCLUSION In this paper, a novel method that utilizes depth information to enhance change detection performance is proposed for a groundbased stereo hyperspectral imaging system. The imaging system makes sure that no registration is required on the data acquired at different times, which enables the depth information to be combined with change detection results easily. The preliminary experimental results obtained on the data acquired by the introduced system validate the performance of the method. 6. REFERENCES [1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
R. J. Radke, S. Andra, O. Al-Kofahi, and B. Roysam, “Image change detection algorithms: A systematic survey,” IEEE Transactions on Image Processing, vol. 14, pp. 294–307, March 2005. T. Celik, “Unsupervised change detection in satellite images using principal component analysis and K-means clustering,” IEEE Geoscience and Remote Sensing Letters, vol. 6, pp. 772–776, October 2009. A. Schaum and A. Stocker, “Long-interval chronochrome target detection,” in Proc. International Symposium on Spectral Sensing Research, 1998. C. Wu, B. Du and, L. Zhang, “A Subspace-based change detection method for hyperspectral images,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 6, no. 2, April 2013. A. Schaum and A. Stocker, “Hyperspectral change detection and supervised matched filtering based on covariance equalization,” in Proc. SPIE, vol. 5425, pp. 77-90, 2004. J. Tian, H. Chaabouni-Chouayakh, and P. Reinartz, “3D building change detection from high resolution spaceborne Stereo Imagery,” IEEE International Workshop on MultiPlatform/Multi-Sensor Remote Sensing and Mapping (M2RSM), pp. 1-7, January 2011. J. Tian, J. Leitloff, and P. Reinartz, “Region based forest change detection from cartosat-1 stereo imagery,” ISPRS 2011 Hannover Workshop, International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 14-17 June 2011. G. R. Dini, K. Jacobsen, F. Rottensteiner, M. Al Rajhi, and C. Heipke, “3D building change detection using high resolution stereo images and GIS database,” ISPRS Congress, International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. XXXIX-B7, 2012. K. Vongsky, M. J. Mendenhall, M. T. Eismann, and G. L. Peterson, “Removing parallax-induced changes in hyperspectral change detection,” IEEE International Geoscience and Remote Sensing Symposium (IGARSS), pp. 2012-2015, July 2012.
B. Reddy and B. N. Chatterji, “An FFT-based technique for translation, rotation, and scale-invariant image registration,” IEEE Transactions on Image Processing, vol. 5, no. 8, pp. 1266-1271, 1996. [11] R. Lark, “A reappraisal of unsupervised classification, 1: correspondence between spectral and conceptual classes,” International Journal of Remote Sensing, vol. 16, no. 8, pp. 1425-1443, 1995. [12] J. A. Richards, Remote sensing digital image analysis: An introduction, 3rd Edition, Springer-Verlag, NY, 1999. [13] R. Zabih and J. Woodfill, “Non-parametric local transforms for computing visual correspondence,” Third European Conference. Computer Vision (ECCV), vol. 801, pp. 151– 158, May 1994. [10]
(a) dataset-1, first acquisition
H. Hirschmüller and D. Scharstein, “Evaluation of stereo matching costs on images with radiometric differences,” IEEE Transactions on Pattern analysis and machine intelligence, vol. 31, pp. 1582-1599, 2009. [15] T. Beeler, B. Bickel, P. Beardsley, B. Summer, and M. Gross, “High-quality single shot capture of facial geometry,” in Proc. of ACM SIGGRAPH, vol. 29, no. 3, pp. 40:1-40:9, 2010. [16] X. Mei, X. Sun, M. Zhou, S. Jiao, H. Wang, and X. Zhang, “On building an accurate stereo matching system on graphics hardware,” IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp. 467-474, 2011. [14]
(b) dataset-1, second acquisition
(c) dataset 1, change ground truth
(d) dataset-1,disp. map first acquisition
(e) dataset 1, disp. map of second acquisition
(f) dataset 1, SAD change detection cost map
(g) dataset-2, first acquisition
(h) dataset-2, second acquisition
(i) dataset 2, change ground truth
(j) dataset-2, disp. map of first (k) dataset-2, disp. map of second (l) dataset-2, SAD change detection cost acquisition acquisition map Figure 2. Acquisition images, ground truths and disparity maps for used datasets
(a) (b) (c) Figure 3. Results of the Depth-Change method a) for dataset 1, b) for dataset 2, c) Global ROC curves for both datasets
