a high performance missing pixel reconstruction ...

2nd. International Conference on Applied and Theoretical Information Systems Research, December 27-29, 2012, Taipei, Taiwan

A HIGH PERFORMANCE MISSING PIXEL RECONSTRUCTION ALGORITHM FOR HYPERSPECTRAL IMAGES Jin Zhou, Signal Processing, Inc., USA [email protected] Chiman Kwan, Signal Processing, Inc., USA [email protected] Bulent Ayhan Signal Processing, Inc., USA [email protected]

ABSTRACT In many image processing applications, pixels may be corrupted or simply missing. In some other cases, pixels may be randomly deleted in order to save bandwidth during transmission. It is important to develop high performance algorithms that can reconstruct those corrupted or missing pixels. In this paper, we will summarize our research effort in developing a high performance reconstruction algorithm for reconstructing missing pixels in hyperspectral images. Experiments using actual images clearly demonstrated that our algorithm can achieve high reconstruction performance even in high missing rates as high as 95% or 99%. Keywords: Hyperspectral images, missing data reconstruction, matrix completion

1. INTRODUCTION 1.1 Data with Missing Information In social networks, links between subjects may be missing. In some images, pixels may be corrupted or simply missing. In some surveillance applications, pixels may be randomly deleted to save network bandwidth. For example, a low cost unmanned air vehicle running surveillance operations may not have enough onboard computational power to perform sophisticated image compression. A simple way to save bandwidth is to randomly select some pixels and send them over to the ground station, which will then reconstruct the missing pixels using high performance computers.

1.2 Matrix Completion In all of the above mentioned scenarios, the missing data locations are known. Conventional methods use interpolation from neighboring pixels to fill in the missing data. However, interpolation can only utilize local information. If the original data have 1


some inherent structures such a repetitive patterns (houses, roads, etc.), then conventional interpolation based methods may not yield good performance. Recently, matrix completion algorithms in [3] and references therein have been proposed to fill in missing data in images. We evaluated a number of these algorithms and realized that these methods do not perform well for images, as one fundamental assumption (low rank condition) was not satisfied in images.

1.3 Paper Organization This paper is organized as follows. In Section 2, we will describe a local approach to matrix completion. Section 3 summarizes our experimental results using hyperspectral images. Different missing rates were investigated. Peak-signal-to-noise ratio (PSNR) was used as a performance metric. Another performance metric is the performance of anomaly detection using receiver operating characteristics (ROC) curves. We conclude our paper in Section 4.

2. THEORY Input Image

LxL blocks

block vectors

LxL

L L

Matrix Completion

Output Image

Figure 1. Signal flow of our algorithm Our approach is a local approach where local blocks (8x8) are relatively smooth and hence a matrix formed by combining all the vectorized blocks will become low rank. For low rank matrices, one can apply some transformation (wavelet, DCT, etc.) to generate sparse representations. Matrix completion can then be performed on each block. The signal flow of our approach is summarized in Fig. 1. An input image with missing data,128x128 for instance, is first divided into many LxL blocks (L=8, for example), which are then vectorized. The missing pixels in the vectors are then filled in by matrix

2


completion, which is summarized in Fig. 2. The completed vectors are then merged to form a complete image. To complete the missing data in the vectors, we adopted a well-known decomposition model as shown in Fig. 2. That is, each vector is represented as the product of a dictionary and a sparse vector. Researchers have used this representation in hyperspectral image processing [1][2], denoising and compressive sensing [5], face recognition and target recognition [6], and references therein. N

K

N

N

is zero Incomplete vectors

Sparse Coefficients

Dictionary

Completed Vectors

Figure 2. Sparse matrix completion We chose Orthogonal Matching Pursuit (OMP) as the solver for generating sparse vectors. We also compared OMP with another sparsity solved called FOCUSS [5]. We found that OMP is more robust in our simulations. Details of OMP can be found in [5]. Advantages of our algorithm are summarized below: • High performance (low root mean square error (RMSE) and high peak signal-tonoise ratio (PSNR)) • Utilize information within each image; no need of outside information • Graceful degradation when missing rate is high • Our method works well even in 99% and 99.9% missing data cases • Applicable to LADAR, EO, and hyperspectral images of various sizes • Parallelizable and suitable for real-time application using FPGA/GPU • Can perform simultaneous denoising and inpainting

3. EXPERIMENTAL RESULTS 3.1 Data This paper summarizes the results of our matrix completion algorithm mentioned in Section 2 for hyperspectral images from the Air Force Wright Patterson Lab. There are two types of images: type B and type C. Type B refers to the case where missing pixel pattern is the same for all bands; type C is the case where missing patterns are independent of bands. We observed that reconstruction performance using our algorithm is very good, even for 95% missing data in type B and 99% missing data in type C cases. This is due to our advanced matrix completion algorithm and high redundancy in spatial and spectral domain. The sample image (48th band) is shown in Fig. 3. The dimensions (width x height x bands) of each images are: 800 x 1024 x 124. 3


We use the following sampling scheme to the image and then apply image reconstruction based on matrix completion: • Data type B (missing pixel pattern is the same for all bands) and C (missing pattern is independent). • Missing rate A 50%, B 90%, C 95%, D 99%, E 99.9% and F 75%.

Figure 3. The 48th band of the AF hyperspectral image

3.2 Reconstruction Results Results of image reconstruction are shown in Table 1 and Fig. 4. We observed that AF data have excellent reconstruction performance. For instance, even in 99.9% missing rate, type C data can achieve 33 dB in PSNR and type B data can achieve 27 dB in PSNR. One reason is that AF data were taken from ground and have high resolution (high spatial correlation between pixels). Another reason is that AF data are more homogeneous since most regions are grass and trees. Table 1. Reconstruction performances of AF images under different missing rates (columns) and different sampling strategies (rows). PSNR (dB) B C

A (50%)

B (90%)

C (95%)

D (99%)

E (99.9%)

F (75%)

42.567 43.654

37.276 43.417

35.939 42.950

32.669 36.033

27.249 33.485

40.863 43.588

4


(a) (b) Figure 4. Reconstruction performances in terms of PSNR (a) type B images (b) type C images. Reconstructed images for one of the bands (48th) are shown below for type B (Fig. 5) and type C (Fig. 6).

B_A (50% missing)

B_F (75% missing)

B_B (90% missing)

B_C (95% missing)

5


B_E (99.9% missing) B_D (99% missing) Figure 5. Reconstructed images of 48th band for type B data of the AF hyperspectral image. Visual inspection indicates that the reconstruction performance is acceptable up to 95% missing pixels.

C_A (50% missing)

C_F (75% missing)

C_B (90% missing)

C_C (95% missing)

6


C_D (99% missing) C_E (99.9% missing) Figure 6. Reconstructed images of 48th band for type C data of AF hyperspectral image. Visual inspection indicates that the reconstruction performance is acceptable up to 99% missing pixels.

3.3 Anomaly Detection Using Reconstructed Images In the AF images, there are 2 small targets which are shown in Fig. 7. We applied Kernel RX algorithm [4] to the recovered images to detect the targets. In addition, we also applied the algorithm to the original image (no missing data) for comparison.

Figure 7. Ground truth targets in the AF hyperspectral image. Fig. 8 ~ Fig. 9 show the ROC comparison plots. In each plot, there are also filtered curves, meaning that the anomaly detection results are filtered with 3x3 smoothing filters. In general, filtered results yielded better performance than those without filtering. From the plots we observe the following: • For AF type B images, when missing rate is equal or less than 90%, the anomaly detection performance of recovered image is comparable to that of original image. • For AF type C images, when missing rate is equal or less than 99%, the anomaly detection performance of recovered image is comparable to that of original image.

7


(a) B_A (50% missing)

(b) B_F (75% missing)

(c) B_B (90% missing)

(d) B_C (95% missing)

(e) B_D (99% missing) (f) B_E (99.9% missing) Figure 8. AF image results (type B missing case). Comparison of ROC of anomaly detection results from ground truth image (green lines) and recovered images (blue lines).

8


(a) C_A (50% missing)

(b) C_F (75% missing)

(c) C_B (90% missing)

(d) C_C (95% missing)

(e) C_D (99% missing) (f) C_E (99.9% missing) Figure 9. AF image results (type C missing case). Compare of ROC of anomaly detection results from ground truth image (green lines) and recovered images (blue lines).

9


4. CONCLUSIONS We presented a locally approach to matrix completion. Actual hyperspectral images were used in our evaluations. Our algorithm yielded high performance in terms of PSNR and ROC curves for anomaly detection. No loss of anomaly detection performance when 90% of pixels are missing for Case B (same pixels are missing in all bands) and 99% of pixels are missing in Case C (pixels are missing randomly in different bands).

5. ACKNOWLEDGEMENTS This research was supported by Office of Naval Research under contract N00014-12-C0079. We would like to thank Dr. Michael Eismann of the AF Wright Patterson Lab. for providing the hyperspectral image used in this paper.

6. REFERENCES [1] C. Chang, Hyperspectral Imaging, Kluwer Academic/Plenum Publishers, 2003. [2] C. Kwan, B. Ayhan, G. Chen, C. Chang, J. Wang, and B. Ji, “A Novel Approach for Spectral Unmixing, Classification, and Concentration Estimation of Chemical and Biological Agents,” IEEE Trans. Geoscience and Remote Sensing, accepted, October, 2005. [1] E . J. Candes and B. Recht, “Exact matrix completion via convex optimization”, Found. of Comput. Math, 2008, 9:717–772. [4] H. Kwon and N.M. Nasrabadi, "Kernel RX-Algorithm : A Nonlinear Anomaly Detector for Hyperspectral Imagery," IEEE Transactions on GeoScience and Remote Sensing, vol. 43, 2005, pp. 388-397. [5] M. Elad, Sparse and Redundant Representations, Springer New York, 2010. [6] T. Tran, “Locally Adaptive Sparse Representation for Detection, Classification, and Recognition,” presentation at ARL, June 2010.

10