792
IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. 6, NO. 2, APRIL 2013
Non-Uniform Random Feature Selection and Kernel Density Scoring With SVM Based Ensemble Classification for Hyperspectral Image Analysis Sathishkumar Samiappan, Saurabh Prasad, Member, IEEE, and Lori M Bruce, Senior Member, IEEE
Abstract—Traditional statistical classification approaches often fail to yield adequate results with Hyperspectral imagery (HSI) because of the high dimensional nature of the data, multimodal class distribution and limited ground truth samples for training. Over the last decade, Support Vector Machines (SVMs) and Multi-Classifier Systems (MCS) have become popular tools for HSI analysis. Random Feature Selection (RFS) for MCS is a popular approach to produce higher classification accuracies. In this study, we present a Non-Uniform Random Feature Selection (NU-RFS) within a MCS framework using SVM as the base classifier. We propose a method to fuse the output of individual classifiers using scores derived from kernel density estimation. This study demonstrates the improvement in classification accuracies by comparing the proposed approach to conventional analysis algorithms and by assessing the sensitivity of the proposed approach to the number of training samples. These results are compared with that of uniform RFS and regular SVM classifiers. We demonstrate the superiority of Non-Uniform based RFS system with respect to overall accuracy, user accuracies, producer accuracies and sensitivity to number of training samples. Index Terms—Ground cover classification, hyperspectral imagery (HSI), multi-classifier systems (MCSs), random feature selection (RFS), support vector machines (SVMs).
G
I. INTRODUCTION
ROUND cover classification is a challenging and an important problem in remote sensing applications. Hyperspectral imagery (HSI) provides a detailed description of ground-cover materials ranging from visible to infrared regions of the electromagnetic spectrum. Such a wide spectral range of information has the potential to yield higher classification accuracies. The key to the design of a powerful classification system lies in extracting pertinent features from the high-dimensional data and employing classifiers to exploit those features. Maximum Likelihood (ML), a traditional supervised pattern classification approach, often fails to classify HSI data accurately because of (a) the high dimensionality of features, (b) Manuscript received May 15, 2012; revised August 17, 2012 and December 11, 2012; accepted December 13, 2012. Date of publication January 17, 2013; date of current version May 13, 2013. S. Samiappan is with the Mississippi State University, Geo Systems Research Institute, Starkville, MS 39759 USA (e-mail:
[email protected]). S. Prasad is with the University of Houston, Electrical and Computer Engineering Department, Houston, TX 77004 USA (corresponding author, e-mail:
[email protected]). L. M. Bruce is with the Department of Electrical and Computer Engineering, Mississippi State University, Mississippi State, MS 39762-9571 USA (e-mail:
[email protected]). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/JSTARS.2013.2237757
multimodal distributions and (c) limited ground truth availability. In order to solve the problem of high dimensionality, there are several existing approaches based on the concepts of dimensionality reduction and feature selection. Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA) and Stepwise-LDA (S-LDA) are popular dimensionality reduction techniques [33]. Feature selection can also be performed using metrics such as Bhattacharya Distance (BD), Jeffries-Matusita (JM), entropy etc. The Gaussian ML classifier assumes that the classes are Gaussian distributed. This is a limitation for majority of practical HSI datasets. Algorithms based on Gaussian Mixture models [1] have been proposed in the past to accommodate multi-modal distributions. An alternative approach that has become more popular recently with HSI data is the use of Support Vector Machines (SVM) [4]. Finally, there are techniques to solve the limited ground-truth availability such as sample interpolation and adaptive classifiers [2], [3]. In recent work, to improve the performance of conventional single classifiers, Multi-Classifier Systems (MCS) have been developed [5]–[8]. MCS are often referred as ensemble classifier systems, and they potentially perform better than single classifiers when diversity is established among the classifiers. The diversity among the classifiers can be established in different ways [9]–[11]. Prasad et al. [6], [7], demonstrated that, with an MCS setup with ML as classifier, the performance can be improved when compared to single classifiers, and there was a potential to further improve such a system by incorporating nonlinear SVM classifiers. Recently, a MCS based on Random Feature Selection (RFS) [12] proposed by Waske et al. and a dynamic subspace approach [13] proposed by Min et al. were shown to perform well with HSI data. Techniques such as random forests [14] and RFS perform well because they create diversity among the classifiers by re-sampling the spectral bands at the inputs of the classifier. As proposed in [10] diversity can also be created in other ways. In [15], Breiman has demonstrated the diversity creation by re-sampling, strategies related to [15] such as bagging [16] and boosting [17] are also proved to be effective. The selection of features in [12] is a uniform random feature selection (RFS). In our recent work [18], we explored the possibilities of using a non-uniform RFS (NU-RFS) based MCS with SVMs. We found that a diverse classifier ensemble for a classification problem need not always come from a RFS as proposed in [12], [13]. In [18] we demonstrated that NU-RFS can provide better performance than uniform RFS. In this study, as an extension, we present a fully automated MCS with NU-RFS using SVM. It is assumed that a diverse set of features leads to higher
1939-1404/$31.00 © 2013 IEEE
SAMIAPPAN et al.: NON-UNIFORM RANDOM FEATURE SELECTION AND KERNEL DENSITY SCORING
classification accuracies. Although the diversity can be defined in many ways [10] for the purposes of this study, a diverse set of spectral bands is defined as the following 1) Bands are selected from multiple spectral regions across the entire spectrum of signature. 2) Cross-correlation between selected bands is minimized. The approach proposed in this paper combines the following methods to create diversity within a pool of classifiers and to ensure that strengths and weaknesses of individual classifiers are incorporated into the final decision making: a) re-sampling features in the data through RFS; b) manipulation of input features through NU-RFS; c) manipulation of output classes through scores computed from Kernel Density estimation. The approach uses a spectral band grouping [28] to perform NU-RFS and a decision fusion strategy based on kernel density scores. To verify the effectiveness of this approach, we performed experiments to compare overall accuracies of SVM, RFS, NU-RFS, SVM with kernel density fusion, and NU-RFS with kernel density fusion. The sensitivity of the above mentioned approaches to the number of samples required to train them is also studied in this work. This paper is organized as follows. Section II provides a review of SVM, MCS, and RFS for SVM and possible extensions. Section III describes proposed kernel density based scoring system for fusion in a MCS, the proposed classification system based on NU-RFS, and band grouping. Section IV discusses the experimental setup and provides results. Finally, Section V summarizes the observations and provides concluding remarks. II. BACKGROUND A. Support Vector Machines The effectiveness of SVMs for HSI data has been shown in [4] and has gained popularity over the last decade. They often provide high classification accuracies compared to other non-parametric and statistical approaches. SVM classifiers are particularly useful to classify heterogeneous classes with a limited number of training samples. A detailed tutorial of SVMs can be found in [19]. SVMs are intrinsically designed as binary classifiers; however multi-class SVM classifiers can be constructed by using original SVMs as basic blocks. One—vs-all and hierarchical tree based approaches are popular techniques for constructing multi-class SVM classifiers. A more detailed explanation for constructing multi-class SVMs can be found in [20], [21]. In this paper a non-linear SVM with a Gaussian Radial Basis Function (RBF) kernel has been used. There are two parameters for an RBF kernel: and . We estimate these parameters using cross-validation and grid search. B. Multi-Classifier System The concept of combining the predictions of multiple classifiers to produce a single classifier has been proposed by various researchers in the past [22], [23]. In the literature, this concept is referred as ensemble classifiers or MCS. The resulting MCS is generally more accurate when compared to the individual classifiers that form MCS. An effective MCS is one where the individual classifiers in the MCS are accurate and make their classification errors on different parts of the input
793
space. Combining the predictions of identical classifiers will not have any improvement so it is useful only when there is a disagreement among the individual classifiers. In [24], Krogh et al. proved that overall classification error can be divided into a quantity which is the average generalization error of each classifier and a quantity proportional to the disagreement among the classifiers. From [25], [26] it can be concluded that an ideal MCS should consist of classifiers that have the highest disagreement possible. Bagging [15] and boosting [17] are very popular methods used to create diversity among classifiers. In Bryll et al. [16], an improved approach called attribute bagging was introduced. This followed the development of many wrapper based MCS approaches. Each classifier is trained with independently randomly selected feature subsets. The outputs are expected to be diverse and can be combined to form a final decision. Breiman [27] introduced a decision tree (DT) based classification approach with Random Forests (RF), Min [13] proposed a dynamic subspace approach and Waske proposed construction of SVM ensemble using RFS [12]. These are some of the approaches that are inspired from the basic idea of bagging and boosting classifiers successfully used on hyperspectral applications. In [36] Jacobs proposed an approach with mixture of experts followed by [37]. In [38] S. Kumar et al. demonstrated the effectiveness of this technique with hyperspectral data with binary classifiers for a multiclass problem. In their work, partitioning of groups of classes is achieved by binary classifiers at different levels. III. PROPOSED APPROACH In this paper, we propose a SVM-based MCS with unequal number of features being selected from different spectral regions resulting in a non-uniform random feature selection. A. Preliminaries The hyperspectral dataset is assumed to have classes, each represented as . is the number of samples in . Samples in are denoted as where means sample of the class . Samples from different classes can be grouped together to form a super class and is represented as . A feature vector is dimensional and each feature is represented by , i.e., . We define the normalized distance between any two set of samples with respect to its feature vector as and , where is the distance between two classes and and defined in (3) & (4) is the distance between two sets and . B. Proposed Non-Uniform RFS Strategy In RFS based multi-classifier system [12], a subset of features are selected by random sampling from a complete set of features whose indices tend to follow uniform distribution. Fig. 1(a) illustrates two examples of equally likely uniformly distributed spectral band feature selection where has highly correlated bands as compared to . An obvious way to avoid this situation is, as shown in Fig. 1(b) to divide the spectrum uniformly into smaller regions and performing feature selection within each region and concatenating the selected features. Outcome of this approach depends on the choice of number of partitions and partition boundaries. Features can still be correlated
794
IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. 6, NO. 2, APRIL 2013
Fig. 2. NU-RFS feature selection from original data.
Fig. 1. (a) Two examples of equally likely uniformly distributed spectral band feature selection where d1 has highly correlated band compared to d2. (b) Example of uniform partitioning of spectral band(shown in blue). (c) Non-uniform partitioning of spectral bands with uniformly distributed feature selection per partition.
with this approach. As one progresses their way along the spectrum, the bands in a hyperspectral signature are typically more highly correlated if they are adjacent and the rate of change in the correlation of neighboring bands varies. An intelligent way of partitioning the spectrum would be to place the partition at a point in the feature set where the correlation of neighboring bands changes drastically. This will result in a non-uniform partitioning of spectral bands and bands selected from these non-uniform regions are expected to be less correlated. This is shown in Fig. 1(c). To obtain an optimal set of partition boundaries, we use a band grouping strategy. In [28], the authors proposed an intelligent spectral partitioning technique that groups highly correlated bands into distinct contiguous subspaces and then used those partitions with a MCS. An intelligent (non-random) band-grouping was performed to partition the spectrum into
subsets. In this approach, the diversity among the classifiers is gained by breaking up the spectrum into smaller groups. The region boundaries are automatically selected based on a bottom-up band-grouping strategy. In this approach, starting with the first band, each successive band is added to the group—if this addition does not change the performance metric employed, then the growth of that group is stopped and a new group is started, resulting in a contiguous partitioning of the spectra. The metric employed for band-grouping in this work is the product of Bhattacharya distance and correlation. For more details about band grouping, the readers can refer to [28], [30]. In the proposed approach, the feature space is divided into distinct but contiguous regions in such a way that in each region, the class separation is maximized and statistical dependence is minimized by band grouping. Let be the size of each region, be the total number of features in the data and be number of features that are selected from each region which is directly proportional to . Then, total number of features selected for each classifier is (1) Fig. 2 illustrates this set up. Since uniform RFS is performed in each region separately, this approach can be thought of as a piece-wise uniform RFS. Since the HSI data has high correlation between consecutive bands, there is a good chance of consecutive bands getting grouped into different classifiers when using uniform RFS in a MCS. These highly correlated bands would clearly affect the diversity of the ensemble, and then result in reduced robustness of the MCS approach. However, in the proposed NU-RFS, features for individual classifiers in the resulting MCS are drawn in a non-uniform fashion thereby creating greater diversity among the classifiers compared to selecting features from a uniform random selection. Experimentally, we observed that this approach can result in better ensembles where features are less correlated, owing to the fact that the probability of features that are spectrally close to each other being sent to different learners is very low. The above said is applied to each classifier in the MCS separately unlike [12]. With initial experiments presented in [18], we found that
SAMIAPPAN et al.: NON-UNIFORM RANDOM FEATURE SELECTION AND KERNEL DENSITY SCORING
795
Fig. 3. Proposed NU-RFS based Multi-Classifier System.
the size of the region plays an important role in the performance of overall classification. As recommended in [29], the . The proposed MCS system is shown in Fig. 3. The random subspace selection demonstrated in [12], [13], [29] for MCS aims to create diversity among classifiers. The aforementioned techniques propose to construct ensembles by bagging and boosting variants. We believe the optimal subset to create maximum diversity need not come from a uniform RFS because there is a very good chance that the nearby spectral bands get grouped into the same classifier. This is clearly the case of classifiers having correlated features. This situation of similar grouping of features can sometimes affects the diversity by forcing the classifiers to commit similar errors. By using the proposed approach described in Fig. 3, we attempt to alleviate this issue. The proposed approach is compared against regular SVM, RFS and NU-RFS using band grouping only. C. Uniform Voting and Kernel Density Decision Fusion NU-RFS produces a group of features to be trained by each classifier. Each of these sets of features has a unique class separation capability since they form a different combination of the original feature set. In order to make use of this uniqueness in our system, we estimate a set of scores for each classifier that is proportional to the ability of the classifier to classify each class from all the other classes. For example, if there are classifiers and the data has classes then we generate a score matrix of size . These scores are computed by kernel density estimation across all the features. Oh et al. proposed an approach to estimate the class separation [31] to perform hand writing recognition. Our approach is inspired from the algorithm proposed in [31]. After performing NU-RFS, we have a set of training data for each classifier. A probability density for a class for a feature vector is estimated. A probability distribution for the class can be computed by (2)
where, is the kernel function and is the smoothing parameter. We have tested the rectangular, normal, triangular and epanechnikov kernel functions [34] in the proposed system. Let be the distance between any two classes and be the distance between any given class and all other classes can be computed by (3) and (4) respectively. (3) (4) where are class distributions of and respectively. When there is complete overlap between distributions, (3) gives a minimum and no overlap gives maximum. i.e., when there is complete overlap cannot distinguish two classes whereas it can distinguish the best when there is no overlap. Thus it defines the ability of to differentiate between any two classes of and . Equation (4) is computed for every class then values are averaged over all the selected features resulting in an array of scores of separability of each class with respect to the selected features. These scores are sorted in descending order where higher the score, higher the ability to classify a class. This is shown in Fig. 3 as compute class score. This process is repeated for every classifier in MCS resulting in a score matrix representing the ability of each classifier to distinguish a particular class from . We denote these scores as . Now, each row of this matrix corresponds to the ability of each classifier to distinguish classes where higher the value of , higher the chance of distinguishing that class from all other classes . Although, estimating class probability density function is a harder problem than classification, the aim of estimating these scores is to get a coarse estimate of separation which can be used during decision fusion. After estimating the score matrix, the actual classification is performed with all the SVM classifiers in MCS resulting in class labels for every test sample from each classifier. Let be
796
IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. 6, NO. 2, APRIL 2013
number of test samples with number of classifiers in MCS, then the resulting class labels can be represented as a matrix. Each column of this matrix holds the prediction of each test sample from different classifiers in MCS. In the hard decision fusion scenario, the final classification decision can be obtained by a majority vote over each classifier. The final decision of test sample can be obtained from by (5) (5) (6) Fig. 4. Sensitivity of various algorithms to number of training samples with AVIRIS Indian Pines data (error bars corresponds to 95% confidence intervals).
Mathematically, mode gives the most frequently occurring event. By means of majority vote, we get a hard decision fusion. This only uses the predictions of classifiers in MCS. The voting scheme described in (5) is uniform voting. i.e., each classifier in MCS has equal strength in deciding the final class label. We propose a voting mechanism based on scores where the strength of each classifier is modified according to its ability to classify a particular class from all other classes . This can be achieved by creating a modified class label column matrix for each test sample based on corresponding . This is done by appending with an array of length . The elements of the appended array will have the class label corresponding to highest , where length of can vary depending on as given in (6). From our experiments with various datasets, we arrived at (6). This is then used to perform majority vote. The decisions of MCS are not modified when . These scores will bias the majority voting decision based on the strengths and weaknesses of each classifier. IV. EXPERIMENTAL SETUP AND RESULTS A. Experimental Dataset and Setup We have used three hyperspectral datasets representing different analysis tasks—two datasets representing an agricultural problem, where classes are vegetation cover types and the third representing an urban classification problem. The first experimental HSI dataset employed was acquired using NASA’s AVIRIS sensor and was collected over northwest Indiana’s Indian Pine test site in June 1992 [32]. The image represents a vegetation-classification scenario with 145 145 pixels and 220 bands in the 400 to 2450 nm region of the visible and infrared spectrum. This dataset has 16 classes. The second experimental HSI dataset was acquired over north Mississippi’s Blackbelt Experiment Station agricultural test site in June 2008. The dataset has seven classes—each representing chemical stress on a corn-crop [35]. The corn crop, grown under controlled conditions was induced with varying degrees of chemical stress. The crop was sprayed with seven different concentrations of Glufosinate herbicide diluted with water, where the seven classes or concentrations were (control), 1/32, 1/16, 1/8, 1/4, 1/2, and 1 times the labeled rate of the herbicide concentrations. This dataset is acquired by using handheld Analytical Spectral Devices (ASD) sensor resulting in HSI datasets with 2151 bands. Since all seven classes in this
dataset represent the same species under varying degrees of stress, it makes for a very challenging classification problem. The third dataset used has 102 spectral bands acquired by the ROSIS sensor over Pavia, Northern Italy. This data has 9 classes. The classes are Water, Trees, Asphalt, Self-Blocking Bricks, Bitumen, Tiles, Shadows, Meadows and Bare soil. For this dataset, considering the very high number of samples from each class, the model selection is conducted on a subset of training samples rather samples from each class. These model parameters are used to train the SVM classifiers. The classification is performed using SVM with Gaussian RBF kernel for all experiments [35]. Model selection for the SVM is performed by using cross validation and grid search. For all the datasets the RBF parameters C and are estimated by selecting 10% of training samples from each class and performing a grid search by using cross validation except Pavia dataset where we used 5% of training data for model selection. We compute a confusion matrix for every classification problem and then estimate user, producer and overall accuracies. B. Results With the AVIRIS Indian Pines Dataset Experimental results demonstrate the superiority of the proposed approach compared to SVM and RFS. The study of overall accuracy for various numbers of training samples is shown in Fig. 4. At 10% training, NU-RFS with kernel density based fusion achieved an overall accuracy of 93.7% with rectangular kernel, and RFS and NU-RFS achieves 81.5% and 80.3% respectively. Interestingly, SVM with kernel scoring performs better than RFS and Band grouping based NU-RFS. The proposed Kernel Scoring based NU-RFS outperforms other approaches by 10%. The maximum overall accuracy achieved is 97.3% with 50% training. For all our experiments, we have used an ensemble size of . We observed that increasing the ensemble size does not provide any significant improvement beyond 8. This is similar to an observation made by Waske et al. [12] when using simple RFS. The study of the performance of various algorithms with respect to sample size follows a very interesting trend. The proposed approach clearly outperforms other techniques that are compared. In Fig. 4, rectangular kernel is used to compute the density. Fig. 5 shows the comparison of various kernel functions with respect to number of training samples used for training.
SAMIAPPAN et al.: NON-UNIFORM RANDOM FEATURE SELECTION AND KERNEL DENSITY SCORING
797
Fig. 5. Sensitivity of proposed approach with different kernel functions to number of training samples with AVIRIS Indian Pines data.
Fig. 7. Producer accuracies of different classes of Indian Pines data with various algorithms.
Fig. 6. User accuracies of different classes of Indian Pines data with various algorithms.
In this case, the rectangular and triangular kernels perform almost equally well compared to normal and epanechikov. The error bars shown are corresponding to 95% confidence intervals. From experimentation, it is found that the performance of the classifier increases with increasing initially and it decreases after reaching a particular value. In order to maintain uniformity among various experiments, the value of is used, datasets. From this yields the best performance for all the three of the classiexperimentation, it is found that the performance fier increases with increasing initially and it decreases after reaching a particular value. Figs. 6 and 7 illustrate the user accuracies (UA) and producer accuracies (PA) for each class in Indian Pines dataset. We observe a consistent improvement ( 2–40%) in both user and producer accuracies throughout all classes when employing the proposed kernel density based scoring approach. This is expected, as the confusion between the classes is reduced via the proposed scoring approach. The standard deviation is shown as error bars for user and producer accuracies. The deviation is approximately 0.8% for both user and producer accuracies. C. Results With the Corn Stress Dataset The study of sensitivity of various classifiers against different training sample sizes reveals an interesting pattern. As is observed with Indian Pines, the proposed approach handles the
Fig. 8. Sensitivity of various algorithms to number of training samples with corn stress data.
small sample size situation better than other approaches. Fig. 8 shows a comparison study of the overall accuracy versus the number of training samples. Systems based on NU-RFS exhibit a 1.5 to 3% increase in overall performance. It is worth pointing out that the performance of the kernel scoring NU-RFS algorithm is above 99% with a sample size of 10 samples/class, where single SVM and original RFS algorithms produce an accuracy of approximately 93% and 95% respectively. Fig. 9 shows the performance of different types of kernel used. Rectangular kernel performs better than other kernels with small training sample size. Figs. 10 and 11 illustrate the user and producer accuracies for each class in the Corn stress dataset. A similar increase in the user and producer accuracies is observed as with the Indian Pines data. The standard deviation is shown as error bars for user and producer accuracies. The deviation is approximately 0.1% for both user and producer accuracies. Tables I and II show the confusion matrices for classification without feature selection and with Kernel scoring NURFS (Triangular kernel function) with 10% training data, respectively. Both user accuracies (UA) and producer accuracies (PA) are improved with the proposed feature selection. Overall accuracies of other feature selection approaches and other kernels are shown in Figs. 8 and 9.
798
IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. 6, NO. 2, APRIL 2013
TABLE II SVM CLASSIFICATION WITH KERNEL SCORING NURFS
Fig. 9. Sensitivity of proposed approach with different kernel functions to number of training samples with corn stress data.
Fig. 12. Sensitivity of various algorithms to number of training samples with Pavia data.
D. Results With the Pavia, Italy Dataset Fig. 10. User accuracies of different classes of corn stress data with various algorithms.
Fig. 11. Producer accuracies of different classes of corn stress data with various algorithms.
TABLE I SVM CLASSIFICATION WITH NO FEATURE SELECTION
The experimental results with the Pavia, Italy dataset show an improvement in overall classification accuracy compared to other algorithms. Fig. 12 shows the performance of the proposed approach under various percentage of number of training samples. Kernel Density based NU-RFS achieves a gain of 7% and also it performs well under limited training samples. Both Kernel Density based approaches combined with SVM and NU-RFS shows superior performance over all the other approaches, and this shows the effectiveness of the proposed decision fusion approach. Fig. 13 illustrates performance with different kernel functions. Figs. 14 and 15 shows the user and producer accuracies for each class of Pavia dataset. Water, trees, bitumen, tiles and bare soil classes gained an improvement of 1 to 5%. This improvement can be seen from other kernel scoring techniques without feature selection. The standard deviation is shown as error bars for user and producer accuracies. The deviation is approximately 0.2% for both user and producer accuracies. V. DISCUSSIONS AND CONCLUSION An SVM based MCS with NU-RFS is developed in this work for hyperspectral classification. The overall accuracies are significantly higher compared to regular SVM based single classifiers and Uniform RFS based MCS. NU-RFS seems to handle the small sample size situation better than other MCS techniques in our comparison study. SVMs are known to better handle small sample size situations compared to statistical classifiers like ML. However, we have observed that single SVM classifier too suffers the curse of dimensionality. The number of
SAMIAPPAN et al.: NON-UNIFORM RANDOM FEATURE SELECTION AND KERNEL DENSITY SCORING
799
situations and hence it will be interesting to explore the possibilities of using this with semi-supervised learning for datasets with few ground truth points. We are testing these aspects in ongoing work. REFERENCES
Fig. 13. Sensitivity of proposed approach with different kernel functions to number of training samples with Pavia data.
Fig. 14. User accuracies of different classes of Pavia data with various algorithms.
Fig. 15. Producer accuracies of different classes of Pavia data with various algorithms.
features that we selected for each classifier is consistent with previous RFS implementations [12], [29]. We also conducted experiments with different values of and where we selected more features, the accuracy improved in some regions, though the impact was marginal. The user and producer accuracies with the proposed approach show a consistent improvement when compared to other conventional approaches. We believe that a study using better decision fusion strategies such as the Linear Opinion Pool (LOP) may yield further improvements, because, it will provide a soft fusion by using the distance between samples to the hyper plane of SVM. It is important to note that the proposed NU-RFS with the kernel density scoring particularly performs well with small sample size
[1] S. G. Beaven, D. Stein, and L. E. Hoff, “Comparison of Gaussian mixture and linear mixture models for classification of hyperspectral data,” in Proc. IEEE IGARSS, 2000, vol. 4, pp. 1597–1599. [2] B. Demir and S. Erturk, “Increasing hyperspectral image classification accuracy for data sets with limited training samples by sample interpolation,” in Proc. 4th Int. Conf. Recent Advances in Space Technologies, 2009, 2009, pp. 367–369. [3] Q. Jackson and D. A. Landgrebe, “An adaptive classifier design for high-dimensional data analysis with a limited training data set,” IEEE Trans. Geosci. Remote Sens., vol. 39, pp. 2664–2679, Dec. 2001. [4] F. Melgani and L. Bruzzone, “Classification of hyperspectral remote sensing images with support vector machines,” IEEE Trans. Geosci. Remote Sens., vol. 42, pp. 1778–1790, Aug. 2004. [5] J. A. Benediktsson, C. Garcia, B. Waske, J. Chanussot, J. R. Sveinsson, and M. Fauvel, “Ensemble methods for classification of hyperspectral data,” in Proc. IEEE IGARSS 2008, pp. I-62–I-65. [6] S. Prasad and L. M. Bruce, “A robust multi-classifier decision fusion framework for hyperspectral, multi-temporal classification,” in Proc. IEEE IGARSS 2008, pp. II-273–II-276. [7] S. Prasad and L. M. Bruce, A Divide-and-Conquer Paradigm for Hyperspectral Classification and Target Recognition Optical Remote Sensing. Berlin Heidelberg: Springer, 2011, vol. 3, pp. 99–122. [8] C. Mingmin, K. Qian, J. A. Benediktsson, and R. Feng, “Ensemble classification algorithm for hyperspectral remote sensing data,” IEEE Geosci. Remote Sens. Lett., vol. 6, pp. 762–766, Oct. 2009. [9] M. S. Haghighi, A. Vahedian, and H. S. Yazdi, “Creating and measuring diversity in multiple classifier systems using support vector data description,” Elsevier Applied Soft Computing, vol. 11, pp. 4941–4942, Dec. 2011. [10] R. Ranawana, “Intelligent multi-classifier design methods for the classification of imbalanced data sets—Application to DNA sequence analysis,” Ph.D. dissertation, Univ. of Oxford, Oxford, U.K., 2007. [11] G. Brown, J. Waytt, R. Harris, and X. Yao, “Diversity creation methods: A survey and categorisation,” J. Information Fusion, vol. 6, 2005. [12] B. Waske, S. van der Linden, J. A. Benediksson, A. Rabe, and P. Hostert, “Sensitivity of support vector machines to random feature selection in classification of hyperspectral data,” IEEE Trans. Geosci. Remote Sens., vol. 48, pp. 2880–2889, Jul. 2010. [13] Y. Jinn-Min, K. Bor-Chen, Y. Pao-Ta, and C. Chun-Hsiang, “A dynamic subspace method for hyperspectral image classification,” IEEE Trans. Geosci. Remote Sens., vol. 48, pp. 2840–2853, Jul. 2010. [14] J. Ham, C. Yangchi, M. M. Crawford, and J. Ghosh, “Investigation of the random forest framework for classification of hyperspectral data,” IEEE Trans. Geosci. Remote Sens., vol. 43, pp. 492–501, Mar. 2005. [15] L. Breiman, “Bagging predictors,” Machine Learning, vol. 24, pp. 123–140, Aug. 1996. [16] R. Bryll, R. G. Osuna, and F. Quek, “Attribute bagging: Improving accuracy of classifier ensembles by using random feature subsets,” Pattern Recogn., vol. 36, pp. 1291–1302, 2003. [17] Y. Freund and R. E. Schapire, “Experiments with a new boosting algorithm,” in Proc. 13th Int. Conf. Machine Learning, Bari, Italy, 1996. [18] S. Samiappan, S. Prasad, and L. M. Bruce, “Automated hyperspectral imagery analysis via support vector machines based multi-classifier system with non-uniform random feature selection,” in Proc. IEEE IGARSS, Vancouver, Canada, 2011. [19] C. J. C. Burges, “A tutorial on support vector machines for pattern recognition,” Data Mining and Knowledge Discovery, vol. 2, pp. 212–167, 1998. [20] D. J. Sebald and J. A. Bucklew, “Support vector machines and the multiple hypothesis test problem,” IEEE Trans. Signal Process., vol. 49, pp. 2865–2872, 2001. [21] C.-W. Hsu and C.-J. Lin, “A comparison of methods for multiclass support vector machines,” IEEE Trans. Neural Networks, vol. 13, pp. 415–425, 2002. [22] E. Alpaydin, “Multiple networks for function learning,” in Proc. IEEE Int. Conf. Neural Networks, 1993, vol. 1, pp. 9–14. [23] R. T. Clemen, “Combining forecasts: A review and annotated bibliography,” Int. J. Forecasting, vol. 5, pp. 559–583, 1989. [24] A. Krogh, “Neural network ensembles, cross validation, active learning,” in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 1995.
800
IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. 6, NO. 2, APRIL 2013
[25] D. W. Opitz, “Generating accurate and diverse members of a neuralnetwork ensemble,” in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 1996. [26] D. W. Opitz et al., “Actively searching for an effective neural-network ensemble,” Connection Science, vol. 8, pp. 337–353, 1996. [27] L. Breiman, “Random forests,” Machine Learning, vol. 45, pp. 5–32, 2001. [28] S. Prasad and L. M. Bruce, “Decision fusion with confidence-based weight assignment for hyperspectral target recognition,” IEEE Trans. Geosci. Remote Sens., vol. 46, pp. 1448–1456, 2008. [29] T. K. Ho, “The random subspace method for constructing decision forests,” IEEE Trans. Pattern Anal. Machine Intell., vol. 20, pp. 832–844, 1998. [30] C. Simin, R. Zhang, W. Cheng, and H. Yuan, “Band selection of hyperspectral images based on Bhattacharyya distance,” WSEAS Trans. Inf. Sci. and App., vol. 6, pp. 1165–1175, 2009. [31] I.-S. Oh, J. S. Lee, and Y. S. Ching, “Analysis of class separation and combination of class-dependent features for handwriting recognition,” IEEE Trans. Pattern Anal. Machine Intell., vol. 21, pp. 1089–1094, 1999. [32] Purdue University, link (on September 29, 2011) [Online]. Available: https://engineering.purdue.edu/~biehl/MultiSpec/hyperspectral.html [33] R. Duda, P. E. Hart, and D. G. Stork, Pattern Classification, 2nd ed. Chicester, U.K.: Wiley, 2006. [34] V. A. Epanechnikov, “Non-parametric estimation of a multivariate probability density,” Theory of Probability and Its Applications, vol. 14, pp. 153–158, 1967. [35] M. A. Lee, S. Prasad, L. M. Bruce, T. R. West, D. Reynolds, T. Irby, and H. Kalluri, “Sensitivity of hyperspectral classification algorithms to training sample size,” in Proc. IEEE WHISPERS 2009, Grenoble, France, 2009. [36] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, “Adaptive mixtures of local experts,” Neural Computing, vol. 3, pp. 79–87, 1991. [37] R. A. Jacobs and M. I. Jordan, “Adaptive mixtures of local experts and the EM algorithm,” Neural Computing, vol. 6, pp. 79–87, 1991. [38] S. Kumar, J. Ghosh, and M. M. Crawford, “Hierarchical fusion of multiple classifiers for hyperspectral data analysis,” Pattern Analysis Applications, pp. 210–220, 2002, Springer Verlag.
Sathishkumar Samiappan (M’12) received the B.E. degree in electronics and communication engineering from Bharathiar University, Coimbatore, India, in 2003 and the M.Tech degree in computer science, with a major in computer vision and image processing, from Amrita University, Coimbatore, India, in 2006. Since 2009, he has been working toward the Ph.D. degree in electrical and computer engineering at Mississippi State University, Starkville, MS. Until 2009, he was a Lecturer in the Department of Electronics and Communication Engineering, Amrita University, Coimbatore, India. Since 2009, he has been a Graduate Research Assistant with Geosystems Research Institute and Graduate Teaching Assistant with the Department of Electrical and Computer Engineering at Mississippi State University, Starkville, MS. His research interests include big data problems, pattern recognition, image processing, machine learning and hyperspectral image classification.
Saurabh Prasad (S’05–M’09) received the B.S. degree in electrical engineering from Jamia Millia Islamia, India, in 2003, the M.S. degree in electrical engineering from Old Dominion University, Norfolk, VA, in 2005, and the Ph.D. degree in electrical engineering from Mississippi State University, Starkville, MS, in 2008. He is an Assistant Professor in the Electrical and Computer Engineering Department at the University of Houston (UH), and is also affiliated with UH’s Geosensing Systems Engineering Research Center and the National Science Foundation (NSF) funded National Center for Airborne Laser Mapping (NCALM). He is the Principal Investigator/Technical-lead on projects funded by the National Geospatial Intelligence Agency (NGA), National Aeronautics and Space Administration (NASA), and Department of Homeland Security (DHS). His research interests include statistical pattern recognition, adaptive signal processing and kernel methods for medical imaging, optical and SAR remote sensing. In particular, his current research work involves the use of information fusion techniques for designing robust statistical pattern classification algorithms for hyperspectral remote sensing systems operating under low-signal-to-noise-ratio, mixed pixel and small training sample-size conditions. Dr. Prasad is an active Reviewer for the IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, the IEEE GEOSCIENCE AND REMOTE SENSING LETTERS and the Elsevier Pattern Recognition Letters. He was awarded the GRI’s Graduate Research Assistant of the Year award in May 2007, and the Office-of-Research Outstanding Graduate Student Research Award in April 2008 at Mississippi State University. In July 2008, he received the Best Student Paper Award at IGARSS’2008 held in Boston, MA. In October 2010, he received the State Pride Faculty Award at Mississippi State University for his academic and research contributions. He was the Lead Editor of the book entitled Optical Remote Sensing: Advances in Signal Processing and Exploitation Techniques (2011).
Lori M. Bruce (S’90–M’96–SM’01) received the B.S., M.S., and Ph.D. degree in electrical and computer engineering from the University of Alabama, Hunstville, and the Georgia Institute of Technology, Atlanta. She is the Associate Dean for Research and Graduate Studies in the Bagley College of Engineering at Mississippi State University. Dr. Bruce has been a Faculty Member for 14 years, during which she has taught approximately 40 engineering courses at the undergraduate and graduate level. Her research in image processing and remote sensing has been funded by the Department of Homeland Security, the Department of Energy, the Department of Transportation, the National Aeronautics and Space Administration, the National Geospatial-Intelligence Agency, the National Science Foundation, the United States Geological Survey, and industry, resulting in over 100 peer reviewed publications and the matriculation of more than 75 graduate students (25 as major professor and more than 50 as thesis/dissertation committee member). Dr. Bruce is an active member of the IEEE Geoscience and Remote Sensing Society, and she is a member of the Phi Kappa Phi, Eta Kappa Nu, and Tau Beta Pi honor societies, and prior to becoming a faculty member, she held the prestigious title of National Science Foundation Research Fellow.
