Speeding Up Multi-class SVM Evaluation via Principle ... - CiteSeerX

1

Speeding Up Multi-class SVM Evaluation via Principle Component Analysis and Recursive Feature Elimination Hansheng Lei, Venu Govindaraju, Fellow, IEEE, Matthew J. Beal

Abstract Support Vector Machines (SVM) have been shown to yield state-of-the-art performance in many pattern analysis applications. Feature selection methods for SVMs are often used to reduce the complexity of learning and evaluation. In this article we propose to combine a standard method, Recursive Feature Elimination (RFE), with Principal Component Analysis (PCA) to produce a multi-class SVM framework for evaluation speedup. In addition, we propose using the Leave-One-Out error to guide the feature elimination and determine the minimum size of feature subset. RFE together with PCA is able to compress the size of SVM classifier model and speed up the SVM evaluation significantly. Experimental results on the MNIST benchmark database and other commonly used datasets show that RFE and PCA can speed up the evaluation of SVM by an order of magnitude while maintaining comparable accuracy. Index Terms Support Vector Machine, Principle Component Analysis, Recursive Feature Elimination, Multi-class Classification

2

I. I NTRODUCTION The Support Vector Machine (SVM) was originally designed for binary classification problems [1]. The SVM Classifier maximizes the margin between classes by selecting a minimum number of Support Vectors (SVs), which could be determined by solving a Quadratic Programming(QP) optimization problem. The training of the SVM, dominated by the QP optimization, used to be very slow and lacked scalability. Researchers have spent considerable effort in speedup of the QP problem and enhancement of the SVM scalability [23], [19], [20]. The bottleneck lies in the kernel matrix. Suppose we have N data points for training, then the size of the kernel matrix is N × N . This had been a challenge for SVM until the Sequential Minimum Optimization (SMO) was proposed [20]. The space complexity of SVM training was reduced to O(1). Thus, the training problem is solved. Recently, with SMO, SVM has been used in many application areas including machine learning, pattern recognition and data mining [27]. However, the fundamental SVM still requires improvement its evaluation aspect [2], [7]. In this paper, we propose a method for SVM evaluation enhancement via Principle Component Analysis (PCA) and Recursive Feature Elimination (RFE). PCA is an orthogonal transformation of the coordinate system that preserves the Euclidean distance of the original points (each point is considered to be a vector of features or components). In the PCA transform, the energy of points are concentrated in the first few components. This leads to dimension reduction. Feature selection has been extensively studied, especially for the applications such as gene selection from microarray data, where a common situation is the large number of genes (thousands) but a small number of samples (hundreds). RFE in the context of SVM has led to promising results for feature selection [9]. In our work, we have used the SVM at the feature selection stage to improve feature selection to improve the SVM evaluation itself. This paper is organized as follows. In section 2, we provide a brief background of SVM and related work in speedup of SVM evaluation. In section 3, we propose Feature Reduced SVM (FR-SVM) for the multi-class problem by incorporating PCA and RFE. We present a method to automatically determine the minimum size of the feature subset using the Leave-One-Out (LOO) error bound as the guide in section

3

4. Experimental results on benchmark datasets are reported in section 5 and our conclusions are presented in section 6.

II. SVM

BACKGROUND

In this section, we briefly describe the basic concepts of binary SVM and how it can be extended for multi-class evaluation. We also discuss methods to speed up the SVM evaluation.

A. Binary Support Vector Machines The basic SVM classifier can be expressed as: g(x) = w · φ(x) + b,

(1)

where input vector x ∈