7th Annual International IEEE EMBS Conference on Neural Engineering Montpellier, France, 22 - 24 April, 2015
Implementation of Feature Extraction Methods and Support Vector Machine for Classification of Partial Body Weight Supports in Overground Robot-Aided Walking Joana Figueiredo, Cristina P. Santos, Eloy Urendes, José L. Pons, Senior Member IEEE, Juan C. Moreno, Member, IEEE
Abstract— A general need for Wearable Robots (WRs) is to perform optimal selection of information for control and monitoring during daily assistance with an autonomous systems. A good feature selection algorithm is key to perform automated estimation of the mode of locomotion under environmental variations. Ambulatory body weight support (BWS) systems can be combined with WRs to provide safe ambulation and support in overground walking in cases of lower limb paralysis. This study aimed to develop a support vector machine (SVM) model for binary and multiclass classification that performs gait pattern recognition for different values of partial BWS during overground robot-aided walking. The principal component analysis (PCA) and kernelbased PCA (kPCA) were applied to improve the classification performance. As a result, the combination of temporal and kinematic features showed to improve the accuracy in the discrimination of gait patterns in healthy patients (88%). In SVM multiclass classification the “one-against-one” approach showed to have a more stable performance (true positive and true negative rate are consistent) than “one-against-all” approach and also lower computational cost both for training and SVM’s decision making. I. INTRODUCTION A number of studies have demonstrated that patients with neurological injury (such as stroke and spinal cord injury) can improve their walking through gait training [1]. In this sense, body weight support (BWS) are commonly used during walking rehabilitation and can be combined with wearable robot (WR) to provide safe ambulation and support in overground locomotion [1], [2]. In order to facilitate automated recognition of gait patterns related to abnormal biomechanical function and to assess the performance of patients during gait training with robotic technologies, diverse machine learning approaches can be applied, in particular the support vector machine (SVM) [3], [4], [5]. This classifier is characterized by the following properties: globally optimal solutions are feasible, avoiding the over-fitting in the training process [6], [7]; has the ability to minimize both structural and empirical risk leading to better generalization to classify for new [3], [4].
*This work is partially supported by: the Commission of the European Union. FP7-ICT-2013.2.1-611695 (BioMot). Joana Figueiredo, Cristina P. Santos are now with the Industrial Electronics Department, University of Minho, Portugal (e-mail:
[email protected],
[email protected]). J.C. Moreno, Eloy Urendes, J.L.Pons are now with the Neural Rehabilitation Group, Cajal Institute, Spanish National Research Council (e-mail:
[email protected],
[email protected],
[email protected]).
978-1-4673-6389-1/15/$31.00 ©2015 IEEE
According to previous gait classification studies, the best performance of the SVM is achieved when using the radial basis function (RBF) kernels [3], [4]. Additionally, the choice of parameter C, which trades off misclassification of training examples against simplicity of the decision surface, becomes critical for good performance [8]. SVM has been proposed for multiclass classification with the so-called “one-against-one” (OAO) and “oneagainst-all” (OAA) approaches [4], [9], [10]. The OAA approach constructs one SVM model per class, distinguishing the samples of one class from all remaining classes, while the OAO approach constructs one SVM model for each pair of classes [11]. Feature extraction methods, such as linear principal component analysis (PCA) and the kernel-based PCA (kPCA), are needed before classification to select the significant information, from spatial-temporal, kinematic and kinetic parameters, that distinguishes the classes [6], [12]. Also, cross-validation methods can be applied to evaluate the generalization ability of the classifier [4], [12]. The goal of this study is to perform an evaluation of the influence of partial levels of BWS in overground robot-aided walking in healthy humans through the recognition of temporal and kinematics features with binary and multiclass classification, comparing OAA and OAO approaches, and testing PCA and kPCA feature extraction methods. A general need for WRs is to perform optimal selection of information for control and monitoring during daily assistance. In this study we start addressing this general goal through of selection of features that have strongest discrimination power to recognize different levels of support. II. METHODS This section will introduce a brief description of data collection and will describe all procedures adopted for construction of binary and multiclass SVM models, followed by performance evaluation measures of these models. All methods were implemented in MATLAB® (Mathworks). A. Acquisition of Gait Data In this study, a bilateral WR was used in actuated hip, knee and ankle joints. This WR is synchronized with an ambulatory partial BWS system (Figure 1). This device supports the user via harness and produces a predefined healthy-like gait pattern. The WR was adjusted to match each of exoskeleton joint to the center subject’s anatomical joint centres. We refer to [13] for a complete technical description.
763
Seven healthy subjects (mean age 25.2 ± 3.1, mean height 175.2cm ± 5.6cm) performed walking trials under three conditions of BWS: 30%, 50% and 70% with constant speed. Participants were asked to walk 3 meters over ground in a straight line without external human assistance.
Figure 1- Bilateral WR associated to an ambulatory partial BWS system.
B. Determination of Features As a disease or trauma can affect diverse gait parameters, it is important to consider multiple features for gait classification [14]. From the recorded gait data (sagittal plane) and automated recognition of gait events [15]) it was possible to extract 8 temporal features and 15 kinematic features (5 parameters for each joint based on the joint angles and the gait events). Thus, 23 features for each observation and for each class (30%, 50% and 70% of BWS) were computed, as shown in Table I (mean and standard deviation) for 8 temporal and kinematics features of the hip joint. TABLE I. MEAN ± STANDARD DEVIATION OF TEMPORAL AND HIP KINEMATIC FEATURES
Stride duration (s) Stride/min Step duration (s) Cadence (step/min) Single support (%) Double support (%) Stance (%) Swing (%) Peak flexion (º) Peak extension (º) Range of angle Angle in heel strike (º) Angle in toe off (º)
BWS 30% 4±0.01 15±0.02 1.96±0.03 30.6±0.4 20.94±1.6 30.03±1.4 80.09±1.3 19.92±1.4 36.2±1.0 5.3±0.9 30.8±1.5 34.3±0.8 18.4±1.4
BWS 50% 4±0.01 15±0.03 1.98±0.02 30.34±0.37 21.22±1.3 29.29±1.1 79.24±0.9 20.8±0.86 36.1±0.8 4.7±0.5 31.4±1.2 34.8±0.7 17.4±1.0
BWS 70% 4±0.01 15±0.03 1.98±0.03 30.29±0.45 21.65±1.1 29.05±0.9 79.08±0.9 20.95±0.9 36.1±0.9 4.4±0.7 31.7±1.3 34.9±0.7 16.9±1
C. Methods of Extraction of Features In order to identify the most significant features out of the 23 extracted features, both a PCA and kPCA were implemented. As a baseline for comparison, it was used the situation without any feature extraction method (below referred to as “No”). The goal of the PCA is to find an optimal linear transformation that represents the data in a least square sense [12]. Thus, it yields a set of orthogonal bases in a new coordinate system and determines the covariance matrix through the single vectors decomposition technique. Then, the directions of maximum variance in the training data that correspond to the principal components are obtained [6],
[12]. Subsequently, the dimensional reduction of d (in this study d is an parameter that corresponds to the number of preserved features) is performed, keeping the first d principal components that retain the most variance of the data. The kPCA first maps the gait data of a nonlinear space into a higher-dimensional feature space through a kernel function. Then, the linear PCA method is applied, as above described, in order to extract the most significant gait features, i.e., the first d kernel principal components [6]. In this study a 2-degree polynomial kernel was selected, since this function kernel achieves best performance than linear or RBF kernels [6], [16], [17]. According to Liang and Lee [17], the data projections for even-degree polynomial kernels tend to make the clusters linearly separable. Also, they concluded that with degree 2, the kernel provides a best separation of clusters and this turns the kernel more flexible [17]. According to [3], [4], the combination of different types of features allows to achieve a best classification, when compared to the situation in which only a type of feature is considered. Consequently, to ensure the presence of temporal and kinematics features in the classification, PCA and kPCA were employed in the temporal and kinematics features separately, and then the extracted features were combined to form a data set for training and test of the SVM models. D. SVM Classifier For the implementation of binary and multiclass classification, the SVM classifier was used. The aim of this classifier is to find an optimal hyperplane, from various possible hyperplanes, that separates all classes of the dataset [18]. Nevertheless, for nonlinear data, in our case with a gait process, the data space can only be separated by a kernel function [3], which, as previously discussed, in this study was the RBF kernel. The performance of the SVM is affected by parameters of the RBF kernel (C and sigma). Therefore, as recommended by Hsu et al. [19], a grid-search was employed combined with a four-fold cross-validation (CV) scheme to find the best values of these parameters. In summary, the implemented algorithm computes separately the binary (30% and 50% of BWS) and multiclass (30%, 50% and 70% of BWS) classification. For both classifications, any of the feature extraction methods (“No”, PCA or kPCA) can be selected. After extracting the temporal and kinematics features a grid-search is executed to find C and sigma parameters, which are applied to build the SVM model. Before, a four-fold CV is implemented for selection of the training data. In the multiclass classification, it is also possible to select between both OAA and OAO approaches. Finally, the SVM classifier is validated and its performance evaluated with the test data. E. Evaluation of the Performance of the SVM Classifier The SVM models are evaluated according to three dimensions, namely: accuracy, sensitivity and specificity, since they present distinct information about the classifier performance [2], [4]. In addition and in order to assess the classifier independently of the distribution a priori of the classes [20],
764
the area under the curve (AUC) was also computed. The AUC has been recommended, since it presents higher convergence than accuracy and represents the average sensitivity across all possible specificities [5], [8].
It can be noticed that the values of the evaluation parameters (Acc, Sens, Spec and AUC) showed in Table III correspond to the average of all values obtained in each partition of multiclass classification.
III. RESULTS
IV. DISCUSSION
The results presented in this section correspond to the metrics for evaluating the SVM performance: accuracy (Acc), sensitivity (Sens), specificity (Spec), AUC and to the values of the parameters C and sigma (also represented as σ), used in the RBF kernel, both for the binary and multiclass classifications.
A. Binary Classification The statistic results in Table I indicates that there is a similarity in mean values of temporal and kinematic features for the three-classes (although only 5 kinematic features are shown, the others 10 features of knee and ankle joints also are similar for three classes), and consequently the partial levels of BWS in overground robot-aided walking in healthy humans provides similar biomechanical output. Thereby, a more exhausting classification process is required to find a boundary decision that separates the three classes. According to Table II it is possible to verify that the kinematics features contribute more to the success of the walking classification than the temporal features. The performance of the SVM is higher considering exclusively kinematics features (accuracy of 81%, specificity of 88%, AUC of 81%) than considering exclusively temporal features (accuracy 75%, specificity of 75%, AUC of 75%). Indeed, this result is consistent with [3], [6]. Also, it can be noticed by combining the two types of features the performance was increased (accuracy of 88% against accuracy of 75% and 81% for temporal and kinematics features, respectively), as concluded by [3], [4], [6], [8]. Thus, combining data types in this case enriches information about the process. Besides, as mentioned by Begg and Kamruzzaman [3], the number of features is an important aspect for the success of the classifier, as we verify from ours results the best performance is obtained when 9 kinematics features and 4 temporal features are selected. Analysing Table II also indicates that the feature extraction methods have a positive influence in the performance of the SVM. The values of accuracy, sensitivity, specificity and AUC are higher with PCA and kPCA than when “No” feature extraction method. Additionally, note that the principal components extracted by PCA probably have more variance than the kernel principal components selected by kPCA due to the best classifier performance with PCA (accuracy 88%, sensitivity of 81%, specificity of 88% and AUC of 88%). However, this conclusion does not agree with [6], [21] that suggested the kPCA for gait analysis.
A. Binary Classification Table II shows the results obtained through binary classification that distinguish the patterns of the gait training between BWS of 30% and BWS of 50%. As already mentioned, the influence of the features extraction methods in the performance of classifier was studied. Moreover, we examined which kind of features are more relevant, either 8 temporal features (below called as T) or 15 kinematic features (below assigned as K), through the SVM test. Afterwards, different combinations of both features were attempted, over several tests through manipulation of the d parameter. Results show that the best performance of the classifier is achieved when 9 kinematics features and 4 temporal features are selected. Lastly, several tests were realized with different values of k-fold and it was verified that four-fold is the more appropriate k-fold in CV scheme for studied situation. TABLE II. RESULTS FOR THE BINARY CLASSIFICATION
“No” PCA kPCA
T
K
C
σ
0 8 4 4
15 0 9 9
0.8 0.3 0.4 0.6 0.3
0.4 3 0.6 0.5 7
Acc (%) 72 81 75 88 84
Sens (%) 75 75 75 81 79
Spec (%) 69 88 75 88 1
AUC (%) 72 81 75 88 74
B. Multiclass Classification We investigated which method for features extraction is more appropriate for multiclass classification. The results are presented in Table III. A similar procedure to the adopted for binary classification shows that is more appropriate to combine 9 kinematics features and 4 temporal features, coupled with the use of a four-fold in CV process. Besides, we evaluated the SVM performance with the approaches OAA and OAO, analysing the number of support vectors (# SV) and the processing time (recorded in seconds with a processor of 1.7 GHz), to compare these approaches in terms of the complexity of the training process and the SVM’s decision making.
OAO
OAA
TABLE III. RESULTS FOR MULTICLASS CLASSIFICATION
“No” PCA kPCA “No” PCA kPCA
C
σ
2.5 2.8 2 8 2.2 0.5
7.7 7.2 0.6 2.7 7.2 4.5
Acc (%) 73 86 85 73 85 86
Sens (%) 89 91 1 87 89 82
Spec (%) 58 53 50 73 73 73
AUC (%) 70 72 73 71 75 76
# SV 43 50 62 38 38 38
Time (s) 127.8 123.5 200.9 84.1 70.1 62.1
B. Multiclass Classification Comparing all strategies of feature extraction showed again that the dimensional reduction of features turns the SVM more accurate, whereby the classification can be effectively improved by identifying the relevant variables. Also, it is observed that the principal components extracted by PCA and kPCA provide a similar performance to the classifier. Thus, it can be concluded that both methods have similar efficiency for pattern recognition of three-classes. Nevertheless, this result disagrees with the considerations of Wu et al. that argue that the nonlinear PCA provides more useful information about gait data [6]. Relatively to OAA and OAO approaches, according to Table III, it was verified that in all situations the approach OAO is more appropriate in terms of accuracy, sensitivity
765
and specificity. Indeed, model training with OAA approach leads to a specificity value very low and is more correct using a model with similar rate of true positive and true negative (sensitivity and specificity consistent), than a model with high rate of true positive and a low true negative rate. Besides, the AUC, a criterion more convergent than accuracy [5], recommends the utilization of the OAO approach. The former result is in agreement with conclusions of a detailed comparison elaborated by [10], [19]. Also, these approaches were compared in terms of the computational complexity of the training process and SVM’s decision making. Table III shows that training with OAA approach created more support vectors and consequently presented the highest processing time, since the training time of a SVM model increases linearly with the number of training samples (the support vectors number) [11]. Thus, the SVM model created through OAA approach has more complexity in making a decision and presents an exhaustive training process, as showed in [11], and thus these considerations reinforce the preference for the OAO approach. V. CONCLUSION This study presented two accurate algorithms for pattern recognition of the levels of support over ground during robot-aided walking. The algorithms, applying binary and multiclass classification with SVM, are valid to identify different classes of partial BWS. In particular, the OAO approach for SVM multiclass classification showed improved performance in terms of accuracy and stability, and also lower computational cost both for training and SVM’s decision making. For both binary and multiclass classifications it was verified that appropriate combination of gait variables is determinant for the success of the pattern recognition. In this study the best performance was achieved when 4 temporal features and 9 kinematic features were used. In conclusion, this study showed that robot-aided walking with ambulatory partial BWS system noticeably alters multiple spatiotemporal and biomechanical gait related features, which is substantial to treat this function in patients with gait disorders due to neurological injury. The implementation of feature extraction methods, as PCA and kPCA, is fundamental to reveal relevant features for pattern recognition of types of support over ground. Further work will consist on the application of this methodology to other control requirements, such as the recognition of levels of actual guidance force and patient contribution to the movement. Also, this study needs to be extended to consider additional available signals (e.g. bioelectric signals), to eventually improve the outcome of the classifier.
[3]
R. Begg, R and J. Kamruzzaman, “A machine learning approach for automated recognition of movement patterns using basic, kinetic and kinematic gait data,” Journal of biomechanics, vol. 38, pp. 401-8, 2005.
[4]
H. Lau, K. Tong and H. Zhu, “Support vector machine for classification of walking conditions of persons after stroke with dropped foot,” Human movement science, vol. 28, pp. 504-14, 2009.
[5]
A.M.S. Muniz, H. Liu, K. E. Lyons, R. Pahwa, W. Liu, F.F. Nobre, and J. Nadal, “Comparison among probabilistic neural network, support vector machine and logistic regression for evaluating the effect of subthalamic stimulation in Parkinson disease on ground reaction force during gait,” Journal of biomechanics, vol. 43, pp. 720-6, 2010.
[6]
J. Wu, J. Wang and L. Liu, “Feature extraction via KPCA for classification of gait patterns,” Human movement science, vol. 26, pp. 393-411, 2007.
[7]
Y. Saeys, I. Inza and P. Larrañaga, “A review of feature selection techniques in bioinformatics,” Bioinformatics, vol. 23, pp. 2507-17, 2007.
[8]
D. Lai, P. Levinger, R. Begg, W. Gilleard and M. Palaniswami, “Automatic recognition of gait patterns exhibiting patellofemoral pain syndrome using a support vector machine approach,” IEEE Transactions on Information Technol- ogy in Biomedicine, vol. 13, pp. 801-817, 2009.
[9]
D. Janssen, W. Schöllhorn, K. M. Newell, J. Jäger, F. Rost and K. Vehof, “Diagnosing fatigue in gait patterns by support vector machines and self-organizing maps,” Human movement science, vol. 30, n.º 5, pp. 966-975, 2011.
[10]
E. L. Allwein, R. E. Schapire and Y. Singer, “Reducing Multiclass to Binary: A Unifying Approach for Margin Classifiers,” Journal of Machine Learning Research, vol. 1, pp. 113-141, 2000.
[11]
C. Hsu and C. Lin, “A comparison of methods for multiclass support vector machines,” IEEE transactions on neural networks, vol. 13, pp. 415-425, 2002.
[12]
M. Yang, H. Zheng, H. Wang and S. McClean, “Feature selection and construction for the discrimination of neurodegenerative diseases based on gait analysis,” em Proceedings of 3rd international ICST conference on pervasive computing technologies for healthcare, 2009.
[13]
E. Urendes, G. Asin, R. Gómez , R. Ceres and J. Pons, “An integrated lower limb exoskeleton and body weight support system: design and implementation,” WebRob, 2014.
[14]
J. Rueterbories, E. G. Spaich, B. Larsen and O. K. Andersen, “Methods for gait event detection and analysis in ambulatory systems,” Medical engineering & physics, vol. 32, pp. 545-52, 2010.
[15]
N. Chia, S. Lambrecth , E. Urendes, J. C. Moreno and J. L. Pons, “Real-Time detection of gait and stance events from WR embedded sensors.,” em WeRob 2014, Baiona, 2014.
[16]
E. Kokiopoulou and Y. Saad, “PCA and kernel PCA using polynomial filtering : a case study on face recognition,” University of Minnesota, 2004.
[17]
Z. Liang and Y. Lee, “Eigen-Analysis of Nonlinear PCA with Polynomial Kernels,” pp. 1-22, 2010.
[18]
V. Vapnik, The Nature of Statistical Learning Theory, New York: Springer, 1995.
[19]
C. Hsu, C. Chang and C. Lin, “A Practical Guide to Support Vector Classification,” National Taiwan University, vol. 1, pp. 1-16, 2010.
[20]
V. López, A. Fernández and F. Herrera, “On the importance of the validation technique for classification with imbalanced datasets: Addressing covariate shift when data is skewed,” Information Sciences, vol. 257, pp. 1-13, 2014.
[21]
A. Kertész-Farkas, S. Dhir, P. Sonego, M. Pacurar, et al., “Benchmarking protein classification algorithms via supervised cross-validation,” Journal of biochemical and biophysical methods, pp. 1215-1223, 2008.
REFERENCES [1]
[2]
T.G. Hornby,C. R. Kinnaird, C. L. Holleran, et al., “Kinematic, Muscular, and Metabolic Responses During Exoskeletal-, Elliptical-, or Therapist-Assisted Stepping in People With Incomplete Spinal Cord Injury,” Physical Therapy Volume, vol. 92, n.º 10, pp. 12781290, October 2012. L. H. Sloot, M. M. van der Krogt and J. Harlaar, “Self-paced versus fixed speed treadmill walking,” Gait & posture, vol. 39, pp. 478484, 2014.
766
