Robust Classification of Objects using a Sparse ... - Semantic Scholar

Robust Classification of Objects using a Sparse Detector Sensor M. Yeasin,1 D.J. Russomanno,1 S.M. Sorower,1 M. Smith,1 and J.S. Shaik2 1 Department of Electrical and Computer Engineering University of Memphis, Memphis TN 38152 {myeasin, drussmnn}@memphis.edu 2 Department of Pathology and Immunology Washington University in St. Louis, St. Louis, MO 63108 Abstract: This paper emphasizes techniques for broadscale classification of objects sensed by a prototype unattended ground sparse detector sensor. A wide range of machine learning techniques were applied to create models of three broad classes of objects, such as humans, humans wearing large backpacks, and nonhumans using data obtained from the sparse detector sensor in a laboratory environment. Fusion of models was performed based on a measure of diversity among classifiers to improve the robustness and also the accuracy of the models. Empirical analysis on 230 sample datasets shows up to a 91.74% accuracy (10fold cross validation) in classifying three broad classes of objects of interest and shows very promising scores on other various performance indices.

1. Introduction By definition, a sparse detector sensor is an imaging device that has a relatively sparse detector array as compared to typical imaging sensors. Sparse detector sensors may be a low-cost alternative to traditional, high resolution imaging sensors, which use dense focal plane arrays, for object classification. Size, cost, and power restrictions preclude the use of traditional imagers in applications that require widespread deployment of the sensors and in scenarios in which the sensors must be regarded as disposable. Classification of sparse detector sensor data is of extreme interest when building inexpensive, ground sensors for many government agencies. But robust classification is challenging due to the paucity of information that can be used to model various objects that may be sensed. The design of an unattended ground sparse detector sensor for broad-scale classification has been prototyped in our laboratory [1]. This prototype sensor is being designed and evaluated in response to the need to monitor trails that provide routes for drug smuggler traffic, as well as other applications in which broadscale classification from data acquired from unattended ground sensors is of high interest. Unattended ground sensors that can reliably distinguish between humans and animals are critically needed. For example, it is typical for drug smugglers to use 70 to 100 lb backpacks to transport marijuana on foot along known trails across national borders.

In our application domain, such trails are abundant, but most of the routes are known by authorities. However, the routes have many travelers that are not of interest, including animals and humans that do not fit the profile of traveling in groups with large backpacks. In typical deployments, a sensor would be placed near a trail having a width of approximately 3 feet. These sensors can be located next to trails where vegetation can be used to hide placement. The trails are located in areas that are considered open range. Wild horses, cattle, deer, large cats, dogs, rabbits, and pigs are just a few of the animals that use the same trails as humans moving through the area. To address the classification of data obtained from a sparse detector sensor this paper summarizes some preliminary empirical analysis using a variety of individual classifiers, ensemble classifiers, and the fusion of classifiers to develop robust models of objects of interest. The fusion of classifiers was performed based on measures of diversity as opposed to random choices. In-depth performance evaluation of the models was performed to show the utility of each classifier in modeling such sparse data and to potentially identify a classifier that might be a candidate for implementation in a deployable version of the prototype sensor. The design of the sparse detector sensor and Web services software to discover, query, and task the sensor in a network-centric environment have been presented in other works [1, 2]. However, those papers did not provide the detail on a wide range of machine learning techniques for classifying data from the sensor as presented in this paper. The remainder of the paper is organized as follows: Section 2 describes the assembly of the sensor and data acquisition, respectively, as previously described in Russomanno et al. [1]. That material is described again here to provide context for the subsequent sections of this paper. Section 3 summarizes the theoretical foundations of the proposed machine learning methods for classifying objects. Section 4 presents the results of empirical analysis and Section 5 concludes the paper and provides future directions.

2. Sparse Detector Sensor The CX-RVMS retro reflective infrared sensor [3] as shown in Figure 1, was used as the principal sensing

element to construct a prototype sparse detector sensor to obtain signature data in our laboratory for further analysis. The CX-RVMS was selected due to its suitability for both laboratory and controlled-field testing. This sensing element has a 1 ms or less response time, 5 m sensing range, consumes less than 40 mA, and can operate from -25 to 60 °C.

Figure 1: CX-RVMS reflective photoelectric sensing element (all units mm) [1] As previously described in [1], the prototype sparse detector sensor was assembled by placing 16 CXRVMS retro reflective sensing elements at 5 inch intervals in a vertical configuration. Each sensing element can be regarded as an optical trip wire and was fixed to a supporting pole with reflectors mounted on an opposing pole to provide the break-beam curtain. Each CX-RVMS sensing element was interfaced to a USBDIO-32 digital input board using a simple voltage divider breadboard circuit to provide the required 0Vto-5V output. The input board was subsequently connected to a host computer via a USB interface. Figure 2 (left panel) illustrates the configuration in which sensing element B0 denotes the first optical trip wire placed at 5 inches off the ground through sensing element C7, which is placed at approximately 80 inches off the ground. Figure 2 (right panel) is a photograph of the pole with the sensing elements interfaced to a host

computer in the lab. The driver software for the digital input board interfaced to the sensor uses a 32-bit DLL compatible with any Windows programming language. A C/C++ program was developed to acquire signature data assuming a single object traverses through the sensor’s optical curtain for initial testing purposes. Therefore, data acquisition started when a break-beam event first occurred and continued at a sampling rate of approximately 10ms until none of the sensing elements detected a break in their beam. The C/C++ program writes a single ASCII file for each object detection event as a string of 1s and 0s corresponding to no-break and break, respectively, for sensing element B0 through C7 as the object passes through the sensor’s optical curtain. Each of the 16 sensing elements is sampled in parallel, which provides data for a ‘16 x I’ matrix. Variable ‘I’ is the number of samples taken for each sensing element and will be constant within a single file, that is, the same number of samples will be taken for sensing elements B0 through C7 for a given object’s data acquisition. However, ‘I’ will vary among files as it depends on the specific object and the interval of time in which at least one sensing element detected a break-beam event, that is, ‘I’ depends on the speed, size, configuration, and other variables of the object. Figure 3 is a visualization of the optical trip wires for a typical break-beam pattern for a human with a small backpack. For the testing reported in this paper, a total of 230 datasets were acquired from the sensor. Figure 4 represents the creation of a simple image from the break-beam patterns for a variety of objects including: a) a human without a backpack, b) a human with a school backpack, c) a human with a large backpack, d) a small dog, e) a human walking a small dog; and f) an office chair. Recall these images were produced from 16 optical trip wires at 5 inch intervals. C7 C6 C5 C4

BREADBOARD I/F CIRCUIT

C3 C2 C1 C0 16 B7 B6 B5 B4 B3 B2 B1 B0

Figure 2: Prototype sparse detector sensor [1]

3.1. Data Representation

Figure 3: Optical trip wire break-beam pattern [1] In the case of multiple objects traversing through the sensor as in Figure 4 e), our data acquisition program required a continuous break of at least one optical trip wire to acquire the data for both objects in one file. Multiple objects as in Figure 4 e) were not used as datasets for the classification algorithms reported later in this paper. A number of samples were acquired for each object type consisting of different strides, postures, orientations, speed, etc.

The data representation module has data processing and feature selection modules. Data processing: The sensed data is divided into categories (e.g., person with large backpack, person without backpack, small animals, etc.) based on the types of objects of interest. The number of samples obtained from the sensor in each category varies from one class to the other (e.g., person with backpack may have more samples than person with out backpack). Feature extraction: As previously described, each object sample is a ‘16 x I’ matrix. Principle component analysis (PCA) was used to project the data onto a lower dimensional space to form feature vectors of equal length and to remove redundancies from the data. Each data sample was projected from ‘16 x I’ to ‘16 x 10’ and was used as a feature vector for modeling various objects. At least 85% of the energy of the signal was preserved in the transformation. Each projected sample was further arranged in lexicographical order to convert the ’16 x 10’ matrix into ‘1 x 160’ vectors. The 160 dimensional feature vectors were used to model various objects of interest (e.g., human with backpack, etc.). 3.2. Modeling Objects

a)

b)

c)

d)

e)

f)

Figure 4: Images created from break-beam patterns acquired from the sparse detector sensor: a) human without backpack, b) human with small backpack, c) human with large backpack, d) small dog, e) human walking a small dog, and f) office chair [1]

3. Classification of Objects The classification of sparse detector sensor data summarized in this paper consists of two main modules: i) data representation, and ii) modeling of objects. In previous work [1], the design of experimental setup and the feasibility of classification of sparse sensor data was discussed. This paper performs in depth performance evaluation and fusion of classifiers based on a measure of diversity.

A diverse set of classifiers from the WEKA toolbox [4] were used for modeling the objects such as human, human with large backpack, and small animals. In particular, individual classifiers (i.e., sequential minimal optimization, function-based classifier, etc.), ensemble classifiers (i.e., bagging, boosting, random forest, etc.), and fusion of multiple classifiers have been used for robust modeling of objects and their performance was compared. The following subsections summarize the concepts and the utility of ensemble classifiers and fusion of classifiers for completeness before presenting their application on the sparse detector sensor data. 3.2.1. Ensemble classifiers Ensemble classifiers are more accurate than a single classifier as they can combine the outputs of multiple “beast weak” classifiers. Intuitively, it is easy to find correct “rules of thumb” for classification. However, it is hard to find a single, highly accurate prediction rule. If the training examples are few (which is the case in the context of this initial research) and the hypothesis space is large, then, there are several equally accurate classifiers [5]. The hypothesis space does not contain the true function, but it has several good approximations. Exhaustive global search in the hypothesis space is expensive so we can combine the predictions of several locally accurate classifiers. The key idea behind ensemble classification is as follows: given training sample S, generate multiple hypotheses, h1, h2, …, hL, determine corresponding weights w1, w2,

…, wL and classify new points according to ∑l wl hl > θ. A whole range of ensemble classification is possible by combining models obtained from different learning algorithms or the same learning algorithm trained in different ways and can be fused using various schemes, such as, majority vote, weighted (confidence of classifier) vote, weighted (confidence in classifier) vote, learned combiner and fusion based on diversity among various hypotheses. Effective ensembles must have accurate and diverse components. In this paper, Bagging, Boosting, LogitBoost, and Random Forests were used. Bagging and Boosting: Bootstrap aggregating (bagging) is a meta-algorithm to improve classification and regression models in terms of stability and classification accuracy [6]. In general, bagging reduces variance and helps to avoid overfitting. Since this method averages several predictors, it is not useful for improving linear models. Bagging will rarely increase error and potentially can reduce the variance part of the error. Note that bagging works better as a source of diversity only when the classifier is unstable [5]. Boosting is similar in overall structure to bagging, except that one keeps track of the performance of the learning algorithm and forces it to concentrate its efforts on instances that have not been correctly learned [7]. Instead of choosing the training instances randomly using a uniform distribution, one chooses the training instances in such a manner as to favor the instances that have not been accurately learned. After several cycles, the prediction is performed by taking a weighted vote of the predictions of each classifier, with the weights being proportional to each classifier’s accuracy on its training set. Schapire’s “Strength of Weak Learnability” theorem [8] proves that hB has improved performance over h1. Boosting algorithms are considered stronger than bagging on noise free data. Boosting can reduce variance and bias part of the error and is susceptible to noise. Boosting increases margins; however, there are strong empirical indications that bagging is much more robust than boosting in noisy settings [5]. For this reason, Kotsiantis et al. [9] built an ensemble using a voting methodology of bagging and boosting ensembles that give better classification accuracy. The LogitBoost algorithm [10] uses Newton steps for fitting an additive symmetric logistic model by maximum likelihood. Random Forests: The Random Forest is a classifier that consists of many decision trees and outputs the class that is the mode of the classes output by individual trees [11]. Diversity is obtained by randomly choosing attributes at each node of the tree and then using the attribute that provides the highest level of learning. The importance of this cannot be overstated as the performance of the Random Forests algorithm is linked to the level of correlation between any two trees in the

forest. The more the correlation increases, the lower the overall performance of the entire forest of trees. For many datasets, Random Forests provide the following characteristics: i) a highly accurate classification, ii) accommodate a very large number of input variables, iii) estimate the importance of variables in determining classification, iv) generate an internal unbiased estimate of the generalization error as the forest building progresses, v) include a good method for estimating missing data and maintain accuracy when a large proportion of the data are missing, vi) provide an experimental way to detect variable interactions, vii) balance error in class population unbalanced datasets, viii) compute proximities between cases that are useful for clustering and detecting outliers, ix) can be extended to unlabeled data, and x) provide fast learning. While the ensembles yield better results compared to individual classifiers, the fusion of classifiers can further improve the overall performance. 3.2.2. Classifier fusion There are various ways to combine the output prediction of the base classifiers. One of the popular and simplest methods uses a voting scheme. In voting, each classifier is provided equally weighted votes towards a particular classification and the class label with majority votes is assigned to the test input. Also, weighted voting in which classifiers are assigned weights according to their generalized performance toward a particular classification task is very popular. It is believed that weighted voting is more effective than majority voting. Generalized stacking [12] is also used to combine classifiers which use more sophisticated algorithms to learn how to combine the outputs of a set of classifiers. Various versions of stacking (e.g., stacking with linear regression, radial basis function, etc.) and versions of voting (e.g., majority voting, voting with average probability, etc.) from WEKA [4] were used for the fusion of classifiers in our work. The classifiers were chosen based on their diversity measure as a candidate for fusion. 3.2.2.1 Measure of diversity among classifiers Different classifiers are designed based on different prior assumptions and following different mathematical procedures to model an underlying function. It is quite possible for different classifiers to produce different results (i.e., make mistakes at different data points) on the same dataset even though the accuracy for different classifiers is the same. It is expected that fusion of classifiers can produce better models compared to the individual classifiers if the fused classifiers are different (i.e., independent, orthogonal, and complementary). The theoretical framework leads to diverse multiple classifier system architectures.

For a set of classifiers, a good choice of diversity measures is based on entropy (E) and is defined as in [13]: E=

1 N

N

∑ j =1

L 1 ⎧L ⎫ min ⎨∑ y j ,i , L − ∑ y j ,i ⎬ L ⎢ ⎥ i =1 i =1 ⎩ ⎭ L − ⎢ ⎥ −1 ⎣2⎦

in which L= total number of classifiers, N = number of data instances, yj,i = 1 if ith classifier correctly classifies jth data point and 0 otherwise. The higher the value of E indicates increased diversity, with 1 being the highest possible diversity and 0 indicating no diversity. There are a few other diversity measures for a set of classifiers which are based on variance measure such as Kohavi-Wolpart (KW) variance [13]. KW measure is defined as follows: 1 N KW = ∑ l ( z j )( L − l ( z j )) NL2 j =1 in which l ( z j ) =

∑

L

i =1

y j ,i and the other symbols have

the meaning as defined earlier. This expression calculates the averaged variance of the Bernoulli variable y with values 0 for incorrect and 1 for correct classification of each object.

4. Empirical Analysis and Evaluation All the classifiers applied to the sparse detector sensor data were evaluated using N-fold cross validation and using various performance metrics such as receiver operating characteristic (ROC) curves [14, 15]. Classifiers produce a confusion/contingency matrix, which show four entities: True positive (TP), True negative (TN), False positive (FP), and False negative (FN). Based on these quantities, a number of performance indices such as precision (P = TP/(TP +FP)), recall (R = TP/(TP +FN)), True positive rate (TPR = TP/(TP+ FN)), and False positive rate (FPR = and FP/(FP +TN)) can be defined. In addition, Fmeasure is defined based on the harmonic mean of precision and recall F-measure: F = 2*Recall*Precision/(Recall + Precision). The plot of precision vs. recall (or TPR vs. FPR) is generally known as the ROC curve. The area under the ROC curve is also used a measure to compare various models. The ROC curve graphically represents the trade-off between false positive and true positive rates of different classifiers. Classifiers are typically plotted by varying a threshold (or cutoff) on the output probabilities of the scheme. 4.1. Classification Results and Performance Metrics Two hundred thirty data samples were collected using the setup described in Section 2. The labels and number of samples for each class are as follows for the initial analysis: Class 1: Human with large backpack (60), Class 2: Human with small/without backpack (135), and Class 3: Miscellaneous objects (35 samples).

Table 1 summarizes the accuracy (using 10-fold cross validation) and RMS error of various classifiers applied to the sparse sensor data. Among the individual classifiers, the REP Tree (RET) recorded the highest accuracy (89.13%). The ensemble classifiers improved the overall accuracy. Both the bagging- and boostingbased methods recorded similar accuracy (91.3%). The fusion of classifiers scheme recorded (91.74%) accuracy, marginally improving the performance. Table 1: Performance of different classifiers using 3 classes (human with large backpack, human with small/without backpack, and miscellaneous objects) Classifiers BN SMO Individual REPT Classifiers J48 MP MCC Ensemble LB Classifiers BG RF StkCRBFLBBGRF StkCLRFusion of LBBGRF Classifiers StkRCBGLBRF AP-BGLBRF MV-BGLBRF

Accuracy 87.83% 83.04% 89.13% 86.96% 79.57% 70.87% 91.30% 91.30% 88.70% 91.30%

RMSE 0.2783 0.3386 0.2606 0.2923 0.3255 0.4008 0.2139 0.2193 0.2651 0.2227

91.74%

0.1998

91.74%

0.2146

91.74% 91.74%

0.2131 0.2347

The following abbreviations were used to summarize the results in tabular form: BN–Bayes Network; SMO– Sequential Minimal Optimization; REPT REP Tree; J48– J48 Decision Tree (C4.5); MP–Multilayer Perceptron; MCC–Multilayer Classifier; LB– LogitBoost; BG–Bagging; RF–Random Forest; StkCRBF-LBBGR–StackingC with RBF Network using LB, B, and RF; StkCLR-LBBGRF–StackingC with Linear Regression Network using LB, BG, and RF; StkRC-BGLBRF–Stacking with Random Committee using BG, LB, and RF; AP-RFBGLB–Vote with Average Probability using RF, BG, and LB; MV-BGLBRF–Vote with Majority Voting using B, LB, and RF; RMSE–Root mean squared error. The following observations can be made from Table 1. It is easy to note that ensemble classifiers, in general, performed better than the individual classifiers and the fusion of classifiers marginally improved the accuracy over ensemble classifiers. The fusion of classifiers recorded the lowest RMS error indicating that model was the least sensitive to outliers. The choice of which classifiers to fuse was made based on their diversity scores. Table 2 summarizes

these scores for various combinations of classifiers. From Table 2 it is easy to see that the second group: Bagging (BG), LogitBoost (LB), Random Forest (RF), has the highest diversity measure values (high entropy and low KW). This justifies the choices of classifiers for the fusion reported in Table 1. Expectedly, the fusion of those classifiers yields the highest accuracy and better performance measures (see Table 4). To further illustrate the utility of the diversity measure in fusion, a simple pair-wise measure of diversity was also used. It was found that group 6 has the lowest KW value (pairwise entropy is undefined). The classification accuracy for this set of classifiers is marginally inferior compared to group 2, but has a lower computational cost than group 2. This provides a trade-off between accuracy of the model and computational time which may be an issue while deployment of the system. Table 2: Diversity measures among classifiers (based on 10-fold cross validation for the 3 class problem) N Classifiers Accuracy Entropy 1 REPT BN J48 2 LB BG RF 3 BG LB REPT 4 BN BG LB 5 BG LB 6 BG RF 7 LB RF 8 LB REPT

89.13% 87.83% 86.96% 91.30% 91.30% 88.70% 91.30% 91.30% 89.13% 87.83% 91.30% 91.30% 91.30% 91.30% 91.30% 88.70% 91.30% 88.70% 91.30% 89.13%

Diversity KW

0.347826

0.077295

0.217391

0.048309

0.173913

0.038647

0.217391

0.048309

-

0.021739 0.043478

-

0.043478

-

0.043478

-

Table 3: Performance analysis of ensemble classifiers Classifier LB

BG

RF

CLASS CLASS1 CLASS2 CLASS3 CLASS1 CLASS2 CLASS3 CLASS1 CLASS2 CLASS3

TPR 0.85 0.956 0.857 0.833 0.963 0.857 0.8 0.933 0.857

FPR 0.035 0.147 0 0.029 0.158 0 0.053 0.179 0

F 0.872 0.928 0.923 0.87 0.929 0.923 0.821 0.906 0.923

ROCA 0.973 0.956 0.966 0.96 0.94 0.941 0.956 0.939 0.949

While the accuracy and RMS error provides some indication of the performance of the models, to have an

in-depth performance evaluation it is imperative to understand the utility and efficacy of the models. Tables 3, 4 and 5 summarize the in-depth performance analysis of different ensemble classifiers and fused classifiers, respectively. Individual classifiers are not reported here as they have relatively poor performance. Table 4: Performance evaluation of fused classifiers Classifiers StkCRBFLBBGRF StkCLRLBBGRF StkRCBGLBRF APBGLBRF MVBGLBRF

CLASS CLASS1 CLASS2 CLASS3 CLASS1 CLASS2 CLASS3 CLASS1 CLASS2 CLASS3 CLASS1 CLASS2 CLASS3 CLASS1 CLASS2 CLASS3

TPR 0.867 0.948 0.857 0.867 0.956 0.857 0.9 0.941 0.857 0.883 0.948 0.857 0.867 0.956 0.857

FPR 0.041 0.137 0 0.035 0.137 0 0.041 0.116 0.005 0.041 0.126 0 0.035 0.137 0

F ROCA 0.874 0.935 0.928 0.926 0.923 0.899 0.881 0.983 0.931 0.974 0.923 0.982 0.893 0.977 0.93 0.956 0.909 0.964 0.883 0.984 0.931 0.97 0.923 0.972 0.881 0.916 0.931 0.909 0.923 0.929

Table 5: Confusion Matrix for best ensemble classifier: (Bagging) and Fusion Classifier (StkCLR-LBBGRF) Classified As a = CLASS1 b = CLASS2 c = CLASS3

LogitBoost (LB) a b c

StkCLR-LBBGRF a b C

51 6 0

52 6 0

9 129 5

0 0 30

8 129 5

0 0 30

The following observations are made from Tables 3 and 4. It is easy to note that ensemble classifiers were quite effective in modeling the three different classes under consideration. Among the ensemble classifiers bagging, boosting, and random forest methods have very good TPR, FPR, F measures and high area under the ROC indicating robustness of the model under varying operating conditions. It should also be noted that the fusion of classifiers based on diversity has not significantly improved the accuracy but it has improved other performance metrics across the board. To further analyze the results, the confusion matrix for the ensemble and fused classifiers is presented in Table 5.

5. Conclusions The main focus of this paper was to provide a thorough performance evaluation of a wide range of machine learning techniques in the context of a novel application using a sparse detector sensor to monitor drug smuggling trails. Empirical analysis on models of various objects of interest (humans with large backpacks, humans with no or small backpacks, and

miscellaneous objects, such as animals and other test objects) were conducted using data obtained from the sensor. The individual models obtained using various learning techniques were also fused based on diversity to improve the accuracy and robustness of the models. From the empirical analysis, it was observed that the algorithms considered were capable of discriminating from break-beam signature data obtained from the sparse detector sensor. Analysis on a preliminary 230 sample dataset show up to a 91.74% accuracy (10-fold cross validation) in detecting humans wearing large backpacks and also show very promising scores on other performance metrics. Empirical analysis show that the ensemble based classifiers outperformed the individual classifiers. It was also noted that the fusion of classifiers based on diversity has improved the robustness and performance metrics. Several issues regarding the further development and deployment of the sparse detector sensor with respect to object classification are ongoing research. The issues include accommodating multiple objects simultaneously passing through the optical curtain, objects walking fast, walking sideways, and other plausible orientations. In addition, work is underway to determine the optimal number of sensing elements/detectors and their configuration for object classification while reducing the overall cost of the sensor.

[3] [4]

[5]

[6] [7]

[8]

[9]

[10]

6. Acknowledgements Funding for this work was provided in part by agreement W911NF-05-2-0019 between the U. of Memphis and the U.S. Army’s Research Lab (ARL), as well as funds from the Herff College of Engineering at the U. of Memphis. This paper does not necessarily represent the position of the U.S. Government.

[11] [12] [13]

References [1] D. J. Russomanno, M. Yeasin, E. Jacobs, M. Smith, and M. S. Sorower, "Sparse detector sensor: profiling experiments for broad-scale classification," in Proceedings SPIE-Defense and Security Symposium: Unattended Ground Sea, and Air Sensor Technologies and Applications X, vol. vol. 6963, Orlando, Florida, 2008, 11 pages. [2] D. J. Russomanno, Y. Tritenko, and Q. Qiu, "A Web service interface for an unattended ground

[14] [15]

sparse sensor detector," in Proceedings of the International Conference on Semantic Web and Web Services. Las Vegas, Nevada, 2008, 6 pages. Anon, "CX-RVM5/D100/ND300R Data Sheet," Ramco Innovations, W. Des Moines, IA, 2007. I. H. Witten and E. Frank, Data Mining: Practical Machine Learning Tools and Techniques, 2nd ed. San Francisco: Morgan Kaufmann, 2005. T. G. Dietterich, "An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, boosting, and randomization," Machine Learning, vol. 40, pp. 139-158, 2000. L. Breiman, "Bagging predictors," Machine Learning, vol. 24, pp. 123-140, 1996. Y. Freund and R. E. Schapire, "Experiments with a New Boosting Algorithm," Proceedings of Intl. Conf. on Machine Learning, pp. 148-156, 1996. R. E. Schapire, "The strength of weak learnability," Machine Learning, vol. 5, pp. 197 227, 1990. S. Kotsiantis and P. Pintelas, "Combining Bagging and Boosting," International Journal of Computational Intelligence, vol. 1, pp. 324-333, 2004. J. Friedman, T. Hastie, and R. Tibshirani, "Additive Logistic Regression: a Statistical View of Boosting," The Annals of Statistics, vol. 28, pp. 337-374, 2000. L. Breiman, "Random Forests," Machine Learning, vol. 45, pp. 5-32, 2001. D. H. Wolpert, Stacked generalization, vol. 5: Pergamon Press, 1992. L.I. Kuncheva and C. J. Whitaker, "Ten measures of diversity in classifier ensembles: limits for two classifiers," in Proc. of IEE Workshop on Intelligent Sensor Processing, vol. 1. Birmingham, UK, 2001, pp. 10-16. T. M. Mitchell, Machine Learning. Singapore: McGraw-Hill Book, 1997. N. M. Adams and D. J. Hand, "Improving the Practice of Classifier Performance Assessment," Neural Computation, vol. 12, pp. 305-312, 2000.