THE EFFECT OF THE USED RESAMPLING TECHNIQUE AND

IADIS International Conference Applied Computing 2006

THE EFFECT OF THE USED RESAMPLING TECHNIQUE AND NUMBER OF SAMPLES IN CONSOLIDATED TREES’ CONSTRUCTION ALGORITHM Ibai Gurrutxaga, Olatz Arbelaitz, Jesús M. Pérez, José I. Martín, Javier Muguerza Dept. of Computer Architecture and Technology, University of the Basque Country M. Lardizabal, 1, 20018 Donostia, Spain

ABSTRACT In many pattern recognition problems, the explanation of the made classification becomes as important as the good performance of the classifier related to its discriminating capacity. For this kind of problems we can use Consolidated Trees´ Construction (CTC) algorithm which uses several subsamples to build a single tree. This paper presents a wide analysis of the behavior of CTC algorithm for 20 databases. The effect of two parameters of the algorithm: number of samples and the way subsamples have been built has been analyzed. The results obtained with Consolidated Trees have been compared to C4.5 trees executing 5 times a 10 fold cross validation. The comparison has been done from two points of view: error rate (accuracy) and complexity (explanation). Results show that, for subsamples of 75% of the training sample, Consolidated Trees achieve, in average, smaller error rates than C4.5 trees when they are built with 10 or more subsamples and with similar complexity, so, they are better situated in the learning curve. On the other hand, the method used to build subsamples clearly affects to the quality of results achieved with Consolidated Trees. If bootstrap samples are used to build trees the obtained results are worse than the ones obtained with subsamples of 75% from the two points of view: error and complexity. KEYWORDS

Classification trees, accuracy, comprehensibility, resampling, multiple classifier system.

1. INTRODUCTION In many pattern recognition problems from the real world, the explanation of the made classification becomes as important as the good performance of the classifier related to its discriminating capacity. Diagnosis in medicine, fraud detection in different fields, customer fidelization, resource assignation, etc, are examples of this kind of applications. The explanation of why a classifier has labeled an example with a particular class, is important, but explanation or comprehensibility is useful in a much wider sense. For example, users more easily trust in comprehensible classifiers, experts can validate and confirm them, algorithms can be more easily refined and classifiers can show us an overall representation of the problem they are intended to solve (Craven 1996). Decision trees (Quinlan 1993) are among the set of classification techniques that are considered comprehensible but they have a problem; they are too sensitive to the sample used in the induction process. This feature of decision trees is called unsteadiness or instability (Dietterich 2000a, Chawla 2002). To face up this problem, several algorithms based on decision trees, such as bagging (Breiman 1996, Bauer and Kohavi 1999), boosting (Freund and Schapire 1996), Random Subespace Method (Ho 1998) and different variants of them (Chawla et al. 2002), with greater discriminating capacity and steadiness have been developed. In all the algorithms mentioned before the final classifier is a combination of different trees with different structures so, they lose the explaining capacity. The result of our research work in this area is the development of an algorithm for building trees, denominated Consolidated Trees’ Construction (CTC) algorithm. Our aim has been to design a new algorithm for building a single tree that improving the performance of standard classification trees, reduces

83

ISBN: 972-8924-09-7 © 2006 IADIS

the complexity and maintains the explaining capacity. The result of this methodology is a single agreed tree, built using different subsamples of the training set. This paper presents a wide analysis of the behavior CTC algorithm has, and it has been compared to the well-known C4.5 algorithm (Quinlan 1993). The results are centered, as usual, in the error rate, but complexity is also analyzed in order to assure that Consolidated Trees are not too complex to be comprehensible. Our goal is to propose a comprehensible classifier as an alternative to multiple classifiers that achieve improvements in accuracy at the expense of explanation. The paper proceeds with the description of the methodology for building Consolidated Trees (CT), Section 2. In Section 3, the description of the data sets and the experimental set-up is presented. The results of our experimental study are discussed in section 4. Finally, Section 5 is devoted to summarize the conclusions and future research.

2. CTC LEARNING ALGORITHM Consolidated Trees’ Construction algorithm (CTC) uses several subsamples to build a single tree. This technique is radically different from bagging, boosting, etc. The consensus is achieved at each step of the tree’s building process and only one tree is built. The different subsamples are used to make proposals about the feature that should be used to split in the current node. The split function used in this work is the gain ratio criterion (the same used by Quinlan in C4.5). The decision about which feature will be used to make the split in a node of the Consolidated Tree (CT) is accorded among the different proposals. The decision is made by a voting process sample by sample. The criterion used to divide a node in this experimentation is simple majority among all the samples. Based on this decision, all the subsamples are divided using the same feature. The iterative process is described in Algorithm 1. The algorithm starts extracting a set of subsamples (Number_Samples) from the original training set. The subsamples are obtained based on the desired resampling technique (Resampling_Mode). Decision trees’ construction algorithms, usually divide the initial sample in several data partitions. In our algorithm, LSi contains the data partitions created from each subsample Si. Algorithm 1. Consolidated Trees’ Construction Algorithm (CTC) Generate Number_Samples subsamples (Si) from S with Resampling_Mode method. CurrentNode := RootNode for i := 1 to Number_Samples LSi := {Si} end for repeat for i := 1 to Number_Samples CurrentSi := First(LSi) LSi := LSi - CurrentSi Induce the best split (X,B)i for CurrentSi end for Obtain the consolidated pair (Xc,Bc), based on (X,B)i, 1 ≤ i ≤ Number_Samples if (Xc,Bc) ≠ Not_Split Split CurrentNode based on (Xc,Bc) for i := 1 to Number_Samples Divide CurrentSi based on (Xc,Bc) to obtain n subsamples {S1i, … Sni} LSi := {S1i, … Sni} + LSi end for else consolidate CurrentNode as a leaf end if CurrentNode := NextNode until ∀i, LSi is empty

84

IADIS International Conference Applied Computing 2006

The pair (X,B)i is the split proposal for the first partition in LSi. X is the feature selected to split and B indicates the proposed branches or criteria to divide the data in the current node. Xc is the feature obtained by a voting process among all the proposed X. Whereas Bc is the median of the proposed Cut values when Xc is continuous and all the possible values of the feature when Xc is discrete. When a node is consolidated as a leaf node, the a posteriori probabilities associated to it are calculated averaging the a posteriori obtained from the data partitions related to that node in all the subsamples. The used resampling technique and the number of subsamples used in the tree’s building process are important aspects of the algorithm (Skurichina et al. 2002). There are many possible combinations for the Resampling_Mode: size of the subsamples ⎯100%, 75%, 50%, etc; of the original training set ⎯, examples can be drawn with replacement or without replacement, stratified or not, etc. We have experimented with many possible options and with the analysis of the preliminary results, some important clues in the kind of Resampling_Mode that should be used came out: the subsamples must be large enough (at least 75%) so that they do not lose too much information of the original training set. The fact that subsamples are stratified, so that the a priory probability of the different categories of the class variable is maintained, is also important. In previous works (Pérez et al. 2004a, 2004b) we have seen that the behaviour of CTC algorithm is satisfactory in situations that require subsampling. Once the consolidated tree has been built, it works the same way a decision tree does.

3. EXPERIMENTAL METHODOLOGY Twenty databases of real applications have been used for the experimentation. Most of them belong to the well-known UCI Repository benchmark (Newman et al. 1998), widely used in the scientific community. Table 1 shows the characteristics of the databases used in the comparison. Table 1. Characteristics of the 20 databases used in experimentation. Breast-W Heart-C Hypo Lymph Credit-G Segment210 Iris Glass Voting Hepatitis Soybean-Large Sick-Euthyroid Liver Credit-A Vehicle Breast-Y Heart-H Segment2310 Spam Faithful

Examples

Variables

Classes

699 303 3163 148 1000 210 150 214 435 155 290 3163 345 690 846 286 294 2310 4601 24507

10 13 25 18 20 19 4 9 16 19 35 25 6 14 18 9 13 19 57 49

2 2 2 4 2 7 3 7 2 2 15 2 2 2 4 2 2 7 2 2

The Faithful database is a real data application from our environment, centered in the electrical appliance's sector, and does not belong to UCI. In this case, we try to analyze the profile of customers during time, so that a classification related to their fidelity to the brand can be done. This will allow the company to follow different strategies to increment the number of customers that are faithful to the brand, so that the sales increase. In this kind of applications, the use of a system that provides information about the factors taking part in the classification (the explanation) is very important; it is nearly more important to analyze and explain why a customer is or is not faithful, than the own categorization of the customer. This is the information that will help the corresponding department to make good decisions. The CTC methodology has been compared to the C4.5 tree building algorithm Release 8 of Quinlan (1993), using the default parameter settings. Both kinds of trees have been pruned using the pruning

85

ISBN: 972-8924-09-7 © 2006 IADIS

algorithm of the C4.5 R8 software, in order to obtain two systems with similar conditions. We can not forget that developing too much a classification tree leads to a greater probability of overtraining. The analysis of the behavior the algorithm has been done for several values of some parameters: - 12 values have been explored for Number_Samples (N_S): 3, 5, 10, 20, 30, 40, 50, 75, 100, 125, 150, 200. For 18 of the databases the experimentation has been done for whole range of N_S parameter. There are two databases, spam and faithful, where the maximum value explored has been limited to 50 to minimize computational cost. - As we mentioned in section 2, many options can be selected for Resampling_Mode parameter. To obain the results presented in this paper, two options have been used: 75% without replacement and stratified (T=75%) and 100% with replacement or bootstrap sampling (Boot). The methodology used for the experimentation is a 10-fold stratified cross validation (Hastie et al. 2001). Cross validation is repeated five times using a different random reordering of the examples in the data set. In each fold of the cross validation, 200 bootstrap subsamples and 200 subsamples of 75% are built. Figure 1 shows a schema of the generation of subsamples. For each one of the 50 fold’s generated in each database, a C4.5 tree will be built based on the traning sample associated to the fold. On the other hand, 12 consolidated trees (1 for each value of N_ S parameter) are built using subsamples of T=75% and other 12 using bootstrap samples. As a consequence, approximately 1,000 C4.5 trees and 24,000 CT trees have been built for the experimentation presented in this paper. Samples Test 1/10

Subsamples

Training 9/10

10-fold CV

5 times

•• •

•• •

•• •

T=75

Boot

•• •

200

•• •

200

•• •

200

•• •

200

•• •

200

•• • •• •

200

Figure 1. Schema of subsample generation to build C4.5 trees and CT trees with two options for Resampling-Mode parameter (75% and bootstrap).

The used methodology has given us the option to compare 5 times groups of 10 CTC and C4.5 trees (50 executions for each instance of the analyzed parameters). In each run we have calculated the average error and the complexity of the trees, estimated as the number of internal nodes of the tree.

4. RESULTS We have done a complete analysis of the behavior of CTC algorithm when changing the parameters mentioned in Section 3. We first present results obtained with T=75% samples, i.e. the size of the subsamples used to build Consolidated Trees has been 75% of the original training set. Table 2 shows the average error obtained by C4.5 and CTC for each database. The names of the different databases can be found in the first column. In the next column the error obtained by C4.5 is shown and the next 12 columns show the error obtained by CTC for different values of N_S (from 3 to 200). Note that, due to computational cost, the last two databases have results for just 7 values of N_S (from 3 to 50). That is why two average values are shown in the last two rows. In the row labeled as Avergae20 the average error of all the databases is shown, while Average18 shows the average error of the first 18 databases. In each row, if the

86

IADIS International Conference Applied Computing 2006

error for C4.5 is lower than any of the twelve error values obtained with CTC the error of C4.5 is bolded; for CTC algorithm, all the error values lower than the error for C4.5 are bolded otherwise. Table 2 shows that CTC is able to achieve, with at least one value of N_S parameter, lower error rates than C4.5 in 14 out of 20 databases. Moreover in 10 of these 14 databases the error rate of CTC is lower for most of N_S values. Examining Figure 2, where values of Average20 and Average18 are plotted, we obtain more information about the overall behavior of CTC algorithm: we can state that in general the error achieved by Consolidated Trees decreases as the N_S parameter increases. If we look to results in Table 2 we can observe that the error rate of Consolidated Trees built with 150 or 200 subsamples (the maximum analysed) is lower than the error rate of C4.5 in 12 out of 18 databases. Anyway, it seems that the error reduction caused by incrementing the number of used subsamples is not significant for values above 50 or 75. Significant differences could not be found by making a statistical significance paired t-test (Dietterich 1998, 2000b) to compare the overall behavior of the algorithm when using 50 subsamples and 200 subsamples. Since computational cost is higher as the number of used subsamples grows, it seems sensible to recommend a N_S value near 50 although any value above 10 achieves in average higher accuracy than C4.5 in our experimentation. We can also observe that Average20 and Average18 do not show qualitative differences, so we can expect that the reduced experimentation made for two databases do not affect overall results. Table 2. Comparison of results for 20 databases. C4.5 and CTC (75%) are shown with values for N_S from 3 to 200.

Breast-W Heart-C Hypo Lymph Credit-G Segment210 Iris Glass Voting Hepatitis Soybean-Large Sick-Euthyroid Liver Credit-A Vehicle Breast-Y Heart-H Segment2310 Spam Faithful Average20 Average18

C4.5 5.63 23.96 0.71 20.44 28.50 13.61 5.75 31.55 3.41 20.29 11.02 1.96 35.31 14.51 27.61 25.81 21.02 3.24 7.25 1.48 15.15 16.35

03 5.77 25.41 0.73 21.34 28.02 13.99 5.61 29.11 3.32 19.49 12.40 2.24 35.16 14.80 27.93 28.40 21.50 3.52 7.46 1.50 15.39 16.60

05 5.66 24.49 0.73 21.13 28.18 13.52 5.75 29.21 3.41 21.55 11.84 2.25 34.69 15.03 28.21 28.33 21.08 3.31 7.59 1.50 15.37 16.58

10 5.63 23.77 0.74 19.67 28.18 13.51 5.48 30.15 3.50 20.99 11.84 2.25 33.28 14.72 27.70 27.20 21.44 3.54 7.56 1.51 15.13 16.31

20 5.52 22.85 0.72 19.91 28.20 12.76 4.68 29.94 3.36 21.11 11.29 2.34 35.68 15.04 27.53 27.14 21.22 3.56 7.55 1.47 15.09 16.27

CTC (T=75%) 30 40 50 75 5.49 5.60 5.63 5.60 22.92 24.02 23.75 23.55 0.72 0.72 0.73 0.73 19.65 19.93 20.06 20.18 28.40 28.12 28.02 27.60 13.13 12.86 11.52 12.28 4.54 4.41 4.28 4.14 30.22 30.13 29.61 29.43 3.41 3.41 3.41 3.36 20.97 21.08 21.25 20.31 11.71 11.30 10.88 10.67 2.30 2.32 2.33 2.34 33.55 34.01 33.59 34.64 15.01 14.98 14.98 15.16 28.01 28.60 28.56 28.53 26.99 26.93 26.72 26.43 21.49 21.29 20.83 21.03 3.54 3.56 3.59 3.33 7.36 7.55 7.43 1.49 1.50 1.49 15.05 15.12 14.93 16.22 16.29 16.10 16.08

100 5.58 23.42 0.74 20.18 27.68 12.28 4.28 29.85 3.36 20.58 10.67 2.42 35.26 14.75 28.13 26.50 21.31 3.44

125 5.60 23.36 0.74 20.19 27.56 12.09 4.14 30.16 3.36 21.25 10.68 2.39 35.80 15.06 27.69 26.44 21.02 3.50

150 5.60 23.16 0.73 20.31 27.72 11.71 4.01 30.06 3.36 20.95 10.88 2.35 34.82 14.95 27.60 26.44 20.89 3.49

200 5.58 23.48 0.73 20.19 27.80 12.09 4.28 30.00 3.36 20.70 10.74 2.38 34.35 15.06 27.42 26.43 20.96 3.57

16.14 16.17 16.06 16.06

Regarding to complexity we must remind that a too complex tree can cause loss of comprehensibility. As argued in the introduction many applications need classifiers with explaining capacities. In order to assure that Consolidated Trees keep this capacity complexity has been measured. The made analysis shows that CTC algorithm builds slightly more complex trees than C4.5 does. We have observed that Consolidated Trees’ complexity increases together with the number of subsamples used to build the tree, but this average complexity is stabilized in a similar way error rate does. In the presented experimentation a C4.5 tree has, in average, 17 internal nodes, while CTC trees’ average complexity is stabilized in about 20 internal nodes. So,

87

ISBN: 972-8924-09-7 © 2006 IADIS

we can assure that as long as C4.5 trees are simple enough to be understood by a human, so are Consolidated Trees. 16,70

15,50 CT T=75%

15,40

C4.5

C4.5

16,50 16,40

15,20

Error

Error

15,30

CT T=75%

16,60

15,10

16,30 16,20

15,00

16,10

14,90

16,00

14,80 3

5

10 20 30 Num ber Sam ples

40

15,90

50

3

5

10

20

30

40

50

75

100

125

150

200

Num ber Sam ples

Figure 2. Average error for C4.5 and CTC (T=75%) algorithms. 20 databases N_S=3 .. 50, left side and 18 databases N_S=3 .. 200, right side.

These facts lead us to say that CTC algorithm is a better choice than C4.5 since it induces more accurate and not much more complex trees. Taking into account that this error reduction is achieved without losing explaining capacity, we argue that CTC can be an alternative option to boosting and bagging algorithms in environments where comprehensibility is important. On the other hand we have done the same analysis with Consolidated Trees induced from bootstrap subsamples. This option seemed promising because the well-known bagging algorithm uses bootstrap samples and achieves good quality results. The obtained results, both error and complexity, have been compared to the results obtained with C4.5 and CTC (T=75%). The most significant results can be observed in Figure 3 where average values for 18 data bases are shown. On the left side of the figure we can observe that the behaviour of the error related to the number of bootstrap subsamples used to build the trees is similar to the one obtained with T=75% subsamples, but the error is always higher in the former. This causes a high number of subsamples to be needed in order to improve the error rate achieved by C4.5 (75 subsamples in contrast to 10 subsamples needed with T=75%). 17.30

70,00

CT Boot

16.90

Error

60,00

C4.5 CT T=75%

16.70 16.50 16.30

Rel. Dif. Complexity

17.10

50,00

CT Boot

40,00

CT T=75%

30,00 20,00 10,00

16.10

0,00

15.90 3

5

10

20

30

40

50

75

Num ber Sam ples

100

125

150

200

3

5

10

20

30

40

50

75

100

125

150

200

Num ber Sam ples

Figure 3. Comparison of error (left side) and complexity (right side) for C4.5, CTC T=75% and CTC bootstrap.

On the right side of Figure 3 we can observe results obtained for complexity. Relative differences of CTC compared to C4.5 are shown. As mentioned before, it can be observed that Consolidated Trees built with T=75% samples are slightly more complex than C4.5 trees. But quite surprisingly Consolidated Trees built with bootstrap samples are considerably more complex than C4.5 trees. We can observe that trees built with

88

IADIS International Conference Applied Computing 2006

more than 40 subsamples are 60% more complex than C4.5 trees. This kind of tree can be too complex to understand in some particularly complex domains, restricting the potential capabilities of the decision tree model. It is not easy to explain the reasons that lead Consolidated Trees induced from bootstrap samples to perform so poorly. The higher subsample size can lead to more complex trees although the complexity increase is higher than what we expected. The number of different cases in each subsample (63% of the total for bootstrap and 75% for T=75%) can probably affect the error rate, but it is still a supposition that must be studied. It is clear that bootstrap subsamples must be discarded in Consolidated Trees’ construction process in benefit of stratified samples of 75% size of the original training set.

5. CONCLUSIONS AND FURTHER WORK This paper presents a wide analysis of the behavior of CTC algorithm. In one hand the experimentation has been done for a large set of databases: 20, and, on the other hand, the effect of two parameters of CTC algorithm has been analyzed. For the number of samples (Number_Samples parameter) 12 values, from 3 to 200, have been used and added to this, we have experimented with two different ways for building subsamples (Resampling_Mode parameter): stratified samples with size of 75% of the training sample and bootstrap samples. The results obtained with Consolidated Trees (built with different numbers and kinds of subsamples) have been compared to C4.5 trees. To make this comparison we have executed 5 times a 10 fold cross validation. Having into account that many real world problems need an explanation as well as high accuracy in the classification the comparison has been done from two points of view: error rate and complexity. Remember that too complex trees lose the explaining capacity. Results show that for subsamples of 75% Consolidated Trees achieve, in average, smaller error rates than C4.5 trees when they are built with 10 or more subsamples. These Consolidated Trees have similar complexity to C4.5 trees, so, they are better situated in the learning curve. Therefore we can state that Consolidated Trees are better classifiers than C4.5 trees. When the value of N_S increases, the error rates achieved with Consolidated Trees decrease whereas the complexity of the built trees increases. Anyway both parameters are stabilized once a certain number of subsamples is used: it seems that the error reduction caused by incrementing the number of used subsamples is not significant for values above 50 or 75 and the structure is stable if 10 or more subsamples are used. Since computational cost is higher as the number of used subsamples grows, it seems sensible to recommend a N_S value near 50. The method used to build subsamples clearly affects to the quality of results achieved with Consolidated Trees. If bootstrap samples are used to build trees the obtained results are worse than the ones obtained with T=75% from the two points of view: error and complexity. For error, the overall behavior is similar but always higher and for complexity the trees are much more complex, so, they lose the explaining capacity. As further work, we are thinking on experimenting with other possibilities for the Resampling_Mode parameter (different amount of information or variability of the subsamples). The first results address us to try with percentages greater than 75% and without replacement making a bias/variance analysis in order to study the origin of the error. Other interesting possibility is to generate new subsamples dynamically, during the building process of the CT, where the probability of selecting each case is modified based on the error (similar to boosting). At the moment CTC algorithm has not been compared to multiple classifiers obtained with techniques such as bagging or boosting because of the lack of explaining capacity of this kind of classifiers. Domingos (1998) proposes a way of obtaining explanation from bagging. At the moment we are comparing Consolidated Trees to this proposal.

ACKNOWLEDGEMENTS The work described in this paper was partly done under the University of Basque Country (UPV/EHU) project: 1/UPV 00139.226-T-15920/2004. It was also funded by the Diputación Foral de Gipuzkoa and the European Union.

89

ISBN: 972-8924-09-7 © 2006 IADIS

We would like to thank the company Fagor Electrodomésticos, S. COOP. for permitting us the use of their data (Faithful) obtained through the project BETIKO.

REFERENCES Bauer E., Kohavi R., 1999. An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants. Machine Learning, 36, 105-139 Breiman L., 1996. Bagging Predictors. Machine Learning, 24, 123-140 Chawla N.V. et al. 2002. Distributed Pasting of Small Votes. LNCS 2364. Multiple Classifier Systems: Proc. 3th. Inter. Workshop, 2002, Italy, 52-61 Craven M.W. Extracting Comprehensible Models from Trained Neural Networks. PhD thesis, Department of Computer Sciences, University of Wisconsin - Madison, 1996. Dietterich T.G., 1998. Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms. Neural Computation, 10(7), 1895-1924 Dietterich T.G., 2000a. Ensemble Methods in Machine Learning. LNCS, 1857. Multiple Classifier Systems: Proc. 1st. Inter. Workshop, 2000, Italy, 1-15 Dietterich T.G., 2000b. An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization. Machine Learning, 40, 139-157 Domingos P., 1998 Knowledge discovery via multiple models. International Data Analysis, 2, 187-202. Freund, Y., Schapire, R. E., 1996. Experiments with a New Boosting Algorithm. Proceedings of the 13th International Conference on Machine Learning, 148-156 Hastie T. et al., 2001. The Elements of Statistical Learning. Springer-Verlang (es). ISBN: 0-387-95284-5 Ho T.K., 1998. The Random Subspace Method for Constructing Decision Forests. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(8), 832-844. Newman D.J. et al., 1998. UCI Repository of Machine Learning Databases. http://www.ics.uci.edu/~mlearn/MLRepository.html, University of California, Irvine, Dept. of Information and Computer Science. Pérez J.M. et al., 2004a. Behavior of Consolidated Trees when using Resampling Techniques. Proceedings of the 4th International Workshop on Pattern Recognition in Information Systems (PRIS-2004), Porto, Portugal, 139-148. Pérez J.M. et al., 2004b. Analysis of structural convergence of Consolidated Trees when resampling is required. Proceedings of the 3rd Australasian Data Mining Conference (AusDM04), Cairns, Australia, 9-21. Quinlan J. R., 1993. C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers Inc.(eds), San Mateo, California Skurichina M. et al., 2002. Bagging and Boosting for the Nearest Mean Classifier: Effects of Sample Size on Diversity and Accuracy. 2364. Multiple Classifier Systems: Proc. 3th. Inter. Workshop, 2002, Italy, 62-71

90