Published in: F. Roli, J. Kittler and T. Windeatt (Eds.), Proc. 5th Int. Workshop on Multiple Classifier Systems, Cagliari, Italy Springer-Verlag, LNCS, 3077, 2004, 1–15
Classifier Ensembles for Changing Environments Ludmila I. Kuncheva School of Informatics, University of Wales, Bangor Bangor, Gwynedd, LL57 1UT, United Kingdom
[email protected]
Abstract. We consider strategies for building classifier ensembles for non-stationary environments where the classification task changes during the operation of the ensemble. Individual classifier models capable of online learning are reviewed. The concept of “forgetting” is discussed. Online ensembles and strategies suitable for changing environments are summarized. Keywords: classifier ensembles, online ensembles, incremental learning, non-stationary environments, concept drift.
1
Introduction ”All things flow, everything runs, as the waters of a river, which seem to be the same but in reality are never the same, as they are in a state of continuous flow.” The doctrine of Heraclitus.
Most of the current research in multiple classifier systems is devoted to static environments. We assume that the classification problem is fixed and we are presented with a data set, large or small, on which to design a classifier. The solutions to the static task have marvelled over the years to such a perfection that the dominance between the classification methods is resolved by a fraction of percent of the classification accuracy. Everything that exists changes with time and so will the classification problem. The changes could be minor fluctuations of the underlying probability distributions, steady trends, random or systematic, rapid substitution of one classification task with another and so on. A classifier (individual or an ensemble)1 , if intended for a real application, should be equipped with a mechanism to adapt to the changes in the environment. Various solutions to this problems have been proposed over the years. Here we try to give a systematic perspective on the problem and the current solutions, and outline new research avenues. The paper is organized as follows. Section 2 sets the scene by introducing the concept of changing environment. Online classifier models are presented in Section 3. Section 4 details some ensemble strategies for changing environments. 1
Unless specified otherwise, a classifier is any mapping D :
