Concept Versioning: A Methodology for Tracking ... - Semantic Scholar

Concept Versioning: A Methodology for Tracking Evolutionary Concept Drift in Dynamic Concept Systems Manfred Klenner and Udo Hahn1 Abstract. Technical domains are affected by continuous change as a reflection of technological progress. Correspondingly, concept descriptions of terminological knowledge bases for such domains must be adjusted to these dynamics. In this paper we propose a concept versioning methodology which accounts for the evolutionary adaptation of already established concepts according to the gradual shaping of new standards. The approach is based on the provision of qualitative progress models for the given domain, measures for the prediction and evaluation of the significance of progress, and a representation and update scheme for concept version management.

1

INTRODUCTION

Technical domains, such as information, hi-fi, or automobile technology, share a common feature, namely that their underlying concept systems undergo continuous change. These dynamics are exhibited either by the substantial augmentation of concept systems in terms of concept innovations (mirroring the introduction of new products, technologies, etc.) or by the evolutionary adaptation of already established concepts according to the gradual shaping of new standards that evidence continuous technological progress in the field. This latter type of change is illustrated, e.g., by growing main memory capacities for personal computers (with ranges from 64 to 256 KB in the early 80’s to today’s 1 to 4 MB of internal storage). The learning problem with respect to these time-varying concepts is usually referred to as concept change ([1], [2]) or concept drift ([3], [4]). Why bother at all about concept drift phenomena? The problem becomes acute for the long-term engineering of non-toy knowledge bases covering rapidly changing domains. Usually, technical standards are frozen in terms of static value restrictions or integrity constraints in order to assure valid reasoning as a basis, e.g., for up-todate recommendations (what’s good?) and evaluations (what’s dated or overpriced?). However, when technical standards continuously evolve in the lifetime of a knowledge base these dynamics must be directly incorporated at the knowledge representation level in order to preserve its inferential adequacy. As an alternative to manual updates of integrity constraints by some human knowledge engineer we propose an automatic procedure for dealing with these concept drift phenomena. It produces a generational stratification of the underlying level of generic concepts in terms of concept versions -each stage of technical development is represented in terms of a version-specific constraint expression defining a particular "standard" holding at that stage of development; single instances are then related to their associated concept version.

1

CLIF - Computational Linguistics Research Group, Albert-Ludwigs-Universität Freiburg, Friedrichstr. 50, D-79098 Freiburg i. Brsg., Germany. email: {klenner, hahn}@coling.uni-freiburg.de

 1994 M. Klenner and U. Hahn ECAI 94. 11th European Conference on Artificial Intelligence Edited by A. Cohn Published in 1994 by John Wiley & Sons, Ltd.

The version tracker introduced in this paper forms part of the concept learning component of a natural language text understanding system that processes product announcements and reviews from various information technology magazines. Given this application context we may factor out the following general requirements for an appropriate concept versioning methodology: • Texts from information technology magazines usually contain only positive examples of the (goal) concept to be learned. This precludes a learning strategy which is exclusively based on a similarity-based approach, since the continuous generalization of a goal concept is not sufficient. As concept drift phenomena affect the entire value range of attributes -- new permitted values must be added while obsolete ones have to be removed (e.g., consider the drift from 64-256 KB to 1-4 MB of internal storage for PCs) -- facilities for discrimination learning must supplement generalization-driven procedures in order to generate different, non-overlapping value ranges for each concept version. This requirement contrasts with previous work (e.g., [3], [4]), where preclassified positive and negative examples are available for the determination of concept drift. • Due to on-line processing requirements (texts are continuously fed into the system as they become available), we are committed to an incremental learning mode. Thus, versioning decisions in our approach are tentative, since they always rely upon incomplete data sets. The capability of performing revisions of onceestablished version boundaries therefore constitutes an integral part of our concept drift model. This contrasts with the assumptions underlying the experiments in [5] and [6], where the complete training sets are given once and for all and learning proceeds under closed data sets. • In accordance with previous approaches, we maintain the most recent concept description which specifies the currently valid standards in the domain. Additionally, we need to preserve the developmental stages of a concept in order to prevent prior and obsolete instances from being abandoned in the course of concept drifts. As a consequence, we require that concept versioning must not affect the concept class of an instance as a whole, but that a history-preserving, generational stratification has to be imposed on it yielding a tri-partite description in terms of a common generic concept, its partition into several concept versions, each covering a set of associated instances. This contrasts with work as described in [7], where destructive updates are performed overwriting prior information, as well as in [1] and [2], where only the most recent concept description is maintained. Based on these considerations our concept drift model requires positive examples, allows incremental processing and adaptive revisions of once-established versions, while it preserves exhaustive classification of instances under the covering generic concept.

2

CONCEPT VERSIONING: AN OVERVIEW

As the most recent texts are entered into the natural language processor, evolutionary changes in a dynamic concept system are signalled when the text parser [8] tries to assign values to some attribute of an instance that are inconsistent with the value range constraints set up by the concept intended to classify that instance. Consider Fig. 1 which depicts numerical value restrictions for the attributes weight and hard disk size of the concept notebook1 (the "worst" and "best" values allowed are enclosed in square brackets). Although the parser has strong linguistic evidences to consider LTE lite/25 as a special kind of notebook, the assignment of 5 lb. to the weight attribute contradicts the associated value restriction of the notebook1 concept. The new notebook LTE lite/25 weighs 5 lb. ...

weight: [12-8 lb.] hard disk: [20-84 MB]

weight: 5 lb. hard disk: ...

Structure of an Instance

Sample Instance

attributesi: clock frequency: valuei: InstVali value: 25 MHz g-spacei: [Bottomi-Topi] g-space: [20-25] diff-valsi: DiffValsi diff-vals: 3

Figure 1. A new instance occurring with inconsistent attribute values relative to the value range constraints set up by the classifying concept

The system may react either by adapting the description of the already given concept class (generalizing the violated conditions), or by creating a new, versioned concept class as a successor to the currently valid one. But care must be taken to balance both alternatives. Generalization, on the one hand, must be constrained so as not to produce timeless concepts, in order to assure that no overgeneralized concepts be created. Versioning, on the other hand, must be constrained so as not to apply too many times, in too short intervals, since otherwise a large amount of only slightly differing concept versions would result, with representation structures suffering to discriminate between really significant developmental steps. In order to cope with these requirements three major constructs will be introduced in this paper: a predictive progress model of the domain, which specifies in qualitative terms the foreseeable developmental directions of change in the underlying domain, an empirical progress measure which is sensitive to the actual development of attribute values, and a significance criterion which actuates the aforementioned balance between generalization and versioning. As long as the significance criterion is not fulfilled only generalization is allowed to occur. This process leads to the successive refinement of the diagnostic data indicative of the domain’s continuing progress. Once the significance criterion is satisfied by a new instance, a versioning process is triggered and the results of quantitative developmental analysis are used to estimate a significance threshold for the new concept version, thus starting the diagnostic cycle again.

3

fringe of the concept hierarchy in terms of concrete subclasses (in our domain, products such as LTE lite/25, Unix) classified by generic concept classes. (Note that the level of instances in this intensional reading is strictly distinguished from the extensional level of individuals as direct representations of countable, uniquely identifiable entities in the domain.) Each concept description of an instance (cf. Fig. 2) consists of a distinguished name and a list of associated attributes, to each of which so-called attribute facets are attached, which are particularly relevant for the versioning procedure. We distinguish between an attribute’s value, indicating the actual value for that attribute, and its g-space, describing the most specific generalization space which is spanned by a bottom and a top value; gspace is successively built from the first instance of a version up to the most recent one (relative to a quality ranking discussed below) and locally maintained by each instance. Finally, the attribute facet diff-vals specifies the number of different values encountered so far for that attribute with respect to the current version.

KNOWLEDGE REPRESENTATION BACKGROUND OF CONCEPT VERSIONING

We assume the underlying terminological layer of the knowledge base to consist of frame representations for concept definitions. These concepts fall into three main categories: generic concepts carry the description of an entire concept class (e.g., notebook, printer, operating system), concept versions represent different generations of such a single concept class in terms of technical standards holding at a certain stage of development (i.e., the result of the methodology described in this paper), and instances characterize the outer

values

facets

Figure 2. Concept description of instances

As far as the family of frame-style, classification-based concept languages [9] is concerned, the subsumption relation establishes a specialization hierarchy which relates concept classes among each other and to their associated instances, but usually no relation holds among the instances of the same generic concept. As it turns out to become crucial for efficient version management to locate the proper position of any incoming instance relative to the previously analyzed ones (cf. section 8 for an in-depth discussion of these update mechanisms), we define a complementary partial ordering among these instances. It is based on a measure that scores each instance’s quality (see Def.-1) and thus establishes a predecessorsuccessor ordering among all those instances which are classified by the same generic concept, according to their fitting into the global quality scale. Let inst* denote the new, currently analyzed instance and att be one of its relevant attributes (relevance information (see section 4) is accessible via its classifying generic concept, genc, using the function RelAtts). Let Val(inst*, att) be an access function returning the value for attribute att of instance inst*. We then define a simple quality metric that expresses the distance of inst* to the unique root instance whose attribute values are taken to be the least favorable ones for the corresponding generic concept (root has to be carefully determined by a human knowledge engineer on the basis of empirical data; it can be considered a kind of bottom-line instance all whose values are unlikely to be undercut by yet another instance, thus serving as a reference point from which improvements can be reasonably assessed): Global_Quality(inst*) :=

∑

att ∈ RelAtts ( genc )

Val ( inst* , att ) − Val ( root, att ) Val ( root, att )

This measure is global in scope, since the more distant an instance is from root the "better" it is on a qualitative scale. Definition-1. Global quality measure

Machine Learning

474

M. Klenner and U. Hahn

4

THE PROGRESS MODEL

The progress model - similar to a domain theory in EBL approaches [10] - captures in qualitative terms the regularities of foreseeable developments in a technical domain. It is currently restricted to scalar quantitative attribute dimensions and estimates, if possible, innovation directions for technical features. The human domain expert in charge of its acquisition must specify two Boolean variables, viz. whether an attribute of a generic concept is relevant for versioning or not and whether future attribute values will increase or decrease compared with its current top value (cf. Fig. 3). scope: computer (also for all subclasses, e.g., portable) predictions: (1) clock frequency: increasing [relevant] (2) main memory size: increasing [relevant] (3) weight: nil [irrelevant] Figure 3. Fragment of the progress model

MeanDevStep(inst, att) :=

The consideration of relevance accounts for the obvious fact that various conceptual attributes differ in their potential to contribute to versioning processes (e.g., the speed of a processor is a relevant attribute, while its size is not relevant). In order to ease the knowledge acquisition process the predictions of the progress model are each associated with the most general concept to which they apply. Their scope is thus determined by the classifier mechanisms underlying terminological reasoning [9]: each statement related to some concept class also holds for its subclasses unless its constraints are overwritten for more specialized ones. This is the case, e.g., for portable, a subclass of computer, where the inherited entry for weight is modified (decreasing replaces nil).

5

MEASURING PROGRESS: THE MEAN DEVELOPMENTAL STEP

Two types of knowledge, predictive and empirical, come into play when progress actually occurs in a technical domain. The first one relates to the domain’s progress model which captures in qualitative terms the main direction of changes to come. Knowledge relating to the actual dynamics of change is only available from empirical data continuously flowing into the system. The determination of real progress in our approach then consists of combining the predictive qualitative estimates as expressed in the progress model with the actual development as indicated by empirical data in terms of quantitative attribute values. First, consider Def.-2 for the abstraction from increasing or decreasing change modes of attribute values. Let att denote an attribute shared by two concept descriptions, con1 and con2. Let genc be the generic concept that classifies both, con1 and con2. The relation exceeds ( >ex ) is then defined by: Val(con1, att) >ex Val(con2, att) :⇔  VVal ( con1 , att ) > Val ( con2 , att )incr if, genc( att , )  if, genc( a , ˙ tt )  VVal ( con1 , att ) < Val ( con2 , att )decr The predicate incr (decr) is true iff the progress model contains an entry for att in genc such that it predicts "increasing" ("decreasing") values for future instances; otherwise it is false. Definition-2. Ordering of values in terms of the exceeds relation

Given predictive knowledge about the developmental direction of an attribute’s value domain in the progress model and given two concrete attribute values, the exceeding one represents an explicit

Machine Learning

technical developmental step compared with the exceeded one. We may rephrase this simple observation in more general terms as follows. Let the values for some attribute of instances, all classified by the same concept (version), be ordered according to the exceeds relation. (In particular, this ordering yields the top value, exceeding all other values of this set, and the bottom value which is exceeded by all others; cf. the g-space facet in Fig. 2.) We may now call any difference between two successive values in that value list a single developmental step. The mean developmental step of a value collection can then be determined as stated in Def.-3. The computation of the mean developmental step yields a factor which, on the one hand, describes the observable behavior of a distribution of attribute values (its mean growth rate). On the other hand, it can also be used to predict future value developments by way of heuristic projection, as will be discussed in the next section.

475

 Top ( inst, att ) − Bottom ( inst, att ) DiVa(inst, if att) 1>  DiVa ( inst, att ) − 1  V undefined else The function DiVa yields the number of different values for an attribute att maintained under the diff-vals facet by the selected instance inst, while Top and Bottom access the top and bottom values, resp., specified in the attribute’s g-space facet. Definition-3. Mean developmental step

6

PREDICTING PROGRESS: THE PREDICTED TOP VALUE

The processing of any new instance causes the mean developmental step, the model’s major construct for the prediction of progress, to be updated subsequently. This update even takes into consideration the values of the version-triggering instance (that particular instance whose values fulfil the significance criterion (cf. Def.-5), thus causing a versioning process actually to be started). That decision is crucial for the quality of predictions, since taking into account new trends as derivable from the most recent top instances improves the predictions to be made for the next version generation. Additionally, as a projection heuristic, we incorporate the number of developmental steps of the current version into the estimate of the minimal upper bound for any new version, hence the slightly modified expression for the mean developmental step in Def.-4. The underlying assumption is that new versions with non-linear growth rates should differ from previous ones through proportionally lower/higher value ranges, while the number of developmental steps should be approximately the same for each version (a strong first-shot assumption, that may, nevertheless, be overridden by subsequent computations to account for unexpected or biased growth rates). The basic construct for guiding the predictions of any value development is the predicted top value, PTV (see Def.-4), whose definition reflects the considerations from above. A PTV is maintained for every relevant attribute of some concept underlying the versioning mechanism; it fixes a minimum value expansion that must be reached by (at least) one instance before versioning is allowed to occur. Unless this condition is fulfilled for every relevant attribute of the concept version, or if a value for a relevant attribute of a new instance is below the PTV, only generalization may occur. At the very beginning of the learning cycle (i.e., for the initial knowledge base), PTVs are usually set equal to the top values of the available concept class description; later on they are managed by the version tracker once a versioning process has occurred.


Let inst be an immediate predecessor (according to the quality metric as defined in Def.-1) of inst*, a version-triggering instance. The function Best yields the most favorable of two values w.r.t. the exceeds relation; we may then determine BestVal := Best( Val(inst*, att), Top(inst, att) ). The computation of the predicted top value, PTV, w.r.t. some (increasing) attribute att for a newly formed concept version, succV, is based on the following criterion: PTV( succV, att ) := Top( inst, att) + BestVal − Bottom ( inst, att ) ( DiVa ( inst, att ) − 1 ) DiVa ( inst, att ) Definition-4. Predicted top value (PTV)

7

EVALUATING PROGRESS: THE SIGNIFICANCE CRITERION

In the previous two sections we have specified the major building blocks for evaluating real progress in a dynamic domain. They can now be assembled in terms of a significance criterion on the basis of which version-triggering instances will be recognized. In order to determine the version significance of any new instance, inst*, one must assess the deviation between the predicted top values of the currently valid concept version, currV, and inst* for any attribute value of inst* exceeding the corresponding PTV. Relating each relevant attributes’ deviation with the mean developmental steps holding for currV and summing up the values for these ratios gives a useful measure for the significance of the violations. Let (according to the quality metric) inst be the predecessor of inst*, currV be the concept version classifying inst, and genc be the generic concept classifying currV. The function PredTopVal accesses the value of PTV of version currV for its attribute att. IF

∀att ∈ RelAtts ( genc ) : Val ( inst* , att ) ≥ ex P redTopVal ( currV, att )

THEN Significance (inst*) :=

∑

att ∈ RelAtts ( genc )

PredTopVal ( currV, att ) − Val ( inst* , att ) MeanDevStep ( inst, att )

ELSE Significance (inst*) := 0 Definition-5. Version significance

Since the estimates of the value ranges of any new version rely upon few data items only, the concept system faces the danger of possible "version oscillation", i.e., it may be affected by instances that only "slightly" violate the newly established PTV, but nevertheless continuously trigger new versioning processes. We thus specify the significance criterion as a kind of delay mechanism. It requires that the values of a version-triggering instance, inst*, must exceed the PTVs of the most recent version for all relevant attributes, on average, by the mean developmental step. This is implied by the additional requirement that a violation of relevant attributes is version-significant iff Significance(inst*) ≥ | RelAtts(genc) | (cf. Def.5). One might also conceive less rigid criteria for larger numbers of relevant attributes, e.g., subsets of relevant attributes whose values float into the same direction. Anyhow, this criterion yields an adaptive threshold, since the values of the mean developmental steps will be successively refined as new instances become available.

8

MANAGING PROGRESS

It is crucial for our approach that the most recent version needs not

Machine Learning

476

necessarily classify the currently considered instance. If the most recent version turns out to be inadequate other versions are tried for that instance, moving backwards in the direction of the predecessors of the currently considered version. The classification of instances is based solely on the fulfillment of criteria specified by the integrity constraints of the versions. Therefore, our learning and classification scheme is not a temporal, but predominantly a qualitative one (usually, but not always, top scores in the attribute dimension and temporal recency nevertheless coincide). In order to guarantee proper, i.e., almost non-overlapping value partitions based on the quality of instances, the following constraints have been set up: • For the most recent version, generalization of relevant attributes is allowed only if the deviations are in conformity with the predicted direction of progress in the domain (i.e., only “exceeding“ values are allowed); • For the remaining versions, generalization of relevant attributes is not allowed at all. As a consequence, when two or more relevant attributes have to be considered for versioning and the violating values of a new instance refer to at least two relevant attributes of at least two different versions, so-called transition instances are formed and grouped under their generic concept without further relating them to any concept version. In general, these restrictions were chosen to avoid large overlapping value ranges resulting from an exclusively temporal classification and learning criterion (however, minor overlaps may result from the versioning process, cf. section 9). The instances sorted back to a prior version directly influence the values for the mean developmental steps of that version’s attributes (the value of diff-vals increases). Consequently, the value for the mean developmental step decreases (see Def.-3) while the significance value increases accordingly (see Def.-5). This way, formerly version-insignificant instances might then become version-significant (cf. [11] for an illustration of such a propagation effect). Therefore, we need mechanisms for the re-evaluation of instances. Two questions then arise: Which instances need to be re-evaluated and what is the correct evaluation space? At this point the benefits of the ordering of instances according to global quality, become evident. The proper location of an instance in the global quality ranking simultaneously discloses its corresponding set of predecessor and successor instances on the quality scale. Thus, the correct generalization space for evaluating any new instance is accessible via that instance’s immediate predecessor, considering its g-space facet (it contains the generalization space successively built from all predecessor instances). At the same time, the instances yet to be re-evaluated are supplied by all the successor instances. If a newly integrated instance itself is not version-significant then only one of its successor instances may possibly become version-significant, but none of its predecessors. The reason is that the successor instances are, by definition (see Def.-1), of a "better" quality, which is a prerequisite to be of "higher" version significance.

9

EXPERIMENTAL RESULTS

We tested the version tracker on training sets for personal computers, portables, printers, hard disks and monitors (altogether about 600 instances). The test sets were taken from information technology magazines covering the time period from 1987 to 1993. The instances were entered into the system in chronological order by year, while an arbitrary (we chose an alphabetical) ordering was effective for each annual set. The chronological ordering by year (or any other comparable large-scale ordering) is justified by the online character of the system and its incremental learning mode.


We here concentrate on the sample set for portables comprising 141 instances. Table 1 shows the resulting concept versions w.r.t. the relevant attributes for the complete training set. The versioning system generated 3 versions, classified 124 instances properly and encountered 17 transition instances. Trying to escape from subjective face validity judgments, we performed a preliminary statistical evaluation of the concept drift model using a discriminance analysis from the SPSS package. We thus measured the fit of instances classified by concept versions. The difference between both approaches lies in the fact that the version tracker determines version boundaries incrementally, with the classifier actually relating instances to concept versions, while discriminance analysis computes the validity of some a priori assignment of instances to concept versions and thus judges the discriminative power of the assignment decisions. The overlap of the membership assignments between the version tracker (VT) and discriminance analysis (DA) is stated in the bottom line of Table 1. These data indicate that reasonable version boundaries were determined by the incremental procedure underlying our concept drift model. Table 1. Final partitions for portable and results of the statistical evaluation

clock frequency main memory size covered instances overlap (VT vs. DA)

version 1 [4.7 - 10] [128 - 768] 34 100 %

version 2 [10 - 20] [640 - 2048] 76 89.4 %

version 3 [20 - 33] [2048 - 8000] 14 95.9 %

10 CONCLUSIONS AND OUTLOOK Concept versioning as a machine learning methodology can be considered a kind of instance-based empirical learning that is located between two well-known paradigms, viz. learning from examples and learning by observation, especially incremental conceptual clustering. As in learning from examples, the system is provided with preclassified instances of some goal concept. The preclassification of the instances results from natural language text analysis (as in “the new notebook LTE lite/25“) and therefore yields positive instances only. As in the standard approach to learning from examples, a single goal concept has to be learned, but unlike that approach an additional intermediary concept layer must be acquired, viz. the different versions of a goal concept. This extended concept formation task shares a somewhat superficial similarity with the problem of determining classes in conceptual clustering. The problem of whether to generalize a given class (version) or to bring up a new one is common to both learning techniques. Usually, this is achieved in conceptual clustering by means of a quality measure which maximizes intra-class similarity and inter-class dissimilarity. As far as concept versioning is concerned, maximizing inter-class dissimilarity seems inappropriate, since versions represent successive developmental stages of a common concept, not completely distinct concept classes. As a consequence, entirely different quality measures must be supplied for concept versioning and conceptual clustering. Additionally, in the area of conceptual clustering instances never come preclassified and usually more than one (intermediary) layer has to learned. It thus turns out that concept versioning is neither a complicated form of learning from examples nor a simplified case of (incremental) conceptual clustering, but constitutes a special concept learning task with a characteristic set of requirements (see section 1). These requirements have led us to results which may be summarized from two perspectives. From a concept learning point of view, we have introduced a model of concept versioning that only requires positive examples, allows incremental processing and adaptive revi-

Machine Learning

477

sions of once-established version generations, while it preserves exhaustive classification of instances under the covering generic concept. Since many of our requirements were motivated by applicational constraints of real-world knowledge base maintenance, we may also consider the results worked out as a contribution to knowledge acquisition methodology. Actually, we propose to automate the knowledge engineering process as far as the on-going evolution of a concept system in a dynamic domain is concerned. The proposed methodology only requires a human knowledge engineer to supply a progress model that contains the foreseeable developmental directions of changes in the domain -- this includes the distinction of relevant from irrelevant attributes for technological progress. Any other criteria (global quality, measures for predicting and evaluating the significance of progress, the representation and management of progress) are domain-independent and constitute our contribution to automated concept drift tracking methodology (for a more elaborated treatment, cf. [12]) Besides broadening the empirical basis of our model, we are currently planning to incorporate qualitative attributes into the model. This is an obvious extension which not only must account for the principles and versioning criteria for qualitative attribute dimensions, but also for combinations of qualitative and quantitative ones. Note, however, that “quality“ in technical domains usually boils down to numerically-valued attributes. Future work will also concentrate on the validation of the concept version descriptions (e.g., by comparing the results from the version tracker to those of appropriately tuned non-incremental clustering procedures or relating machine-generated versions to those provided by human domain experts) and the application of that methodology to other dynamic domains (e.g., automobile or hi-fi technology). The versioning model has been implemented in Quintus Prolog and runs on SUN SPARCStations. An implementation using the LOOM representation system [13] is under way. Acknowledgments. The work reported in this paper is funded by a grant from DFG (grant no. Ha 2097/2-1). We also like to thank the reviewers for their detailed suggestions to improve the paper.

REFERENCES [1] [2] [3]

S. Carey, Conceptual Change in Childhood, MIT Press, Cambridge/MA, 1985. P. Thagard, ‘Concepts and conceptual change’, Synthese, 82, 255-274, (1990). J.C. Schlimmer and R.H. Granger, ‘Beyond incremental processing: Tracking concept drift’, AAAI-86: Proc. 5th National Conf. on Artificial Intelligence, 502507, Morgan Kaufmann, Los Altos/CA,1986. [4] P. Langley, ‘A general theory of discrimination learning’, In D. Klahr; P. Langley and R. Neches (Eds.), Production System Models of Learning and Development, 99-161, MIT Press, Cambridge/MA, 1987. [5] P.L.Bartlett, ‘Learning with a slowly changing distribution’, COLT’92: Proc. 5th Workshop on Computational Learning Theory, 243-252, 1992. [6] D.P. Helmbold and P.M. Long, ‘Tracking drifting concepts by minimizing disagreements’, Machine Learning, 14 (1), 27-45, (1994). [7] L. DeRaedt and M. Bruynooghe, ‘Belief updating from integrity constraints and queries’, Artificial Intelligence, 53, 192-307, (1992). [8] U. Hahn, ‘Making understanders out of parsers: Semantically driven parsing as a key concept for realistic text understanding applications’, International Journal of Intelligent Systems, 4 (3), 345-393, (1989). [9] R. MacGregor, ‘The evolving technology of classification-based knowledge representation systems’, In J. Sowa (Ed.), Principles of Semantic Networks, 385400, Morgan Kaufmann, San Mateo/CA, 1991. [10] T.M. Mitchell; R.M. Keller and S.T. Kedar-Cabelli, ‘Explanation-based generalization: a unifying view’, Machine Learning, 1 (1), 47-80, (1986). [11] U. Hahn and M. Klenner, ‘A version tracker for dynamic technical domains as a tool for automated knowledge base evolution’, FLAIRS-94: Proc. 7th Annual Florida AI Research Symposium, 1994. [12] U. Hahn, and M. Klenner, Concept Versioning, CLIF-Report 7/94, AG Linguistische Informatik/Computerlinguistik (CLIF), Universität Freiburg, 1994. [13] R. MacGregor and R. Bates, The LOOM Knowledge Representation Language. ISI/RS-87-188, ISI, University of Southern California, 1987.
