NOEMON: An Intelligent Assistant for Classifier Selection Myra Spiliopoulou and Alexis Kalousis and Lukas C. Faulstich and Theoharis Theoharisy Institut f¨ur Wirtschaftsinformatik, Humboldt-Universit¨at zu Berlin, Spandauer Str. 1, D-10178 Berlin, Email:
[email protected] Institute of Computer Science, University of Athens, Email:
[email protected] 1 Institut f¨ur Informatik, Freie Universit¨at Berlin, Email:
[email protected] 2 yInstitute of Computer Science, University of Athens, Email:
[email protected] 3
Abstract. The selection of an appropriate classification model and algorithm is crucial for effective knowledge discovery on a dataset. For large databases, as they are common in data mining, such a selection is necessary, because the cost of invoking all alternative classifiers is prohibitive. This selection task is impeded by two factors: First, there are many performance criteria, and the behaviour of a classifier varies considerably with them. Second, a classifier’s performance is strongly affected by the characteristics of the dataset. Classifier selection implies mastering a lot of background information on the dataset, the models and the algorithms in question. An intelligent assistant can reduce this effort by registering background information and knowledge of the analyst and inducing helpful suggestions from them. In this study, we present such an assistant, NOEMON. For each registered classifier, NOEMON measures its performance for a collection of datasets. Rules are induced from those measurements and accommodated to a knowledge base. Then, the suggestion on the most appropriate classifier for a dataset is based on the similarity of the dataset with existing ones and on the performance of the classifiers for the latter. Feedback from applying the proposed classifier to the dataset is used to extend the knowledge base and further tune NOEMON. Keywords. Classifier selection, classifier comparison, multidimensional metrics, dataset morphology
1 Introduction The machine learning field is evolving since a long time and has given us a variety of models and algorithms to perform various tasks, like prediction, classification, clustering etc. For the task of classification, which is also the focus of this paper, the analyst has a variety of models and algorithms at her disposal. She must select among them the ones that better match to the morphology and the special characteristics of the problem at hand. This selection is one of the most difficult problems since there is no model or algorithm that performs better than all others independently of the particular problem characteristics [Aha92, Sal91, S+ 91, WK89]. Each algorithm has a “selective superiority” [Bro95], 1 Work
of this author has been supported by the Greek General Secretariat for Research & Technology (project 28) 2 Work of this author has been supported by the German Research Society, Berlin-Brandenburg Graduate School on Distributed Information Systems (DFG grant no. GRK 316) 3 Work of this author has been partially supported by the Greek General Secretariat for Research & Technology (project 28)
ENE
ENE
i.e. it is better than the rest for specific types of problems. This happens because each algorithm has a socalled “inductive bias” [Mit97] caused by the assumptions it makes in order to generalize from the training data to unseen examples. Hence, the analyst must posses a lot of experience to be able to identify the most appropriate algorithm for the morphology of the problem at hand. The process of selecting the appropriate models and algorithms is described thoroughly in [BS97]. The model of an algorithm actually defines the “search space” or “hypothesis space”, such as k-DNF or kCNF forms, linear discriminant functions, rules etc. The algorithm searches this space for the right hypothesis, i.e. for the hypothesis that better fits to the data. The algorithm determines the order of visiting the states in this space. For example, between two algorithms that both search in DNF space, one might start the search from DNF forms that contain the complete set of features, while the other might start from sets consisting of only one feature. Hence, the wrong choice of algorithm may result in a slow convergence
towards the right hypothesis, or may even end at a suboptimal solution due to a local minimum. A wrong choice of model can have a more severe impact: A hypothesis appropriate for the problem at hand might be ignored because it is not contained in the model’s search space. The knowledge discovery process is an iterative one, as depicted in Fig. 1. The analyst must first select the right model for the classification task to be performed and, within it, the right algorithm, whereby the special morphological characteristics of the problem must always be taken into account. The algorithm is then invoked and its output is evaluated. If the evaluation results are poor, the process is repeated from a previous stage with new selections. Model selection
Algorithm selection
Algorithm invocation
Evaluation
Results
Figure 1 Model and task selection for knowledge discovery
The human effort in this trial-and-error procedure is apparent. A plethora of commercial and prototype systems with a variety of models and algorithms is put at the analyst’s disposal. However, the selection among them is left to the analyst. Recently, several efforts have been made to facilitate the selection of the appropriate model and algorithm.
In VBMS [R+ 87], an architecture is proposed that tries to predict which of the available algorithms will perform better for a given classification problem. It uses problem characteristics like the number of examples and the number of features. The system actually produces new knowledge for each new problem it faces, by associating problem characteristics with algorithm performance. The main disadvantage of VBMS is that it is trained as new classification tasks are presented to it; this makes it quite slow. Moreover, a single performance criterion is being considered, namely execution time. In SBS [PB95], the problem of model and algorithm
selection is observed as a search in the meta-space of possible models of representation and in the metaspaces of the possible ways to traverse each one of the possible models. Movements in those meta-spaces are performed by the dedicated operators implemented in SBS. SBS does not produce new knowledge and can only utilize existing knowledge that can be given in the form of preconditions in the operators. The analyst must encode this knowledge explicitly. In MCS [Bro95], the selection of models and algorithms from a pool of available ones is performed on the basis of existing knowledge from the expert, encoded in the form of rules. However, those rules are incorporated into the system and cannot be extended. Schaffer proposes a brute force method for selecting the appropriate algorithm [Sch93]: All available algorithms are executed for the problem at hand and their accuracy is estimated using cross validation. The one achieving the highest score is selected. The disadvantage of this method is the high demand on computation resources. An interesting approach is proposed in [GB95], aiming at the automatic derivation of rules to guide algorithm selection. The approach is based on the characteristics of the data. They define a set of characteristics that are expected to affect the performance of the algorithms. Then, they invoke machine learning techniques to create models that associate the characteristics with the accuracy of the algorithms. The main advantage of this approach is the automated procedure of producing new knowledge on the expected performance of each new algorithm. However, the method has only been used in a limited number of problems. Moreover, accuracy is the sole performance measure. When multiple performance criteria are desirable, the problem of mapping the performance results into comparable scalar values becomes apparent. In [NS97], the “Data Envelopment Analysis” (DEA) methodology has been proposed as a solution to this problem. In this study, positive performance metrics like accuracy, and negative ones, like training time, are combined into a single performance ratio, called “efficiency”. A common weighting scheme is not required; it is computed on the basis of any among the algorithms under inspection. Despite the intuition that such a weighting scheme would be biased, it is shown that an objective comparison of the algorithms is possible [NS97]. However, a methodology of using the comparison results to select the most appropriate algorithm for a new problem is not proposed. In this study, we propose NOEMON, an intelligent assistant for model and algorithm selection in the context of the classification task. NOEMON keeps a set of registered classification models and algo-
rithms, jointly called “classifiers” hereafter. Their performance is measured according to a number of criteria. Performance results are retained for a collection of known datasets, the morphological characteristics of which are recorded and constitute a “morphology space”. For a new dataset, NOEMON suggests the most appropriate classifier, deciding on the basis of morphological similarity between the new dataset and the existing collection. The actual performance results for the new dataset are used to extend the knowledge of NOEMON on the behaviour of the registered classifiers. Feedback from the analyst is incorporated as grading on the suggestions made by NOEMON.
Second, we incorporate the DEA methodology as described in [NS97], to map the multidimensional performance values produced for several performance criteria into an one-dimension interval [0; 1]. As noted above, though, we go beyond the simple comparison of classifiers for known datasets: NOEMON suggests a classifier for an unknown dataset. NOEMON models the classifier selection problem as a knowledge discovery task. The morphological characteristics we establish and the performance measurements we obtain result in the formulation of a training space, on which machine learning models can be used to assess rules for classifier selection.
2 Overview of the NOEMON Project We propose NOEMON, a mixed initiative system intended to perform knowledge discovery and to assist the expert in driving the discovery process. Assistance concerns the selection of the most appropriate classification model and algorithm for a given problem. NOEMON will supply the expert with suggestions based on its knowledge on the performance of the models and algorithms for related problems. This knowledge is being accummulated in a knowledge base and is updated as new problems are being processed and feedback from the analyst is obtained.
NOEMON has two components, the “Assistant” and the “Problem-Solver” or “Solver”, as shown in fig. 2. The Assistant suggests the classification model and algorithm to the analyst using knowledge from an expandable knowledge base and (optional) instructions from the analyst. Based on the selected algorithm, the Solver resolves the problem input and returns the knowledge discovery results to the analyst. Based on the obtained results, the user provides further feedback, e.g. in terms of a grading for the suggested model and algorithm. Instructions Feedback
NOEMON o o
Problem
)
NOEMON builds on previous experience from two aspects. First, we adopt a similar approach to [GB95, R+ 87] for the acquisition of the initial knowledge, which is refined whenever new classification activities take place. However, NOEMON explicitly aims at the definition of a minimal and complete set of morphological features affecting performance, and at the establishment of a morphology space. Moreover, its goal is not the comparison of classifiers for a known problem, but the suggestion of a classifier for an unknown one.
2.1 The Architecture of NOEMON
Suggested Classifier
Assistant
KD results Solver
Figure 2 The components of NOEMON
Among the two components of NOEMON, the Solver is a rather trivial task. It invokes the classification algorithms approved by the analyst or suggested by her explicitly, and it returns the results to her. Depending on the algorithm and the interaction it requires or allows, this task is more or less interactive. The Assistant component uses two collections, a collection of classification models and algorithms to select from and a collection of already processed problems, upon which the selection is based. In our first prototype, we have registered the models and algorithms of the MLC++ system [Koh96]. The problem collection will be initiated as a set of publicly available databases, which have already been studied using various models and algorithms, and for which knowledge discovery results are publicly available. 2.2 The Assistant Component of NOEMON The Assistant is a more complicated component. Its internal architecture is depicted in Fig. 3. Its core is the knowledge base used by the SELector (NOEMONSEL) to suggest a classification model and algorithm for the input problem. Note that in its current version, NOEMON-SEL performs both selections in one step. The reason is that different algorithms conforming to
the same classification model can have very different performance behaviour. We indicate a classification model and algorithm jointly as “classifier” hereafter.
P-3. Execution time
The establishment and refinement of the knowledge base is performed by the RULE GENerator (NOEMON-RULEGEN). It induces rules concerning the performance behaviour of classifiers for problems that have been processed thus far. New knowledge is induced when a new classifier is registered to NOEMON. The existing knowledge is refined whenever a new classification problem is being processed. In that case, rules that lead to the suggestion of a particular classifier are assigned a grade, the value of which depends on whether the analyst approved or disapproved the suggestion.
One may claim that information concerning resource demand, i.e. memory and/or disk space, is not provided by all classifiers. However, by monitoring the execution of the classifier, resource demands can be measured. Some classifiers should not applied to certain datasets at all. This can be modeled
Data on the performance behaviour of classifiers are retained and managed by the Training Space Manager module (NOEMON-TRAIN). This module owes its name to the fact that it maintains data used to weight up and rank the classifiers in terms of performance for already processed problems. These performance data consititute a multidimensional space, the “Training Space”. Its dimensions are the “morphological” characteristics that describe a problem, and the quality parameters of the performance metric. As problems are being processed, a collection of problem “morphologies” is being established. In the subsequent sections, we describe the modules of the Assistant.
3 A Training Space for Classifiers NOEMON-TRAIN is the module collecting and maintaining the data, from which knowledge is induced and decisions are taken. Those data concern the performance of the classifiers registered with NOEMON for a collection of already processed problems. To describe the data and the vector space they constitute, we have to describe the (a) criteria used to model performance and (b) the characteristics used to model the problems. Typically, classification problems are described in terms of the characteristics of the input dataset. We adhere to this approach and use therefore the term “dataset” instead of “problem” hereafter. 3.1 Performance Criteria We define the “performance” of a classifier as a multidimensional measure comprised of the following scalar measures: P-1. Accuracy P-2. Training time
P-4. Resource demand
by assigning an infinite resource demand to it. Among the criteria included in our list, accuracy measures performance in positive terms, while time and resource demand are negative criteria measuring “cost”. We use the joint term “performance” criteria for them. It should be noted that this list is not claimed to be complete or ideally representative. It is rather intended as a first basis of our experiments. Other criteria may be incorporated. Since we have more than one criterion, the performance space is multidimensional. However, we intend to experiment with the “DEA” methodology, as proposed in [NS97] for the computation of scalar values for multi-criteria metrics. In the referenced study, the “Data Envelopment Analysis” (DEA) is used to compare a number of classifiers for a set of datasets using the accuracy, the testing time and the training time as criteria [NS97]. Notably, the methodology is transparent to the use of criteria with positive or negative impact on the performance, as are the accuracy and the execution time respectively. Hence, this approach is appropriate for NOEMON-TRAIN. One disadvantage of the DEA approach is that different versions of DEA rank the same classifiers differently [NS97]. This is indeed a problem for the selection of the best classifier for one known dataset. However, NOEMON-TRAIN rather suggests a classifier for an unknown dataset, based on the performance of all classifiers on all known datasets. For this, it is initially sufficient to compute consistent performance values using any arbitrary DEA version. At a later phase, we can statistically study the impact of different DEA versions on the decisions made by NOEMONTRAIN. 3.2 Modelling the Dataset Morphology As “dataset morphology” we define a set of structural characteristics that jointly determine the performance of a classifier on a dataset. Several characteristics has been recorded in the literature as having an impact on the performance of a classifier. They include characteristics like the number of attributes and the type (nominal or continuous) of each one, the number of classes, the number of examples in the learning set and
Collection of datasets
Registered Classifiers
TRAINing Space Manager
Performance Criteria
Training Space
RULE-GENerator o o
)
Knowledge Base
Problem database
SELector
SUGGESTED CLASSIFIER
Figure 3 The environment and the modules of the NOEMON Assistant
their distribution in the classes, the noise etc. Some classifiers can be mislead by irrelevant or redundant, i.e. functionally dependent, attributes. Hence, criteria such as the number of irrelevant or redundant attributes could also be considered. Statistical measures like the correlation of attributes and classes, the skewness and the kurtosis, as well as informational measures like the entropy of a class have also been suggested. The problem of specifying the appropriate dataset characteristics that adequately characterize the performance of a classifier can be observed as a feature extraction problem. We need the set of features with the highest discriminatory power. It is desirable to keep this set as small as possible. However, it must be large enough to ensure that no two datasets with the same morphology have different performance values for the same classifier. In Table 1, we present the set of “morphological characteristics”, with which we will start our experiments. Note that for some of the features, e.g. the number of tuples, the use of logarithmic scale might be appropriate. As a first step, we will use a coarse measure of distribution for the morphological features depicted in the second column of Table 1, namely the average value. As a next step, the minimum, maximum and standard deviation values will also be taken into account. 3.3 Establishing the Training Space The morphology characteristics constitute a space, in which each dataset morphology is a vector. For each
such vector, we can compute the performance of each classifier registered with NOEMON. According to the performance criteria of subsection 3.1, the performance is also a vector. However, by applying the DEA methodology, we can map this space into the interval [0; 1]. The space comprised of pairs of morphologies and performance values is called the “Training Space”. We can conceive it as a two-dimensional space, the horizontal dimension X of which describes the (multidimensional) morphology space, while the vertical dimension Y contains the performance values of the classifiers. Each point in this space is annotated with the classifier which produced that point. According to these observations, for each dataset morphology x and list of classifiers C1; : : :; Ck , the performance values for the morphology are the points (x; y1 ); : : :; (x; yk ) in the Training Space. The best classifier for x is an Ci such that yi yj (w.r.t. the total ordering on the interval [0; 1]) for each j = 1; : : :; k. Thus, a partial ordering on the set fC1; : : :; Ckg is induced. This implies that there is not necessarily a single best classifier for a given morphology. NOEMON-TRAIN establishes the Training Space by applying each classifier registered in NOEMON on each dataset of the morphologies’ collection. Performance values corresponding to objective criteria, like execution time, can be assigned automatically. For the evaluation of the performance value for subjective criteria, interaction with the expert is required.
Number of: nominal attributes continuous attributes tuples classes
Distribution of: examples in the classes distinct values in the attributes missing values in the attributes
Fraction of: attributes by classes
Table 1 The characteristics constituting a dataset morphology
The establishment of the Training Space is thus a resource consuming procedure. However, it needs only be performed once per classifier, so that NOEMON can make assessments on its performance. Assessments on the performance of a classifier for an unknown dataset are based on the performance of the classifier for known datasets in the Training Space. The operability of NOEMON depends therefore on the density of the morphology space. Despite the large number of publicly available datasets, the morphology space is very large and its density will grow very slowly. Therefore, we are going to investigate the possibility of generating artificial datasets and apply the classifiers on them, in order to fill the morphology space. In that case, the performance results for an artificial dataset should be replaced with the results for a real dataset, as soon as one becomes available. To extend the Training Space, each new dataset input to NOEMON becomes incorporated into the collection of morphologies. Then, NOEMON-TRAIN invokes all registered classifiers against it. Note that this operation is independent of the suggestions on the most appropriate classifier for this dataset, made by the NOEMON-SEL, as described in the next section. The activities of NOEMON-TRAIN are performed at off-peak times and are indended to increase the density of the morphology space using real data. If NOEMON-TRAIN discovers that a classifier yields different performance results for two datasets with the same morphology, it must undertake actions to reestablish a proper Training Space. The obvious option is to expand or refine the set of morphological characteristics. We will also investigate methodologies of modelling the Training Space in such a way that small performance differences between datasets of the same morphology are tolerated.
NOEMON, the NOEMON-SEL module consults the knowledge base for it, triggers the relevant rules and suggests the most appropriate classifier(s). In our first version of NOEMON, NOEMON-SEL will use a nearest neighbour rule based on the geometrical proximity of morphologies in the morphology space. Hence, NOEMON-SEL can be applied directly on the Training Space, and NOEMON-RULEGEN can be omitted. We describe this approach below, and we then show how NOEMON-RULEGEN will be next incorporated in this context. 4.1 The Notion of Morphological Similarity We define similarity among datasets in terms of geometrical proximity in the morphology space. We introduce the notion of “threshold distance” TDi in the range of each morphological characteristic Ci: values within the threshold distance are “geometrically close”. Then, we observe the morphology space as a conventional Euclidean space. Definition 4.1 Let M1 ; : : :; Mn denote the characteristics constituting a dataset morphology and let TD1 ; : : :; TDn denote their threshold distances. Two datasets D; D0 with respective morphologies < m1 ; : : :; mn > and < m01 ; : : :; m0n > are “similar” or “close (to each other)” if the following inequality holds for each i = 1; : : :; n:
jm ? m0 j TD i
i
i
2 To suggest the most appropriate classifier(s) for a dataset, NOEMON-SEL needs its dataset morphology of D. For this, the morphological characteristics shown in Table 1 should be extracted from the dataset in a preprocessing step.
4 Suggesting a Classifier for a New Dataset
4.2 Finding the Nearest Neighbours of a Dataset
The next module of the Assistant, NOEMONRULEGEN, is responsible for extracting rules from the Training Space and incorporating them into a knowledge base. Then, for each new dataset input to
Using the notion of morphological similarity defined in Def. 4.1, NOEMON-SEL identifies the datasets close to D, denoted as D1 ; : : :; Dk . It selects the best classifier(s) of each one into a list < C1; : : :; Cl >. In
general, l k, since the same classifier may be the best one for more than one dataset morphologies. For each i = 1; : : :; k, let di be the euclidean distance between D and Di in the morphology space, and let yi;j be the performance value of Cj for Di (j = 1; : : :; l). Then, NOEMON-SEL computes the k y . After this “rank” of Cj for D as rankj = i=1 di;j i ranking, the most appropriate classifier(s) for D is the one (or are those) with the highest rank.
P
What should happen if there is no dataset morphologically similar to D? In that case, NOEMON-SEL cannot draw any conclusions on the appropriateness of the available classifiers for D. The only option available is to invoke NOEMON-TRAIN for D, i.e. apply all classifiers to it. In this case, NOEMON cannot assist in the classification task for D. However, after the incorporation of D into the Training Space, NOEMON-SEL can assist in the processing of datasets with a morphology similar to D. 4.3 Obtaining Feedback from the Analyst The activities of NOEMON-SEL, as described thus far, are unsupervised. However, the feedback of the analyst is invaluable, especially if the suggestions of NOEMON-SEL are based on performance results for artificial datasets. Therefore, we intend to extend the Training Space by a dimension G of grade values. The expert may then assign a grade gi;j 2 [?1; 1] to each classifier Cj suggested by a NOEMON-SEL for a dataset Di , for which a performance value yi;j is recorded. Then, the value used for Cj during ranking is gi;j yi;j instead of yi;j . This solution is rather inflexible, because the analyst can provide feedback in a much richer form than simple numbers. In particular, the feedback can take the form of rules concerning a group of morphological characteristics or a group of classifiers. Those rules should be incorporated in the knowledge base of NOEMON. 4.4 The Knowledge Base After completing an initial suite of experiments with the simplified version of NOEMON and after obtaining feedback on its behaviour, we will work on establishing the knowledge base and implement NOEMON-RULEGEN to generate rules for it. The rules of the knowledge base will include:
Rules on (some) classifiers and/or morphologies, given explicitly by the analyst Constraints on the applicability of classifiers to
certain types of datasets, as specified by the classifier implementors or the analyst
Specifications on the settings of the parameters affecting the Training Space, such as threshold distances and settings concerning the application of the DEA methodology Preferences concerning the performance measures and their relative weights Rules extracted by NOEMON-RULEGEN from the Training Space NOEMON-RULEGEN will act as a knowledge discovery module over the Training Space, employing machine learning techniques for classification (or clustering).
5 Conclusion We have presented an overview of the Assistant component of the NOEMON project, which is designed to suggest a classification model and algorithm for a given dataset. NOEMON bases its suggestions on the establishment of a Training Space of performance data for different dataset morphologies, and on the similarity between the new dataset and the existing ones. We model the datasets as vectors in a morphology space, aiming at a minimal and complete set of morphological characteristics that affect performance. To alleviate the problem of multi-dimensional performance measures, we adhere to the exploitation of the DEA methodology, which has already been used to map multidimensional performance values for classifiers into a scalar value interval. We have decided for an initial set of performance criteria, which includes positive measures, as accuracy, and negative ones, like processing time and resource demand. In the first version of its implementation, NOEMON will suggest a classifier for a dataset by ranking the classifiers that perform best for morphologically similarly datasets. The feedback of the analyst on this suggestion will be modeled in the form of grading. In the next version, this simple selection module will be augmented by a machine learning component that establishes a knowledge base from the performance measurement results and the analyst’s feedback. This will turn the selection procedure into a machine learning (classification or clustering) task. We are currently working on the first version of NOEMON, by executing performance measurements for the classifiers in the MLC++ library [Koh96] for a growing collection of publicly available datasets. The NOEMON-SEL module responsible for the suggestion of a classifier is being implemented. Our first experiments will concern the setting and tuning of the
parameters affecting the morphological proximity between datasets.
Acknowledgments We like to extend our thanks to Felix Naumann for his helpful comments.
References [Aha92] D.W. Aha. Generalizing from case studies: A case study. In 9th Int. Machine Learning Conf., 1992. [Bro95] C.E. Brodley. Recursive automatic bias selection for classifier construction. Machine Learning, 20:63–94, 1995. [BS97] C.E. Brodley and P. Smyth. Applying classification algorithms in practice. Statistics and Computing, 1997. to appear. [GB95] J. Gama and P. Brazdil. Characterization of classification algorithms. In Pinto C. Fereira and N. Mamede, editors, Progress in AI, 7th Portugese Conf. in AI (EPIA’95), pages 83–102. Springer Verlag, 1995. [Koh96] Ron Kohavi. Data mining using MLC , a ma. In Tools with AI, chine learning library in C 1996. [Mit97] T. Mitchel. Machine Learning. MacGraw Hill, 1997. [NS97] Gholamreza Nakhaeizadeh and Alexander Schnabl. Development of multi-criteria metrics for the evaluation of data mining algorithms. In KDD’97, pages 37–42. AAAI Press, 1997. [PB95] F. J. Provost and B. G. Buchanan. Inductive policy: The pragmatics of bias selection. Machine Learning, 20:35–61, 1995. [R+ 87] L. Rendell et al. Layered concept learning and dynamically variable bias management. In 10th Int. Joint Conf. on Artificial Intelligence, pages 308– 314, 1987. [S+ 91] J. W. Shavlik et al. Symbolic and neural computation : An experimental approach. Machine Learning, 6:111–114, 1991. [Sal91] S. Salzberg. A nearest hyperrectangle learning method. Machine Learning, 6:251–276, 1991. [Sch93] C. Schaffer. Selecting a classification method by cross-validation. Machine Learning, 13:135–143, 1993. [WK89] S.M. Weis and I. Kapouleas. An empirical comparison of pattern recognition, neural nets and machine learning classification methods. In 11th Int. Joint Conf. in Artificial Intelligence, 1989.
++
++
