ORGANISING KNOWLEDGE REFINEMENT OPERATORS

Appears in

Veri cation and Validation of Knowledge Based Systems, Copyright c Kluwer Academic Publishers. All rights reserved.

1999

ORGANISING KNOWLEDGE REFINEMENT OPERATORS Robin Boswell and Susan Craw School of Computer and Mathematical Sciences The Robert Gordon University Aberdeen, AB25 1HG rab,[email protected]

Abstract

Keywords:

Knowledge re nement tools seek to correct faulty knowledge based systems (KBSs). The goal of the KrustWorks project is to provide a source of re nement components from which specialised re nement tools tailored to the needs of a range of KBSs can be built. Central to the toolkit is a set of generic re nement operators and a representation language for KBS rules. The language abstracts those properties of rule elements which determine whether they can take part in rule chaining, and how they can be re ned. It is used to organise the re nement operators so that each can be applied to similar rule elements found in dierent KBSs. This organisation facilitates the re-use of re nement operators when new KBSs are encountered, and the development of new ones. The toolkit was evaluated by applying it to a KBS shell signi cantly dierent from the ones previously encountered. The rule elements from this shell could be represented within the existing hierarchy, but it was necessary to add some new re nement operators. A KrustTool was then able to x bugs actually occurring in the application, which con rms the usefulness of the re nement operators, and of our approach to organising and applying them. Knowledge Re nement, Knowledge Management, Re nement Operators

149

1.

INTRODUCTION

Knowledge re nement is the process of changing the knowledge in a knowledge-based system (KBS) in reaction to evidence that the KBS is not producing correct solutions. The KBS is evaluated on tasks for which the correct solution has been provided by an expert. Those where the KBS and expert solutions dier provide evidence that the KBS is faulty. Knowledge re nement tools each perform the same general steps. The tool is presented with a faulty KBS and some evidence of faulty behaviour; often this evidence consists of examples that the KBS fails to solve correctly, together with the correct solutions. The re nement tool reacts to a piece of evidence by undertaking the following three tasks: blame allocation determines which rules or parts of rules might be responsible for the faulty behaviour; re nement creation suggests rule modi cations that may correct the faulty behaviour; and re nement selection picks the best of the possible re nements. The goal of re nement is that the re ned KBS correctly solves as many of the examples as possible, with the expectation that novel examples will also have an improved success rate. Most knowledge re nement systems are designed to work with KBSs developed in a single language (Ourston and Mooney, 1994; Richards and Mooney, 1995; Murphy and Pazzani, 1994a), or a particular shell (Ginsberg, 1988). However, it is wasteful to develop re nement tools for individual languages and shells. Instead, we are developing re-usable re nement components that can be applied to a variety of KBS environments. These include re nement operators, re nement lters, and evaluation functions. In this paper we concentrate on a central component: the set of re nement operators. Our aim is to organise a small set of re nement operators so that they can exploit similarities in behaviour between rules in a range of KBS shells. As a result, it should be possible to re-use these operators when a new KBS is encountered, and to develop new operators relatively easily by amending existing ones. In section 2, we describe the particular approach taken by our re nement tools. Section 2.1 explains how the knowledge required to carry out the various re nement tasks can be represented in a generic form, so that the tools can be written in a KBS-independent way, thus widening their applicabilty and facilitating their re-use. The core of the paper is section 3 which concentrates on the particular task of re nement implementation. It develops a hierarchical knowledge representation which abstracts the relevant properties of rule elements, and shows how this hierarchy is used to organise our toolkit of re nement operators. Section 4 150

demonstrates the power of our approach by showing how the re nement operators were successfully applied to a new and signi cantly dierent type of KBS. The remaining sections describe related work and conclusions. 2.

THE KRUSTTOOL APPROACH TO REFINEMENT

This paper is based on our experience with the Krust re nement system (Craw, 1996). We now refer to re nement tools that apply the basic mechanism of the original Krust system as KrustTools. We are developing a KrustWorks framework from which an individual KrustTool for a particular KBS will be assembled; i.e. there is not one unique KrustTool. Figure 1 shows a KrustTool performing the operations highlighted above for a single re nement example. The KBS's problemsolving is analysed, and blame is allocated to the knowledge that has taken part in the faulty solution, or which failed to contribute to the solution as intended. In the KrustTool approach, the next step, re nement creation, is divided into two stages: the generation of abstract re nements, and the implementation of these re nements as actual rule changes. These abstract re nements specify that a rule or condition be specialised or generalised, or that its priority be increased or decreased. At the implementation stage, knowledge speci c re nement operators implement the abstract re nements by making actual repairs to the rules. KrustTools are unusual in proposing many faults and generating many repairs initially, and so a KrustTool applies lters to remove unlikely re nements before any re ned KBSs are implemented. It then evaluates the performance of these re ned KBSs on the training example itself and other examples that are available. 2.1

A GENERIC TOOLKIT

We seek to create a generic re nement toolkit, providing tools to perform the various re nement steps shown in gure 1. These tools should be able to exploit common features shared between dierent KBS, and thus avoid duplicating eort. We can exploit these commonalities by building common representations for 1. the knowledge content of rules, and 2. the reasoning process. The relevant aspects of the knowledge content of rules are two-fold: the ways in which conditions and conclusions can be re ned, and the 151

Faulty Knowledge Base Refinement Example

ALLOCATE BLAME

GENERATE REFINEMENTS Filter

Refinement Operator Toolset

IMPLEMENT REFINEMENTS

All training examples

Filter

Refined Knowledge Base

Figure 1

EVALUATE KBSs & CHOOSE BEST

The Operation of a KrustTool

ways in which they can match, causing rule-chaining. These features determine the design of our knowledge representation language for rules, or knowledge hierarchy. KrustTools use this language to build an internal representation of the static knowledge contained in rules which we call the knowledge skeleton. Secondly, the aspects of the reasoning of a KBS which are relevant to re nement are the ways in which the rules actually red, leading to an incorrect conclusion, and the alternative ways in which they might have red to lead to a correct conclusion. We represent this ring behaviour in an extension of a proof-tree which we call a problem graph, the details of which appear in (Craw and Boswell, 1999). Here we simply note that, to provide the knowledge needed in the construction of the problem graph, the knowledge hierarchy must take account of the matching behaviour for rule elements as well as their re nement operators. Figure 2 relates the knowledge skeleton and problem graph to the re nement steps of a KrustTool. The tool performs the following steps. 1. The tool translates the KBS's rules into its internal representation language, forming the knowledge skeleton. 2. The tool is given a training example, for which the KBS gives an incorrect solution, together with information about how the KBS reaches its conclusions, normally in the form of an execution trace. The tool uses the information to build a problem graph. 152

3. The tool proceeds with re nement generation, implementation and testing using as input the problem graph, knowledge skeleton, and (at the evaluation phase) other training examples. The diagram shows how the knowledge skeleton organises both the matching functions that are used to create the problem graph, and also the re nement operators that are used for re nement implementation. Refinement Example Problem Graph Refinement Algorithm

Knowledge Hierarchy

Propose

Matching Functions

Implement

Execution Trace

KRUSTTool

Knowledge Skeleton

Refinement Operators

Translators

==> f-0 (fact) => Activation 0 FIRE 1 not-artic: f4 ==> f-5 (not-artic) FIRE 2 weight: f3,f4 ==> f6 (total-weight 1) FIRE 3 isa-car: f2 =>> f-7 (vehicle car)

KBS

Figure 2

3.

The KrustTool and KBS Processes

ORGANISING THE OPERATORS

The output of the re nement generator is one of a xed set of abstract re nements, which must then be implemented. We seek to do this in a generic way, using a set of non KBS-speci c re nement operators. We must therefore devise a knowledge representation for KBS rules whose structure is based on those aspects of the rule's behaviour which are relevant to re nement. We can then create a set of re nement operators designed to apply to the various terms in the structure, so that the organisation of the operators will mirror that of the knowledge hierarchy. 3.1

DEVELOPING THE HIERARCHY

The aim of re nement is to cause certain rules which previously succeeded to fail, and vice versa. This can be achieved by modifying individual conditions within rules so that they fail rather than succeed, and so on. However, we note that certain kinds of conditions, such as 153

assignments, can never fail. This property will naturally be represented within our hierarchy. We call conditions which can succeed or fail, or, to put it another way, can be true or false, tests. The second important property of rule elements, the capacity to be involved in rule chaining, must also be represented within our hierarchy. We call rule elements which can be involved in chaining, goals. We are solely concerned with the matching behaviour between the condition of one rule and the conclusion of another, so our de nition of goals is independent of the direction of rule chaining. The organisational requirements of re nement operators have thus led us to create an initial knowledge hierarchy as shown in gure 3. Note that a goal can be true or false, so appears as a sub-class of test. Knowledge Element

Test Can be true or false

Goal Figure 3

Non-test

Can be involved in chaining

Initial subdivision of knowledge elements

We now proceed to develop the knowledge hierarchy further, and de ne re nement operators for the dierent classes in the hierarchy. In general, we continue to partition the classes until each leaf node represents a class of rule element with a well-de ned set of associated re nement operators, and, where applicable, matching functions. 3.2

REFINING TESTS

The function of a re nement operator is to take an abstract re nement, of the form \specialise condition" or \generalise condition", and implement it as an actual change. The purpose of the re nement is to change the behaviour of the condition for a particular re nement case. Moreover KrustTools agree with other re nement programs in preferring conservative re nements. Consequently, \specialise condition" actually means \implement the minimal specialisation to the condition so that it fails when the KBS is run on the re nement case", and similarly for generalisation. We therefore need to de ne re nement operators which perform such specialisations and generalisations for the various types of test we encounter in KBSs. We have already de ned one kind of test: the goal, whose truth value in a rule is determined by the ring of other rules. There are also tests 154

whose truth is determined locally, such as comparisons. We consider re nement operators for non-goals rst, as these are simpler.

Re ning non-goals. The only test we have so far needed to de ne

which is not a goal is the comparison. Comparisons are de ned to be equations or inequalities. The adjust value operator re nes a comparison in a minimal way by \balancing" it so that in the case of generalisation it just succeeds, and in the case of specialisation, it just fails.

Re ning Goals. We rst show that it is possible to implement re ne-

ment operators for the goal class itself, then go on to de ne sub-classes of goal, with more specialised operators. To do so, we have to consider the distinguishing feature of goals: their chaining behaviour. All types of goal succeed if their arguments match the arguments of facts in working memory, the details of the matching process depending on goal type. For example, the condition (enlist ?Student ?Org) matches the fact (enlist student42 army). The matching procedure for this type of goal states that two goals match if they have the same key-word, and the corresponding arguments unify. Matching variables with constants is trivial in this instance, but becomes more interesting if we add metaknowledge to describe the type of the variable. The following example shows how meta-knowledge about an ordered attribute can be used in re ning a goal condition. military deferment(Student) :enlisted(Student, army, [medium,long]).

This rule states that a student obtains military deferment if he enlists in the army for a period lying in the range medium to long. An expert may suggest this rule should re for a case where the period of enlistment is short. To generalise the condition, we make use of meta-knowledge which states that the enlistment period is an ordered attribute with values short, medium, long. The above rule's condition can therefore be generalised to enlisted(Student, army, [short, long]). This re nement operator was applied to a particular goal type, but it was based purely on the way meta-knowledge constrains the matching of goals, not on any property of the particular goal type. It will therefore be inherited by the sub-classes of goal de ned in the next section.

Sub-classes of Goal. There are a variety of goal types which may

be found in dierent KBSs. We identify four sub-classes of goal, distinguished by their chaining behaviour. However, these sub-classes also 155

dier in the way in which they can be re ned, so the partitioning allows us to introduce and organise further re nement operators. consist of a keyword followed by arguments; e.g. the literal colour(sky, blue, light). These do not oer any further opportunity for re nement, unless they contain additional terms as sub-expressions. Such nested terms are discussed below in section 3.3.

Ordered Terms Prolog

are a sub-class of Ordered Term where the keyword is the attribute and the object and value form the remaining two arguments; e.g., colour(sky, blue).

OAV Triples

consist of a keyword followed by a series of attribute-value pairs, e.g. weather(sky, blue, temperature, 40, humidity 5%). An AV Tuple contrasts with an Ordered Term in that the signi cance of each value item is determined not by its position, but by the preceding attribute name. For each keyword, a xed set of attributes must be de ned, though not all the attributes need be used in a rule condition. The variable number of conditions in an AV Tuple allows us to create additional re nement operators associated with this goal type. AV Tuples can be generalised by removing attribute-value pairs, and specialised by adding them.

AV Tuples

take any other goal as an argument; e.g., not(colour(sky, blue)). Negations can be re ned by applying any operator ap-

Negations

plicable to their argument, but reversing its \direction". In other words, a negation is specialised by generalising the goal within it, and vice versa. 3.3

NESTED TERMS

Many rule elements, including those described so far, consist of a single element from the hierarchy, but a rule element can consist of an arbitrarily deep structure. For example, the negation mentioned above takes a goal as an argument. These recursive style rule elements have also proved useful for more complex knowledge formats found in some KBS languages. For example, the following Clips and PowerModel conditions both select students whose debt is less than $1000. Clips version PowerModel version

(student ?name &: (< ?debt 1000)) ?student.debt < 1000;

The Clips condition above will be represented as an Ordered Term with a keyword and two arguments, where the second argument is itself a 156

comparison: (student ?name (< ?debt 1000)). The PowerModel condition will be represented as an OAV Triple, where the value argument is a comparison. When a KrustTool implements re nements to such nested terms, the Ordered Term and OAV Triple operators will be applied to the terms as a whole, and comparison operators will be applied to the comparisons. 4.

EVALUATION

One criterion for evaluating the eectiveness of our organisation of re nement operators is the ease with which they can be applied to new KBS shells. One such shell which we were able to use for evaluation is the product formulation expert system (Pfes). This diers from Clips, PowerModel and Prolog, the models for our approach, in that it is intended for solving design or formulation problems, whereas the expert systems to which we have applied KrustTools have typically solved classi cation problems. Because of its purpose, Pfes has a number of features distinguishing it from some or all of the other shells. The result is not a single class value, but a set of data called a formulation, or in more familiar terms, a recipe. The shell is forward-chaining. The shell makes use of a data structure called an agenda. This is an ordered untyped list which items may be added to or read from. It is used to communicate data between rules, where for example, one rule may generate data, a second may lter it, and a third may use the data that remain. The complex nature of the result was not a problem, since Palmer and Craw, 1995, have already shown that KrustTools can make use of multiple results from a KBS. The forward-chaining has a bearing on the construction of the problem-graph, but we have seen in section 3.1 that it is irrelvant to the organisation and application of re nement operators. The third factor proved more interesting. We show below how we were able to represent Pfes agendas within KrustTools. A further feature of the particular Pfes application which we used was the availability of large numbers of examples. This permitted us to create new inductive re nement operators, which are described below. The Pfes KBS we used for our evaluation was Zeneca Pharmaceuticals' tablet formulation system (Tfs). Tablet formulation is a complex synthesis task, and Tfs is one of the few knowledge based formulation systems in regular commercial use. To create a KrustTool for Pfes, 157

as for any other new KBS, we had to perform two tasks: associate KBS rule elements with existing elements in the knowledge hierarchy, adding new elements to the hierarchy if required; and add further re nement operators if required. Many of the rule elements in Pfes could be immediately identi ed with standard rule elements already found in Krust's knowledge hierarchy. For example, the Pfes condition attribute-has-value specification budget ?budget can be represented as an OAV Triple, and the Pfes condition is ?value greater-than 0 can be represented as a comparison. We next consider agendas, for which the representation was less obvious. 4.1

THE REPRESENTATION OF AGENDAS

Agendas did not initially appear to correspond to any term in the hierarchy. However, it is possible to consider the role of the agenda in a declarative rather than a procedural way; in other words, we can regard the command to add an item to an agenda as an assertion that the item is on the agenda. From this viewpoint, Pfes agendas can be seen as a mechanism for storing attribute-value data, so that they can be represented within the existing hierarchy as Ordered Terms, as the following examples demonstrate. Not all agendas have the same semantics, but the number of dierent possibilities actually employed within Pfes applications is fairly limited. Figure 4 shows an example based on the student loan domain (we use this domain rather than the pharmaceutical one for reasons of commercial con dentiality and ease of understanding). Here, student-enrollment-agenda is a list of students, where each student is followed by his or her period of enrollment. The gure shows the contents of an agenda at some point during the running of Pfes, together with the Pfes rule elements that write to and read from the agenda, and the KrustTool representation of these elements as Ordered Terms. Note that a conclusion that writes to the agenda, and the corresponding condition that reads from it, have the same KrustTool representation, though the two appear dierent in Pfes. This ensures that the existing matching functions for Ordered Terms provide the correct chaining behaviour for agenda-related conditions and conclusions. 4.2

NEW REFINEMENT OPERATORS

We have shown that the existing hierarchy was able to accommodate all the rule elements used in Pfes. However, we found that the set of re nement operators needed extending. Since Pfes rule elements could be represented in the hierarchy, we were able to apply the existing re158

Snapshot of student-enrollment-agenda

student10 2 student18 4 student20 3

Pfes Read/Write Operations on student-enrollment-agenda Conclusions: 1) add ?student to-bottom-of student-enrollment-agenda 2) add ?years to-bottom-of student-enrollment-agenda Conditions:

1) ?student is-on student-enrollment-agenda 2) ?years is-after ?student on student-enrollment-agenda

KrustTool representation of each operation 1) on-agenda(student-enrollment-agenda, ?student) 2) agenda-unlabelled-attribute(student-enrollment-agenda, ?student, ?years)

Figure 4

An agenda and its associated operations

nement operators. However, for some tasks, we found that these were insuÆcient, and we needed to create new inductive operators. Our original design assumed that few training cases would be available, so our initial operators drew their evidence from a single re nement example. However, in the case of Pfes, many examples were available, so we investigated the contribution that could be made by operators which learned from many examples at one time. The approach may be summarised as follows (Boswell, 1998). 1. Select as positive examples those examples that exhibit a particular fault, and as negative examples those that do not. 2. Identify features distinguishing these two groups. 3. Use these features to guide re nement implementation. It proved necessary to add four inductive operators Each operator is applicable to a dierent existing rule element type, thus illustrating how our structuring of the re nement operators made it simple to add and apply new operators as required. The rst operator is described in detail, the rest more brie y. inductive adjust value

re nes comparisons in a similar way to

adjust value, but it learns from all available examples, not just

from the re nement example. The operator examines the traces 159

for all examples where the comparison had the same behaviour (success or failure) as for the re nement case. It then contrasts those where the system conclusion was correct from those where the comparison was blamed for an incorrect system conclusion. From these two classes of traces, it induces a new threshold value for the comparison which will correct the behaviour of all the faulty examples, while not interfering with the correct examples. applies to failed OAV Triples, where the attribute has numeric type. The operator creates a new fact to match the failed goal, taking into consideration any subsequent comparisons applied to the Value element.

inductive add fact

uses a least squares approximation to correct a faulty arithmetic expression.

inductive change formula

is a specialised operator, applicable to rules which re multiple times, thus adding a sequence of items to an agenda. The operator makes multiple copies of the original rule, adding extra conditions to each copy. The eect is that the new rules put the same items on the agenda as the original rule, but in a dierent order.

inductive split rule

Since the extra trace information required by these operators is represented in a KBS-independent form, as a problem graph, the operators are applicable to other KBSs, not just to Pfes. 5.

RELATED WORK

Our knowledge hierarchy describes the rule-elements within a KBS, and so it may be regarded as an ontology for rule-elements which tells a KrustTool which re nement operators may be applied to each element. Our aim is therefore similar to that of Fensel et al., 1997. In their work on problem-solving methods (PSMs), they devise ontologies for both tasks and methods. These ontologies state the abilities of each PSM, and the requirements of each task, thus allowing an adapter to select a method appropriate for a given task. This approach allows PSMs to be written in a task-independent way, and so to be applied to a range of tasks. KrustWorks' re nement operators therefore correspond to the PSMs, and the knowledge hierarchy forms a PSM ontology. It indicates the requirements of each operator by stating the rule element types to which the operator may be applied. This allows the re nement operators to be written in a KBS-independent way, and so applied to a range of KBSs. However, KrustWorks diers from the work of Fensel et al in 160

that it is devoted to the single task of re nement, so there is no need to create a task ontology. KrustTools are distinguished from other re nement tools by their ability to re ne KBSs written in a variety of shells. KrustTools' organisation of re nement operators according to abstract properties of knowledge elements is simply not relevant to tools such as Clips-R (Murphy and Pazzani, 1994b) whose target is Clips knowledge bases, or Either (Ourston and Mooney, 1994) and Forte (Richards and Mooney, 1995) whose target is Prolog KBSs. However, a comparison can be made between the re nement operators oered by these tools. They all use operators which delete rules and conditions, and which specialise and generalise comparisons. In addition, all the tools possess inductive operators for condition addition. Either uses ID3 to induces new rule conditions from example properties. Clips-R creates new rule conditions by copying existing Clips patterns from elsewhere in the KBS and generalising them. Forte uses a path- nding algorithm to learn new rules which expresss complex relationships in terms of simpler ones. What all these inductive operators have in common is that they operate at the level of rules. In contrast, all but one of the KrustTools' inductive re nement operators are linked to particular rule element types, re ecting the importance within KrustTools of identifying the signi cant properties of rule element types. 6.

CONCLUSIONS

Most re nement tools apply to a single KBS environment. We have presented an alternative approach to re nement which uses generic representations for the knowledge contained in rules, and for the reasoning process of a KBS. This has enabled us to build a set of KBS-independent tools, applicable to a variety of KBS environments. In this paper, we have concentrated on the re nement operators within the KrustWorks tool-kit. Our generic forms of knowledge representation allow us to organise these operators so that they may be applied to a variety of KBSs, exploiting similarities of behaviour in syntactically-diering rule elements. This facilitates operator re-use, and the development of new operators. This toolkit was based on our experience developing KrustTools for Clips, PowerModel and Prolog KBSs. We have validated our approach by applying KrustWorks to a signi cantly dierent KBS shell, Pfes. Our experience with Pfes has shown that it was relatively straightforward to develop a KrustTool for this new KBS, and to add and apply new operators when required. Moreover, the tool was then 161

able to perform successfully in debugging and maintaining this KBS. For the simpler bugs, the re nements corresponded exactly to those made by the developers; for more complex changes, the KrustTool found alternatives that were judged acceptable by the domain expert. Acknowledgments

The work described in this paper is supported by EPSRC grant GR/L38387, awarded to Susan Craw. We also thank IntelliCorp Ltd. and Zeneca Pharmaceuticals for their contributions to the project. References

Boswell, R. (1998). Knowledge Re nement for a Formulation System. PhD thesis, School of Computer and Mathematical Sciences, The Robert Gordon University. Craw, S. (1996). Re nement complements veri cation and validation. International Journal of Human-Computer Studies, 44(2):245{256. Craw, S. and Boswell, R. (1999). Representing problem-solving for knowledge re nement. In Proceedings of the Sixteenth National Conference on Arti cial Intelligence. In press. Fensel, D., Motta, E., Decker, S., and Zdrahal, Z. (1997). Using ontologies for de ning tasks, problem-solving methods and their mappings. In Plaza, E. and Benjamins, R., editors, Knowledge Acquisition, Modeling and Management, Proceedings of the 10th European Workshop (EKAW97), pages 113{128, Sant Feliu de Guixols, Spain. Springer. Ginsberg, A. (1988). Automatic Re nement of Expert System Knowledge Bases. Research Notes in Arti cial Intelligence. Pitman, London. Murphy, P. M. and Pazzani, M. J. (1994a). Revision of production system rule-bases. In Cohen, W. W. and Hirsh, H., editors, Proceedings of the Eleventh International Conference on Machine Learning, pages 199{207, New Brunswick, NJ. Morgan Kaufmann. Murphy, P. M. and Pazzani, M. J. (1994b). Revision of production system rule-bases. In Cohen, W. W. and Hirsh, H., editors, Proceedings of the Eleventh International Conference on Machine Learning, pages 199{207, New Brunswick, NJ. Morgan Kaufmann. Ourston, D. and Mooney, R. (1994). Theory re nement combining analytical and empirical methods. Arti cial Intelligence, 66:273{309. Palmer, G. J. and Craw, S. (1995). Utilising explanation to assist the re nement of knowledge-based systems. In Proceedings of the 3rd European Symposium on the Validation and Veri cation of Knowledge Based Systems (EUROVAV-95), pages 201{211. 162

Richards, B. L. and Mooney, R. J. (1995). Re nement of rst-order hornclause domain theories. Machine Learning, 19(2):95{131.

163