The Mincer Metaphor for Problem-Solving Methods

The Mincer Metaphor for Problem-Solving Methods: Making Assumptions for Reasons of Eciency Dieter Fensel Remco Straatman Frank van Harmelen y University of Amsterdam Department of Social Science Informatics (SWI) Roetersstraat 15 1018 WB Amsterdam the Netherlands E-mail: [email protected]

Abstract

In this paper we present the following view on problem-solving methods: Problemsolving methods describe an ecient reasoning strategy to achieve a goal by introducing assumptions about the available domain knowledge and the functionality of the task. These assumptions characterize a problem-solving method.

1 Introduction The concept of a problem-solving method (PPM) is present in a large part of current knowledgeengineering frameworks (e.g. generic tasks [Chandrasekaran, 1986], Role-limiting Methods [Marcus, 1988, Puppe, 1993], kads [Schreiber et al., 1993], method-to-task approach [Musen, 1992], Components of expertise [Steels, 1990], gdm [Terpstra et al., 1993]). The actual form of problem-solving methods diers between frameworks, but in general a PSM describes which reasoning-steps have to be performed, and which type of domain-knowledge is needed to perform a task. Still, a problem-solving method abstracts from implementation details and from a speci c domain. A diagnostic method may require a fault model (i.e., a speci c type of domain knowledge) but the method should be described independent from any application domain such as diagnosis of cars, electronic circuits etc. Furthermore problem-solving methods are described at the knowledge-level [Newell, 1982]. A knowledge-level description describes reasoning in terms of goals to be achieved, elementary reasoning operators, and knowledge needed to achieve these goals and abstracts from details concerned with implementing the system. Problem solving methods are used in a number of ways in knowledge-engineering:  Supported by the Netherlands Computer Science Research Foundation with nancial support from the Netherlands Organization for Scienti c Research (NWO) y Current aliation is: AI Group, Department of Mathematics and Computer Science, Free University of Amsterdam

1

 as a guideline to acquire problem-solving knowledge from an expert;  a description of the essence of the reasoning process of the expert and the knowledge-based

system;  to enable exible reasoning by selecting methods during problem solving;  as a skeletal description of the design model of the knowledge-based system. That is, the knowledge base is developed by structure preserving design and its structure re ects that of the problem-solving method. The concept problem-solving method has greatly stimulated research in the knowledgeengineering community and has resulted in libraries and shells of problem-solving methods [Benjamins, 1993, Breuker & Van de Velde, 1994, Chandrasekaran et al., 1992]. However a question that, according to us, has not been answered clearly is the relation between problemsolving methods and eciency1. Most descriptions of PSM frameworks do point to PSMs as being related to eciency, however no framework makes this relation explicit. Others claim to have no concern for eciency since their problem-solving methods are only used to capture the expert's problem-solving behavior.2 According to us these points of view miss the main point of problem-solving methods: providing the desired functionality in an ecient fashion. A desirable method describes not just any realization of the functionality, but one which takes into account the constraints on the reasoning process. The constraints have to do with the fact that we do not want to achieve the functionality in principle but rather in practice3 in a reasoning system that has limitations in available resources such as space and time. When this relation between PSMs and eciency is ignored or kept as an implicit notion, both the selection and design of PSMs cannot be performed in an informed manner. In the design of PSMs this will result in a design process that has no explicit representation of one of its goals. Such a design process cannot justify its decisions or guarantee that its products are acceptable. In the selection of a PSM the same holds: ignoring one of the criteria will lead to uninformed decisions. After stating the claim that PSMs provide functionality in an ecient way, the next question then is: how do PSMs achieve this? In our view, the way problem solving methods achieve ecient realization of functionality is by making assumptions. The assumptions put restrictions on the context of the problem-solving method, such as the domain and the possible inputs of the method or the precise de nition of the functionality (i.e., the goal which can be achieved by applying the problem-solving method). These restrictions enable correct reasoning to be performed in an ecient manner. For instance, a method for obtaining the cheapest correct We may want to use the term computationally adequacy here since the term eciency tends to invoke thoughts of symbol-level optimizations. However we think that trying to make a problem solvable in a tractable way (a knowledge-level concern) is not fundamentally dierent from symbol-level optimization: the basic goal, dealing with limited resources, is the same; the level of abstraction diers. 2 When people in knowledge engineering make this claim they forget the fact that experts also have to solve the task given their real-life limitations. In fact a large part of expert-knowledge, such as heuristics, is concerned exactly with ecient reasoning given these limitations. Actually, the conceptualization of a domain from an expert diers from the conceptualization of a novice as the conceptualization of the former re ects the learning process which yields eciency in problem solving. Making the assumptions underlying these conceptualizations and the way they lead to ecient reasoning explicit is precisely the purpose of our work. 3 \Rules of substantive rationality that are not backed by executable algorithms are a worthless currency" [Simon, 1981] 1

2

design is starting with an empty design an extending this until it is correct. This method requires the cost-function to be monotonic in the size of the design to work properly. When one selects this method, one makes the assumption that this restriction on the cost-function holds. Analysis of the context of the system has to determine whether the assumption actually holds. The role assumptions play in the ecient realization of functionality suggests that the process of designing PSMs must be based on them. [Akkermans et al., 1993, Wielinga et al., 1995] introduce a general approach that views the construction process of problem-solving methods for knowledge-based systems as an assumption-driven activity. A formal speci cation of a task is derived from informal requirements by introducing assumptions about the problem and the problem space. This task speci cation is re ned into a functional speci cation of the problem-solving method by making assumptions about the problem-solving paradigm and the available domain theory. Further assumption are introduced in the process of de ning an operational speci cation of the method. A task is decomposed into declaratively described subtasks and the data control ow between these subtasks are de ned. We will use this approach as a general framework and try to make it more concrete. Our focus lies thereby on assumptions which are related to eciency of a problem-solving method. We propose to view the process of constructing a PSM for a given function as the process of incrementally adding assumptions that enable ecient reasoning. This way of constructing PSMs does not consist merely of steps that re ne the previous structure of the reasoning system, as is sometimes suggested in literature. Typically, addition of new assumptions that make the method useful changes its structure. We think our view on designing PSMs is more in accordance with observed practice in the construction of problem-solving methods. Summarizing, we want to make the following three claims in this paper:  Problem solving methods are concerned with ecient realization of functionality. This is an important characteristic of PSMs and should be dealt with explicitly.  Problem solving methods achieve this eciency by making assumptions about their context. It is important to make explicit these assumptions to reason about problem-solving methods.  The process of constructing problem solving methods is assumption-based. During this process assumptions are added that facilitate ecient operationalization of the desired functionality. Note that our point of view does not disallow any of the previously mentioned uses of PSMs. The PSM description still identi es what domain-knowledge needs to be elicited (probably even in more detail), and we can also still elicit methods from experts, as long as we make explicit all aspects of the method. In fact, the more precise description of the assumptions of a method, enables another use of PSM descriptions. The description of PSMs used in building a system can serve as a design rationale of the resulting reasoning system. Such a rationale can be used both for explanation and for maintenance of the system. The assumptions of a problem-solving method de ne the relation between the method and the domain knowledge which is required by it. They handle the domain dependency of a problem-solving method in domain-independent terms. The assumptions can be viewed as an index of a method since a method can only be chosen if its assumptions are ful lled by domain knowledge. We can then view the assumptions as proof obligations for the domain knowledge, or an assumption can be viewed as an elicitation task. That is, the assumptions 3

de ne goals for the knowledge acquisition process. Making the assumptions of a problem-solving method over the domain knowledge explicit is a way to deal with the interaction problem. The interaction problem [Bylander & Chandrasekaran, 1988] states that domain knowledge cannot be represented independently of how it will be used in reasoning. Vice versa one could state that a problem-solving method and its speci c variant cannot be constructed independently of assumptions about the available domain knowledge. Developing reusable problem-solving methods as well as reusable domain theories requires therefore:  the explicit representation of the assumptions of the method over the domain knowledge.  the explicit representations of properties of the domain knowledge which can be related to these assumptions. Ontologies (i.e., meta theories of domain theories) are proposed as a means to explicitly represent the commitments of a domain theory ([Wielinga & Schreiber, 1994, Schreiber et al., 1995]). Ontologies introduce generic terminologies which are instantiated by a domain theory. These generic terminologies can be viewed as representations of the ontological commitments of a domain theory. The rest of the paper is structured as follows: In section 2 we de ne in more detail what a PSM consists of and the relation between its parts. In section 3 we further elaborate on our point of view that problem-solving methods provide ecient functionality by imposing requirements based on two example methods. In section 4 we discuss some implications of our viewpoint on the process of developing and reusing problem-solving methods and in section 5 on problemsolving methods itself. Finally in section 6 we discuss the dierences between our view and that proposed in the knowledge engineering literature, software engineering, and complexity analysis. We conclude in section 7 by outlining further research tasks.

2 A Framework for Representing Problem-Solving Methods We will present a framework for representing the dierent aspects of a problem-solving method. Our framework does not drastically deviate from earlier work on PSMs, but rather clari es the relations that hold between the ingredients of a PSM and presents some required extensions.4 In particular we extend the description of PSMs with a description of their costs, and describe the relations between function, operationalization, assumptions, and costs of a PSM. A PSM describes how to achieve a goal in terms of problem-solving behavior. For this reason a PSM consists of two main parts: a functional speci cation and an operational speci cation together with its assumptions and a cost description. The relation of these parts to the environment of the PSM is the following: the functional speci cation should match the goal description of a task and the assumptions should hold for the domain knowledge. Furthermore, the cost described in the cost description should be acceptable for this task. Figure 1 depicts the parts of a PSM and their relations. The functional speci cation, PSMfunctional , is a declarative description of the input-output behavior the PSM was designed for. In this way, the functional description can be seen as a description of what can be achieved by the PSM. The operational speci cation, PSMoperational , describes how to realize the functionality in a reasoning system. The assumptions, PSMassumptions \... when one searches the literature for a precise de nition of problem-solving methods, it becomes clear that there are dierent views." [Wielinga et al., 1993] 4

4

goal

matches

is-acceptable-for

cost description

Problem-solving method

functional specification is-realised-by

has-cost operational specification is-based-on assumptions

hold-true-in

domain knowledge

Figure 1: The PSM and its environment describe conditions under which the structure described in the operational description will achieve the functionality. The relation between functional speci cation, operational speci cation, cost description, and assumptions is essential for understanding PSMs. Given that the requirements hold, the reasoning system de ned by the operational speci cation will exhibit the input-output behavior speci ed in the functional speci cation. One can view the assumptions as the \missing-pieces" in the proof that the behavior of the operational speci cation satis es its goal. When selecting a PSM these assumptions become proof obligations for the correctness and eciency proof of the system. The cost description, PSMcost , describes the costs that are associated with using this PSM (or more speci cally, with PSMoperational ) to achieve the functionality. The cost of a PSM denotes the amount of resources (computing space and time, user interaction, etc.) needed by the method. Given the functional description, the assumptions and the cost description, we are able to express exactly what the utility of the PSM is. We can express what the method provides, namely the functionality described in PSMfunctional , and what it expects in return, namely that which is described in PSMassumptions and PSMcost . Also we describe how we achieve this, namely by the reasoning system described in PSMoperational. We now will elaborate on the speci cation of the four parts of a PSM. The functional description describes the input-output behavior of the PSM. The simplest form of functional description is a list of input-output tuples. However, in practice, this will often not be possible or feasible because of the size of such a speci cation. In practice, some form of mathematical representation of the relation between input and output is needed. In the paper, we use syntactically extended rst-order logic for this purpose. We 5

use syntactical extensions like sets but from a semantical point of view the usual modeltheoretical semantics of rst-order logic can be applied (see [Kifer et al., 1993] for more details). An example for a functional speci cation is given in section 3. We do not want to go into detail on this topic, as the operational description of PSM is the focus of our paper5 . The operational speci cation consists of inferences and the data- and control- ow between them. Our operational speci cation is based on kads [Schreiber et al., 1993]. The inferences specify the reasoning steps that are used to accomplish the functionality of the method. They are described by their input/output relation and can be achieved by a method or a primitive inference. The roles are stores that are used to act as input and output of the inferences. Roles are either domain views in which case they get their values from the domain knowledge, or internal roles in which case they get their values from other inferences. The data- ow connects inputs and outputs of inferences to the roles. Finally, the control of a PSM describes the ordering of execution of the inferences. In this paper we will use a procedural representation for this, based on quanti ed dynamic logic [Harel, 1984, Kozen, 1990]. Therefore, the building blocks of our language are formulas and programs. Atomic formulas are formulas over the contents of roles, and complex formulas can be built in the usual manner. There are two types of primitive programs: (i) a formula followed by a test operator \?", and (ii) programs that assign the output of an inference action to a role: output-role := quantor output-role inference-action(role ; : : :; role ):

Quantors can be m and  selecting either m or all instantiations of the inference action. Complex programs are built up in the usual manner by sequence ;, non-deterministic choice [, and loop . Again, as with our restriction to kads, we do not think that this particular choice for control language is of much importance in this paper. Inferences can be realized by either methods (i.e., the description of a method can be hierarchically composed) or primitive inferences. These primitive inferences are atomic reasoning steps which are not decomposed any further. A primitive inference consists of a description of its inputs and outputs, and the relation between the input and output. In fact one can think of primitive inferences as a special type of problem solving method which has no operational speci cation. The assumptions of a method are both necessary and sucient criteria for the application of the method. The assumptions can de ne restrictions on the possible input of the method, and availability and properties of domain knowledge. Examples of input assumptions are for example the fact that the requirements for a design must not con ict, or that an input-list must be sorted according to some criteria. Example of assumptions on domain knowledge are the availability of heuristics that link violated constraints to possible repair actions, or that a preference relation must describe a complete ordering. In [Fensel, 1995] a number of assumptions were identi ed for an existing problem-solving method. In section 3 we will summarize some of this work. A survey on assumptions used by problemsolving methods in diagnosis can be found in [Benjamins, 1993]. As the assumptions describe properties of the domain knowledge, which is represented in rst-order logic Other work in the AI literature provides ample proposals for notations of the functional description of a reasoning system. See e.g. [Levesque, 1984] for a generic example, and [tenTeije & vanHarmelen, 1994] for a speci c notation for diagnostic reasoning. 5

6

(with some conservative extensions), meta-logic seems a good candidate language for formally specifying the assumptions. The cost of a method will be expressed in terms of the computing time needed by the inferences and the control of the method, as described in PSMoperational . In this paper we will not go into detail on how to measure the eort. We want to mention however, that in the case of typical AI-applications we should be less interested in in the worst case behavior, because a signi cant part of the applied methods are of a heuristic nature which do not improve the worst-case behavior. The worst case is precisely the case where the heuristics do not bring any improvement. Also, average-case behavior analysis often introduces assumptions about the problem distribution which are hardly justi able. Therefore, we want to look at the typical-case complexity and assume the expert as oracle who can provide us with typical cases. Note that we are not necessarily looking for absolute values for the cost descriptions, but rather want to be able to make qualitative judgments on, and comparisons between methods6 . In this section we have described the dierent parts we think are essential in describing PSMs. In section 5 we will discuss the formal relationship between the parts in more detail.

3 Why Are Problem-Solving Methods Necessary? An Example We use the task parametric design and the problem-solving method propose & revise to illustrate the main points of our paper. This choice is in uenced by the fact, that the point of view which is presented here arose from the experiences gathered in the Sisyphus-II project [Schreiber & Birmingham, 1996] aiming at comparing dierent approaches to knowledge engineering. Sisyphus-II de ned the task of con guring a vertical transportation system (an elevator) which was originally described in [Marcus, 1988, Marcus et al., 1988]. [Marcus, 1988] and [Marcus et al., 1988] developed the problem-solving method propose & revise for parametric design to solve this con guration task. In the following, we formally de ne the task parametric design. Then, we de ne a problemsolving method generate & test which can be used to solve this task in principle. This method can be derived straightforwardly from the task speci cation. As this method is very inecient, we derive in a third step a more ecient problem-solving method propose & revise. Propose & revise restricts the task and makes additional assumptions about available domain knowledge in order to gain eciency. The purpose of this section is neither to de ne the appropriate way to specify the task parametric design nor the problem-solving method propose & revise. We only use both to illustrate our ideas on problem-solving methods.

3.1 A Formal Description of Parametric Design

A parametric design problem can be de ned by a problem space, requirements, constraints, and a preference (see [Tank, 1992] for more details). The problem space describes the space which contains all possible designs. The de nition of the problem space is domain-speci c knowledge. Further, a nite set of requirements is assumed to be input delivered by the user. These requirements must be ful lled by a desired design artifact. A design which ful lls all 6

A cost model for kads conceptual models is explained in [Straatman & Beys, 1995].

7

requirements is called a desired design. In addition to the case-speci c user input, a nite set of constraints model additional conditions for a valid design artifact. These constraints are domain knowledge describing the regularities in the domain in which the design artifact is constructed. A design which ful lls all constraints is called a valid design. A design which is desired and valid is called a solution. The preference de nes a preference function on the problem space and can be used to discriminate between several solutions. In the case of parametric design, a design artifact is described by a set of attribute-value pairs. Let a nite set of parameters (i.e. attributes) A = A1 ; : : :; An be given. For each attribute, we assume a set of possible values r1; : : :; rn which de ne its range. We can now use rst-order logic to de ne the task.

De nition 1 (Problem Space) A problem space is a signature S (de ning variables, con-

stants, typed function symbols and typed predicate symbols) together with a standard interpretation I over that signature.

De nition 2 (Requirements and Constraints) The set of requirements and constraints are represented by two sets of well-formed formulas R and C over the signature S . Example We regard the case of con guring an elevator. As attributes are given

the number of persons that can be transported by the elevator and the space which is required by it. We represent the rst attribute by the variable persons and the second attribute by the variable space. X = fpersons; spaceg. The range restriction are rpersons = f1p; : : :; 20pg and rspace = f0:0m2; 0:5m2; 1:0m2; : : :; 20:0m2g. As function symbol, we provide halve : rpersons ! rspace used to de ne a constraint, and interpretation by: halve : rpersons ! rspace with halve(Xp) = 0:5  Xm2 We also provide three binary predicate symbols ,, and =, interpretation as usual. After de ning the problem space, we can now de ne requirements and constraints. The client comes up with two requirements: 1. The elevator should be able to transport 8 persons at the same time. 2. The elevator should not require more than 5m2 space. We model the requirements by the set of formulas R = fpersons  8p; space  5:0m2g As part of the domain knowledge, we assume the following constraint: An elevator needs 0:5m2 for each person which it should be able to transport at the same time. We model this constraint by the set of formulas C = fspace = halve(persons)g Desired designs, valid designs, and possible solution are de ned by means of variable substitutions and models. As we apply the standard de nition of a model we refer to [Lloyd, 1987] for its de nition.

De nition 3 (Possible designs, desired design, valid design, solution)

The set of all possible ground substitutions de nes the set of all possible designs. A desired design is a ground substitution  for which holds I is a model of R. A valid design is a ground substitution  for which holds I is a model of C . A solution is a ground substitution  for which holds I is a model of (R [ C ).

8

It is easy to see that the set of possible solutions is the intersection of the sets of all desired and valid designs. Example With the de nitions of the previous example we get desired designs = f j 8  (persons)  20p ^ 0:0m2  (space)  5m2g, valid designs = fj(persons) = halve((space))g, and solutions = fj(persons) = 8p ^ (space) = 4:0m2, or (persons) = 9p ^ (space) = 4:5m2, or (persons) = 10p ^ (space) = 5:0m2g. By applying the preference P, an optimal solution is selected out of all solutions. De nition 4 (Preference) The preference P is a partial function on all possible designs.

De nition 5 (Optimal solution) An optimal solution is a solution s with: :(9s0 2 solutions j P (s) < P (s0 )) Example We assume possible designs ordered by their space. That is, a possible

design is preferred over another possible design if it requires less space. Therefore we get  with (persons) = 8p ^ (space) = 4:0m2 as optimal solution.

In general, several optimal solutions could exist and one can further distinguish whether the user gets all of them or a non-deterministic selection of some of them. One could extend this de nition by introducing priorities on requirements and constraints, or by distinguishing between constraints which must always hold and constraints which should hold etc. but this is beyond the scope of this paper.

3.2 A Non-Ecient Solution by Generate & Test

A straightforward operationalization of the declarative task speci cation can be achieved by applying a variant of generate & test. This method can be viewed as applying an uninformed theorem prover to our speci c type of task. The method de nes four dierent inferences and four dierent types of knowledge which are required by it. The inference structure of this method is given in gure 2.  The inference action generate requires knowledge which describes what possible designs are. X 2 possible design $ X 2 problem space.  The inference action R-test requires knowledge which describes what desired designs are. X 2 desired design X 2 possible design ^ 8Y (Y 2 requirements ! fulfills(X; Y )).  The inference action C-test requires knowledge which describes what valid designs are. X 2 valid design X 2 possible design ^ :9Y (Y 2 constraints ^ violates(X; Y )).  The inference action select requires knowledge which evaluates solutions, i.e., knowledge which describes what a preferred design is. X 2 optimal solution X 2 solution ^ :9Y (Y 2 solution ^ P (X ) < P (Y )). 9

problem space

generate

possible design

constraints

C-test

R-test

valid design

desired design

requirements

inference data-flow solution input or output internal role domain-view

select

preference

optimal solution

Figure 2: Inference structure of generate & test The advantage of this method is that it clearly separates the dierent types of knowledge which are included in the functional speci cation of the parametric design task. On the other hand, it is precisely this separation which prevents the development of an ecient problem-solver. The knowledge about what is a correct and good solution is clearly separated from the generation step and there is no feedback from the results of the test and evaluation step. The use of the results of the test and evaluation step as guidance for the generation step are precisely the main improvements which are incorporated in the propose & revise method. We now have to complete the method description by de ning its control. Again we do this in a straightforward manner (see gure 3). The control ow speci es the following reasoning process:  First, all possible designs are derived.  Second, all valid designs are derived.  Third, all desired designs are derived.  Fourth, valid and desired designs are intersected.  Fifth, an optimal solution is selected. The sequence of the second and third step is arbitrary and could also be speci ed as parallel activities. It is clear that this method is not very ecient as it has to derive and test all possible designs (i.e., the complete problem space). On the other hand, the method is able to 10

possible design := :possible design generate(possible design); valid design := :valid design C-test(valid design; possible design); desired design := :desired design R-test(desired design; possible design); solution := valid design \ possible design; optimal solution := 1:optimal solution select(optimal solution; solution) Figure 3: Control ow 1 of generate & test achieve the functionality speci ed by the task if the problem space is nite. In regard to in nite problem spaces one has to regard the following. First, at the knowledge level in its original sense one should abstract from all computational concerns like limited space of computation time by describing a completely rational agent. Therefore, it is not at all clear whether an in nite search space should be regarded as a problem. In fact the problem of tractability of the method arises not only for in nite search spaces but already for nite spaces as the size of the space increases exponentially with the number of parameters and their ranges. Dealing with the size of the search space therefore immediately leads to the consideration of limited rationality7 . The signi cant problem of this method arises not only in the case of in nite search spaces but already for nite search spaces. Already for realistic settings with nite search spaces no computational agent can be implemented which realizes the method. Second, one can transform each in nite search space into a nite one by making some pragmatic assumptions. In the example we used the two parameters elevator space and number of persons who should be transported at the same time as problem space. By taking the size (together with a minimal grain size for its measurement) and the population of the city in which the elevator should be used as upper bounds for the parameters we could de ne a nite search space. In general, it should always possible to transform a in nite search space into a nite one by making domainspeci c assumptions. These domain-speci c assumptions improve the eciency of our method by reducing the search space. Third, one could think of reducing the functionality of the method. In the task description (see de nition 5) we required that an optimal solution should be found. A weaker de nition of the functionality of the method is to require that an acceptable solution is a solution which has a preference higher than some threshold t.8 De nition 6 (Acceptable solution) An acceptable solution is a solution s with P (s) > t. The speci cation of the inference action select then becomes: X 2 acceptable solution X 2 solution ^ P (X ) > t 7 Decision procedures with perfect rationality try to nd an optimal solution whereas decision procedures with limited rationality re ect also on the costs to nd such an optimal solution. 8 We see here also the problem of using worst-case analysis: In the worst case it takes the same eort to nd an optimal solution (i.e., a global optimum) as in de nition 5, or an acceptable solution as de ned in de nition 6.

11

The technique of weakening the task de nition to improve the eciency of the computation is commonly used. A well-known example from the problem area of model-based diagnosis is the single-fault assumption (cf. [deKleer & Williams, 1987]). It assumes that the symptoms of a device are caused by one fault. This can be used to improve the eciency of the methods, but disables these methods to deal with situations where the device was suering from several faults. The weakened functionality of de nition 6 enables us to de ne a new control ow for the method which allows the method to deal with in nite problem spaces (see Figure 4). The

repeat

possible design := m :possible design generate(possible design); valid design := :valid design C-test(valid design; possible design); desired design := :desired design R-test(desired design; possible design); solution := valid design \ possible design; optimal solution := 1 :acceptable solution select(acceptable solution; solution) until :;(acceptable solution)? Figure 4: Control ow 2 of generate & test sequence of the four inference actions is repeated until an acceptable solution has been found. The inference action generate derives m possible designs per step which are further on treated by the two test steps and the select step. If generate works randomly we can for each given probability 1 >  > 0 guarantee that the method nds a solution (if there exists one) in nite time with probability > 1 ? . The average amount of time increases with . Making the search space nite by introducing domain-speci c assumptions or reducing the functionality by weakening the solution criteria transforms generate & test into a method which can solve the problem in principle. Still we cannot expect to get an agent which solves this task in a realistic amount of time by implementing the method. Actually, we have not really described a problem-solving method but rather a kind of uninformed theorem prover. Arbitrary generated designs are tested whether they are desired, valid, and preferred. Still, we have an operational description of achieving the goal. From a point of view where one does not want to care about eciency, this could be a legal point to describe the essence of the reasoning process of a system which solves the task. For example, [Rouveirol & Albert, 1994] de ne a knowledge level model of machine learning algorithms by applying the generate & test schema and in [Bredeweg, 1994] it is used to de ne a top-level view on the diagnostic task.

3.3 An Ecient Solution by Propose & Revise

The three main advantages of generate & test as it is developed above are: 12

 It requires only the knowledge given by the functional speci cation.  The four types of knowledge (regarding the requirements also as knowledge) are clearly

separated and each inference uses precisely one knowledge type. The description of the problem space is used in the generation step, the requirements and the constraints are used in two test steps and the preference is used in the select step.  Its inference structure is cycle-free. That is, its operational speci cation does not contain feedback loops which introduce non-monotonicity into the reasoning process. Generate & test leads to a precise and clear distinction of dierent conceptual types of knowledge and de nes the dynamic behavior of the problem-solving process in a highly understandable manner. On the other hand, these advantages are precisely the reasons which cause the inecient problem-solving behavior. The problem-solving method propose & revise, which we discuss in this section, adds eciency to the problem-solving process by regarding the given properties of generate & test as disadvantages:  generate & test requires only the knowledge given by the functional speci cation: An expert includes feedback based on experience from solving earlier design problems.  In generate & test the knowledge about what is a desired, correct, and preferred solution is clearly separated from the generation step: A much more clever strategy is to use these knowledge types to guide the generation step of possible designs. An expert has learned which design decisions led to desired, valid, and preferred solutions and which did not. Therefore, expertise compiles test knowledge into the generation step. New types of knowledge arise which enable the ecient generation of solutions.  There is no dynamic feedback in generate & test from the results of the test and evaluation step of a given design: If a design derived during problem solving is not a solution, a new design is derived in the next step. Dynamic feedback would include the reported shortcomings of the rst proposed design as guidance for its modi cation by the next derived design. The use of the test and evaluation knowledge as guidance for the generation step by new types of knowledge and the use of the feedback of the test step as input for the generation step are precisely the main improvements which are incorporated into the propose & revise method. In a pessimistic manner this can be expressed as destroying the clear conceptual distinctions of generate & test. Optimistically, this can be viewed as introducing new types of knowledge which glue these dierent aspects together and which add expertise. Another improvement comes in by dierentiating the generate step into two dierent activities. The propose step derives an initial design based on the requirements and the revise step tries to improve an incorrect design based on the feedback of the test step. It therefore uses the meta-information that this design is incorrect as well as the reported constraint violations of test as knowledge source. As the generate step is split up into two dierent activities the requirement for available design knowledge has to be reformulated:  The propose step requires propose knowledge which enables it to derive desired designs using the requirements as input. X 2 desired design Rs = requirements ^ propose(Rs; X ) 13

 The revise step requires repair knowledge which enables it to x constraint violations of

desired designs. Additionally it uses the reported violations to guide the repair process. Revise delivers an acceptable solution as its output.

X 2 acceptable solution Z 2 acceptable solution

X 2 desired design ^ violations(X; ;) X 2 desired design ^ violations(X; V s) ^ repair(X; V s; Z )

Figure 5 shows the inference structure of propose & revise. The third inference action test requires constraints to check desired designs for violations. As output it derives for each desired design the set of constraints which are violated by it.

violations(X; V s)

X 2 desired design ^ V s = fY 2 constraintsjviolates(X; Y )g

Propose & revise requires a number of additional assumptions to justify its I/O behavior against generate

revise knowledge

revise

acceptable solution

violations

requirements

inference data-flow

propose

desired design

test

constraints

propose knowledge

input or output internal role domain-view

Figure 5: Inference structure of propose & revise the speci ed task. Propose & revise no longer contains an R-test. That is, the designs are no longer checked whether they meet the requirements. Propose is assumed to derive already desired designs instead of possible designs as delivered by the generate step. It is also assumed that the revise step delivers designs which are desired (assumptions about the domain speci c repair knowledge) or that requirements violations which are not xed by it must be accepted (weakening the functionality of the method). Additionally, propose & revise no longer contains a selection of a solution using the preference. That is, it is either assumed that the propose step as well as the revise step deliver acceptable (or optimal) solutions or that the functionality of the task is reduced to nding just a solution. The propose knowledge enables the derivation of designs which meet the user requirements, which have the highest preference, and which contain only a few constraints violations. Propose 14

already derives desired designs instead of possible designs. The test step only has to check the constraint violations. User requirements are ful lled or if not they cannot be ful lled. In addition, the step has to deliver the violated constraints. Given the violations, the revise step has x knowledge which allows an ecient modi cation of the design which repairs the violations and change the design in a way such that it still meets the user requirements possible and it still leads to most preferable designs. The revise step tries to improve an incorrect design based on the feedback of the test step: It uses the meta-information that this design is incorrect as well as the reported violation of test as knowledge source. If we take a closer look at revise by distinguishing several sub-steps of it we would see that the test inference appears also as sub-step of revise (cf. [Fensel, 1995]). After applying some repair rules on an invalid design, revise has to check whether the given violations are overcome and whether no new violations are introduced by applying the repair rules. Again, test knowledge which was originally separated from the generation step appears now as sub-activity of it. The revise step causes the main eort in precisely specifying the behavior of the method as well as its main computational eort. The actual eciency of the method therefore heavily relies on the quality of the repair rules which are required by revise but also on the propose knowledge. The propose knowledge is responsible for ensuring preferred desired designs which require less repair activities. The main point of the method in gaining eciency is not so much to get rid of the R-test and selection step but to reduce the search space from the set of all possible designs (i.e., the complete problem space) to the set of preferred desired designs (see gure 6) which should be nearly valid. Regarding the restructuring of the inference structure by propose & revise, the possible designs = problem space

preferred and desired designs

preferred designs

valid designs desired designs

solutions search space of generate & test search space of propose & revise

Figure 6: Topography of the search space possible control ows I and II are shown in gure 7. Control ow I tries to nd an acceptable solution in one attempt (i.e., the assumption is that this can be done) whereas control ow II includes a loop of propose, test, and revise until an acceptable solution is found. The propose step as well as the revise step glue types of knowledge together which were treated separately by generate & test. These new knowledge types de ne strong assumptions about domain knowledge required by the method. The only reason for doing this is trying to gain eciency. That is, we assume that the \re ned" problem-solving method propose & revise will be able to nd a solution faster than generate & test (or a better solution in the 15

Control ow 1 of propose & revise desired design := m :desired design propose(desired design; requirements); violations := :violations test(violations; desired design); acceptable solution := 1:acceptable solution revise(acceptable solution; desired design; violations);

Control ow 2 of propose & revise

repeat

desired design := m :desired design propose(desired design; requirements); violations := :violations test(violations; desired design); acceptable solution := 1 :acceptable solution revise(acceptable solution; desired design; violations); until :;(acceptable solution)? Figure 7: Dierent control ows of propose & revise

same amount of time). Therefore, developing problem-solving methods means to blur conceptual distinctions and to introduce assumptions about new types of domain knowledge for reasons of eciency. The pure and very clear separation of four types of knowledge in generate & test is destroyed by trying to press parts of the test and evaluation knowledge into the generation step in order to improve its eciency and therefore the eciency of the complete method. Furthermore, this is only one possible transformation of propose & revise. We still have made the assumption, that propose and revise works with complete designs. But as discussed in [Marcus et al., 1988] it could also make sense to regard repair activities as soon as possible. That is, instead of proposing a complete design which then is repaired, we can also incrementally develop a design and repair at each step where a constraint violations occurs. Actually we have not yet exploited the fact, that we specify propose & revise for a subclass of design tasks, i.e. for parametric designs. A natural decomposition of the entire design artifact is provided by the parameters which describe it. In each propose step we can assign one or some parameters a value and then we can apply revise to these incomplete designs before we propose the next parameter values. This divide & conquer strategy with intermediate repairs requires that the constraints do not interact much (see [Marcus, 1988]). Otherwise, we always have to redo older repair activities when new constraints violations are reported for a further parameter. Actually it is the same assumption which underlies all approaches on hierarchical task decomposition. It is only useful if the subtasks in dierent steps to not interact heavily. The stepwise derivation of incomplete designs requires the introduction of the new inferences select-parameter and check16

completeness. This stepwise derivation of the design leads to even more \structure altering" transformations of generate & test and propose & revise which result in even more eciency improvements.

3.4 Resume

All in all we can conclude that propose & revise provides the same or less functionality as generate & test. It makes stronger assumptions to achieve this functionality. Also, it is much harder to understand in detail than generate & test. Especially the revise step requires several levels of re nement to de ne it precisely (see [Fensel, 1995]) and \the non-monotonic nature of the Propose and Revise method is dicult to capture in intuitively understandable theories."[Wielinga et al., 1995]. Given this we must face the fact that the only reason why we still would prefer propose & revise is for reasons of eciency. The rst draft title of this paper was \stepwise re nement of problem-solving methods". However, it should be clear that this cannot be guaranteed. The process of constructing a PSM need not at all be a re nement process. Instead, the conceptual structure could be changed in order to allow ecient reasoning. We will examine this issue more deeply in a successor paper.

4 The Development Process of Problem-Solving Methods The development of problem-solving methods is a model construction process. As we aim on reusing generic model components we can apply a process model of component-based design (see [Top & Akkermans, 1994]). It consists of three main activities:  Specify: The intention of the user is translated into explicit (i.e., formally described) requirements. The outcome of this phase is a functional speci cation of the system or more speci c a competence theory of the problem-solving method. Already in this phase unrealistic expectations on the competence of the system get transformed into more realistic requirements on computational agent. Still, the main aspect of this phase is the proper de nition of the functionality of the system and not its computational realization. Outcome of this phase is a task de nition and a functional speci cation of a problemsolving method.  Construct: An operational problem-solving method is constructed which describes how the functionality is achieved in an ecient manner. Assumptions are introduced over domain knowledge required by the method. As already discussed this can lead to changes (weakening) of the functionality but the main aspect of this phase is not the functionality but the development of a way to achieve the functionality. Outcome of this phase is an operational speci cation of a problem- solving method together with its assumptions and costs.  Assess: The operational speci cation and its functionality must be compared with the required functionality. Possibly, this assessment can result in a set of violated requirements. These violated requirements are input of the specify and construct steps in the next iteration of the process. As process model of these three sub-activities we get a cyclic iteration of the three activities which corresponds to process models in software engineering [Boehm, 1988, Floyd et al., 1989], 17

knowledge engineering [Morik, 1987, Neubert & Studer, 1992], or component-based design [Breuker & Van de Velde, 1994]. Speci c characteristics are:

 Generic descriptions of the functionality of problem-solving methods as de ned for diagnostic tasks can be used as input for the speci cation step.  Generic schemas and transformation operations as sketched in this section can be used as input for the construction step.

The control strategy of the entire process could be a variant of propose & revise9 . An initial functional and operational speci cation is proposed and the violated requirements (or violated assumptions) are used as input for the next iteration of the process.

4.1 The Role of an Operational Speci cation of a Problem-Solving Method: The Mincer Metaphor

An operational speci cation of a problem-solving method translates a goal description (i.e., a task) into a set of assumptions on domain knowledge required to solve the task eciently. It consists of inference actions, the data ow between the inferences, and the control ow de ned on these inferences. It describes an abstract algorithm which achieves a goal if its assumptions are ful lled. The algorithmic description is not complete with regard to its elementary steps. The inference actions are described functionally only. Therefore, it is not speci ed how these inference actions can be computed. At a given level of granularity, the algorithmic realization of an inference action is viewed as internal detail which should not be speci ed. The operational speci cation de nes therefore an intermediate level. It allows to specify as much algorithmic aspects as necessary without enforcing a complete algorithmic de nition. From our point of view an operational speci cation of a problem-solving method should not be viewed as a high-level design model of the knowledge-based systems. The development process of ecient problem solver is not a structure-preserving transformation process. Instead, the structure of the operational speci cation is changed by introducing new assumptions or by strengthening earlier ones. We expect similar transitions from the speci cation of a problem-solving method to a design model which describes an ecient implementation. Therefore, we expect neither that the inferences of the problem-solving methods correspond to separate buildings blocks of the design model nor do we assume that the control de ned by the problem-solving method correspond to the actual control ow of the implementation. We will illustrate the problem by an example given in [Eriksson et al., 1995]. [Eriksson et al., 1995] describe and compare dierent problem-solving methods. One of these methods is the board-game method which can be used to solve one-player board games but also assignment problems as the room-allocation problem of Sisyphus-I [(ed.), 1994]. When de ning a move inference action of the board-game method applied to the towers-of-Hanoi game they discuss the following alternative:  The possible-move inference delivers only strictly legal moves.  The possible-move inference delivers potential -but not necessarily strictly legal- moves, and a second inference action contradiction prunes illegal states by having knowledge over illegal states. 9

This idea originates from Annette ten Teije and Joost Breuker.

18

[Eriksson et al., 1995] state the following dilemma: The latter con guration of the boardgame method is computationally less ecient than the former, because an implementation must generate and reject contradicting states that are irrelevant for the goal. However, for certain games, the con guration of the board-game method in terms of both possible-move and contradiction simpli es the software engineering considerably. They use the cannibals and missionaries problem as an example where possible-move describe the transportation and contradiction describes illegal states of the locations. Another example is chess where possiblemove could de ne the possible moves for each piece and contradiction rules out possible moves according to the current situation of the board. Each inference deals with completely dierent types of knowledge. Again for eciency reason this is not at all a clever implementation strategy. Keeping the conceptual structure of the problem-solving method identical with the structure of the implemented system leads either to inecient implementations or speci cations which are hard to understand as they re ect eciency aspects which are related with implementation issues. Gaining eciency cannot only be achieved by local optimization of single inference steps but by global optimizations which change the conceptual structure of the problem-solving method. Therefore the operational description does not deliver a high-level design model of the knowledge-based system. As a consequence the question may arise: what is the purpose of an operational description?. From our point of view an operational description of a problem-solving method gives the rationale of the assumption it makes about the required domain knowledge. The conceptual description of the method de nes the translation of the goal in the assumptions about knowledge required to achieve it (see gure 8). The conceptual description justi es the necessity to make these assumptions in order to achieve a level of eciency in ful lling the goal. Goal

assumption 1 assumption n

Goal’

Figure 8: The mincer metaphor The development process of the operational description of a problem solving method is the process which derives these assumptions. The development process starts with operational descriptions with little concern about eciency and little assumptions like the generate & test 19

version in section 3. Then, in a stepwise process additional assumptions are introduced and the conceptual structure of the method is changed accordingly. The cascade of operational descriptions with increasing concern about eciency delivers the rationale of the assumptions. It reports at which stage of the development process the assumptions were introduced and which consequences they have. Therefore, a single operational description and the transition process which led to it describe the role of an assumption in achieving the goal eciently. As the development process of appropriate problem-solving methods is a modeling activity we do not assume a linear development process. Instead, we assume a cyclic process of introducing, modifying, and revising assumptions. By documenting this process, we also keep the rationale of the assumptions (see [Parnas & Clements, 1986, Landes & Studer, 1995]). The precise de nition of the task heavily in uences the computational complexity of the problem and the assumptions which have to be made in order to achieve some level of eciency. Therefore the construction process of problem-solving methods is not only a process of developing operational speci cations and their assumptions but it can also include the revision of the functionality (and therefore the task which can be solved by the method) to a subproblem of the original given problem. In section 3 we used the example of reducing the task from nding an optimal solution to nding an acceptable solution. Developing ecient problem-solving methods can therefore be viewed as a dual process of introducing assumptions about domain knowledge and of weakening the task. In the latter case, the functionality of the method is directly weakened. But also the former case can be viewed as a process of weakening the functionality of the method as it can only applied to case where the assumptions hold. Therefore, its functionality is restricted by introducing assumptions. Naturally this viewpoint on designing PSMs brings up questions about the relation to implementation design:  Essentially the process of going from the conceptual model to the implementation will be the same as that of going from the initial PSM model to the acceptable one. Both processes consist of making transformations, based on assumptions.  However, the information that is used during implementation will be a superset of the information that is available during the design of the conceptual model. The additional information available during implementation design concerns information about aspects such as operating systems, machine limitations etc.  Also, the type of transformations will be dierent. Whereas all transformations in PSM design are between PSM descriptions, implementation transformations can be between PSM descriptions and implementation language as well. We do not see the necessity or feasibility of structure preserving design for constructing an implementation. Structure preserving design is not necessary since the process makes explicit the design steps and their assumptions. This enables explanation and modi cation of the implementation, which are the main reasons for structure-preserving design. We do not think structure-preserving design is feasible, since a lot of decisions cannot be made during PSM construction because of the restricted information available at this level. The information available during implementation design is a superset of that available during the PSM design in two ways. First, information is added that was not present during PSM design (e.g. the speci c implementation language). Second, information that was present in an abstract manner at during PSM design, may be present in a more speci c form during 20

implementation design. An example of this is the notion of computing time, which is expressed at an abstract level during PSM design (e.g. complexity measures) but can be made more speci c (e.g. estimates of CPU-seconds) during implementation design. Note that because of this, the design process assumes that the decisions made during PSM design either do not con ict with possible implementation design decisions, or that these decisions have more impact.

4.2 ADAM: Assumption-Driven Adaptation of problem-solving Methods

We view the construction of problem-solving methods as an assumption-based process. The general idea is that construction of a PSM is a process of transforming an initial, purely functional, PSM description into an operational one by making assumptions. We will call this process adam (Assumption Driven Adaption of problem-solving Methods). We envision a construction process that starts out by making an analysis of the problemspace of the task. Based on this, a PSM description is made. This consists of a functional speci cation based on the problem space and a \weak" operational speci cation. That is, the operationalization is based of the functional speci cation and has a general \weak" problem solving strategy. The subsequent steps consist of transforming the PSM into a more acceptable PSM in terms of the requirements (computing time, space and other resources). The process stops when an acceptable PSM is reached. The parametric design example in section 3 can be seen as an example of this. First we analyze the problem space, which leads to the creation of a rst PSM, generate & test. By making additional assumptions we arrive at an acceptable method, propose & revise. This process is depicted in gure 9. The transformations will be

problem space

f

f

o

o

a

a

psm #1

psm #2

f

functional specification

o

operational specification

a

assumptions

f

...

o a psm #n

problem solving method transformation

Figure 9: The PSM design process aimed at improving the current PSM with respect to eciency (or other resource use). A transformation will add assumptions to the PSM, and can change the operational description, the functional description, or both. In the knowledge engineering literature the problem of designing an appropriate problemsolving method for a given application is mostly viewed as a selection or hierarchical design problem.  Selecting an appropriate problem-solving method out of a library of problem-solving methods [Breuker & Van de Velde, 1994].  Constructing a problem-solving method by hierarchical re nement of its inferences (i.e., subtasks) [Terpstra et al., 1993]. 21

Both approaches view the development process of an appropriate problem-solving method for a given application as a routine design problem. In the rst case, the problem-solving method can be selected out a set of given methods and in the second case the problem of designing a method can be solved by hierarchical design. We take a more pessimistic point of view as what we want to provide as starting point are general schemes of problem-solving methods de ning their functionality and an operational description with very weak assumptions (like partially interpreted schemes and their assertions in the PARIS system [Katz et al., 1989]). For instance for propose & revise there exist many variants with slightly dierent assumptions and behavior (see [Fensel, 1995]). Putting all variants into a library would lead to non-manageable libraries. Therefore, we can provide only a general schema covering all these variants. In addition, modeling operators [Van de Velde, 1994] are provided which transform these schemes into more concrete description of problem-solving methods by introducing new assumptions. We aim on supporting the creative process of developing problem-solving methods by these modeling operators. [Aben, 1995] developed the same idea for a library of elementary inference actions. Instead of storing e.g. the inference action select or all its variants in a library [Aben, 1995] provides general schemas of such elementary inferences and operators which can be used to modify their precise de nitions. As such a development process is not mainly a process of hierarchical re nement of inferences but a process which can lead to drastic changes of the overall inference structure and the control we cannot rely on supporting only hierarchical re nement as done in other approaches. The functionality and the assumptions of a method can be used as an index during the development process. The functionality of the method is relevant for a goal-driven development as it de nes the range of tasks to which the method is applicable. The assumptions of the method are relevant for a development process driven by the available domain knowledge as this knowledge must ful ll these assumptions.

5 Problem-Solving Methods As Product A problem-solving method relates a goal (a task) with required domain knowledge to achieve the task (see gure 1). As discussed before, the description of a problem-solving method consists again of two parts: (1) a speci cation of its functionality which de nes the link from the task to the method. (2) An operational description of the method speci es a way to realize the functionality. Part of the operational speci cations are assumptions about the domain knowledge required to realize the functionality in an ecient manner. These assumptions de ne the link of the method to the domain knowledge. The external and internal relationships of a problem-solving method de ne three dierent types of proof obligations. A fourth type of proof concerns the eciency of the chosen method. First, the external relation between the goal (i.e., the task) and the functionality of the problem-solving method has to be established. One has to ensure that the functionality of the method is strong enough to ful ll the goal. Representing the functional description and the goal by predicates over input and output one has to ensure that:

j= 8I 8O(precondition(I ) ! (psmdec (I; O) $ goal(I; O)))

Second, the external relation between the method and the domain knowledge has to be estab-

lished. One has to ensure that the domain knowledge ful lls the assumptions of the method. 22

Depending on the type of assumptions we get either:

domain knowledge j= assumptions; or domain knowledge j6= :assumptions In the rst case we enforce that the assumption is ful lled. In the second case, we enforce that the assumption is not violated by the domain knowledge. Third, the internal relationship between the declarative and operational descriptions of the method has to be established. One has to ensure, that given the assumptions the operational descriptions describes a way to achieve the functionality. As the description of the operational speci cation requires a logic over states, we use dynamic logic ([Harel, 1984, Kozen, 1990]) to formalize this obligation:

j= 8I : (precondition(I ) ^ assumptions) ! hpsmopiTrue j= 8I 9O : (precondition(I ) ^ assumptions) ! [psmop]psmdec(I; O) The rst obligation ensures the termination of the program and the second obligation ensures that the desired functionality is provided. Fourth, a statement about the eciency of the method has to be made. In the ideal case, we want to have that, given the assumptions, each other operational description requires at least the same eort as the chosen one to achieve the functionality of the method. A simpler obligation is to proof a statement about the expected eciency of a chosen method. Related work is done by [Poeck & Puppe, 1992] who discuss ecient variants of problem-solving methods for assignment tasks; [Eriksson et al., 1995] who did a worst-case analysis for some problem-solving methods; and [Straatman & Beys, 1995] who developed a calculus to express computational properties of problem-solving methods. [Aben, 1993] presents a framework for describing primitive inferences which has parallels with our framework for PSMs. The inferences are described in terms of preconditions, body, and postconditions. The postcondition would correspond to our functional description, the precondition to our assumptions, and the body to our operational speci cation. One dierence is that our operational description takes control into account, which (of course) is absent when describing primitive inferences. Also no notion of cost is present in this framework.

6 Related Work In this section we will brie y discuss some related work from various research elds. We will start by comparing our work with other work in knowledge-engineering. After this we will take a look at related work in software engineering and complexity theory.

6.1 Related work in Knowledge Engineering

We brie y discuss the dierences between our approach and other work on problem-solving methods in the literature. Relevant is rst of all the work by Newell [Newell, 1982], whose \knowledge level hypothesis" is the basis for much of the current work on problem-solving methods. The major dierence between our work and the view of Newell is that Newell was concerned with ideal rationality as an idealized concept. The knowledge level is concerned with purely functional speci cations, whereas we agree with Simon that \Rules of substantive 23

rationality that are not backed by executable algorithms are a worthless currency" [Simon, 1981]. A re nement of Newell's work that does take eciency into account is the work on bounded rationality (e.g. [Russell & Subramanian, 1995]). An important notion there is the utility of a reasoning process, where the bene t of reasoning is charged against its costs. Such notions also play a role in our framework (in particular the use of the cost model in section 2). However, the notions in the work on bounded rationality are rather general, and, for instance, do not consider the introduction or re nement of domain speci c assumptions in order to increase utility. Although not treated in the work of Newell, much of the work that is based on Newell's knowledge-level hypothesis does indeed take eciency into account10. Work on such theories about problem-solving methods as role-limiting methods [Marcus, 1988], generic tasks [Chandrasekaran, 1986, Chandrasekaran et al., 1992], the method-to task approach [Musen, 1989], generalized directive models [Terpstra et al., 1993, vanHeijst et al., 1992], components of expertise [Steels, 1990] and kads [Wielinga et al., 1992, Schreiber et al., 1994] all deal with the matter of eciency in one way or another. However, many of these approaches treat eciency implicitly. We will brie y sketch the dierences between our approach and these works. role-limiting methods are clearly aimed at providing ecient problem solvers. Unlike our approach, this concern is so strong that no separation is made between eciency and functionality. In their terminology, a problem-solving method is \an algorithm which determines how domain-speci c knowledge is used for solving problems." [Puppe, 1993]. From our point of view a problem-solving method de nes a class of possible algorithms by describing their functionality, their common assumptions and their expected eciency. Originally, role-limiting methods were building blocks that solved an entire task. Currently, work is done to extend the application scope of role-limiting methods by breaking them down into components with smaller grain size [Poeck & Gappa, 1993]. A complete problem-solver for a given task can then be con gured out of these elementary components. Therefore, the selection problem of appropriate components becomes much more serious. In addition, the complete problem solver is now the result of a con guration activity. Selecting a component and con guring a complete problem solver involves two questions: The functionality of the methods must t to the given task (or subtask) and their assumptions about domain and task must be ful lled in order to enable eciency. We assume that signi cant progress in this direction can only be achieved by systematic explication of underlying assumptions of these methods and their relation to the expected eciency as we propose in our paper. The work on generic tasks is very close in many respects to our own, and the same types of knowledge are distinguished: a domain independent speci cation of the task, subtask decomposition knowledge, search-control knowledge and method selection knowledge which roughly corresponds to the assumptions in our approach. Also shared with the generic task approach is the belief that the task structure is only meant as an analytic tool and that it should not be taken as the structure of the design or implementation. The main dierence between our approach and generic tasks is that we have argued against the hierarchical nature of the Actually, the introduction of eciency aspects at the knowledge level can already be found as a side aspect in [Newell, 1982]. He discusses the case where several actions lead to the same goal or several goals have to be achieved. In this context, he makes a short remark: \If the agent has general goals of minimizing eort, minimizing cost, or doing things in a simple way, these general goals select out a speci c action from a set of otherwise equipotent task-speci c actions." [Newell, 1982]. 10

24

decomposition process of tasks into subtasks. This same point (the lack of belief in hierarchical decomposition as a process to obtain realistic problem-solving methods) also characterizes the main dierence with the work in the method-to task approach. With generalized directive models we share the notion of assumptions that must be introduced during the construction of a problem-solving methods. These assumptions are embodied in the GDM formalism in the form of applicability conditions on the grammar rules that perform the transformations on the problem-solving method. However, the nature of the GDM grammar enforces a hierarchical and local decomposition process, which would not be able to capture the transformations that were required in section 3 to obtain propose & revise. The components of expertise approach shares our concern with the pragmatics of problem-solving methods, and distinguishes a conceptual and a pragmatic viewpoint on methods (see [Steels, 1990]). At the conceptual level the general type of inference of the system is described. The pragmatic level can be used to express limitations as of time and space or of observation. Therefore, the pragmatic level introduces the idea of bounded rationality in the components of expertise framework. A problem-solving method is a "knowledge-level characterization" of the problem-solving process of the system. As in role-limiting methods and in dierence to our point of view a problem-solving method is viewed as an algorithm: "A method is an algorithm. It contains a series of activities and a control ow de ned over these activities." [Steels, 1993]. [Van de Velde, 1991] re nes the principle of rationality into a two-step procedure. In a rst step knowledge is con gured into a model of the actual task set to the agent (which is called knowledge use level in [Steels, 1990]). In the second step the principle of rationality can then be applied within the boundaries of the task model. A problem-solving method is speci ed by three components in this framework: a task decomposition into subtasks, a competence theory (i.e., a de nition of the functionality of a problem-solving method), and a specialized principle of rationality (also called control). Obviously, this kind of rationality is connected with the eciency of the problem solver. The dierence is that we make explicit the role of assumptions in this process. In kads, the concern with the eciency of problem-solving is less prominent than in components of expertise. The connection between problem-solving methods and computational eciency is proposed as epistemological and computational adequacy [Schreiber et al., 1991]. A problem-solving method speci es role limitations in the use of knowledge. The combinatorial explosion of inferences in unstructured knowledge bases is thus prevented [Wielinga et al., 1993]. Thus, the assumption is that epistemological role limitations as described by a knowledge-level model are connected to computational access limitations. This point was further worked out in [Akkermans et al., 1993, Wielinga et al., 1995]. There, the construction process of problemsolving methods is also seen as an assumption driven activity, and a formal speci cation is derived from informal requirements by introducing assumptions about the problem and the problem space. Our work is based on the work by [Wielinga et al., 1995]. In their work, PSM construction is seen as re nement of competence theories. A competence theory expresses the relation between input and output of a task. This process is seen as consisting of three steps: 1. Speci cation of the problem-space and of the requirements for the solution. This yields a required competence theory T0 for the PSM. 2. Conceptual re nement of this competence theory introducing the intermediate task and domain vocabulary based on assumptions regarding the available 25

domain theory and the ways the task goal is achieved by the method. This leads to a re ned competence theory T1 of a PSM. 3. Operationalization of this re ned competence theory to inference structures and associated control regimes that are suciently detailed and practical to act as a basis for KBS design. This entails further assumptions of an operational nature and yields an operational speci cation T2 of a PSM. ([Wielinga et al., 1995], page 2{3) Besides the high degree of correspondence our work also diers from this work in some aspects:  introduction of eciency as an explicit requirement of this design process.  making explicit the relation between the assumptions, the functional, cost, and operational speci cation of the PSM.  instead of re ning competence theories and transforming these into operational speci cations in separate steps, we view the process as transformation of entire PSMs. As can be seen in section 4, every step in the design process can change any or all parts of the PSM. We have already argued that this process is not viewed as a process of stepwise re nement.

6.2 Related Work in Software Engineering

Work related to the development process we described is done by the software engineering community under the names program development by stepwise re nement or program transformations. In both cases, a mostly formal speci cation of the functionality of a program is transformed, in a number of steps, into an eective and ecient description of a program which achieves this functionality. In software engineering this is viewed as an activity during the design or implementation of system. In the case of KBS the eciency aspect already is an important issue during the analysis and speci cation process. Therefore, ideas used for program development in software engineering can be applied for specifying ecient problem solver for computational hard problems. [Wirth, 1983] describes program development as a process of stepwise re nement. 11 He uses the development of a program for 8-Queens problem as example. As in our paper he starts with a variant of generate & test which can achieve the functionality in principle but not in practice. More ecient versions are achieved by stepwise re nement. "In each step a given task is broken up into a number of subtasks" [Wirth, 1983] or in other words: "In each step, one or several instructions of the given program are decomposed into more detailed instructions." [Wirth, 1983]. Having lots in common with this approach the main dierences to our approach lie mainly in two aspects. First, we view the development process of ecient problem solvers not as a process of stepwise re nement of its sub-steps but of its assumptions. Second, during the development of the problem solver the functionality is weakened explicitly (e.g., from an optimal to an acceptable solution) or assumptions are introduced which limit the system to cases where these assumptions are ful lled. Therefore, we get eciency not by algorithmic re nement but by introducing assumptions on powerful domain knowledge and by weakening the task. 11

The paper originally appeared in 1979.

26

The main issue in work on program transformation is the aspect of correctness (see e.g. [Bauer et al., 1987]). That is, one wants to ensure that an implementations ful lls a formal speci cation. We have already mentioned that this concept of correctness has to be changed in our approach as we do assume modi cations (i.e., restrictions) of the functionality during the transformation process. A very interesting approach in the area of program transformation is KIDS (Kestrel Interactive Development System, see [Smith, 1990, Smith & Parra, 1993]) which provides automated support for the development of correct and ecient programs from formal speci cations. The program development is divided into six activities from which four of them are relevant in our context:  Development of a domain theory. The result of this activity provides the terminology of an application domain together with the available domain knowledge necessary to specify the functionality of the desired system.  The creation of a functional speci cation. The domain theory is used to de ne the goal which should be achieved by the system.  Apply a generic search strategy. Prede ned library routines, the divide-and-conquer strategy, global search strategies (like chronological backtracking or branch-and-bound), and local search strategies like hill-climbing are provided to de ne an operational but inecient speci cation of the system.  Optimize the search strategy. Local optimization techniques like simpli cation, partial evaluation, etc. are provided to improve the eciency of the sub-steps of the search method. In our paper, we also started with the general search strategy generate & test and re ned it to propose & revise by introducing eciency aspects. Again, the main dierence is that in KIDS the transition from an inecient to an ecient realization does not change the functionality of the system. Parts of the search space are pruned and search heuristics are used which do not change the completeness of the search in regard to the possible solutions. A further dierence concerning the life cycle of the development process is that in knowledge engineering the task and the problem-solving method is used as a guideline for acquiring and modeling the domain knowledge. The assumptions of a method and the terms which where used to de ne the task (like problem space, requirements, constraints, preference in our example in section four) de ne the goals of the acquisition process of domain knowledge. The dierence is a consequence of the domain-independent formulations of tasks and methods which should enable their reuse in dierent application domains. In KIDS, the domain model is necessary to de ne the tasks.

6.3 Related Work in Complexity Analysis

The dierence between an expert and novice is often not that only the expert can solve the cases but lies in the fact that only the expert can solve it eciently. This expert knowledge is the empirical foundation for work on problem-solving methods. The attempt is to abstract and to generalize these heuristic strategies in order to make them reusable for dierent domains. Problem-solving methods and their assumptions try to add eciency to the reasoning process. The assumptions of a method de ne goals for the knowledge acquisition process. Vice versa 27

it allows the expert to introduce his domain-speci c problem-solving knowledge which enables ecient problem solving. A large part of the problem types tackled by problem-solving methods are at least NP-hard (e.g. scheduling, design, diagnosis). This means that there is no hope of nding a method that will solve all cases in polynomial time. [Rich & Knight, 1991] even de ne AI as \... the study of techniques for solving exponentially hard problems in polynomial time by exploiting knowledge about the problem domain." Most problem-solving method in knowledge engineering describe a heuristic strategy to tackle problems for which no polynomial algorithms are known. To overcome this problem, there are basically three general approaches:  Introducing assumptions about the domain knowledge (or the user of the system) which reduces the complexity of the part of the problem which is solved by the problem-solving method. For example, if the preference function for a solution has only one optimum, a limited search strategy like hill climbing can nd the best solution. 12  Weakening the desired functionality of the system and reducing therefore the complexity of the problem. An example of this type of change is no longer requiring an optimal solution, but one which is acceptable (see section 3).  Making heuristic assumptions which should improve the average behavior of the knowledge-based system. Heuristic knowledge is either knowledge that speci es an ordering on dierent search alternatives to be examined during problem solving or knowledge which structures the search space by excluding non-promising regions. The ordering should be such that preference will be given to alternatives that, on average, will lead to success more often than the other alternatives. The addition of heuristic knowledge can change the average-case behavior of the method but not the complexity class. [Bottcher & Dressler, 1993] introduces a number of heuristics to improve the average search process for model-based diagnosis. Propose-and-revise uses the assumptions that a strategy which proposes initial designs and locally re nes them until no constraints violations can be found is more ecient than a method which tries to nd a correct solution in one step. The rst two approaches tackle the problem by solving a dierent, namely more restricted, problem. There is a dierence between these approaches however: In the rst approach this restriction is already part of the domain so the required functionality does not change. The second approach explicitly changes the functionality. The three approaches also dier in terms of the assumptions they introduce. The rst approach introduces assumptions about properties of the domain. The second approach introduces assumptions about the overall goal of the system. The assumption is that the reduced functionality still is acceptable to reach the goal of the system. The third approach assumes the existence of heuristic knowledge in the domain that enables ecient average case behavior. Studying these assumptions and restrictions and their in uence on the eciency de nes a link to the work in complexity analysis. [Nebel, 1995] proposes dierent strategies to deal with highly complex problems. We have translated his proposals into our framework: 1. Natural restriction of the problem: If an application domain is given, one can introduce assumptions about problem instances (input) which bound the parameter which causes the complexity by a constant. In terms of complexity analysis the domain knowledge or the user of the systems are used as oracles who solves complex parts of the problem. 12

28

2. Enforcing restrictions: Application-independent systems like knowledge representation systems as KL-ONE cannot rely on application-speci c assumptions. Here, eciency can only achieved by reducing the functionality of the system. In the case of KL-ONE the representation formalisms has to be restricted in order to guaranty polynomial behavior. 3. Approximations: Instead of looking for the optimal solution the functionality of the system is reduced to nd an approximation. (See e.g. [vanHarmelen & tenTeije, 1995] for an example of such an approach in diagnostic reasoning). 4. Reconsidering the Problem: For example, heuristic or case-based diagnosis can behave much better from a computational point of view than consistency-based diagnosis [Reiter, 1987] which is NP-hard. Otherwise, heuristic or case-based diagnosis \work only if there exists experience with the technical device one wants to diagnose, which is not necessary for the consistency based approach." [Nebel, 1995] 5. Solving small instances: Finally one could accept combinatorial explosion and use heuristics and search space structuring to prevent unnecessary eort. Our work on problem-solving methods can be discussed along these ve strategies. (1) introduces assumptions over the input, (2) and (3) weaken the functionality of the KBS, and (5) includes heuristics and strategies to improve the search process. They strategies used by propose & revise as discussed in the paper include:  either the weakening of the functionality (see 3),  or the introduction of strong assumptions about the available expertise (see 4). One can either assume that the repair knowledge of propose & revise is always strong enough to nd an optimal solution (i.e., one assumes (4)). Or one can make a weaker assumption about this knowledge by reducing the goal to nd an acceptable solution, i.e., an approximation as proposed by (3).

7 Conclusion Our paper tries to answer three questions: What are problem-solving methods? Why are they necessary? How can we get them? In a nutshell, we provided the following answers:

What is a problem-solving method? A PSM is a mincer which translates a declarative

goal descriptions into a set of assumptions about domain knowledge required to achieve the goal in an ecient manner. Why do we need problem-solving methods? The dichotomy of a declarative goal description and an ecient implementation must be bridged by a level where one rationalizes an ecient problem solver, that is, a problem solver with limited resources. A part of expertise is knowledge about achieving goals under bounded rationality. An operational description of a problem-solving method de nes the appropriate level to elicit, acquire, interpret, and model this kind of knowledge. How do we get problem-solving methods? The development process of a problemsolving method is not a re nement of its conceptual structure. Actually it is a re nement process of its assumptions. Assumptions over domain knowledge are introduced, 29

strengthened, or modi ed in order to achieve eciency. The conceptual structure of a PSM (its operational description) describes only the relation between the assumptions, the cost and the goal of the method. It is the reason for the assumptions and therefore has no value on its own. The point of view on problem-solving methods as presented in our paper de nes a number of research topics. First, an adequate framework for describing problem-solving methods has to be established:  A formal notation for the functionality of a method is required. A proposal for such a notation for diagnostic methods can be found in [tenTeije & vanHarmelen, 1994].  A logic over states is needed to express the operational speci cation of a method. This language must be able to express control over functional speci ed basic building blocks (i.e., inferences). We can build here on the work on formal languages for KADS (see [Fensel & vanHarmelen, 1994] for a comparison of these languages).  A formal notation for the assumptions is needed. A variant of meta-logic could be used to specify the assumptions of the method.  Finally, a feasible calculus must be provided to specify the computational behavior of a method. [Straatman & Beys, 1995] describes a step in this direction. Second, based on these formal notations a proof calculus is necessary which enables to prove relationships between the dierent parts of the speci cation of a method (see section 5). A rst step into this direction is achieved by [Fensel & Groenboom, 1995] where proof rules are de ned for languages like KARL and (ML)2. Based on these proof rules, automated support by theorem provers is necessary in order to shift proofs from a thing which is possible in principle to realistic enterprise. As the description formalisms includes logic over states like dynamic logic we will investigate the possibility to use theorem provers like KIV [Heisel et al., 1990] developed for program veri cation based on dynamic logic. Third, methods and tools are necessary which support the cyclic development of appropriate problem-solving methods. This includes a library with problem-solving methods schema indexed by their functionality, assumptions and cost, and operations which work on assumptions and derive problem-solving method instantiations. [Van de Velde, 1994] de nes three components of a modeling library: Modeling components which are structures that are useful for the construction of complete models. Generic models that are frames representing a class of complete models. Modeling operators transform a model into another one. Substantiating these ideas seems to be a promising research direction.

Acknowledgments We would like to thank Ameen Abu-Hanna, Joost Breuker, Guus Schreiber, Annette ten Teije, Peter Terpstra and Bob Wielinga for helpful discussions which enabled this paper. We would also like to thank Manfred Aben, Jurgen Angele, Richard Benjamins, Gertjan van Heijst, Rudi Studer, and two anonymous referees for comments on earlier versions of the paper. 30

References [Aben, 1993] M. Aben. Formally specifying re-usable knowledge model components. Knowledge Acquisition, 5:119{141, 1993. [Aben, 1995] M. Aben. Formal Methods in Knowledge Engineering. PhD thesis, University of Amsterdam, Faculty of Psychology, February 1995. ISBN 90-5470-028-9. [Akkermans et al., 1993] J. M. Akkermans, B. J. Wielinga, and A. Th. Schreiber. Steps in constructing problem-solving methods. In N. Aussenac, G. Boy, B. Gaines, M. Linster, J.-G. Ganascia, and Y. Kodrato, editors, Knowledge Acquisition for Knowledge-Based Systems. Proceedings of the 7th European Workshop EKAW'93, Toulouse and Caylus, France, number 723 in Lecture Notes in Computer Science, pages 45{65, Berlin Heidelberg, Germany, September 1993. Springer-Verlag. [Bauer et al., 1987] F. L. Bauer, H. Ehler, R. Horsch, B. Muller, H. Partsch, O. Paukner, and P. Pepper. The Munich Project CIP, vol II: The Transformation System CIP-S. Lecture Notes on Computer Science (LNCS) no 292. Springer-Verlag, Berlin, 1987. [Benjamins, 1993] V. R. Benjamins. Problem Solving Methods for Diagnosis. PhD thesis, University of Amsterdam, Amsterdam, The Netherlands, June 1993. [Boehm, 1988] B.W. Boehm. A spiral model of software development and enhancement. IEEE Computer, 1988. [Bottcher & Dressler, 1993] C. Bottcher and O. Dressler. Diagnosis process dynamics: Holding the diagnostic trackhound in leash. In Proceedings IJCAI`93. Morgan Kaufmann, 1993. [Bredeweg, 1994] B. Bredeweg. Model-based diagnosis and prediction of behaviour. In J. Breuker and W. van de Velde, editors, CommonKADS Library for Expertise Modelling (reusable problem solving components), pages 121{153. IOS Press, Amsterdam, Oxford, Washington, 1994. [Breuker & Van de Velde, 1994] J. A. Breuker and W. Van de Velde, editors. The CommonKADS Library for Expertise Modelling. IOS Press, Amsterdam, The Netherlands, 1994. [Bylander & Chandrasekaran, 1988] T. Bylander and B. Chandrasekaran. Generic tasks in knowledge-based reasoning: The right level of abstraction for knowledge acquisition. In B. Gaines and J. Boose, editors, Knowledge Acquisition for Knowledge Based Systems, volume 1, pages 65{77. Academic Press, London, 1988. [Chandrasekaran, 1986] B. Chandrasekaran. Generic tasks in knowledge based reasoning: High level building blocks for expert system design. IEEE Expert, 1(3):23{30, 1986. [Chandrasekaran et al., 1992] B. Chandrasekaran, T. R. Johnson, and J. W. Smith. Taskstructure analysis for knowledge modeling. Communications of the ACM, 35(9):124{137, 1992. [deKleer & Williams, 1987] J. H. de Kleer and B. C. Williams. Diagnosing multiple faults. Arti cial Intelligence, 32:97{130, 1987. 31

[(ed.), 1994] M. Linster (ed.). Sisyphus 91/92: Models of problem solving. International Journal of Human Computer Studies, 40(3), 1994. (special issue). [Eriksson et al., 1995] H. Eriksson, Y. Shahar, S. W. Tu, A. R. Puerta, and M. A. Musen. Task modeling with reusable problem-solving methods. Arti cial Intelligence, 1995. To appear. [Fensel & Groenboom, 1995] D. Fensel and R. Groenboom. A formal semantics for specifying the dynamic reasoning of knowledge-based systems. In Proceedings of KEML'96, 1995. [Fensel & vanHarmelen, 1994] D. Fensel and F. van Harmelen. A comparison of languages which operationalize and formalize kads models of expertise. The Knowledge Engineering Review, 8(2), June 1994. [Fensel, 1995] D. Fensel. Assumptions and limitations of a problem-solving method: A case study. In Proceedings of the 9th Ban Knowledge Acquisition for Knowledge-Based System Workshop (KAW'95), Ban, Canada, 1995. [Floyd et al., 1989] C. Floyd, F.-M. Reisin, and G. Schmidt. Steps to software development with user. In Proceedings of the 2nd European Software Engineering Conference ESEC89, September 11-15, Warwick, Lecture Notes in Computer Science, no 387, Springer-Verlag, Berlin, 1989. [Harel, 1984] D. Harel. Dynamic logic. In D. Gabbay and F. Guenthner, editors, Handbook of Philosophical Logic, Vol. II: extensions of Classical Logic, pages 497{604. Reidel, Dordrecht, The Netherlands, 1984. [Heisel et al., 1990] M. Heisel, W. Reif, and W. Stephan. Tactical theorem proving in program veri cation. In M. E. Stickel, editor, 10th International Conference on Automated Deduction, Kaiserslautern, July 1990, Lecture Notes in Arti cial Intelligence, no 449. Springer-Verlag, 1990. [Katz et al., 1989] S. Katz, C. A. Richter, and K. The. Paris: A system for reusing partially interpreted schemas. volume I - Concepts and Models of Frontier Series: Software Reusability. ACM Press, New York, 1989. [Kifer et al., 1993] M. Kifer, G. Lausen, and J. Wu. Logical foundations of object oriented and frame-based languages. Technical Report 93/06, Department of Computer Science, SUNY at Stony Brook, NY, April 1993. [Kozen, 1990] D. Kozen. Logics of programs. In Handbook of Theoretical Computer Science. Elsevier Science Publ., B. V., Amsterdam, 1990. [Landes & Studer, 1995] D. Landes and R. Studer. The treatment of non-functional requirements in mike. In Proc. 5th European Software Engineering Conference ESEC'95, Barcelona, Spain, Sept. 25-28 1995. [Levesque, 1984] H. J. Levesque. Foundations of a functional approach to knowledge representation. Arti cial Intelligence, 23(2):155{212, 1984. [Lloyd, 1987] J. W. Lloyd. Foundations of Logic Programming. Springer-Verlag, Berlin, 2nd edition edition, 1987. 32

[Marcus, 1988] S. Marcus, editor. Automatic knowledge acquisition for expert systems. Kluwer, Boston, 1988. [Marcus et al., 1988] S. Marcus, J. Stout, and J. McDermott. VT: An expert elevator designer that uses knowledge-based backtracking. AI Magazine, Spring:95{111, 1988. [Morik, 1987] K. Morik. Sloppy modeling. In K. Morik, editor, Knowledge Representation and Organisation in Machine Learning, Lecture Notes in Arti cial Intelligence, no 347. Springer-Verlag, 1987. [Musen, 1989] M. A. Musen. An editor for the conceptual models of interactive knowledgeacquisition tools. International Journal on Man-Machine Studies, 31, 1989. [Musen, 1992] M. A. Musen. Overcoming the limitations of role-limiting methods. Knowledge Acquisition, 4, 1992. [Nebel, 1995] B. Nebel. Arti cial intelligence: A computational perspective. In G. Brewka, editor, Essentials in Knowledge Representation. 1995. [Neubert & Studer, 1992] S. Neubert and R. Studer. The keep model. In T. Wetter et al., editor, Current Developments in Knowledge Acquisition. Springer-Verlag, 1992. [Newell, 1982] A. Newell. The knowledge level. Arti cial Intelligence, 18:87{127, 1982. [Parnas & Clements, 1986] D. L. Parnas and P. C. Clements. A rational design process: How and why to fake it. IEEE Transactions on Software Engineering, 12:251{257, 1986. [Poeck & Gappa, 1993] K. Poeck and U. Gappa. Making role limiting shells more exible. In Knowledge Acquisition for Knowledge-Based Systems, Proceedings of the 7th European Workshop EKAW'93, Toulouse, France, number 723 in Lecture Notes in AI, Berlin, September 6-10 1993. Springer-Verlag. [Poeck & Puppe, 1992] K. Poeck and F. Puppe. Coke: Ecient solving of complex assignment problems with the propose-and-exchange method. In Proceedings of the 5th International Conference on Tools with Arti cial Intelligence, Arlington, Virginia, US, 1992. [Puppe, 1993] F. Puppe. Systematic Introduction to Expert Systems: Knowledge Representation and Problem-Solving Methods. Springer Verlag, Berlin, 1993. [Reiter, 1987] R. Reiter. A theory of diagnosis from rst principles. Arti cial Intelligence, 32:57{96, 1987. [Rich & Knight, 1991] E. Rich and K. Knight. Arti cial Intelligence. McGraw-Hill, New York, 2nd edition, 1991. [Rouveirol & Albert, 1994] C. Rouveirol and P. Albert. Knowledge level model of a con gurable learning system. In Proceedings of the European Knowledge Acquisition Workshop (EKAW'94), Hoegaarden, Belgium, number no 867 in Lecture Notes in Ariti cial Intelligence (LNAI), Berlin, September 26-29 1994. Springer Verlag. [Russell & Subramanian, 1995] S. J. Russell and D. Subramanian. Provably bounded-optimal agents. Journal of Arti cial Intelligence Research, 2, 1995. 33

[Schreiber & Birmingham, 1996] A. Th. Schreiber and W. P. Birmingham. The sisyphus VT initiative, editorial special issue. International Journal on Human Computer Studies, 1996. (in press). [Schreiber et al., 1991] A. Th. Schreiber, J. M. Akkermans, and B. J. Wielinga. On problems with the knowledge level perspective. In L. Steels and B. Smith, editors, AISB{91: Arti cial Intelligence and Simulation of behaviour, pages 208{221, London, 1991. SpringerVerlag. Also in: Proceedings Ban'90 Knowledge Acquisition Workshop, J. H. Boose and B. R. Gaines (editors), SRDG Publications, University of Calgary, pages 30.1{30.14. [Schreiber et al., 1993] A. Th. Schreiber, B. J. Wielinga, and J. A. Breuker, editors. KADS: A Principled Approach to Knowledge-Based System Development, volume 11 of KnowledgeBased Systems Book Series. Academic Press, London, 1993. ISBN 0-12-629040-7. [Schreiber et al., 1994] A. Th. Schreiber, B. J. Wielinga, R. de Hoog, J. M. Akkermans, and W. Van de Velde. CommonKADS: A comprehensive methodology for KBS development. IEEE Expert, 9(6):28{37, December 1994. [Schreiber et al., 1995] A. Th. Schreiber, B. J. Wielinga, and W. H. J. Jansweijer. The KACTUS view on the 'O' word. In IJCAI Workshop on Basic Ontological Issues in Knowledge Sharing, 1995. Also in: J. C. Bioch and Y.-H. Tan (eds.). Proceedings 7th Dutch National Conference on Arti cial Intelligence NAIC'95, EURIDIS, Erasmus University Rotterdam, The Netherlands, pp. 159{168, 1995. [Simon, 1981] H. A. Simon. The Science of the Arti cial. The MIT Press, Cambridge, 2nd edition, 1981. [Smith & Parra, 1993] D. R. Smith and E. A. Parra. Tranformational approach to transportation scheduling. In Proceedings of the 8th Knowledge-Based Software Engineering Conference, pages 60{68. IEEE Computer Society Press, Sept. 14{17 1993. (Best Paper Award). [Smith, 1990] D. R. Smith. Kids: A semiautomatic program development system. IEEE Transactions on Software Engineering, 16(9), 1990. [Steels, 1990] L. Steels. Components of expertise. AI Magazine, Summer 1990. [Steels, 1993] L. Steels. The componential framework and its role in reusability. In Jean-Marc David, Jean-Paul Krivine, and Reid Simmons, editors, Second Generation Expert Systems, pages 273{298. Springer-Verlag, Berlin Heidelberg, Germany, 1993. [Straatman & Beys, 1995] R. Straatman and P. Beys. A performance model for knowledgebased systems. In M. Ayel and M. C. Rousset, editors, EUROVAV-95 European Symposium on the Validation and Veri cation of Knowledge Based Systems, pages 253{263. ADEIRAS, Universite de Sovoie, Chambery, 26-28 June 1995. [Tank, 1992] W. Tank. Modellierung von Expertise uber Kon gurierungsaufgaben. In x, Sankt Augustin, Germany, 1992. ISBN 3-929037-05-X. [tenTeije & vanHarmelen, 1994] A. ten Teije and F. van Harmelen. An extended spectrum of logical de nitions for diagnostic systems. In Proceedings of DX-94 Fifth International Workshop on Principles of Diagnosis, 1994. 34

[Terpstra et al., 1993] P. Terpstra, G. van Heijst, B. Wielinga, and N. Shadtbolt. Knowledge acquisition support through generalised directive models. In Jean-Marc David, Jean-Paul Krivine, and Reid Simmons, editors, Second Generation Expert Systems, pages 428{455. Springer-Verlag, Berlin Heidelberg, Germany, 1993. [Top & Akkermans, 1994] J. Top and H. Akkermans. Engineering modelling. In J. Breuker and W. Van de Velde, editors, CommonKADS Library for Expertise Modelling, volume 21 of Frontiers in Arti cial Intelligence and Applications, chapter 12, pages 265{303. IOS Press, Amsterdam, 1994. [Van de Velde, 1991] W. Van de Velde. Tractable rationality at the knowledge-level. In L. Steels and B. Smith, editors, Proceedings AISB'91: Arti cial Intelligence and Simulation of Behaviour, pages 196{207, London, 1991. Springer-Verlag. [Van de Velde, 1994] W. Van de Velde. A constructivist view on knowledge engineering. In Proceedings of the 11th European Conference on Arti cial Intelligence (ECAI'94), Amsterdam, August 1994. [vanHarmelen & tenTeije, 1995] F. van Harmelen and A. ten Teije. Approximations in diagnosis: motivations and techniques. In C. Bioch and Y.H. Tan, editors, Proceedings of the Dutch Conference on AI (NAIC'95), Rotterdam, June 1995. [vanHeijst et al., 1992] G. van Heijst, P. Terpstra, B. J. Wielinga, and N. Shadbolt. Using generalised directive models in knowledge acquisition. In Th. Wetter, K. D. Altho, J. Boose, B. Gaines, M. Linster, and F. Schmalhofer, editors, Current Developments in Knowledge Acquisition: EKAW-92, Berlin, Germany, 1992. Springer-Verlag. [Wielinga & Schreiber, 1994] B. J. Wielinga and A. Th. Schreiber. Conceptual modelling of large reusable knowledge bases. In K. von Luck and H. Marburger, editors, Management and Processing of Complex Data Structures, volume 777 of Lecture Notes in Computer Science, pages 181{200, Berlin, Germany, 1994. Springer Verlag. [Wielinga et al., 1992] B. J. Wielinga, A. Th. Schreiber, and J. A. Breuker. KADS: A modelling approach to knowledge engineering. Knowledge Acquisition, 4(1):5{53, 1992. Special issue `The KADS approach to knowledge engineering'. Reprinted in: Buchanan, B. and Wilkins, D. editors (1992), Readings in Knowledge Acquisition and Learning, San Mateo, California, Morgan Kaufmann, pp. 92{116. [Wielinga et al., 1993] B. J. Wielinga, W. Van de Velde, A. Th. Schreiber, and J. M. Akkermans. Towards a uni cation of knowledge modelling approaches. In Jean-Marc David, Jean-Paul Krivine, and Reid Simmons, editors, Second Generation Expert Systems, pages 299{335. Springer-Verlag, Berlin Heidelberg, Germany, 1993. [Wielinga et al., 1995] B. J. Wielinga, J. M. Akkermans, and A. Th. Schreiber. A formal analysis of parametric design problem solving. In B. R. Gaines and M. A. Musen, editors, Proceedings of the 8th Ban Knowledge Acquisition for Knowledge-Based Systems Workshop, volume II, pages 37.1{37.15, Alberta, Canada, 1995. SRDG Publications, University of Calgary. 35

[Wirth, 1983] N. Wirth. Program development by stepwise re nement. Communications of the ACM, 26(1), 1983.

36