Diagrammatic Reasoning Systems 1 Introduction - CiteSeerX

Diagrammatic Reasoning Systems  Mateja Jamnik July 10, 1996

1 Introduction This paper is a literature survey on existing systems in diagrammatic reasoning. The interest in these systems is due to the intention of the author to automate diagrammatic proofs in mathematics. The characteristics of the system that we aim to implement will be described in more detail in the next section. The importance of diagrams in many domains of reasoning has been extensively discussed by Larkin and Simon in [Larkin & Simon, 1987], who claim that a diagram is (sometimes) worth ten thousand words. They state that the diagrammatic representation explicitly preserves the information about topological and geometric relations among the components of the problem. What they mean by explicitly preserving information is that the information that needs to be conveyed is analogically represented by the diagram. Thus, the information can be directly visualised by humans. The advantage of a diagram is that it concisely stores the information, explicitly represents the relations among the elements of the diagram, and it supports a lot of perceptual inferences that are very easy for humans. It is clear that diagrams have an important role in problem solving. Whether the reasoning is conducted symbolically or whether it is diagrammatic, the goal is to make inferences by manipulating and inspecting the internal representations of information. In the case of diagrammatic reasoning the internal representation of at least some of the information which is particularly describing the spatial relationships, ie. topological or geometric structures, is a diagram. Diagrammatic representation can to some extent be de ned in operational terms. This means that we describe it by the operations that can be performed on a certain structure. The data structure that describes diagrammatic representation can contain only the information that is explicit in the diagram. Furthermore, the operations on diagrammatic data structure can only be the ones that humans can visually (or mentally) perform on the diagram. In a diagrammatic representation, according to Larkin and Simon [Larkin & Simon, 1987]: \... the expressions correspond, on a one-to-one basis, to the components of a diagram describing the problem. Each expression contains the information that is stored at one particular locus in the diagram, including information about relations with adjacent loci." Kulpa claims in [Kulpa, 1994] that: \The eld of diagrammatic data and knowledge representation, and diagrammatic reasoning has recently become one of the most rapidly growing areas of research in arti cial intelligence and related elds of computer science and cognitive science." This is not very surprising due to the fact that visual thinking has been an area studied for a long time, and it has been closely related to the problem solving and human thinking [Arnheim, 1969].  The research reported in this paper was supported by Arti cial Intelligence Department, Faculty of Science and Engineering grant, University of Edinburgh, and the Slovenska Znanstvena Fundacija (Slovenian Scienti c Foundation) supplementary grant.

1

Furthermore, mathematicians have always used diagrams to assist problem solving. Thus, it is surprising that the area of diagrammatic reasoning which deals with the issues of visual representations, visual thinking and the role of diagrams in human thinking has not been paid more attention earlier. Diagrammatic reasoning uses diagrams to search for a solution of a given problem. To achieve this, diagrams need to represent data and knowledge of the related problem domain. Generally, no matter what sort of representation we use for the diagrams, it will be a combination of analogical and propositional knowledge representation. The syntax of analogical (also called direct or homomorphic) representation models the semantics of the problem domain. In the propositional (also called Fregean) representation, parts and relationships of the propositional representation of a problem do not relate to the problem domain. For more information see [Sloman, 1985]. There have been several diagrammatic reasoning systems implemented in the past. They are called diagrammatic reasoning systems, since they use a diagram to aid the search for the solution of the problem. One of the rst ones was Gelernter's Geometry Machine described in the next section. The other systems that I describe in the next section have a common problem domain with Gelernter's Geometry Machine, ie. Euclidean plane geometry. They are all diagrammatic in the sense that they make some use of a diagrammatic representation of the problem. The diagram does not necessarily have to be represented `visually' (ie. by an external representation of a diagram which may be visualised by the user). Instead, some sort of a nonvisual representation is used. In most cases, diagrams are represented by Cartesian coordinates, in some cases by the bitmap or raster matrix, and in some cases the diagrams are in fact visual (ie. the user interface allows the display of a visual image of the diagram). The systems in the following sections are divided according to their problem domain. They will be described according to their architecture and their main features, with particular focus on their use of the diagram. For each system I will give an example of the problem that it can solve in Appendix B. Finally, I shall describe in a few words the similarities and the dierences of the described system to the system that we aim to implement.

2 System for `Informal' Diagrammatic Proofs The system that we propose to implement will prove theorems of a restricted domain of mathematics (input by the user) by geometric operations on the diagram. Thus, we call them diagrammatic proofs. To clarify the concept of a diagrammatic proof refer to Appendix A on page 13 for examples of such proofs. The dierence between the formal logic proof and the diagrammatic proof is that instead of applying the rules of some logic for inference steps, we are going to instruct our system to perform geometric operations on the diagram. These will capture the inference steps of our diagrammatic proof. From the analysis of the examples (presented in Appendix A), three categories of proofs can be distinguished: Category 1: Proofs that are not schematic: there is no need for induction to prove the general case. They need a simple geometric manipulation to prove the individual concrete case represented by the diagram, and need a generalisation at the end to show that this proof will hold 8n. The Pythagorean Theorem 1 is an example of the proof of this category. Category 2: Proofs that are schematic: they require no inductive step to prove the theorem for each individual concrete case of a diagram, but require induction for the nth case (a concrete diagram cannot be drawn for this case unless using some sort of notation for the abstraction of in nity). So we can use the constructive !-rule which will generate a proof for the nth case from the proofs of individual proof instances. For more information on constructive !-rule, see [Baker et al, 1992]. The theorem for Sum of Odd Integers is an example of the proof of this category. Category 3: Proofs that are inherently inductive: for each individual concrete case of the diagram they need an inductive step to prove the theorem. This means that we cannot 2

generate an all-inclusive (complete) diagram for any particular instance (ie. concrete case) of the theorem due to the need for representing an in nity in the diagram. Consequently, we cannot use the constructive !-rule, which requires a few individual instances of proofs to be able then to generalise these to get a general proof for all cases. The theorem for Geometric Sum is an example of the proof of this category. Primarily, it is our aim to automate diagrammatic proofs for examples of Category 2, but we plan to implement diagrammatic theorem proving of examples for Category 1 as well. There are three most obvious functions of the system which we want to implement. These include inputting the theorem to be proved and then diagrammatically proving it. In theorem proving there also needs to be some kind of communication between this diagrammatic component which deals with diagrams and the inference engine which deals with diagrammatic inference steps. This suggests that the system should consist of the following three parts: 1) Inference engine: it is the knowledge base component of the system; it generates strings of goals, subgoals, constraints and geometric operations (instructions) that are passed to the diagram. It is the main component of the system. It inspects the string representing the theorem and gets from the user a suggested shape/object from which to start the diagrammatic theorem proving. It accepts the diagrammatic operations which are to be performed on the diagram from the user, and passes them over as instructions to the diagram. It gets the new state of the diagram from the translation component and it inspects the modi ed diagram. Then it compares the state of the diagram with the goal in the proof of the theorem. It could also recognise syntactic symmetries in the diagram. A submodule of the inference engine contains the vehicle for performing schematic proofs, ie. the part where the constructive !-rule is put into eect. The representation used in the inference engine could be propositional and/or relational. 2) Diagrammatic component: the Cartesian representation of the diagram is formed in this component; this is the internal 2-dimensional or 3-dimensional representation of the shapes and objects. Each operation instructed from the inference engine is put into eect here. It necessarily modi es the diagram. Therefore, the new, created properties of the diagram are introduced here, and reported back to the inference engine which inspects and analyses them. 3) Translation component: it mediates the communication between the inference engine and the diagrammatic component. It superimposes the relational representation over the Cartesian representation, and vice versa, depending which way the communication is going. In the next few sections I describe diagrammatic reasoning system that have been implemented in the past.

3 Problem Domain: Euclidean Plane Geometry

3.1 Gelernter's Geometry Machine

One of the rst systems implemented that used diagrams for reasoning was Gelernter's Geometry Machine [Gelernter, 1963]. The geometry machine operates on statements expressed as strings of characters in some formal logical system. The problem is a statement, and the solution is a sequence of statements. The sequence of solution statements starts from some axiom that the system chooses. Then it continues backward, inferring further statements of the solution based on the existing axioms or other theorems. The nal statement of the solution is the problem itself. Working backwards ensures that the sequence under consideration as a solution indeed terminates in the required theorem. However, the problem-solving tree still has a high degree of branching. To prune the search tree, the geometry machine uses the heuristic properties of the 3

diagram to reject false sequences, as opposed to exhaustively searching for the subgoals that are true. This means that the subgoal hypotheses are tested against measurements of a coordinate diagram, and if the subgoal is false in the diagram, then it is rejected. The geometry machine consists of three components: Syntax computer: it manipulates the formal system by generating strings of hypothesis (premises, subgoals). Diagram computer: the theorem to be proved is represented in a coordinate system. Also, it contains a series of qualitative descriptions of the diagram. Heuristic computer: it is the main component of the system. It compares sequences of strings generated by the syntax computer and their interpretation in the diagram. The heuristic computer rejects sequences not supported by the diagram. Furthermore, it recognises the syntactic symmetries of classes of strings and does modi cations and improvements to the system. Figure 1 shows the ow of control in the geometry machine. It is taken from [Gelernter, 1963]. Note that the syntax computer can communicate with diagram computer and vice versa only through the heuristic computer. Heuristic Computer

Syntax Computer

Diagram Computer

Figure 1: The architecture of the Geometry Machine It is important to note that the system does not generate its own diagram. Rather, the diagram is supplied by the user. This bears some degree of psychological validity, since humans usually draw the diagrams to accompany proving of a theorem by themselves. The diagram is supplied to the geometry machine in the form of a list of possible coordinates for points named in the theorem. The second list speci es points joined by segments. The diagram has two roles. Its negative role is to reject hypotheses (subgoals) proposed by the heuristic computer that are not true in the diagram. In this way the search space is pruned. The positive role of the diagram is to shorten the inference paths by assuming various facts obvious in the diagram as true, ie. it veri es the correctness of simple goals by checking them in the diagram (eg: a certain point lies between two others). To eliminate most of the search time there are two main heuristics introduced to the system. The rst one uses the notion of immediately true properties (eg: equalities of angles) and assigns them a priority, so they are developed before any other subgoals are considered. The second is the notion of a \distance" between each subgoal string and the set of premise strings. Thus when the priority subgoals have been used the system chooses for the next subgoal the one that is \closest" to the premise set. In summary, the signi cance of Gelernter's work is due to considering for the rst time using a diagram to control search for the proof of a theorem. The Geometry machine controls the proof search by using a diagram as a model of the goal to be proved. Thus, it could be referred to as a model checking system, where the model of the problem is a diagram. In the Appendix B on page 15 there is an example of one of the theorems that the geometry machine is capable of proving. It appears to be trivial, but the interesting point about this example is that the system introduces, ie. constructs an auxiliary element to the diagram. 4

3.2 Goldstein's BTP - Basic Theorem Prover

Goldstein in many ways extended Gelernter's Geometry Machine by implementing his diagrammatic reasoning system called BTP - Basic Theorem Prover [Goldstein, 1973]. His system solves problems of a small part of plane Euclidean geometry. Goldstein extends the capabilities of Geometry Machine on three levels:  Mathematical knowledge is represented on a higher level due to the availability of goaloriented programming languages, so that the less work needs to be focussed on the implementational issues.  More computational eciency is achieved by the use of canonical names, which identify synonyms for geometrical entities.  The search for the proof is aided by the heuristic which couples constructions with common speci c purpose which are applicable in the proof. The problem domain of BTP incorporates theorems concerning the congruence of triangles, equality of segments, parallelism of lines, equality of angles and quadrilaterals to be parallelograms. In such a limited domain the theorem prover is still capable to prove considerably complex theorems. See Appendix B on page 16 for an example of a theorem proved by BTP. The input to BTP is a diagram, which is represented in the form of Cartesian coordinates of the points and a list of connections between the points, the hypotheses and the objective, ie. the goal. BTP way to prove theorems is to start with the conclusion and try to get to the hypotheses. It consists of the following components: Consequent knowledge: is represented in terms of strategies. A strategy consists of to-prove line which must match the current goal, an establish line which lists the sub-goals, and a reasons line which summarises the strategy in a sentence. Antecedent knowledge: of the system serves to:  Eliminate synonyms: it renames synonymous statements of a theorem (namely, the same theorem can be worded dierently) into a consistent language that is understood by the theorem prover.  Derive corollaries: in the course of the analysis of the theorem, relevant corollaries of hypotheses and objectives that have been successfully proved are asserted. Experts: for computational eciency strategies that have a common goal are organised into experts (eg: triangle congruence). The goal of the expert is converted into canonical form to avoid the problem of recognising identical elements of the diagram that are expressed with symmetric names (eg: segment AD is equivalent to segment DA). Diagram Filter: diagram is parsed and used to reject goals that are false in the diagram.

3.3 Koedinger and Anderson's DC

[Koedinger & Anderson, 1990] implemented a geometry problem solver called the Diagram Con guration model. The interesting characteristic of this system is that the authors based the con guration of the model of the system entirely on the empirical data from testing how human experts solve these problems. Thus, supported by the empirical evidence they claim that DC reasons the way humans do. The key feature of the system is that it organises its data in perceptual chunks, called diagram con gurations. These are analogical to key features of diagrams that humans recognise when they inspect a diagram. Therefore, during the process of generating a solution path, DC infers the key steps rst, and ignores along the way the less important features of the diagram, ie. the less important inference steps. The Diagram Con guration model (DC) consists of: 5

Diagram Con guration Schemas: is the major knowledge structure of DC. They are asso-

ciated with elementary or more complex geometric structures in the form of clusters of geometry facts (eg: congruent-triangles-shared-side scheme, perpendicular-adjacent-angles scheme). A scheme consists of the following parts: Con guration: storage for a geometric image, ie. a diagram. It is a con guration of points and lines which is part of the geometric diagram. Whole-statement: is a geometry statement referring to the whole of the con guration (eg: 4XY Z  = 4XZW ). Part-statements: are geometry statements referring to the the relationships among the parts of the diagram (eg: 6 Y = 6 Z ). Ways-to-prove: lists subsets of part-statements that are sucient to prove the wholestatement and all of the part-statements. DC's Processing Components: DC consists of three major processing stages: Diagram Parsing: it recognises con gurations in the diagram and instantiates their corresponding schemas. The recognition is done on two levels: low level simple object recognition and high-level plausible con guration hypothesising. Statement Encoding: it deciphers the meaning of the given and goal statements and represents them as part-statements which are tagged either \known" or \desired". Schema Search: using forward and backward inferences, schemas that are possibly true of the problem are iteratively identi ed (ie. the systems searches through possible schemas until the link between the given and a goal statement is found). The main idea of DC is that it uses schemas instead of statements of geometry to plan the search for solution to a problem. In the rst stage, the diagram is parsed and the possible schemas are instantiated. This is done by inspecting the elements of the input diagram and identifying the schemas that are related to particular features of the diagram (for example, the schema for right angle triangles is instantiated if the input diagram contains a right angle triangle). Establishing one schema may enable establishing another, since certain features of the diagram are common to several schemas. No problem solving search is done at this stage, however, the biggest part of the work of the system is done by restricting the solution space by diagram parsing. Figure 2 shows a problem de nition and the solution space of the problem after the diagram parsing and the instantiations of schemas (taken from [Koedinger & Anderson, 1990]). The boxes show the schemas that have been recognised and the lines connect schemas to their part-statements. B

B

B B

B

B A

A

D

Rt-Tri

A

D

D C Rt-Tri

B A

C

D

A

D C Bisected -Ang

Perp-Adj -Angs

C

A

D Cong-TriShared-Side

D

C

Bisected -Seg

A

C Isos Triangle

C

Given:

rt 6 ADB BD bisects 6 ABC Goal:

D midpoint of AC

rt

ADB

rt

CDB

ADB

~ =

GIVEN

CDB

~ CBD ABD = GIVEN

~

AD = DC GOAL

AB

~ =

BC

~

A=

C

Figure 2: DC's Problem De nition and Solution Space After diagram parsing, the given/goal statements of the problem de nition are encoded by tagging them as \known" (or \desired") if they are already part-statements or whole-statements 6

of a certain schema. Finally, to nd the solution the system searches for a path from the givens to the goal statements. Note that the constraints which are listed in ways-to-prove component of the schema have to be met when searching for the solution path. There may be several solution paths. In summary, Koedinger and Anderson's DC system controls search for a solution of a problem by organising the proof search space into smaller spaces which deal with specialised concepts, ie. schemas. These, when identi ed to be related to a problem allow us to apply a smaller set of rules.

3.4 Baker-Plummer and Bailin's &/GROVER

&/GROVER, developed by [Barker-Plummer & Bailin, 1992] is an automated reasoning system which uses information from a diagram to guide proof search. The architecture of &/GROVER system consists of the & automated theorem prover, based on the sequent calculus for Zermelo set theory (see [Bailin & Barker-Plummer, 1993]). GROVER is the diagram interpreting component of the system, which passes the crucial information to prove the theorem from the inspected diagram to the & theorem prover. In the scope of this paper we are mainly interested in the GROVER diagrammatic reasoning component. See Appendix B.3 on page 16 for an example of a theorem that &/GROVER can prove. The architecture of GROVER is shown in Figure 3 (from [Barker-Plummer et al, 1995]). Graphical Editor

ViewRunner

Diagram Local/Global Parsing Formulae Allows user to check results of parse of the diagram Geometry2Logic

Verify Logic

Formulae

Conjecture

Determines goals to prove

Allows user to check resulting strategy

Create Strategy

Verify Strategy

Associate with Diagrams Existential Solve Determine Hypotheses Proof Strategy (induction et al)

Goals

Produces Proofs of Goals

& Proofs

Figure 3: The architecture of GROVER GROVER consists of the following components: ViewRunner: is the graphical editor tool, and an interface to GROVER. It enables users to draw a diagram consisting of fairly elementary components. The diagram is saved in an abstract description of the geometry of the diagram format (eg: describing arcs, circles, arrows, dots, etc.). Geometry to Logic: is an expert system component. It parses the abstract description of the diagram and translates the logical content of the diagram into formulae expressed in the & language. It works in a bottom-up fashion. This means that it rst analyses the objects of the diagram, then relationships between objects and nally the collection of atomic formulae to determine more complex formulae. It is one of the two most important components of the system. 7

Verify Logic: is an inspection tool allowing the user to examine and modify the \logical con-

tent" (ie. logical formulae description) of the input diagram. This description is derived by the Geometry to Logic component from the graphical representation of the diagram. Create Strategy: constructs a sequence of goals which relate the logical formulae determined from the diagram to the conjecture that the user wants to prove. This sequence of goals can then be proved by the & theorem prover. It is the second of the most important components of the system. Verify Strategy: allows user to inspect the sequence of goals generated by the Create Strategy component. If it is decided that they are acceptable, then the sequence is passed to the & theorem prover to verify that they are indeed provable. The main idea of how GROVER works is that the information is extracted from the diagram and translated into logical formulae in the language of & which are then proved by &. Then they are used as additional hypotheses to the main proof of the conjecture. Thus, the formulae that are extracted from the diagram are in fact lemmas, or more technically, cut formulae in the proof of the main conjecture. GROVER is similar to Geometry Machine in that it also uses the diagram as a model of the goal which is to be proved. Moreover, the diagram speci es the subgoals themselves. Therefore, it determines the high-level structure of the proof. Also, it speci es the ordering in which the subgoals are applied. To summarise, in order to prevent a high degree of branching of the proof search tree, GROVER considers only subgoals that are known to be true in the diagram.

4 Problem Domain: Qualitative Physics

4.1 Funt's WHISPER

One of the rst systems that used diagrammatic reasoning applied to qualitative simulation of a physical system behaviour was Funt's WHISPER [Funt, 1980]. It solved problems of motion and stability of objects due to the force of gravity. The input to WHISPER is a diagram of a blocks world structure. Its output is a series of diagrams that denote the events that occur as the structure collapses. WHISPER's main components are: HLR - higher level reasoner: it has the ability to observe the diagram through a parallel processing retina via its perceptual system. It can make changes to the current diagram. HLR contains knowledge about the stability and the motion of falling objects. Diagram: it shows the objects as a block world structure. Its output is a series of updated diagrams that are the consequence of the balancing eect of the structure. Every new diagram in the series is an input to the next stage of the collapse sequence problem solving. So it continues to output the diagrams until the structure is stable. Retina: it is a parallel processor that executes the perceptual primitives, which are its algorithms. It locates objects and their supports. It also demonstrates (ie. visualises) the rotation of objects and can thus determine when an object will come in contact with another object in the structure. It observes the results of the experimental changes made to the diagram by re-drawing procedures. The ow of control in the components of WHISPER is shown in Figure 4 and is adapted from [Funt, 1980]. Reasoning in WHISPER is conducted in the following way: the HLR asks its \perceptual system" to \look at" the vector representation of the diagram with its raster polar representation retina. The retina locates objects and their supports, and the HLR examines the stability of objects shown in the diagram. Unstable objects can rotate or slide. The retina then visualises the rotation or sliding of the dominant objects, and subsequently the diagram is accordingly 8

High Level Reasoner

Questions Update Diagram

Answers to Questions

Retina Map Diagram onto Retinal Processors Diagram

Figure 4: The architecture of WHISPER modi ed. Now the problem solving starts from the beginning until the structure is stable. See one of the examples that WHISPER can solve in Appendix B on page 17.

4.2 Iwasaki, Tessler and Law's REDRAW

Iwasaki, Tessler and Law implemented a system called REDRAW (REasoning with DRAWings) which solves problems of qualitative structural analysis in qualitative physics [Iwasaki et al, 1995]. It combines standard structural analysis with diagrammatic reasoning for qualitative analysis of simple frame structures under load. More precisely, it determines the de ection shape of a building frame structure under load. The authors implemented two versions of REDRAW. REDRAW-I reasons solely with diagrams, whereas REDRAW-II employs equations as well as diagrams. The system was implemented in KEE, a Lisp-based object-oriented knowledge engineering environment. The main goal of this system is to reason in the similar way as the human engineer does. The information does not need to be represented absolutely accurately, but just a sketch of the de ected shape gives the idea of what happens to a frame subjected to a load. The numerical details can be calculated separately. The input to the system is a frame structure and a load. Its output is a de ected shape of the frame. Given the input, REDRAW produces a diagram and the symbolic model for reasoning about non-diagrammatic concepts. According to the structural engineering knowledge of the system, the diagram is modi ed as the load is applied, meeting the constraints applied to the structure. This process propagates through until the nal shape of the frame is stabilised. An example of the problem solved by REDRAW is given in the Appendix B on page 17. The components of REDRAW system are the following: Structure Layer: is a symbolic reasoning component. It contains the structural engineering knowledge about the objects of the structure such as beams, columns, connections, supports, loads. It also contains information about constraints that are imposed on the shape such as equilibrium condition. To use this knowledge it includes heuristics and logical inference rules to control the structural analysis. Diagram Layer: is a diagrammatic reasoning component. It includes the internal 2-dimensional representation of the shape of the frame. The internal representation of the shape is a combination of a bitmap of the points in the picture and a symbolic representation in terms of x-y coordinates. The elements of the graph are lines, points, angles, splines, circles. It also contains the information about the geometric relations, coordinates, directions, and graphical operations such as rotate, bend, translate, smooth, etc. Translator: it mediates the communication between the structure and the diagram layer. For example, the structure layer sends a command or posts a constraint on the diagram layer, which is carried out or checked by the diagram layer. Furthermore, a representation of the 9

beam (or an operation of de ection of a beam) in the structure layer is translated into the representation of the segment (or a transformation of the segment) in the diagram layer. The architecture of REDRAW is presented in Figure 5 and is due to [Iwasaki et al, 1995].

(Deflect B1 :dir down)

TRANSLATOR

STRUCTURE LAYER

DIAGRAM LAYER (Bend B1.pic :y -)

Diagram Representation

Figure 5: The architecture of REDRAW This system appears to be the most similar in many respects to the system that we have in mind. The closest similarity of REDRAW to the system that we aim to implement is that REDRAW solves problems only through the manipulation of the diagram, and is therefore much more intuitive and closer to human reasoning than purely symbolic reasoning. The operations that can be applied to the diagram are determined by the diagram and the information that the diagram layer contains on the purely diagrammatic constraints (such as: you cannot bend a rigid connection). In our case we aim to do the same. Namely, the state of the diagram, ie. the information it contains will tell us about the possible geometric operations that we can conduct on the diagram. The description of REDRAW system in [Iwasaki et al, 1995] does not explicitly state whether the user needs to input the actual diagram as well as the information describing the diagram. This should not be particularly important, since REDRAW deals with very domain speci c problems. It works on the set of problems that are inherently diagrammatic, ie. problems that are described purely by qualitative information.

5 Other Related Work There have been several other researchers, who have implemented diagrammatic reasoning systems that I am going to brie y mention in this section. However, their work does not necessarily relate closely to ours, but is interesting in terms of diagrammatic reasoning features that they use. These systems range through various problem domain. 1. Plane Euclidean Geometry:  A while before Funt implemented his WHISPER [Funt, 1980] system, he implemented another system called CONSTRUCTOR [Funt, 1973] which is a problem solving program handling Euclidean geometry straight-edge and compass constructions. The program input is restricted English description of the problem de nition. Its output is equally, a restricted English description of the instruction steps to draw a gure that has been requested. The program is interesting in that it uses the knowledge from partially constructed gures to see what other speci c instructions need to follow to be able to construct the gure. In this sense it is analogous to the theorem prover use of the results of the previous inferences to be able to make the next inference. The program, however, does not explicitly use the diagram, despite the fact that it reasons about it.  Nevins implemented a geometry theorem prover [Nevins, 1975] which is claimed to be one of the most powerful existing geometry expert system. The key feature of his system is forward reasoning strategy, for which he claims is the way humans 10

think. Certain features of the diagram cue the inference steps, which are made using a number of paradigms. The paradigms are guided by the diagram and can make multiple conclusions. They are capable of making inferences that require multiple steps. In many ways Koedinger and Anderson's DC (see section 3.3) system extends the Nevins model. However, the model does not visually construct the diagram, nor does it use the numerical information from the diagram.  McDougal and Hammond's Polya [McDougal & Hammond, 1993] is a geometry theoremprover. Its input is a list of givens, a goal and a diagram. Its output is a proof which is arrived at after a series of interpretations of plans for visual search and plans for writing proofs. The diagram is described in terms of Cartesian coordinates, marks for segments and marks for angles. For more information see [McDougal, 1993] and [McDougal & Hammond, 1995]. 2. Heterogeneous Logic - ie. a logic that models inferencing between dierent kinds of representations (eg: sentential and diagrammatic):  Barwise and Etchemendy's Hyperproof reasons about the blocks world. An interesting point is that it does so by graphical and symbolic inferencing. It is an interactive tool for proof checking, as opposed to an automated theorem prover. Hyperproof uses a diagram for a concise representation of a complex system aiming to aid human reasoning. The user can take advantage either of conventional symbolic inference rules or graphical inference rules. For more information on Hyperproof see [Barwise & Etchemendy, 1991]. 3. Qualitative Physics:  BITPIC was implemented by Furnas and is a \falling balls" simulation system, which uses purely diagrammatic reasoning method for the simulation. It operates on a raster image through a series of inference rules, which are in fact local picture, pixelneighbouring rewrite rules. A nal output of the system is a series of snapshots of the simulation. For more information see [Furnas, 1990]. Along with all the research that has been done on diagrammatic reasoning there are several journals and conferences that deal speci cally with this topic. Some of them are Journal of Visual Languages and Computing, IEEE Workshops on Visual Languages and IEEE Conferences on Visualisation. The two most signi cant events in this branch of research have been the AAAI Spring Symposium on Reasoning with Diagrammatic Representations in March, 1992 (the working notes were later edited by H.N. Narayanan and published in [Narayanan, 1992]), and the release of the book on Diagrammatic Reasoning by [Chandrasekaran et al, 1995].

6 Discussion If we compare the main functionality features that we aim to implement in our system to the other diagrammatic systems described in previous sections, then we can conclude that the related systems are functionally orthogonal to the one that we are proposing in this paper. Namely, they use diagrams to guide the proof search and to prune the proof search-tree by rejecting subgoals (which are expressed symbolically) that are false in the diagram. Symbolic (in contrast to diagrammatic) inferences then prove the theorem. We, on the other hand want to reverse the role that the symbolic inferencing and the diagram have by designing a system that will use diagrammatic inference steps instead of symbolic ones. Thus, the high level reasoning will be entirely dependent on the diagram and the diagrammatic methods. The propositional statement of the theorem will help guiding our proof search in the sense that it will suggest to us what diagram to start the proof from, and what possible diagrammatic inference steps might be next in the proof. For example, if we are dealing with 11

the theorem, the conclusion of which is n2 , then this is a clear suggestion that the initial diagram for a proof should be a square. Furthermore, if the theorem involves summation of integers, this is a clear indication that the representation of a square should be enumerative, ie. the square should be represented as a collection of counters (dots), rather than being represented metrically. It remains a matter of my research to determine how many of these heuristic methods will be performed by users, how many will be automated, and to what extent these methods can be automated. In general, the area of diagrammatic reasoning is becoming more important, and more research has been done in this direction in the past few years. The importance of diagrams became more apparent, as well as the fact that symbolic evaluation is not the only rigorous and formal way of proving theorems. We hope that by automating the `informal' diagrammatic proofs in mathematics our system will illuminate the signi cance of diagrams in human thinking.

7 Conclusion I have described in this survey paper several diagrammatic reasoning system which have been implemented in the past, that relate to the system that we aim to implement. They all have a common factor of using the diagram, or some sort of diagrammatic representation of the diagram to guide the search for the proof of a theorem in that they either refute subgoals of the proof that are simply false in the diagram, or they help making inference steps of the proofs by suggesting the next subgoal. However, none of the systems that I have described uses only the diagram to make inference steps. All of them make symbolic inference steps and then check their correctness in the diagram. We want to develop a system whose inference steps are going to be diagrammatic operations only. In this way we shall enable the user to see what the result of the inference step is, and thus ultimately nd the proof much more understandable and intuitive.

12

A Examples Theorems for Our Systems The following section shows the examples of the problems that we aim to prove diagrammatically, ie. with application of geometric operations as inference steps on a diagram. They are classi ed into two Categories according to the inherent nature of the proof.

A.1 Example Proof of Category 1

Pythagorean Theorem 1

The Pythagoras' Theorem states that the square above the hypotenuse of a right angle triangle equals to the sum of the squares above its sides. Example is taken from [Nelsen, 1993, page 3]. The mathematical formulation is: a2 + b 2 = c 2

SS SS S b

b

SS  S S  SS SS   SS

a

   a

c

c

b

a

b

a

b

a

Take a square and partition it with two cuts into two squares and two rectangles. Cut the rectangles, which are identical, down the diagonal, so that four identical right angle triangles are formed. Now join back the triangles into a square so that each side of the square is formed from one side of one triangle and the other side of another triangle. Note that in this way the size of the square is preserved and the square in the middle is the square above the hypotenuse. Thus, when we subtract the areas of the four triangles from the original and the new square, it is clear that the sum of the squares above the sides of the right angle triangle (in the original square) equals to the square above the hypotenuse of this triangle (in the new square).

Diagrammatic Proof:

1. Cut a square into 2 rectangles. 2. Cut it again perpendicular to the rst cut so that 2 squares and 2 identical rectangles are created. 3. Cut the rectangles down the diagonal (ie. we now have 4 identical right angle triangles). 4. Join the triangles into a square so that a side of a square is made up of 2 dierent sides of 2 triangles.

A.2 Example Proof of Category 2 Sum of Odd Integers

This example is also taken from [Nelsen, 1993, page 71]. The theorem about the sum of odd integers states the following:

Xn (2i ? 1) = n i=1

13

2

jjjjjj jjjjjj jjjjjj jjjjjj jjjjjj jjjjjj 1 + 3 + 5 + : : : + (2n ? 1) = n2 If we take a square we can cut it into as many L's (which are made up of two adjacent sides of the square) as the size of the side of the square. Note that one L is made out of two sides, ie. (2  n), but the joining vertex of the two sides has been counted twice. Therefore, one L has a size of (2  n ? 1), where n is the size of the square.

Schematic Diagrammatic Proof: 1. Cut a square into n L's, where an L consists of 2 adjacent sides of the square. 2. Cut each L into two segments. 3. For each L, join these two segments one on top of the other length-wise (note that one of the two segments is always one unit longer than the other, thus an L always consists of an odd number of units).

A.3 Example Proof of Category 3 Geometric Sum

The formal de nition of a geometric sum1 is given as follows:

X1 1i = 1 i=1 2

. ..

1 23 1 2

1 24

1

1 22

1 1 2

+ 41 + 81 +


= 1

Take a square of unit size. Cut it down the middle. Now, cut one half of the previous cut square into halves again. This will create two identical squares making up a half of the original square. Take one of these two squares and continue doing this procedure inde nitely. 1

Example is taken from [Nelsen, 1993, page 118].

14

Diagrammatic Proof: 1. Take a square and cut it in half. 2. Take 1 of the halves of the square and cut it in half. 3. Take 1 of the new smaller squares created by the cutting procedure and repeat the rst two steps. 4. Continue doing this forever.

B Examples of Theorems Proved by Various Systems The following sections show some of the example theorems that the systems described in this paper can prove.

B.1 Geometry Machine

Theorem: In a quadrilateral with one pair of opposite sides equal and parallel, the other pair of sides are equal.

Premises

Quad-lateral ABCD Segment BC parallel segment AD Segment BC equals segment AD CA DB AC BD

BA AB DC CD

B

DA CB BC AD A

Goals

Segment AB equals segment CD

I am stuck, elapsed time = 0.88 minute Construct segment DB Add premise segment DB Restart problem

Solution

Segment BC parallel segment AD Premise Opp-side CADB Assumption based on diagram Segment DB Premise Angle ADB equals angle CBD Alternate interior angles of parallel lines Segment BC equals segment AD Premise Segment BD equals segment DB 15

C

D

Identity Triangle ABD Assumption based on diagram Triangle ABD congruent triangle CDB Side-angle-side Segment AB equals segment CD Corresponding elements of congruent triangles Total elapsed time = 4.06 minutes

B.2 BTP - Basic Theorem Prover

This example has been proved by Gelernter's Geometry Machine, however, Goldstein tried to show that his system can deal with the same problems as the Geometry Machine plus several more. The problem is very simple and serves as a demonstration of how the BTP system works.

Statement: The angle bisector is equidistant from the rays of the angle. Hypotheses: SEG DB BISECTS ANGLE ABC SEG DA PERPENDICULAR SEG BA SEG DC PERPENDICULAR SEG BC

A D

Objective: SEG AD = SEG CD Proof: BTP ON GELERNTER, THEOREM 1 STEPS 1 SEG DB = SEG DB 2 ANGLE DBA = ANGLE DBC 3 RIGHTANGLE DAB 4 RIGHTANGLE DCB 5 ANGLE DAB = ANGLE DCB 6 TRIANGLE ADB = TRIANGLE CDB 7 SEG AD = SEG CD

B

C

REASONS ;by identity ;by hypothesis ;by hypothesis ;by hypothesis ;right angles are equal ;by ass ;corresponding sides of congruent ;triangles

QED

B.3 GROVER

I shall not describe the whole of the theorem proving example in GROVER. Instead, the theorem will be stated and the diagram used to guide the proof of the theorem will be demonstrated. In the example, the theorem that I shall describe is the Diamond Lemma. For more details see [Barker-Plummer & Bailin, 1992]. Diamond Lemma is a non-trivial theorem in the theory of well-founded relations. It is stated as: LCRR(x) ^ W FR (x) ! GCRR(x) where LCRR (x) states that the relation R j x (\R restricted to the set x") has the local ChurchRosser property, and GCRR(x) states that the relation R j x has global Church-Rosser property. The diagram used to prove the Diamond Lemma is as follows:

16

!a R

R

R

R

d

e

R*

R* f

!b

!c R*

R* g

R* R* h

B.4 WHISPER

Problem: predict the sequence of events occurring during the collapse of a \blocks world" Input: structure.

Output:

B.5 REDRAW

The diagrams below show a solution sequence generated by REDRAW given the description of the frame and the load that it is subjected to. LOAD3 JOINT3

BEAM3

COL3

JOINT4

COL4

SUPPORT3

SUPPORT4 (a)

(b)

(c)

(d)

17

References [Arnheim, 1969]

Arnheim, R. (1969). Visual Thinking. University of California Press, Berkeley, CA. [Bailin & Barker-Plummer, 1993] Bailin, S.C. and Barker-Plummer, D. (1993). Z-match: An inference rule for incrementally elaborating set instantiations. Journal of Automated Reasoning, 11(3):391{428. [Baker et al, 1992] Baker, S., Ireland, A. and Smaill, A. (1992). On the use of the constructive omega rule within automated deduction. In Voronkov, A., (ed.), International Conference on Logic Programming and Automated Reasoning | LPAR 92, St. Petersburg, Lecture Notes in Arti cial Intelligence No. 624, pages

214{225. Springer-Verlag. [Barker-Plummer & Bailin, 1992] Barker-Plummer, D. and Bailin, S.C. (March 1992). Proofs and pictures: Proving the diamond lemma with the GROVER theorem proving system. In Working Notes of the AAAI Symposium on Reasoning with Diagrammatic Representations, Stanford, USA. [Barker-Plummer et al, 1995] Barker-Plummer, D., Bailin, S.C. and Ehrlichman, S.M., (November 1995). Diagrams and mathematics, Draft copy of an unpublished paper. [Barwise & Etchemendy, 1991] Barwise, J. and Etchemendy, J. (1991). Visual information and valid reasoning. In Zimmerman, W. and Cunningham, S., (eds.), Visualization in Teaching and Learning Mathematics, pages 9{24. Mathematical Association of America. [Chandrasekaran et al, 1995] Chandrasekaran, B., Glasgow, J. and Narayanan, N.H., (eds.). (1995). Diagrammatic Reasoning: Cognitive and Computational Perspectives. AAAI Press / The MIT Press. [Funt, 1973] Funt, B. V. (October 1973). A procedural approach to constructions in euclidean geometry. Unpublished M.Sc. thesis, University of British Columbia. [Funt, 1980] Funt, B. V. (1980). Problem-solving with diagrammatic representations. Arti cial Intelligence, 13:201{230. Reprinted in \Diagrammatic Reasoning: Cognitive and Computational Perspectives", Glasgow, J., Narayanan, N. H., and Chandrasekaran B. (eds.), AAAI Press / The MIT Press, 1995, pages 33-68. [Furnas, 1990] Furnas, G. W. (1990). Formal models for imaginal deduction. In Proceedings of the Twelfth Annual Conference of the Cognitive Science Society, pages 662{669. Lawrence Erlbaum Associates. [Gelernter, 1963] Gelernter, H. (1963). Realization of a geometry theoremproving machine. In Feigenbaum, E. and Feldman, J., (eds.), Computers and Thought, pages 134{52. McGraw Hill. [Goldstein, 1973] Goldstein, I. (1973). Elementary geometry theorem proving. AI Memo 280, MIT. 18

[Iwasaki et al, 1995]

Iwasaki, Y., Tessler, S. and Law, K.H. (1995). Qualitative structural analysis through mixed diagrammatic and symbolic reasoning. In Glasgow, J., Narayanan, N.H. and Chandrasekaran, B., (eds.), Diagrammatic Reasoning: Cognitive and Computational Perspectives, chapter 21, pages 711{ 729. AAAI Press / The MIT Press. [Koedinger & Anderson, 1990] Koedinger, K.R. and Anderson, J.R. (1990). Abstract planning and perceptual chunks. Cognitive Science, 14:511{550. Reprinted in \Diagrammatic Reasoning: Cognitive and Computational Perspectives", Glasgow, J., Narayanan, N. H., and Chandrasekaran B. (eds.), AAAI Press / The MIT Press, 1995, pages 577-625. [Kulpa, 1994] Kulpa, Z. (1994). Diagrammatic representation and reasoning. Machine GRAPHICS & VISION, 3(1/2):77{103. [Larkin & Simon, 1987] Larkin, J.H. and Simon, H.A. (1987). Why a diagram is (sometimes) worth ten thousand words. Cognitive Science, 11:65{99. Reprinted in \Diagrammatic Reasoning: Cognitive and Computational Perspectives", Glasgow, J., Narayanan, N. H., and Chandrasekaran B. (eds.), AAAI Press / The MIT Press, 1995, pages 69-109. [McDougal & Hammond, 1993] McDougal, T.F. and Hammond, K.J. (1993). Representing and using procedural knowledge to build geometry proofs. In Proceedings of the Eleventh National Conference on Arti cial Intelligence, pages 60{65. AAAI Press / The MIT Press.

[McDougal & Hammond, 1995] McDougal, T.F. and Hammond, K.J. (1995). Using diagrammatic features to index plans for geometry theorem-proving. In Glasgow, J., Narayanan, N.H. and Chandrasekaran, B., (eds.), Diagrammatic Reasoning: Cognitive and Computational Perspectives, chapter 17, pages 691{709. AAAI Press / The MIT Press. [McDougal, 1993] McDougal, T.F. (1993). Using case-based reasoning and situated activity to write geometry proofs. In Proceedings of the Fifteenth Annual Conference of the Cognitive Science Society, pages 711{716. Lawrence Erlbaum Associates. [Narayanan, 1992] Narayanan, N.H., (ed.). (1992). AAAI Spring Symposium [Nelsen, 1993] [Nevins, 1975] [Sloman, 1985]

on Reasoning with Diagrammatic Representations: Working Notes. AAAI Press. Nelsen, R.B. (1993). Proofs Without Words: Exercises in Visual Thinking. The Mathematical Association of America.

Nevins, A. (1975). Plane geometry theorem-proving using forward chaining. Arti cial Intelligence, 6:1{23. Sloman, A. (1985). Afterthoughts on analogical representations. In Brachman, R.J. and Levesque, H.J., (eds.), Readings in Knowledge Representation, pages 432{439. Morgan Kaufmann. 19