Graph Decomposition and Its Use for Ontology ...

Graph Decomposition and Its Use for Ontology Verification and Semantic Representation Julia M. Taylor

Victor Raskin

CERIAS & Linguistics Purdue University West Lafayette, IN, USA [email protected]

Linguistics & CERIAS Purdue University West Lafayette, IN, USA [email protected]

Abstract—This paper explores the use of a graph decomposition technique applied to an ontology, designed as a basis for understanding natural language text. It demonstrates how the technique can help with ontology verification, detecting common human errors in concept acquisition, as well as having some ramifications for semantic processing of multiple noun expressions, among other natural language entities. Ontology verification with the help of graph decomposition is illustrated on the phenomena of inverse properties, transitive properties, and hierarchical (ancestor/descendant) properties. The ontology on which the technique has been tested is that of Ontological Semantic Technology, a significantly modified version of Ontological Semantics [5]. Keywords—ontological semantic technology, verification, natural language, graph decomposition

ontology

I.

EXPLOITING THE FORMAL SIDE OF ONTOLOGICAL SEMANTICS Ontological Semantics ([8] and references there) has been motivated primarily by representing natural language meaning comprehensively and deeply rather than not at all or selectively and shallowly. It separates its resources into a language independent property-rich ontology, which reflects the knowledge of the world, and language-dependent lexicons where every sense of every word or phrasal is defined in ontological terms and the knowledge of a specific language is accommodated. At the same time, the approach has used a mathematical foundation and a reasonably clear but loosely defined mechanism (see, however, [8]:7.1.1, 7.1.6) that has been taken for granted but not protected from mis- and abuse in acquisition. Its ontology was definitely more engineering, in the sense of being result-oriented, rather than formal, i. e., fully dedicated to being built in strict accordance with the mathematical rules—and that prevented the school from taking conceptual and processing advantage of its own presumed mathematical properties. The formal ventures within the Ontological Semantics of the 1990s were rare, and even when incorporated into its main body ([8]:8.3.1-8.3.2), they were not implemented to any significant extent. Just about the only mathematical property acted upon was the hierarchical IS-A property on which simple inheritance is based, but even there, the not so rare phenomenon of multiple parenting was never accommodated sufficiently clearly and consistently, and the This research was supported in part by the National Science Foundation Grant No. 1012208 (Division of Computer and Network Systems, Trustworthy Computer Program

availability of a reliable acquisition tool that would preclude human inheritance errors was sporadic. Besides, not to quote Bill Clinton, “what is is” was never clearly defined (cf. the classical [1] and its follow-ups). As a result, the so-called legacy resources [7], resulting from the research efforts and small academic proof-of-concept implementations of the 1990s, remain quite faulty, both formally and, accordingly, substantively. The Ontological Semantic Technology (OST) of the late 2000s [9,5,10], besides modifying and, on occasion, departing from the earlier tenets, has been also increasingly more conscious of its formal foundations and interested in deepening their understanding and significance for adequate meaning representation and processing, such as inferencing and reasoning, where important research outside of the approach has been done on their logical aspects, and while not deep or complex enough for real NL processing needs, it would be unavailable for use by OST unless the latter strictly conformed to its own professed formal properties as well as developing and improving them to its substantive advantage. One major step in this direction was virtual ontology [10] that suggested a totally new perspective on the bifurcation of Ontological Semantic resources into the ontology and lexicons. This paper extends the mathematical foundations of the OST ontology by introducing—or rather adapting for the purpose—the notions of (de)composition and derivation. Seen as a graph (see [4] for the basic background), the ontology is an incredibly complex and interweaving aggregate of vertices and edges. Decomposition slices this graph into multiple graphs, each organized around—or derived for—one property. The paper also shows that decomposition helps to verify certain features of the ontology, detecting and removing human error with regard to these features. It can be used as well for processing multi-noun expressions, or noun chains, where the nouns in question do not provide a hint to the properties that connect them. Graph decomposition helps here both directly and indirectly: the former is achieved by finding the property whose decomposed graph yields the shortest path between the two concepts underlying the constituent nouns; the latter is provided through ontology verification, thus improving the direct decomposition results. The accurate processing of noun chains may come from the background information contained

in the ontology or gathered from elsewhere in and outside of text, and this paper will rely entirely on the ontological information rather than on the linguistic or extralinguistic context, thus providing a predominantly paradigmatic rather than syntagmatic solution. In general, ontology verification is a crucial process, and it has been approached, mostly in formal ontology, for over a decade. A major step forward was OntoClean [2,3,12], a comprehensive system for checking on the intensions of the central principles and types of relations in formal ontology, the major premise being that any deviation from the prescribed rigor would result in crippling the engineering applications of ontology. Realizing that OntoClean is rather rarefied to use practically—and, in fact, has not seen too many practical applications—[11] proposed a somewhat more accessible application model. AEON, for using OntoClean, weaving some elements of natural language analysis into it. Closer to our own enterprise, [13] suggested an experimental model for evaluation an ontology on the basis of its performance in an application. This is indeed our ultimate stance on ontology verification and evaluation. We do, however, believe that quality control is essential in the process of ontology acquisition, long before an application is ready to work and expose errors that should be corrected. Our approach is different also because our ontology parsimoniously supports a set of large lexicons and models deep, comprehensive meaning rather than the shallow semantics that the ubiquitous [6] ascribes to ontologies in an attempt to endear them to the proponents of statistical and machine learning methods. Additionally and most importantly, OntoClean and its spin-offs are concerned with taxonomy, or vertical cleaning. We assume such cleaning to have been accomplished and focus instead on horizontal cleaning, that is the properties other than is-a. Most processing errors in OST and other rule-based NLP systems are due to acquisition errors, especially in the ontology, where a mistake in making a restriction too broad or too narrow, for instance, has serious ramifications, as opposed to a lexicon error that affects only one entry. In the following sections, we demonstrate how graph decomposition works and how it can help with ontology verification and what properties lend themselves to such verification most readily. II.

GRAPH DECOMPOSITION

We treat ontology as a lattice that contains concepts and relationships among them. Structurally, an ontology can be considered to be a directed graph, where the relationships serve as edges. We can then exploit the structure and properties of a graph to verify the intuitive structure of an ontology, constructed from the linguistic point of view. Unlike a typical graph, where the edge has only one meaning per graph, and that is that of a (directional) path, cost, or other single-meaning metric, most edges in the ontology have different meanings, namely those of a property that represents a relationship between the connecting conceptual vertices. Looking at an ontology through a one-property prism can provide valuable insight, as we hope to show in this paper.

Prior to decomposing the ontology, the inherited information of each concept has to be taken into account. A lot of concepts inherit information from their ancestors. A concept may either inherit a property and a filler from its parent, or it may narrow down the filler of its parent using the same property, or it may add a new property that a parent concept does not have. The inherited information can be explicitly restated in the child concepts, for the simplicity of calculation, or be adjusted by the weight depending on the distance from the source of inheritance. Whenever properties themselves are arranged in a hierarchical order, a parent property used in a concept can be legally narrowed down to the child property with the same or narrower filler. We look at an ontology as a composition of graphs, one graph per property, where each graph has a single meaning for its edges. For example, the IS-A graph contains a hierarchy of the concepts; the AGENT graph only connects vertices that are in the AGENT relationship with each other. The ontology is, then, an ordered pair O = (C, P) with a set C of concepts together with a set P of properties, which could be looked at as a two-element ordered subset of C, as each property connects two concepts. In other words, our ontology is a labeled multigraph, formally defined as: (1)       



O = (∑C, ∑P, C, P, s, t, lC, lP), where C is a set of concepts or nodes P is a set of properties or multiset of arcs or edges ∑C is a finite alphabet of the available concept labels ∑P is a finite alphabet of the available property labels s: C P is a map indicating the source node of an arc t: C P is a map indicating the target node of an arc lC: C∑C is a map describing the labeling of conceptual nodes lP: P∑P is a map describing the labeling of property arcs

When a weighted ontology is considered, O = (C, P, w), with nonnegative property weights wst ≥ 0, (s, t)∈C instead of O = (C, P). A. The structure of properties and their effect on decomposition There are two cases that should be looked at in ontology decomposition: when properties are independent and unordered and when they are arranged in a partial order. In the first case, when the properties are unordered, the ontology can be decomposed such that: (2) O={O1, …Oi, …, On}, where Oi = (∑C, pi, C, Pi, s, t, lC, lP), pi ∈∑P and ||P||=n, with i indicating each property. In other words, for every property, a sub-ontology Oi can be created that spans some concepts in the ontology using only the chosen property.

The situation is more difficult in the second case when the properties are ordered, especially as a hierarchy. For example, we may have a property LOCATION with 3 children: PATH, START-LOCATION, END-LOCATION, and we are constructing a LOCATION graph. The question arises: should its children be included on its graph? If we follow (2), then none of the concepts connected through PATH, START-LOCATION, ENDLOCATION will be shown here, but intuitively, they should be, possibly with a slightly different weighting preference.

A. Inverses What is of interest in graph decomposition, from the ontology verification standpoint, is that some errors in inverses can now be detected by looking at the composition of Oi and Oj, where the former is the “direct-property’ graph and the latter its inverse, for instance, INSTRUMENT and INSTRUMENTOF. Figure 1 shows Oi and Figure 2 the composition graph of Oi and Oj.

We then propose the following construction of any OiC such that Pi is a parent of Pj: (3) OiC=Oi⊗ Oj where lP of Oj takes preference over lp of Oi. It is possible to construct a weighted graph instead, where arcs corresponding to Oj and Oi. are included, but with different weights, which are determined at the ontology verification stage:

Figure 1. Property graph, where blue vertices indicate concepts, green vertices indicate property fillers

(4) OiC=Oi ⊗ Oj where both lP of Oj and lp of Oi are included with some weight. Trivially, then, (5) if Pi is a parent of Pj, then Oi = Oj ⊗ X, where X is some graph and Oi is not O. It follows from (5) that if Ok is a subgraph of Oi, then Pk is a child of Pi, and should be seen as such in the ontology O. B. What type of graph are we looking based on source and target Oi can be further described based on the relationships between all pairs of (s, t) with respect to a chosen arc set. In many cases, ontological properties have non-intersecting domains and ranges, thus forming two disjoint sets of S and T, with Oi resulting in a bipartite graph. This situation is especially typical for properties that connect EVENTS and OBJECTS, such as AGENT, BENEFICIARY, etc, and their inverses AGENT-OF, BENEFICIARY-OF, etc. On the opposite side of the spectrum, are decomposed graphs whose domains and ranges are the same set. It is especially interesting to look at these graphs for transitive properties for ontology verification purposes, which is one of the topics in the next section. III.

APPLICATIONS OF GRAPH DECOMPOSITION: ONTOLOGY VERIFICATION In this section, we will illustrate, on a number of property type examples, how graph decomposition may be used to detect acquisition errors. We will briefly look at the cases of inverse properties, transitive properties, and cases where properties are parents with children. We will see how an acquirer can do something very wrong while not committing any error from the point of view of the syntax of the formalism nor from the point of view of its semantics, in the particular case of the acquired material, but it will create a contradictory situation in that part of the ontology.

Figure 2. A property (blue edges) and its inverse (green edges)

A concept like DIG will, of course, have a concept like as its instrument. Correspondingly, SHOVEL will be described as INSTRUMENT-OF for DIG. It may lead to a situation where DIG will be erroneously constructed from the inverse property as the only instrument of SHOVEL or, conversely, that it will be set up as the only concept which is INSTRUMENT-OF for SHOVEL. The decomposed graph on the INSTRUMENT-OF property will, however, show that there are other nodes connected to DIG with this property. It is this disparity that will detect the inverse acquisition errors of this type. While no syntactic errors in the formalism are made in this type of error, a semantic inconsistency is detected: it is true that SHOVEL is indeed INSTRUMENT-OF for DIG and, conversely, DIG does have SHOVEL as an (but not the) INSTRUMENT. The mistake is that there are not the only possible such fillers for their respective properties, and leaving the error uncorrected will lead—and has led—to serious processing mistakes. SHOVEL

B. Transitive properties Let us assume that there is a transitive property Pi in the ontology, resulting in a transitivity chain from A to B to C to D to E, as shown on Figure 3.

The weighted graph decomposition of the hierarchical properties can be used to create a simple solution: create a decomposition of Oi, where i is a parent property, according to (4). If there is a pair (s, t) such that they are connected by more than 1 arc, as shown in Figure 5, flag these nodes as erroneous and alert an ontology engineer.

Figure 3. Transitive property graph, chain of interest is in green

The property connecting these vertices may be, for instance MADE-OF, the inverse of MATERIAL, with A = NECKLACE, B = BEAD, C = GLASS, D = SAND, and E = SILICA. The acquisition

error is between A and B: in frequently occurring cognitive slippage, the acquirer misunderstood the ontological property MADE-OF as an English phrasal, which may have a different sense, that of ‘constructed of’ rather than ‘composed of.’ The error will be hiding in the ARTIFACT branch and not immediately visible in the branch containing natural materials. The graph decomposed on the MADE-OF property will present the whole path to an ontology engineer, who will respond negatively to the implied question whether necklaces are really made of sand or silica by cutting A from at least D and E, as per Figure 4.

Figure 5. Hierarchical properties

Domain decomposition also makes it possible to check if one property could potentially be a parent of another property and alert an ontology engineer. A property Pp is considered to be a parent property of a Property Pc, if there exists OPc that forms a subgraph of OPp, in other words, every vertex of OPc is in OPp and every act of OPc is in OPp, as shown in Figure 6. If the ontology engineer decides that such properties were not meant to be in a parent-child relationship, at least one arc should be created to indicate that such relationship is incorrect.

Figure 4. Transitive property with a mistake

C. Hierarchical properties Some properties in the ontology are structured in a hierarchy, such as LOCATION, START-LOCATION, ENDLOCATION, with LOCATION being the parent of the other two. Any concept that has a parent property can have child properties, but then the child property overrides the parent one. For example, TRANSFER-LOCATION, a descendent of EVENT, can have a START-LOCATION and END-LOCATION, but no longer a LOCATION which EVENT has. Such constraints on the domains can be trivially controlled. It is more difficult to control the range in such cases, especially when something can be in the range of a child property for one concept, and the range of a parent property for a different concept. For example, a CITY can be in the range of END-LOCATION of TRAVEL as well as in the range of LOCATION of OCCUPY. On the other hand, if we look at CITY, it is in the domain of END-LOCATION-OF (range TRAVEL) and LOCATION-OF (range OCCUPY) at the same time. If we go with the trivial domain validation for these properties, we would come up with an error, even though this is a perfectly legal situation. Moreover, CITY could have had LOCATION-OF any EVENT, which does have an error.

Figure 6. Domain decomposition of hierarchical properties

D. Practical applications of ontology verification The results of ontology verification that lead to correction of ontological errors are best seen in the working of noun chains, i. e., N + N*… The reason noun chains are an obvious demonstration of how important the correctness of the ontology is, is due to the fact that such chains do not explicitly provide a property that can connect the nouns. Such a property or several properties can only be found with the help of an ontology. Consider, for example, the noun chain goat milk bottle. No matter what the sentence says, the relationship between milk and bottle and goat and milk will not be specified there. There is also no preposition that can suggest the relationship. Thus, all the work is carried by the ontology and knowledge contained in it. It is possible to interpret these noun chains with the help of graph decomposition. For each pair of concepts corresponding to the words, the shortest path is found in each Oi decomposition, thus resulting in a list of (value, property i) pairs. It is reasonable to suspect that the smallest non-zero

ALKALOID

NITROGEN

AIRPORT

DIRECTION-SIGN

AIRPORT-TERMINAL

PORTER

FAST-FOOD-RESTAURANT BED

MATTRESS

BROILER

STOVE

BURNER

FREEZER

OVEN

DWELLING KITCHEN

REFRIGERATOR

ARMCHAIR

CHAIR-ARM

WAITING-ROOM

SHOPPING-COMPLEX

FOOD-SERVICE-ORGANIZATION

CHAIR

SEAT

IDENTIFICATION-SIGN

RESTROOM

ARTIFACT-LEG

STORE

DINING-ROOM

BASIN

BAY

SEA TOILET

OCEAN

LODGING-CORPORATION

BUILDING-PLACE-PART

WALL

ELECTRICAL-OUTLET

HALLWAY

SERVICE-BUILDING

CEILING NAME-HUMAN

RECEPTION-AREA WINDOW

DIRECTORY

SKYLIGHT

ADDRESS ROOF CONFERENCE-ROOM

BATHROOM SAW GARAGE COOKING-ACCESSORY ANNEX KNIFE ATTIC

BLADE

HOTEL-ROOM RAZOR BASEMENT SHOVEL

BUILDING

ROOM

AX

CLOSET

SHOWER OFFICE-BUILDING BATHTUB

BUILDING-ARTIFACT-PART

FLOOR GEOPOLITICAL-ENTITY

GEOGRAPHICAL-ENTITY PORCH

CONTINENTAL-ENTITY

DOOR

EARTH-SURFACE RAILING PRIVATE-HOME CUP

GEOLOGICAL-ENTITY

SHELLFISH

HANDLE

ELEVATOR SAUCER

VEGETABLE-FOODSTUFF

BALCONY

DAIRY-PRODUCT

DESK

RESEARCH-ENVIRONMENT

EGG

COFFEE-CUP

SOUP

MILK

DRESSER

ORGANIZATION

FISH-MEAT

FRYING-PAN

HAMMER

DRAWER

PAIL SOCIAL-OBJECT

MIRROR TOWN

BODY-OF-WATER

POULTRY

SAUCEPAN

VILLAGE

PUBLIC-SQUARE

PERSONAL-COMPUTER

FAT

MEAT

MICROPROCESSOR

LID

NATION

CITY

MICROCHIP

DISTRICT COUNTY

WATER INTRODUCTION

AIR-VEHICLE

ALTIMETER

ORGANIZATION-DIVISION BIBLIOGRAPHY

CONCLUSION ACADEMIC-PAPER

Figure 7. Fragment of HAS-OBJECT-AS-PART property graph FOOTNOTE

value points to the best property or a set of alternative best properties. For example, it is reasonable to suspect that (1, CONTAINS) will be among the best pairs for anchoring concepts of bottle and milk. Similarly, it is reasonable to assume that (1, ORIGIN) will be among the best pairs for anchoring concepts behind goat and milk, resulting in the following text meaning representation: (bottle(contains(milk(origin(goat))))). TITLE

ARTICLE

ARTICLE-PART

SECTION

TEXT-MANUAL

CHAPTER

Let us suppose that there is also a concept of ALCOHOLICDRINK, and it is CONTAINED-IN a BOTTLE. CONTAINED-IN is an inverse of CONTAINS. If the inverses were not done correctly and the mistake not caught, for instance, with the help of graph decomposition, as shown in the previous section, it is possible that ALCOHOLIC-DRINK would THEN override the concept LIQUID that a bottle can contain. Then, information in the ontology will be such that a bottle cannot contain milk (although milk is still CONTAINED-IN a bottle), which would result in the absence of CONTAINS from the list of properties with the shortest path between milk and bottle and thus prevent the semantic text analyzer from arriving at the text meaning representation above. Similar mistakes can be avoided with the described verification of hierarchical properties in such examples as

Chicago trip (with Chicago—assuming the city sense—being the END-LOCATION or DESTINATION of the trip, rather than a LOCATION). In this case, the list of path-property pairs would consist of only one selection, instead of two, thus removing the ambiguity that the analyzer will signal but that will not really be there. IV.

EMPIRICAL VERIFICATION

The question that we would like to address in this section is how useful graph decomposition turns out to be in a real life system. For the testing scenario, we are using a public domain ontology, accessible through [7] and commonly referred in the literature [3, 6, 8] as the MikroKosmos ontology. We purposefully selected an ontology that had not been touched for years, and the rank-and-file ontology engineers who were most familiar with the ontology had long moved on since. We are thus looking at a sufficiently unfamiliar knowledge resource. The question that we want to address is whether we can estimate the lower bound of changes—i. e., the bare minimum of corrections—that would have to be made to this ontology for it to be useful for natural language applications that we have in mind. It should be noted that such measures are not necessarily an indication of the quality of the ontology, either way, but

rather of the feasible modifications for a particular application. It is assumed here that the hierarchical, or “sortal” correction, addressed in [2-3] is done separately and possibly previously. In addition to the graph decomposition property verification techniques outlined in the section above, another trivial but very important feature should be mentioned. Once the decomposition is accomplished, it is highly convenient to look at the resulting graph to spot inconsistencies and errors practically “at a glance.” For example, it took less than a minute for the authors both experienced knowledge engineers, who had not looked at this ontology for years if at all, to spot an mistake in 1 out of the 5 non-inherited usages of the property SUBJECT in the ontology; about 5 minutes for 27 mistakes in the 74 non-inherited usages of the property HASEVENT-AS-PART. It took about 10 minutes to spot several types of inconsistencies in 465 non-inherited usages of the property HAS-OBJECT-AS-PART. These particular inconsistencies were especially important to catch because they revealed that the ontology was not consistent with regard to mereological principles, thus compromising the integrity of the property, and thus ensuring incorrect interpretations, especially since the property is transitive. The inverse properties were checked in a similar manner, according to the principles outlined in Section III.A above. In less than a minute, property SUBJECT and its inverse SUBJECTMATTER were inspected and an inconsistency in 1 usage out of 11 was discovered. Fig. 7 above is the busiest section of HASOBJECT-AS-PART graph, with inverses marked in blue. The error visibility of transitive properties on these graphs is quite impressive as well, making verification quite expedient. Thus, it took less than 10 minutes to examine the existing transitive links on the same HAS-OBJECT-AS-PART property. There were 31 errors spotted among the 278 links that connected more than 2 nodes. V.

VI. ACKNOWLEDGEMENTS As stated above, this research was supported in part by the National Science Foundation Grant No. 1012208 (Division of Computer and Network Systems, Trustworthy Computer Program). Some of the graphs are results of a software written for this grant by Mr. Blake Self, the resources for which have been maintained by Mr. Adam Hammer of CERIAS at Purdue. REFERENCES [1] [2]

[3]

[4] [5]

[6]

[7]

[8] [9]

[10]

CONCLUSION

In this paper, we explored the graph decomposition of an ontology and illustrated, on the example of the OST technology, how to use it for ontology verification with some ramifications for semantic representation. Ontology verification is crucial when humans and computers collaborate in a hybrid semi-automatic system of knowledge acquisition. This work can be seen as yet another step in increasing the degree of automation in the system for property-rich ontologies. It must be emphasized again that this is used in addition to, not instead of, standard logical inconsistency checks on the ontologies.

[11]

[12]

[13]

Brachman, R. 1983. What IS-A is and isn't. An Analysis of Taxonomic Links in Semantic Networks, IEEE Computer 16:10 Guarino, N. and C. Welty. 2002. Evaluating Ontological Decisions with OntoClean. Communications of the ACM. 45(2):61-65. New York:ACM Press Guarino, N., and C. Welty. 2002. Identity and Subsumption. In R. Green, C. A. Bean, & S. H. Myaeng (Eds.), The Semantics of Relationships: An Interdisciplinary Perspective. Pp. 111-125. Dordrecht: Kluwer. Harari, F. 1969. Graph Theory. Reading, MA: Addison–Wesley Hempelmann, C. F., J. M. Taylor, and V. Raskin 2010. Applicationguided Ontological Engineering, International Conference on Artificial Intelligence, Las Vegas, NE, July Hovy, E.. 2005. Methodologies for the Reliable Construction of Ontological Knowledge. In: F. Dau, M.-L. Mugnier, and G. Stumme (eds), Conceptual Structures: Common Semantics for Sharing Knowledge. Proceedings of the 13th Annual International Conference on Conceptual Structures (ICCS 2005), pp 91–106. Springer Lecture Notes in AI 3596.. KBAE 2002. Knowledge Based Acquisition Editor. CERIAS, Purdue University, W. Lafayette, IN. http://kbae.cerias.purdue.edu:443, login: nlpgroup, password: ch@ng3m3 Nirenburg, S., and V. Raskin 2004. Ontological Semantics. Cambridge, MA: MIT Press. Raskin, V., J. M. Taylor, and C. F. Hempelmann 2010. Guessing vs. Knowing: The Two Approaches to Semantics in Natural Language Processing, Annual International Conference Dialogue 2010, Moscow, Russia Taylor, J. M., V. Raskin, and C. F. Hempelmann 2010. On an Automatic Acquisition Toolbox for Ontologies and Lexicons in Ontological Semantics, International Conference on Artificial Intelligence, Las Vegas, NE, July. Völker, J., D. Vrandecic, and Y. Sure. 2005. Automatic Evaluation of Ontologies (AEON). In Y. Gil, E. Motta, V. R. Benjamins, M. A. Musen, eds. Proceedings of the 4th International Semantic Web Conference (ISWC2005), volume 3729 of LNCS, pp. 716-731. Springer Verlagerlin-Heidelberg, November. Welty, C. and N. Guarino. 2001. Support for Ontological Analysis of Taxonomic Relationships. J. Data and Knowledge Engineering. 39(1):51-74. October,. Welty, C., R. Mahindru, and J. Chu-Carroll. 2004. Evaluating Ontology Cleaning. Proceedings of AAAI-2004. Pp. 311-314. San Jose:AAAI Press. July.