Medical Ontology Validation through Question Answering

Medical Ontology Validation through Question Answering Asma Ben Abacha, Marcos Da Silveira, and Cédric Pruski Ressource Centre for Health Care Technologies (CR SANTEC), Public Research Centre Henri Tudor, 6 avenue des Hauts-Fourneaux, L-4362 Esch-sur-Alzette, Luxembourg {asma.benabacha,marcos.dasilveira,cedric.pruski}@tudor.lu

Abstract. Medical ontology construction is an interactive process that requires the collaboration of both ICT and medical experts. The complexity of the medical domain and the formal description languages makes this collaboration a time consuming and error-prone task. In this paper, we define an ontology validation method that hides the complexity of the formal description languages behind a question-answering game. The proposed approach differs from “classic” logical-consistency validation approaches and tackles the validation of the domain conceptualization. Reasoning techniques and verbalization methods are used to transform statements inferred from ontologies into natural language questions. The answers of the domain experts to these questions are used to validate and improve the ontology by identifying where it needs to be modified. The validation system then performs automatically the ontology updates needed to correct the detected errors. Keywords: Ontology Validation, Natural Language Processing, Question Generation, Medical Domain, RDFS, OWL

1

Introduction

Building and adapting medical ontologies is a complex task which requires a substantial human effort and a close collaboration between domain experts (e.g. health professionals) and ICT engineers. Even if ICT tools and automatic ontology construction techniques are mature enough to support this work [1–3], they provide only partial solutions and manual interventions from ICT experts will always be necessary if a high quality is expected. Ontologies are also the basis for numerous Clinical Decision Support Systems (CDSS) used to support medical activities therefore the quality of the underlying ontologies affects the results of using CDSSs that rely on these ontologies. In consequence, automatically-built medical ontologies (including schema knowledge and individuals description) must be validated by domain experts. However, experts from the medical domain are usually not familiar with ICT formalisms and technologies and must be assisted by knowledge engineers during the validation process, which augments the number of potential errors. If the

2

Asma Ben Abacha, Marcos Da Silveira and Cédric Pruski

validation of the logical and structural aspects of the ontology like inconsistency, incompleteness or redundancy can be done automatically using dedicated tools [4], the validation of the conceptualization is more complex to do and requires relevat tools to assist the domain experts. In this paper, we focus on how to assist Health Professionals (HP) in the validation of the conceptualization of a domain reflected in an ontology (i.e. the adequacy of the ontology with the real world) without requiring a deep knowledge in informatics. The main innovation of this research effort is to provide methods for the validation of ontologies through an interactive question-answering approach. This challenging task involves two main steps: – The generation of questions in natural language from medical ontologies (these questions will be answered by domain experts). – The interpretation of the information acquired from the experts to deduce if the part of the ontology that has been evaluated is valid, invalid or needs further modifications. The remainder of the paper is structured as follows. Section 2 presents related work of the ontology validation field. Section 3 discusses existing criteria for validating ontologies and introduces an overview of our approach. Section 4 presents our method for the generation of natural language questions, while section 5 deals with the interpretation of the provided answers. Section 6 provides a first experimental study of the approach. Section 7 wraps up with concluding remarks and outlines future work.

2

Related Works

Ontology validation is the central point of a big family of approaches interested in evaluating the structural and logical aspects of ontologies. In their work [4], vor der Bruck and Stenzhorn describe a method to validate ontologies using an automatic theorem prover and MultiNet axioms. Recently, the OOPS! system [5] has been proposed. It consists in detecting predefined anomalies or bad practices in ontologies. However, the real world representation dimension, referring to how accurately the ontology represents the domain intended for modelling, is often neglected in existing approaches for ontology validation since this has to be done manually by the experts. In fact, few approaches addressed the validation of the domain-conceptualization side. Some of them focused on interface development to better present large amount of (structured) data without overwhelming users [6], while others used Natural Language Processing (NLP) techniques to interact with domain experts [7]. In this paper, we focus on the second type of approaches and propose a question-answering method based on NLP techniques to validate the functional aspect of the ontology as described in [8]. To our knowledge, the only work that addresses the problem of ontology validation by means of question-answering techniques is the MoKi system presented in [7]. However, the question formulation process do not integrate the fact that

Medical Ontology Validation through Question Answering

3

domain experts (e.g. in the medical domain) are not supposed to be familiar with ICT formalisms and the proposed system still require a substantial intervention of ICT experts.

3

Ontology Validation

In this section we discuss ontology validation criteria and present an overview of our approach. 3.1

Essential Validation Criteria

An ontology defines a set of representational primitives with which to model a domain of knowledge or discourse. The representational primitives are typically classes (or sets), attributes (or properties), and relationships (or relations among class members). The definitions of the representational primitives include information about their meaning and constraints on their logically consistent application [9]. Several criteria are used for the validation of ontologies, some of them address the formal correctness of the ontology such as [10]: – Duplication errors: some elements of the ontology can be deduced from the others. – Disjunction errors: defining a class as a conjunction of distinct classes. – Consistency and coherence: check if the current definitions lead to contradictory conclusions. Another kind of criteria tackles the closeness of the ontology to the modelled domain such as completeness. Gómez-Pérez [10] notes that it is impossible to prove the completeness of an ontology, however, it remains possible to prove the incompleteness of an element of the ontology. In this context, another important issue is the domain-level correctness of the ontology in the context of automatic ontology construction. Automated ontology construction processes are more and more required due to the huge amount of knowledge that must be modelled in some domains. For instance, in the medical field, knowledge doubles every 5 years [11] or even every 2 years [12]. In the scope of this paper we are interested in validating medical ontologies (including schema knowledge and individuals description) without relying on the method used to build them. More precisely, we focus on the domain-level correctness and target existing ontologies that have no formal errors. In this context we try to answer 4 main questions: – Which elements need to be validated? – How to order/rank the elements to be validated? Which validations are independent? Which ones are dependent from each other? – How to validate these elements? – How to make the necessary updates after each validation step?

4


Several criteria can be used to evaluate the quality of the domain conceptualization [13, 14]. We focus particularly on five criteria: – The scope (or fit) of the vocabulary [13]. – The well-ness (fit) of the taxonomy (i.e. the generalization or is-a hierarchy) [13]. – The adequacy of the non-taxonomic relations (i.e. the fit of the semantic relations) [13]. – Coherence and Extensibility [14]: the ontology should be coherent in order to perform inferences that are correct w.r.t. the available definitions. It must be constructed in a manner that any concept addition cannot affect its consistency. – Minimal Ontological commitment [14]: the ontology should have the minimum hypotheses about the real world and should not contain additional knowledge about the domain that it models. 3.2

Proposed Approach

We propose a two-fold approach to validate medical ontologies (cf. figure 1). The first step consists in generating automatically a list of natural language questions from the ontology to be validated (cf. section 4). These questions are submitted to domain experts who provide an agreement decision (Yes/No) and a textual feedback. The next step consists on interpreting expert’s feedback to validate or modify the ontology (cf. section 5). The novelty of our approach is that manual interventions will be made only by HPs who will lead the ontology validation process. ICT experts will be required only when the error cannot be solved automatically. This will increase the quality of exchanges between actors and reduce errors and time consumption.

Fig. 1. Proposed Approach for Medical Ontology Validation

The proposed approach can be used to (i) validate ontologies constructed automatically from medical texts (e.g. clinical guidelines) and also (ii) to re-validate ontologies (constructed manually or automatically), since medical knowledge evolves quickly over time.


4

5

Question Generation from Ontologies

The aim of this step is to build relevant natural language questions from formalized knowledge in order to validate the maximum number of assertions with the minimum number of questions. Since we will use existing tools to verify the correctness of the formalism, we focus our research on validating the following types of ontology statements: – – – – – –

A rdfs:subClassOf B (class A is a subclass of B) P rdfs:subPropertyOf Q (property P is a sub-property of Q) P rdfs:domain D (D is the domain class for property P) P rdfs:range R (R is the range class for property P) I rdf:type A (I is an individual of class A) I P J (the property P links the individuals I and J)

The proposed approach uses manually constructed patterns for each kind of ontology element as described in the following section. 4.1

Pattern-Based Method for Question Generation

We start from the hypothesis that all the elements of a medical ontology must be validated. This involves validating concepts (e.g. Substance), relations between concepts (e.g. administrated for), concept instances (e.g. activated charcoal is an instance of Manufactured Material), relations between concept instances (e.g. chest X-ray can be ordered for Chronic cough) or between concept instances and literals (e.g. “give oral activated charcoal 50g” indicates the dose of the substance to be administrated “50g”). These ontology elements provide the main keywords of the question patterns through the labels of concepts, relations and instances. Several points should be taken into account such as: – How to build relevant natural language questions from ontology elements? – How to take into account the complexity of answering medical questions which can be related to the question type1 ? To answer these questions, we constructed manually question patterns associated to each type of ontological element. A question pattern consists in a regular textual expression with the appropriate “gaps” [15]. For instance, the pattern “Is DOSE of DRUG well suited for PATIENTS having DIS ?” is a textual patterns with 4 gaps: DOSE, DRUG, PATIENTS and DIS. This question pattern aims to validate a drug dose administrated to a patient having a particular disease. Table 1 presents examples of boolean-question patterns. 1

For example, as the same symptoms can refer to different medical problems w.r.t the patient age, country, etc., questions about diagnostics are expected to be more difficult than questions about complementary medical exams (e.g. Ultrasonography, Radiology).

6

Asma Ben Abacha, Marcos Da Silveira and Cédric Pruski Question pattern Example of instance Does a(n) CLASS have a(n) PROP- Does a Symptom have a Measurement ERTY? Method? Does a Treatment have an Administration Route? Is SUB-CLASS a type of CLASS? Is Statistical Evidence a type of Evidence? Is SUB-PROP a type of PROP? Is Primary Treatment a type of Treatment? Does a(n) CLASS1 PROPERTY Does a Medical Exam diagnose a Disease? a(n) CLASS2? Does INSTANCE1 PROPERTY Does Prozac treat Schizophrenia? INSTANCE2? Table 1. Examples of boolean-question patterns

4.2

Question Optimization Strategy

At this level, our main objective is how to build relevant questions from formalized knowledge in order to validate the maximum number of assertions with the minimum number of questions. We propose an optimization strategy relying on the RDFS logical rules in order to rank the questions according to the elements that imply the more changes in the ontology. For instance, if we have the following data: – hasSuitedAntiobioticsType rdf:subPropertyOf hasTreatment – Antibiotics rdfs:subClassOf Treatment – hasSuitedAntiobioticsType rdfs:range Antibiotics and the expert invalidates “Antibiotics rdfs:subClassOf Treatment”, than the property hasSuitedAntiobioticsType cannot be declared as a sub-property of hasTreatment because the hasSuitedAntibioticType relation has not a common range with the property hasTreatment which leads to a formal error w.r.t to RDFS entailment rules. We consider all RDFS entailment rules2 . Table 2 presents some inversed forms of these rules in order to show the impact of invalidating each one of the target ontology statements.

NOT A rdfs:subClassOf B ⇒ NOT A rdfs:subClassOf C s.t. C rdfs:subClassOf B NOT P rdfs:domain A ⇒ NOT P rdf:subPropertyOf Q s.t. Q rdfs:domain A NOT I rdf:type A ⇒ NOT s.t. P rdfs:domain A NOT s.t. P rdfs:range A Table 2. Examples of ontology update rules w.r.t. invalidated elements 2

http://www.w3.org/TR/rdf-mt/#RDFSRules


7

Thus, questions could be ranked in a manner that allows to delete some of the remaining questions if one of the RDFS entailment rules apply. This leads to the following validation order: 1. 2. 3. 4. 5.

5

A rdfs:subClassOf B P rdfs:domain D and P rdfs:range R P rdfs:subPropertyOf Q I rdf:type A IPJ

Feedback Interpretation and Ontology Validation

The second step of our approach is the exploitation of experts’ feedback to validate or modify the target ontology. The ontology to be validated may contain concepts, individuals and relations defined between concepts or individuals. Feedback consists in two main parts: an assertion on the correctness of the target knowledge and a free textual explanation if provided. In the scope of this paper we take into account ontologies that are formally-valid (with no inconsistencies) and focus on the validation of domain conceptualization. In this context, “Yes” answers will have no impact on the ontology. The ontology will be modified on the “No” answers provided by the domain experts. Invalidating an ontology element will have different impacts according to the element type as we have discussed in section 4.2. We use the same RDFS entailment rules to update the ontology. The ontology item invalidated by the expert and the inferred invalidations are deleted from the ontology as well as the questions that were associated to them.

6

Experiments and Discussion

We tested our approach on three different medical ontologies3 : – Caries Ontology (CO)4 – Disease-Treatment Ontology (DTO)5 – Mental Diseases Ontology (MDO)6 In a first step, we studied the number of generated questions according to the number of ontology elements to be validated. Table 3 presents the number of questions w.r.t. the number of classes, properties and instances of each ontology (DTO, MDO and CO) without question optimization. 3

4 5 6

In this paper, we work on medical ontologies in English but our approach can be applied to other languages. CO was developed manually by an expert in the dentistry at CRP Henri Tudor We constructed an OWL translation of the ontology proposed by Khoo et al. [16] http://mental-functioning-ontology.googlecode.com/svnhistory/r19/trunk/ontology/MD.owl

8


Ontology DTO MDO CO

Class Number 49 149 26

Property Number 148 76 266

Instance Number 0 18 13

Total Number of OE 197 243 305

Question Number 165 243 290

Table 3. The number of Ontology Elements (OE) and the number of generated questions for different medical ontologies without optimization

The number of generated questions depends on the ontology size and shows the importance of implementing adequate questions ranking and optimization strategies, in particular for large ontologies. In our experiments, our optimization method works better in case of ontologies with many instances. For the CO ontology, this strategy helps minimizing the number of submitted questions from 290 to 283 questions with only 4 NO answers. For the MDO ontology, our method allows asking 239 questions instead of 243 with only 2 NO answers. In case of ontologies with more NO answers (i.e. more invalid elements), the number of deleted questions will increase. For the DTO ontology, there is no available instances and in the same time there was not “NO answers” given by the expert, so the initial number of questions was conserved. The ontologies used in these experiments were constructed manually and semi-automatically. More experiments should be conducted on automatically constructed ontologies in order to evaluate more accurately the benefits of question optimization. In the case of ontologies with few invalid elements (few NO answers), we are currently working on presentation-level optimizations to reduce the time needed to answer the questions by the experts. In particular, we study two main presentations: question factorization according to an ontology element (concept, relation or individual) and logical chaining (A hasRelation1With B, B hasRelation2With C, etc.). These representations can help medical experts to reduce the time needed to understand and answer the questions. These experiments also showed the need to add other specific types of questions and answers in order to acquire missing information and to enrich the ontology when necessary. For instance, an answer to a question can be YES for one group of patient (e.g. Infant) and NO for another group or under a specific condition (e.g. co-morbidity). Our validation approach can also lead to the isolation of a concept or of a branch of the ontology. We are working on improving our system by adding for the expert the possibility to precise a contextual element or condition that clarifies ambiguous situations. We also work on integrating factual questions in our system in order to add missing information to the ontology. Figure 2 presents our method to exploit factual questions in order to enrich the ontology. Future work will also include the development and the evaluation of our approach when considering more complex OWL semantics (instead of only RDFS).


9

Fig. 2. Improving Question Generation

7

Conclusion

With the rise of automatic ontology construction methods, the problematic of ontology validation interests more and more research efforts, but mostly at a formal level. In this paper, we tackled the problem of ontology validation from a conceptual and a semantic point of view. We proposed a different approach based on question answering to ease the communication between the domain experts and the ontology validation system. Natural language questions are generated from the medical ontology to be validated and presented to domain experts. The acquired answers are then used to update/correct the ontology. The main contribution of this work is the combination natural language processing techniques and logical reasoning in order to support the ontology validation task. This combination provides the means for a non ICT expert (i.e. someone that do not have a deep knowledge in the ontology representation formalism) to validate the elements of the ontology. We continue to work on the optimization of the number of questions and some preliminary outcomes are presented in this paper. We plan to improve our Question Generation module by adding more optimization strategies and question types. Another enhancement could be to model specific domain knowledge as logical (domain-level) rules in order to decrease the number of ontology elements that need to be validated. Future work will also include the development of relevant heuristics for the selection of a small set of relevant questions from big ontologies.

References 1. Liu, K., Hogan, W.R., Crowley, R.S.: Natural language processing methods and systems for biomedical ontology learning. Journal of biomedical informatics 44(1) (February 2011) 163–179 2. Navigli, R., Velardi, P.: From glossaries to ontologies: Extracting semantic structure from textual definitions. In: Proceedings of the 2008 conference on Ontology

10

3.

4.

5.

6.

7.

8.

9. 10. 11. 12. 13.

14. 15.

16.

Asma Ben Abacha, Marcos Da Silveira and Cédric Pruski Learning and Population: Bridging the Gap between Text and Knowledge, Amsterdam, The Netherlands, The Netherlands, IOS Press (2008) 71–87 Ruiz-Mart´ınez, J.M., Valencia-Garc´ıa, R., Fern´ andez-Breis, J.T., S´ anchez, F.G., Mart´ınez-Béjar, R.: Ontology learning from biomedical natural language documents using umls. Expert Systems with Applications 38(10) (2011) 12365–12378 vor der Bruck, T., Stenzhorn, H.: Logical Ontology Validation Using an Automatic Theorem Prover. In: Proceedings of the 2010 conference on ECAI 2010: 19th European Conference on Artificial Intelligence, IOS Press (2010) 491–496 Poveda-villal´ on, M., Su´ arez-figueroa, M.C., G´ omez-pérez, A.: Validating Ontologies with OOPS ! In: Knowledge Engineering and Knowledge Management, EKAW 2012, Springer-Verlag (2012) 267–281 Pohl, M., Wiltner, S., Rind, A., Aigner, W., Miksch, S., Turic, T., Drexler, F.: Patient development at a glance: An evaluation of a medical data visualization. In: INTERACT (4). (2011) 292–299 Pammer, V.: Automatic Support for Ontology Evaluation Review of Entailed Statements and Assertional Effects for OWL Ontologies. PhD thesis, Graz University of Technology (March 2010) Gangemi, A., Catenacci, C., Ciaramita, M., Lehmann, J.: Modelling Ontology Evaluation and Validation. In: Proceedings of the 3rd European conference on The Semantic Web: research and applications, Budva, Montenegro, Springer-Verlag (2006) 140–154 Gruber, T.: Ontology. Encyclopedia of Database Systems (2008) G´ omez-Pérez, A.: Ontology evaluation. In: Handbook on Ontologies. (2004) 251– 274 Engelbrecht, R.: Expert systems for medicine—functions and developments. Zentralbl Gynakol 119(9) (1997) 428–434 Hotvedt, M.O.: Continuing medical education: actually learning rather than simply listening. JAMA 275(21) (1996) 1637–1638 Porzel, R., Malaka, R.: A Task-based Approach for Ontology Evaluation. In: Procceding of ECAI2004 - Workshop Ontology Learning and Population, Valencia, Spain (August 2004) Gruber, T.R.: A translation approach to portable ontology specifications. Knowledge Acquisition 5(2) (June 1993) 199–220 Hearst, M.: Automatic acquisition of hyponyms from large text corpora. In: Proceedings of the 14th International Conference on Computational Linguistics (COLING-1992). (1992) 539–545 Khoo, C.S., Na, J.C., Wang, V.W., Chan, S.: Developing an ontology for encoding disease treatment information in medical abstracts. DESIDOC Journal of Library & Information Technology 31(2) (2011)