Preliminary Steps towards a Knowledge Factory Process Vinay K. Chaudhri, Nikhil Dinesh, John Pacheco SRI International Menlo Park, CA 94025, USA {chaudhri, dinesh, pacheco}@ai.sri.com
Gary Ng
Peter E. Clark
Cerebra San Diego, CA
[email protected]
Vulcan Inc Seattle, WA, USA
[email protected]
Andrew Goldenkranz
Patrice Seyed
Naveen Sharma
Fremont Union High School District
[email protected]
University of Buffalo, Buffalo, NY
[email protected]
Evalueserve, India
[email protected]
Categories and Subject Descriptors I.2.1 [Artificial Intelligence]: Applications and Expert Systems
General Terms Measurement, Documentation, Performance, Design, Experimentation
Keywords Knowledge engineering, ontologies, deductive question answering, knowledge acquisition
1. INTRODUCTION In the fall 2010 issue of the AI Magazine, we reported the design, implementation and evaluation of a knowledge acquisition system called AURA. AURA enables domain experts in Physics, Chemistry and Biology to author their knowledge, and a different set of experts to pose questions against that knowledge. The evaluation results previously reported were from 50 pages each from science textbooks in Physics, Chemistry and Biology. The results were most promising for Biology. Based on those results we undertook a content building effort to capture knowledge from approximately 315 pages (or 20 chapters) of the same Biology textbook [2] and incorporated the resulting content in the electronic version of that book. In this demo/poster session, we will demonstrate the biology knowledge base (KB) created using AURA, the electronic textbook application Inquire, and discuss the knowledge engineering process we used to construct the KB.
2. KNOWLEDGE FACTORY PROCESS The AI magazine paper on AURA reported the question answering performance for KBs created by domain experts under different conditions. The question answering performance, however, leaves the following questions open. If a different set of novel questions is posed, can we expect to get the same performance? To get a truly robust measurement of novel question performance, one would need to pose a series of tests. Such series of tests has never been posed in Project Halo before. A related open question is how many tests with novel questions one needs to perform to ensure that the novel question performance stays approximately the same with each new comparable test? If a novel question set is very different in scope and complexity or poses questions that are outside the capabilities that were tested, the performance will obviously degrade. Copyright is held by the author/owner(s). K-CAP’11, June 26–29, 2011, Banff, Alberta, Canada. ACM 978-1-4503-0396-5/11/06.
To what extent has the textbook knowledge been captured? A problem with any evaluation that is based on question answering performance is that when a question fails, in many cases, the failure can be blamed on a knowledge gap. Assuming one can invest enough knowledge capture effort, can the knowledge gaps be avoided? Is there an inter-domain expert agreement during the knowledge encoding process? To expand the scope of the KB, we need multiple domain experts entering knowledge. This process can fail if there is a lack of consistent conventions for knowledge encoding and each encoder uses a different style. An ideal situation would be to have a process using which given a specific sentence, there is a well defined, and preferably a unique way, to formally represent it in the KB. While such a goal is unachievable in full generality, our hope is to have a closed community of curators who can develop a set of guidelines that can be followed by them to come up with consensus representations. We believe this to be a realizable goal because multiple textbook authors are able to converge on one way of presenting the material in the textbook even if they may disagree in some respects. We call a knowledge entry process a knowledge factory process if it solves the above three problems. In other words, a knowledge factory process is a knowledge entry process that can ensure that the results on the performance of novel questions is consistent across a series of questions that are similar in scope and difficulty, can systematically eliminate knowledge gaps, and leads to consensus representations by multiple domain experts. To the best of our knowledge, no such knowledge capture method exists that meets all three of these goals and our work makes modest steps in that direction. We next present key elements of the design of the knowledge factory process and then discuss our experience in implementing those elements.
2.1 Design of the Knowledge Factory Process Our solution for eliminating the knowledge gaps was to follow a sentence driven encoding strategy. Under this strategy, a domain expert analyzes each sentence for its relevance to question answering. For each relevant sentence, the domain expert represents it using the knowledge entry capabilities of AURA to the extent it can be represented. The goal of this strategy is to systematically work through each sentence so that there is a clear scope for the knowledge that needs to be encoded, and we avoid unintentional gap in the KB.
Our solution to address inter-domain expert agreement problem had two aspects. First aspect was to develop a knowledge engineering process manual that provides explicit knowledge entry guidelines. The manual includes several encoding situations and provides guidelines on what solutions to prefer. Second aspect was to design a collaborative process amongst multiple domain experts so that the representation of each sentence was reviewed by multiple experts to naturally evolve towards a consensus representation. Our collaborative process separates the encoding process into the roles such as planner, encoder, tester: a planner plans the knowledge to be entered, encoder performs the entry, and the tester tests the knowledge. During this factory like pipeline, the representation of each sentence gets reviewed by multiple people and any issues can get discussed and resolved. We designed the collaborative process structure because in a preliminary test it was clear that if the domain experts work in isolation using only the knowledge engineering process manual, they are unlikely to arrive at consensus representations. Our solution for ensuring that the performance of the system does not degrade with each new comparable novel question set was to simply perform a series of tests. It was unclear to us how many such tests will be needed or adequate, so we arbitrarily chose to do three tests. Each test was followed by a period to fix the knowledge gaps before the next test was performed.
2.2 Experience in Using the Knowledge Factory Process The sentence driven process was hugely popular in the project team: the domain experts had a clear plan, and the program management had a clear picture of how far we had progressed. The process also resulted in a set of sentences that could not be encoded either because of difficult ontological modeling problems or because AURA did not support the necessary knowledge acquisition interface. These difficult to encode sentences are the basis of a workshop on Deep Knowledge Representation (See https://sites.google.com/site/dkrckcap2011/). The sentence driven process, however, was not successful in fully eliminating knowledge gaps. In a test with novel questions, we still found knowledge gaps that could have been avoided during the encoding process. We are currently investigating why such gaps arose in spite of a systematic sentence based encoding scheme. The knowledge engineering process manual and the team structure for collaborative knowledge entry were partially effective in ensuring consensus representation. The success has been partial because even though the encoding team was able to reach consensus on the representations, the encoding did not always match the intuitions of a Biology teacher. This is a side effect of the fact that the encoding domain experts are not biology teachers and are located in a different location than biology teachers responsible for making use of the content in the electronic textbook (described in the next section). The collaborative team needs to be extended to add biology teacher expertise.
We performed three test and fix cycles on five chapters of the syllabus. Each test and fix cycle revealed gaps in the KB, and the question answering performance improved with each test. Since the performance did not level during the scope of the three tests, it was difficult to tell whether three tests were adequate. To ensure the sufficiency of these tests, one would need to perform a few more tests until the question answering performance levels off. Exploring that in more detail is a topic for future exploration. A factor that confounds these results is that some questions cannot be answered due to the sentences that cannot be encoded due to knowledge representation and acquisition difficulties. It has been difficult to assess if the results would be different if all the sentences could be encoded.
3. ELECTRONIC TEXTBOOK INQUIRE Our target application, motivating this work, is an “intelligent textbook”, i.e., an electronic version of the Campbell Biology textbook [2] that allows users to interactively browse and explore the material on-line, and ask questions and receive machineinferred answers from the KB, including questions that require inference. A prototype of this application, called Inquire, has already been constructed. There are three primary interfaces to the KB as a student is reading the book: (a) Clicking on a biology term in the book resulting in a page that summarizes information about that term (b) providing a list of suggested questions in response to highlighting (c) asking free-form questions expressed in controlled English. For the first of these, clicking on a biology term results in a concept summary page generated automatically from the knowledge formally represented in the KB. The concept summary pages can be extremely useful to a student in focusing on the essence of a concept, as a quick review guide, and an exam preparation aid. For the second of these interfaces, as a student highlights a section of the textbook, Inquire suggests some questions to the user about the highlighted text. The question suggestion is based on the knowledge that is formally represented in the KB. For example, in a paragraph describing Protein, the system may suggest the following questions: What is the shape of protein? How many polypeptides does Protein have? Users can enter free-form questions through a controlled English interface, allowing students to pose their own questions to the textbook. For example, a student may ask: ``What is the difference between a pentose and a hexose?”
ACKNOWLEDGMENT This work was supported by Vulcan Inc.
4. REFERENCES [1] Gunning D. et. al., Project Halo Update — Progress Toward Digital Aristotle, AI Magazine, Fall 2010, 33-58. [2] Jane B. Reece, Lisa A. Urry, Michael L. Cain, Steven A. Wasserman, Peter V. Minorsky, Robert B. Jackson. Campbell Biology, Pearson Publishing. 2010.
