Extracting Semantic Relationships between Terms: Supervised vs. Unsupervised Methods Emmanuel Morin IRIN 2, rue de la Housini`ere - BP 92208 44322 Nantes Cedex 3 FRANCE
[email protected]
Michal Finkelstein-Landau Math & Computer Science Department Bar Ilan University Ramat Gan 52900 ISRAEL
[email protected]
May 13, 1999
1 Introduction As the amount of electronic documents (corpora, dictionaries, newspapers, newswires, etc.) becomes more and more important and diversified, there is a need to extract information automatically from these texts. In order to extract terms and relations between terms, two methods can be used. The first method is the unsupervised approach, which requires a term extraction module and few predefined types, especially term types, in order to find relationships between terms and to assign appropriate types to the relationships. Works on automatic term recognition usually involve predefinition of a set of term patterns, extraction procedure and a scoring mechanism to filter out non-relevant candidates. Smadja (1993) describes a set of techniques based on statistical methods for retrieving collocations from large text collections. Daille (1996) presents a combination of linguistic filters and statistical methods to extract two-word terms. This work implements finite automata for each term pattern, then various statistical scores for ranking the extracted terms are compared. Unsupervised identification of term relationships is a more complicated task, reported in works from various fields including Computational Linguistics and Knowledge Discovery in Texts. A keyword-based model for text mining is described in Feldman and Dagan (1995). The work suggests to use a wide range of KDD (Knowledge Discovery in Databases) operations on collections of textual documents, including association discovery among keywords within the documents. Cooper and Byrd (1997) reports the TALENT extraction tools, designed to extract and organize lexical networks from named and unnamed relations in the text. Named relations are not necessarily primed with specific relation names that it is looking for, but discovered by exploiting text patterns in which such relationships are typically expressed. The second method is the supervised relation classification system, which requires predefinition of lexicosyntactic patterns as well as manual traverses on outputs of terminologists, in order to find pairs that belong to the predefined relations. Hearst (1992, 1998) reports a method using lexico-syntactic patterns to extract lexical relations between words from unrestricted text. For example, the pattern NP, especially fNP,g forjandg NP (where NP is a noun phrase), and the sentence: (...) most European countries, especially France, England and Spain extract three lexical relations: (1) HYPONYM(France, European country), (2) HYPONYM(England, European country), and (3) HYPONYM(Spain, European country). These relations can then be included in a hierarchical thesaurus. Here, only a single instance of a lexico-syntactic pattern needs to be encountered to extract the corresponding conceptual relation. Other supervised systems use partial syntactic structures by The experiments presented in this paper were performed on a subset of the [REUTERS] corpus, a 0.9-million word English corpus including 5770 news stories.
1
Bootstrap: initial pairs of terms
Corpus
Lexical
preprocessor
Shallow parser + classifier
Lexico-syntactic patterns
Lemmatized and tagged corpus Information extractor
Database of lexico-syntactic patterns
Partial hierarchies of single-word terms
Figure 1: The information extraction system P ROM E´ TH E´ E
using local information for extracting specific relations. L IEP (Huffman, 1995) learns information extraction patterns from example texts containing events. A user can choose which combinations of entities signify events to be extracted. These positive examples are used by L IEP to build a set of extraction patterns. These supervised systems have good performance for information extraction tasks in limited domain. But, the cost of adapting an information extraction system to a new domain can be prohibitive. In order to evaluate the complementarity of these methods, we compared a supervised method with an unsupervised method for the extraction of semantic relationships between terms. The paper is the result of this study. The remainder of this paper is organized as follows. Section 2 presents the supervised system P ROM E´ TH E´ E, and describes the methodology for acquisition of lexico-syntactic patterns. Section 3 presents an unsupervised method that combines ideas of term identification and term relationship extraction for term-level text mining. Section 4 presents the integrated system and experimentation. Finally, section 5 concludes this study.
2 Iterative Acquisition of Lexico-syntactic Patterns We first present the supervised system P ROM E´ TH E´ E for corpus-based information extraction that extracts semantic relations between terms.1 This system is built on previous work on automatic extraction of hypernym links through shallow parsing (Hearst, 1992, 1998). In addition to this previous study, the system incorporates a technique for the automatic generalization of lexico-syntactic patterns that relies on a syntactically-motivated distance between patterns. As illustrated in Figure 1, the P ROM E´ TH E´ E system has two functionalities: 1. The corpus-based acquisition of lexico-syntactic patterns with respect to a specific conceptual relation. 2. The extraction of pairs of conceptual related terms through a database of lexico-syntactic patterns.
Shallow Parser and Classifier A shallow parser is complemented with a classifier for the purpose of discovering new patterns through corpus exploration. This procedure, inspired by Hearst (1992, 1998), is composed of 7 steps: 1. Select manually a representative conceptual relation, for instance the hypernym relation. 2. Collect a list of pairs of terms linked by the selected relation. The list of pairs of terms can be extracted from a thesaurus, a knowledge base or can be manually specified. For instance, the hypernym relation neocortex IS-A vulnerable area is used. 1
For expository purposes of this section, some examples are taken from [MEDIC], a 1.56-million word English corpus of scientific abstracts in the medical domain.
2
3. Find sentences in which conceptually related terms occur. These sentences are lemmatized, and noun phrases are identified. Therefore, sentences are represented as lexico-syntactic expressions2 . Through this simplification process, we have a more generic representation of relevant sentences, and the comparison of these sentences is easier. For instance, the previous relation HYPERNYM(vulnerable area, neocortex) is used to extract from the corpus [MEDIC] the sentence: Neuronal damage were found in the selectively vulnerable areas such as neocortex, striatum, hippocampus and thalamus. The sentence is then transformed into the following lexico-syntactic expression: NP find in NP such as LIST
(1)
4. Find a common environment that generalizes the lexico-syntactic expressions extracted at the third step. This environment is calculated with the help of a measure of similarity and a procedure of generalization that produce candidate lexico-syntactic pattern. For instance, from the previous expression, and at least another similar one, the following candidate lexico-syntactic pattern is deduced: NP such as LIST
(2)
5. Validate candidate lexico-syntactic patterns by an expert. 6. Use new patterns to extract more pairs of candidate terms. 7. Validate candidate pairs of terms by an expert, and go to step 3. Through this technique, lexico-syntactic patterns are extracted from a technical corpus. These patterns are then exploited by the information extractor that produces pairs of conceptual related terms.
Automatic Classification of Lexico-syntactic Patterns Let us illustrate with more details the fourth step of the described algorithm that automatically acquires lexicosyntactic patterns by clustering similar patterns. As indicated in item 3. above, the relation HYPERNYM(vulnerable area,neocortex) instantiate the pattern: NP find in NP such as LIST
(3)
Similarly, from the relation HYPERNYM(complication, infection), the sentence: Therapeutic complications such as infection, recurrence, and loss of support of the articular surface have continued to plague the treatment of giant cell tumor is extracted through corpus exploration. A second lexico-syntactic expression is produced: NP such as LIST continue to plague NP
(4)
Lexico-syntactic expressions (3) and (4) can be abstracted as:3 .
A = A1 A2
A
j
A
k
A
n
with
RELATION (A ; A k >j +1 j
k)
and
B = B1 B2
B0 j
B0 k
RELATION (B 0 ; B 0 ) B 0 with k0 > j 0 + 1 j
k
n
Let Sim(A; B ) be a function measuring the similarity of lexico-syntactic expressions on the following hypothesis:
A and B that relies
2
A lexico-syntactic expression is composed of a set of items, which can be either lemmas, punctuation marks, numbers, symbols (e.g. x, j always requires at least one item between Aj and Ak .
+1
3
win1(A)
win2(A) win3(A)
A = A1 A2 ...... Aj ... Ak ......... An
B = B1 B2 ... Bj’ ........ Bk’... Bn’ win1(B)
win2(B)
win3(B)
Figure 2: Comparison of two lexico-syntactic expressions
Hypothesis 2.1 (Syntactic isomorphy) If two lexico-syntactic expressions A and B indicate the same pattern then, the items Aj and Bj 0 , and the items Ak and Bk0 have the same syntactic function. Let Win1 (A) be the window built from the first through j-1 words, Win2 (A) be the window built from words ranking from j+1th through k-1th words, and Win3 (A) be the window built from k+1th through nth words (see Figure 2). The similarity function is defined as follows:
3 X
Sim(A; B ) =
=1
Sim(Win (A); Win (B )) i
i
i
8 < Win1(A) = A1A2 A ?1 2(A) = A +1 A ?1 : Win Win3(A) = A +1 A
with
(5)
8 Win (B ) = B B B 0 1 1 2 < ?1 0 and Win (B ) = B +1 B 2 ?1 : Win (B ) = B 3 +1 B 0 The function of similarity between lexico-syntactic patterns Sim(Win (A); Win (B )) is defined experi
j
j
k
j
k
n
k
i
j
k
n
i
mentally as function of the longest common string. All lexico-syntactic expressions are compared two by two by previous similarity measure, and similar lexico-syntactic expressions are clustered. Each cluster is associated with a candidate lexico-syntactic pattern. For instance, the sentences introduced earlier generate the unique candidate lexico-syntactic pattern: NP such as LIST
(6)
We now turn to the description of the unsupervised system for extracting relationship between terms. 4
3 Term Level Text Mining The unsupervised system combines ideas of term identification and term relationship extraction for term-level text mining. The overall purpose of the system is to find interesting relationships between terms and to label these relationships. The system uses NLP (Natural Language Processing) techniques including lemmatizing and shallow parsing in order to increase confidence in both extracting terms and identifying relationships. Multi-word term recognition methods are used for finding relationships between terms. In particular, similar association measures that in previous literature (Daille, 1996; Smadja, 1993) were used for term and collocation extraction are implemented in this work for extracting relations between terms. The system requires only few manual definitions and avoids the need to know the relevant lexico-syntactic pattern in advance. 4
For more information on the P ROM E´ TH E´ E system, in particular a complete description of the generalization patterns process, see the following related publication: Morin (1999).
4
Term Extraction As a first stage, the system extracts various term patterns from the corpus, which can be divided to the following families: 1. Simple term patterns: Adjective-Noun (ADJ-N), Noun-Sequence (NSEQ), Noun-Preposition-Noun (NPREP-N) and Proper Name (PN). 2. Syntactic relations: Verb-Object (VB-OBJ) and Subject-Verb (SUBJ-VB). 3. Semantic relations: IsA and HasA. The extraction process is preceded by a module for tagging, lemmatizing and shallow parsing the documents. We used an academic version of Noun- Phrase and Name Parser, which is a redevelopment of Voutilainen (1993).
Term Typing and Filtering This stage is intended to determine which terms become in focus, since the extraction process yields enormous number of term candidates. Using a predefined list of term types, some terms are typed and become in focus regardless of their distributional properties. Others are scored according to classical scoring criteria in order to filter out non-relevant combinations (Daille, 1996; Smadja, 1993). Among the term types defined for this work are: 1. Merger terms: terms containing the substring “merge”, which refer to merger events. For example: merger agreement, merger of airline, and announce merger. 2. Product terms: terms that form a product. For example, the object in a VB-OBJ term where VB = “produce” (oil in produce oil) and the last noun in a N-PREP-N term where the first noun is “production” and PREP = “of” (camera in production of camera). 3. Company-Name terms: proper names containing substrings that tend to appear within company names like “Ltd.”, “Corp”, “Co”, and “Inc”. For example: Lloyds Bank NZA Ltd., and Utah International Inc. The assumption is that finding term types is not difficult using local cues and a predefined list of types. We leave unsupervised identification of new types for future research.
Term Associations and Labeling Associations In this work, relationships between terms are identified according to cooccurrence association calculation. The relationships differ by two factors: 1. Types of cooccurrences: some relations are better identified using term cooccurrences in joint sentences, while for others cooccurrences in joint documents give better results. 2. Types of scores: Mutual Information for example, discriminates in favor of rare events while Log Likelihood behaves in an opposite way, thus different association measures can identify different conceptual relations. Labeling relationships in an unsupervised manner can be done by using frequent “relation words” like verbs and adjectives that are involved in the relationship. In current work, a default label is used to label relations, which is the type of the main term involved in the relationship. For example, associations between CompanyName terms and Product terms are automatically labeled with “Produce”, which is the type of the main terms in those relationships. Examples of relationships between Company-Name terms and Product terms (with Log Likelihood scores) that were found in the corpus are shown in Table 1. 5
Company-Name U.S. Group CPC International Lex Vehicle Lease Ltd. McDonnell Douglas Corp Heinz-UFE Ltd. Alliant Computer System Corp Honda Motor Corp McDonnell Douglas Corp
Product corn car aircraft cereal software car jet
Score 47.07 18.59 15.86 13.68 10.32 9.28 8.50
Table 1: Relationships between Company-Name terms and Product terms Some relations are composed by three items, therefore an incremental association calculation is implemented as well. The Merge relation is an example of a ternary relation in our implementation, since it involves a main Merger term and two Company-Name terms that participates the merger. Subtypes, frequent words that are part of the relationships, can be given in order to refine and specify the global semantic label. For example, subtypes for the Merge relation are: propose, plan, reject, etc., which are verbs that describe the merger (whether it is proposed, planned or even rejected). Examples of Merger relationships between terms (with their Log Likelihood scores) that were found in the corpus are shown in Table 2. Company-Name Alpha Health Systems Corp MTS Acquisition Corp Bristol-Myers Co Comdata Network Inc Electrospace Systems Inc Industries Inc Trust Co
Company-Name REPH Acquisition Co Seton Co SciMed Life Systems Inc Lambert Inc Chrysler Corp Hayes-Albion Corp Independence Ban Corp
Merger merger proposal plan of merger agreement of merger finance merger merger agreement vote on merger complete merger
Score 30.88 30.88 16.43 16.43 16.43 16.43 16.43
Table 2: Relationships between Merger terms and Company-Name terms
4 The Integrated System and Experiments There is a clear distinction between the two systems. The supervised approach extracts explicit relations - the examined relations are known in advance. The unsupervised approach, on the other hand, mostly identifies associations between terms but relation labels are implicit - they may remain unknown. Rather than the two methods are contrasted they are complementary. The supervised method can find only small portion of related terms due to the variety of sentence styles and the inability to find a common environment to all those sentences. Either knowledge base should be consulted or much human effort should be invested in order to bootstrap the system by supplying initial pairs of terms or lexico-syntactic patterns that represent the examined relations. The system is also unable to measure the strength of relationship between terms, except for the frequency of each pair within different patterns. The unsupervised method on the other hand, achieves a much higher coverage in identifying strong relationships between terms but noisy pairs decrease performance. In some cases a relationship can be labeled and be given a subtype, but in many other cases relation labels remain unknown. The integrated system utilizes the advantages of each approach and improves performance with respect to the above mentioned drawbacks. One aspect of the integrated system exploits the high coverage of the unsupervised system for supplying an initial training input to the supervised system. This is an automatic process that reduces human involvement in the supervised system. A second aspect is the mutual feedback of the systems. The unsupervised system can filter out noisy pairs 6
of terms after consulting the output of the supervised system, while the supervised system gets quantitative evaluations of the strength of relations between terms and in some cases also subtypes that refines the global semantic relations it examines.
The Merge Relation A pair of terms belonging to the Merge relation is of the form Merge(CN1 , CN2 ), where CN1 and CN2 are both Company-Name terms that participates some merger event (merger in progress, actual, etc.). The first experiment evaluated the performance of P ROM E´ TH E´ E system as a stand-alone system. In order to bootstrap P ROM E´ TH E´ E, we have manually defined two lexico-syntactic patterns: 1 merger of CN1 with CN2 Dixons Group Plc said shareholders at a special meeting of Cyclops Corp approve the previously announced merger of Cyclops with Dixons 2 merger of CN1 and CN2 Hoechst Celanese was formed Feb 27 by the merger of Celanese Corp and American Hoechst Corp Then, all instances of those patterns were extracted from the corpus, and P ROM E´ TH E´ E incrementally learned more patterns for the Merge relation. The new patterns learned were: 3 CN1 said it complete * acquisition of CN2 Chubb Corp said it completed the previously announced acquisition of Sovereign Corp 4 CN1 said it shareholder * CN2 approve * merger of the two company INTERCO Inc said its shareholders and shareholders of the Lane Co approved the merger of the two companies 5 CN1 said it shareholder approve * merger with CN2 Fair Lanes Inc said its shareholders approved the previously announced merger with Maricorp Inc a unit of Northern Pacific Corp 6 CN1 said it agree * to (acquirejbuyjmerge with) CN2 Datron Corp said it agreed to merge with GGFH Inc a Florida-based company formed by the four top officers of the company 7 CN1 ’s (proposed)? acquisition of CN2 Fujitsu’s acquisition of Fairchild would have given the Japanese computer maker control of a comprehensive North American sales and distribution system and access to microprocessor technology an area where Fujitsu is weak analysts said Using those patterns, 101 pairs of terms (class A) conceptually related have been extracted from the corpus. The second experiment was performed on the integrated system. At first, Merger terms and CompanyName terms (described in section 3) were extracted from the corpus. For the 350 Merger terms (e.g. merger talk, approve merger, merger transaction) and 4500 Company-Name terms (e.g. Texas Bancshare Inc, Bank of England) that were found, a ranked list of 263 conceptually related triples within the Merge relation was generated using an automatic relationship identification module. Each triple included the merger description and two companies. The triples became pairs by leaving only the two related company names to be given as initial training input to the learning system (class C). The P ROM E´ TH E´ E system discovered again patterns 1, 3 ,4, 5, 6, 7, and a new pattern: 8 CN1 said it sign * to (acquirejbuyjmerge with) CN2 Dauphin Deposit Corp said it signed a definitive agreement to acquire Colonial Bancorp Inc
7
C
supervised system (A and B)
B A
11 00 00 11 00 11 00 11 00 11 00 11
unsupervised system (C)
C =28 C =121
C =263
C =23 C =101
Figure 3: Overlap of the pairs of terms for the Merge relation
Using those patterns, 121 pairs of terms (class B) conceptually related have been extracted from the corpus. From the manual definition of two patterns and from the output of the unsupervised system, P ROM E´ TH E´ extracts practically the same lexico-syntactic patterns. Therefore, the output of the unsupervised system is a good alternative to manual definition, enabling to reduce human effort. As illustrated in Figure 3, the systems have only few pairs of terms in common. The low overlap can be explained by:
P ROM E´ TH E´ E system extracts only pairs of terms relative to a sentence, whereas the unsupervised system may extract pairs of terms according to cooccurrences in joint documents (and not only sentences).
Several pairs in classes A and B express the Acquisition relation (which is close to the Merge relation), while pairs in class C refer only to the Merge relation, since the main items in the relations extracted by the unsupervised system are merger terms.
The relatively high precision for class C (72%) is still lower in comparison to classes A and B (92% and 93% respectively). In some cases, since cooccurrence statistics ignore sentence structure, the unsupervised system includes company names that appear in the vicinity of two merged companies by mistake, like Merge(Maricorp Inc, Northern Pacific Corp) in the sentence Fair Lanes Inc said its shareholders approved the previously announced merger with Maricorp Inc a unit of Northern Pacific Corp.
To conclude, this experiment confirms the complementarity of the two approaches to cover the Merge relation.
The Produce Relation A pair of terms belonging to the Produce relation is of the form Produce(CN1 ,NP2 ), where CN1 is a CompanyName term and NP2 is a noun phrase describing a product. The semantic interpretation is that CN1 produces NP2 , but it can also mean that CN1 (distributesjsellsjprovidesjsupplies) NP2 . In contrast to the Merge relation, patterns for the Produce relation can not be easily defined. In general, this relation is usually implicit i.e. it is not clearly represented by lexico-syntactic patterns, except for the trivial pattern: CN1 produces NP2 like in the sentence: Vismara primarily produces a variety of pork products. Therefore, the experiment has been performed on the integrated system, designated to learn lexico-syntactic patterns for this relation. Among the phrases that were used to select Product terms are: 1. “production of NP”: part of a N-PREP-N term, e.g. production of fibre. 2. “producer of NP”: part of a N-PREP-N term, e.g. producer of chocolate. 3. “produce NP”: part of a VB-OBJ term, e.g. produce gas. 8
Company-Name terms were extracted as in the Merge experiment. Relationships between 130 product candidates and 4500 producers were calculated automatically, and finally, the ranked list of 873 pairs of terms was used as initial training input to the P ROM E´ TH E´ system in order to learn lexico-syntactic patterns for the Produce relation. The patterns found at the end of this process were: 1. CN1 is? a NP2 company Fujitsu Ltd is a computer and telecommunications company based in Japan 2. CN1 (isjis aja)? producer of NP2 It owns Code-A-Phone Corp a producer of telephone answering machines 3. CN1 (isjis aja)? maker of NP2 Phelps Dodge which ranks as the largest copper producer in the U.S. last year paid 240 mln dlrs for Columbian Chemicals Co a maker of carbon black which is used in rubber and tires among other products 4. CN1 (isjis aja)? manufacturer of NP2 Tokheim Corp manufacturer of electronic petroleum marketing systems said it expects shipments of Tokheim Convenience Systems its new family of dispensers to improve its sales trend throughout 1987 Unlike the previous experiment, the P ROM E´ TH E´ system did not extract many interesting lexico-syntactic patterns for the Produce relation. Among the 873 pairs of terms identified by the unsupervised system, 176 do not appear within a sentence. This observation shows that expressing relations varies according to the conceptual relation and the corpus they appear in. The most important facts about company events are usually reported in the first description of the document, and detailed facts are described after, not always with reference to the name of the company (but using anaphora). The relation between a “producer” and a “product” is usually implicit, and rarely appears within a sentence but within a document. Among the lexico-syntactic patterns found by P ROM E´ TH E´ , only 72 pairs of terms are extracted with a good precision (79%), apparently since the precision of the bootstrap pairs is rather low (64%), resulted by noisy product candidates, i.e. terms that were typed as products by mistake. An additional difficulty is that in some cases the name of the company includes the name of its product, like in National Computer System Inc expects fiscal year earnings to improve (...). These relations can neither be extracted by the P ROM E´ TH E´ E system nor by the unsupervised system. In this experimentation we have encountered interesting properties that can characterize semantic relations, and a hypothesis regarding the relation between different extraction methods and different semantic relations is suggested. This will be dealt in section 5.
5 Conclusions and Future Work The major conclusion from the joint experimentation is that the methods are complementary. An integrated system has been designed in order to overcome the disadvantages of each method give better results. The difference between the wide variety of lexico-syntactic patterns found within the Merge relation and the very few patterns found within the Produce relation is a result of the characteristics of the chosen corpus. Only important news about company events are typically reported in the beginning of a news story, while more detailed facts are described later, sometimes with no reference to the name of the company (see Riloff (1993) for the same observation). In [REUTERS] corpus, a story begins with important events (merger, acquisition, bankruptcy, etc.), and secondary information like products and people appears later, usually with company anaphora, therefore it is easier to find lexico-syntactic patterns for the explicit relations that appear in those stories. Nevertheless, the unsupervised system can add the missing information by finding associations between terms based on cooccurrences within documents, thus overcoming the anaphora problem. To conclude, the P ROM E´ TH E´ E system can extract more didactic, well- expressed relations and the unsupervised system can find also implicit relations (but labels are not always known). We suggest to implement the integrated system, which contributes in many directions: 9
Reducing manual effort in the supervised system: since the unsupervised system disregards the semantic structure of sentences, it can provide an initial training input that includes a wide range of pairs of terms within the examined relation, thus human effort in reduced.
Achieving better coverage of the examined relation: as the supervised system gets many pairs of terms as training input, it can learn various lexico-syntactic patterns and use them to extract more pairs of related terms.
Finding implicit relations and verifying associations between terms: the unsupervised system can identify pairs of terms that are not easily identified by the supervised system, and then consult the supervised system for filtering noisy pairs of terms.
For future work we would like to experiment in few directions:
As a result of the problems with the Reuters corpus, we would like to apply the integrated system also on a more didactic corpus (e.g. technical reports), in which sentences are longer and varied. The corpus should be large enough to ensure sufficient statistical information and should contain a wide variety of sentence structures and patterns within sentences.
We would like to define more semantic relations and learn their characteristics in order to choose the appropriate parameters and improve performance of the integrated system.
We would like to perform the same experiments reported in this paper using simple anaphora resolution methods in order to identify relations that are not explicitly expressed in the corpus.
References James W. Cooper and Roy J. Byrd. Lexical Navigation: Visually Prompted Query Expansion and Refinement. In Proceedings of the 2nd ACM international conference on Digital libraries, pages 237–246, Philadelphia,PA, July 1997. B´eatrice Daille. Study and implementation of combined techniques for automatic extraction of terminology. In P. Resnik and J. Klavans, editors, The Balancing Act: Combining Symbolic and Statistical Approaches to Language, pages 49–66. MIT Press, Cambridge, MA, 1996. R. Feldman and I. Dagan. Knowledge Discovery in Textual Databases (KDT). In Proceedings of the 1st International Conference on Knowledge Discovery (KDD-95), pages 112–117, August 1995. Marti A. Hearst. Automatic Acquisition of Hyponyms from Large Text Corpora. In Actes, 14th International Conference on Computational Linguistics (COLING’92), pages 539–545, Nantes, France, 1992. Marti A. Hearst. Automated discovery of wordnet relations. In Christiane Fellbaum, editor, WordNet: An Electronic Lexical Database, pages 131–151. MIT Press, Cambridge, MA, 1998. Scott B. Huffman. Learning information extraction patterns from examples. In Workshop New Approaches to Learning for Natural Language Processing at IJCAI’95, pages 127–133, Montreal, 1995. Emmanuel Morin. Using Lexico-syntactic Patterns to Extract Semantic Relations between Terms from Technical Corpus. In Proceedings, 5th International Congress on Terminology and Knowledge Engineering (TKE’99). (to appear), august 1999. Ellen Riloff. Automatically Constructing a Dictionary for Information Extraction Tasks. In Actes, 11th National Conference on Artificial Intelligence (AAAI’93), pages 811–816, Washington, DC, July 1993. Frank Smadja. Retrieving Collocations from Text: Xtract. Computational Linguistics, 19(1):143–177, 1993. A. Voutilainen. Nptool, a Detector of English Noun Phrases. In Proceedings of the Workshop of Very Large Corpora, Association for Computational Linguistics, pages 42–51, Ohio State University, 1993. 10
