Recognizing Confinement in Web Texts Megumi Ohki† Suguru Matsuyoshi† Junta Mizuno‡ Kentaro Inui‡ Nara Insutitute of Science and Technology†
Eric Nichols‡ Koji Murakami† ∗ Shouko Masuda† Yuji Matsumoto† Tohoku University‡
{megumi-o,shouko,matuyosi,matsu}@is.naist.jp {eric,junta-m,inui}@tohoku.ac.jp
[email protected]
Abstract In the Recognizing Textual Entailment (RTE) task, sentence pairs are classified into one of three semantic relations: E NTAILMENT, C ONTRADICTION or U NKNOWN. While we find some sentence pairs hold full entailments or contradictions, there are a number of pairs that partially entail or contradict one another depending on a specific situation. These partial contradiction sentence pairs contain useful information for opinion mining and other such tasks, but it is difficult for Internet users to access this knowledge because current frameworks do not differentiate between full contradictions and partial contradictions. In this paper, under current approaches to semantic relation recognition, we define a new semantic relation known as C ONFINEMENT in order to recognize this useful information. This information is classified as either C ONTRADICTION or E NTAILMENT. We provide a series of semantic templates to recognize C ON FINEMENT relations in Web texts, and then implement a system for recognizing C ONFINEMENT between sentence pairs. We show that our proposed system can obtains a F-score of 61% for recognizing C ON FINEMENT in Japanese-language Web texts, and it outperforms a baseline which does not use a manually compiled list of lexico-syntactic patterns to instantiate the semantic templates.
1 Introduction On the Internet, there are various kinds of documents, and they often include conflicting opinions or differing information on a single topic. Collecting and organizing this diverse information is an important part of multi-document summarization. When searching with a particular query on the Internet, we want information that tells us what other people think about the query: e.g. do they believe it is true or not; what are the necessary conditions for it to apply. For example, consider the hypothetical search results for the query given in (1). You get opinion (2a), which supports the query, and opinion (2b) which opposes it. (1) Xylitol is effective at preventing tooth decay. (2) a. Xylitol can prevent tooth decay. b. Xylitol is not effective at all at preventing tooth decay. A major task in the Recognizing Textual Entailment (RTE) Challenge (Giampiccolo et al. (2007)) is classifying the semantic relation between a Text and a Hypothesis into E NTAILMENT, C ONTRADICTION, or U NKNOWN. Murakami et al. (2009) report on the S TATEMENT M AP project, the goal of which is to help Internet users evaluate the credibility of information sources by analyzing supporting evidence from a variety of viewpoints on their topics of interest and presenting them to users together with the supporting evidence in a way that makes it clear how they are related. A variety of techniques have been successfully employed in the RTE Challenge in order to recognize instances of textual entailment. ∗
Current afflication: Rakuten Institute of Technology
215
However, as far as we know, there have been no studies on recognizing sentences which specify conditions under which a query applies, despite the fact that these relations are useful information for Internet users. Such useful sentences are plentiful on the Web. Consider the following examples of C ONTRA DICTION and E NTAILMENT . (3) a. Xylitol can not prevent tooth decay if it not at least 50%. b. The effect of Xylitol on preventing tooth decay is limited. In example (3a), the necessary condition to prevent tooth decay by Xylitol is “it contains at least fifty percent Xylitol”. That condition is expressed by the phrase in bold in (3a). This sentence informs users that if they want to prevent tooth decay, the products they use must contain a certain amount of Xylitol to be effective. In example (3b), we obtain information on uncertainty of Xylitol’s tooth decay prevention effectiveness from the phrase “is limited”. It tells that Xylitol is not necessarily effective at preventing tooth decay, and thus it is not completely in agreement with or contradiction to the original sentence (1). It is important to recognize the semantic relation shown in (3) because it provides more specific information about the query or specifies the conditions under which the statement holds or does not. This is valuable information for Internet users and needs to be distinguished from fully contradicting or agreeing opinions. We call this semantic relation C ONFINEMENT because it confines the situation under which a query applies. In this paper, we give a language independent definition of the C ONFINEMENT relation in predicate logic and provide a framework for detecting the relation through a series of semantic templates that take logical and semantic features as input. We implement a system that detects C ONFINEMENT relations between sentence pairs in Japanese by instantiating the semantic templates using rules and a list of lexico-semantic patterns. Finally, we conduct empirical evaluation of recognition of the C ONFINEMENT relation between queries and sentences in Japanese-language Web texts.
2 Related Work In RTE research, only three types of relations are defined: E NTAILMENT, C ONTRADICTION, and U NKNOWN. RTE is an important task and has been the target of much research (Szpektor et al. (2007); Sammons et al. (2009)). However, none of the previous research has introduced relations corresponding to C ONFINEMENT. Cross-document Structure Theory (CST, Radev (2000)) is another approach to recognizing semantic relations between sentences. CST is an extended rhetorical structure analysis based on Rhetorical Structure Theory (RST). It attempts to describe the semantic relations between two or more sentences from different source documents that are related to the same topic. It defines 18 kinds of semantic relations between sentences. Etoh and Okumura (2005) constructed a Japanese Cross-document Relation Corpus and defined 14 kinds of semantic relations. It is difficult to consider C ONFINEMENT relations in the CST categorical semantic relations because it focuses on comparing sentences in terms of equivalence and difference between sentences. At first glance, C ONFINEMENT may seem to be defined in terms of difference between sentences, but this approach does not capture the idea of restriction on a sentence’s applicability. Thus, it is beyond the scope of CST. In the field of linguistics, Nakagawa and Mori (1995) discussed restrictions as represented in the four Japanese subordinate clause patterns. Abe (1996) researched the role of quantifiers in quantitative restrictions and the role of “だけ (only).” There is much other researches on expressions representing “confinement” in a sentence in linguistics. These expressions are useful in order to recognize phrases which contradict each other. However, as far as we know, there is no research on the relation of C ON FINEMENT between two sentences in the linguistics literature. The absence of related research makes defining and recognizing C ONFINEMENT a very challenging task.
3 The CONFINEMENT Relation We present the definition of the C ONFINEMENT relation and describe its differences from E NTAIL MENT and C ONTRADICTION . In essence, a pair of sentences is in the C ONFINEMENT relation if either the premise or consequent of the second sentence has a certain condition or restriction, and without such condition or restriction the pair is equivalent to either E NTAILMENT or C ONTRADICTION.
216
Consider an example of C ONFINEMENT setence pair: (2a) and (3a). The statement “it (Xylitol) is not at least 50%” is a condition of the statement “Xylitol can not prevent tooth decay.” It is a C ONTRADICTION if the conditional statement is satisfied. Because the truth value of the whole statement depends on various conditions to be satisfied, it is important to properly define a framework to define them. 3.1 A Logical Definition of CONFINEMENT We present a definition of C ONFINEMENT in predicate logic. We define C ONFINEMENT as a semantic relation between two sentences, where the first sentence corresponds to RTE’s Hypothesis, or the user Query, and the second sentence corresponds to RTE’s Text that has some semantic relation with the Query, which we want to identify. Here we consider sentence pairs where the Query matches the logical pattern ∀x(P (x) → C(x)), where we call P (x) the Premise and C(x) the Consequence. There are many ways of representing sentences as logical expressions, and we think that the logical pattern (∀(P (x) → C(x))) can cover a variety of queries. For example, the sentence “Xylitol is effective at preventing tooth decay.” can be represented as ∀x(isXylitol(x) → effectiveAtPreventingToothDecay(x)). Consider the case where one sentence contains only a Consequence. This case can be regarded as a special case of the above formula. We write such a sentence as ∀x(T → C(x)) showing that the Premise is always True. In this paper, we limit discussion of the C ONFINEMENT relation to the Query matching to the above logical pattern. Recognizing C ONFINEMENT between the Text and the Query having more complex semantic patterns is an area of future work. Here, we split the definition of C ONFINEMENT into subtypes according to: (i) conditions to satisfy in addition to the Premise, and (ii) limitations on the degree of the Consequence. Premise side Additional conditions for achieving the Consequence Explicit constraint Some conditional sentences use an expression correspoinding to logical “only if,” which explicitly means two way conditions as the following formula. ∀x((P (x) ∧ AdditionalCondition(x) → C(x)) (1) ∧(P (x) ∧ ¬AdditionalCondition(x) → ¬C(x))) For example, S1 in Table 1, “Xylitol is effective at preventing cavities only when it is 100%”, explicitly specify that Xylitol is effective if it is 100% and is not effective if it is not 100%. So, we assume the form of the above formula for this type of statement. Implicit constraint This type of sentence specifies an additional condition on the Premise and is represented by the following formula. The Premise needs to be satisfied for the consequence to be achieved. ∀x((P (x) ∧ AdditionalCondition(x) → C(x)) (2) Example S5 in Table 1 says “Xylitol is effective at preventing tooth decay if it is 100%”, which is assumed by Formula (2). S5 does not contain an expression such as “only (だけ)”, which explicitly specifies that C(x) does not hold when an additional condition is not satisfied. One may understand that it implicitly means “Xylitol is not effective at preventing tooth decay if it is not 100%,” but S5 does not structly require this. Consequence side Constraints on the degree of achieving the Consequence There are sentences in partial entailment or contradiction where the degree of achieving of the Consequence is limited. To represent these limitations on the Consequence side, we define a CONFINEMENT relation where the degrees of the Consequence are limited as in Example (3b). We define the following formula to represent these limitations on the Consequence side. ∀x((P (x) → Cr (x)) (3) Cr (x) represents C(x) with additional restriction. For example, S3 in Table 1 says that Xylitol is somewhat effective at preventing tooth decay, which means that there are cases in which Xylitol can not prevent tooth decay. In the case of S3 , Cr (x) is “is a bit effective”. This type of C ONFINE MENT provides valuable information about Xylitol’s limited ability to promote dental hygiene in S3 .
217
All C ONFINEMENTs on the Consequence side are of type E XPLICIT C ONFINEMENT, because they explicitly mean that a part of the Consequence is achieved but no other parts are achieved. 3.2 Semantic Templates We propose a series of semantic templates to classify sentence pairs into one of the C ONFINEMENT relation subtypes we define. The semantic templates take a set of features as input and use their values to categorize the sentence pair. In Section 4, we evaluate the coverage of the semantic templates by classifying a small set of sentence pairs using manually set feature values. In Section 6, we provide more realistic evaluation by using a proposed system to set the feature values automatically and classify sentence pairs as E NTAILMENT / C ONTRADICTION, or C ONFINEMENT. We assume that each sentence consists of a Premise and Consequence, and that each sentence pair which has a C ONFINEMENT relation contains at least one additional condition or one additional limitation as defined in Section 3.1. We know that there are a variety of expressions that indicate the presence of a C ONFINEMENT relation. For example, both “Only 100% pure Xylitol is effective at preventing tooth decay.” and “Xylitol is not effective at preventing tooth decay unless it is 100% pure.” are C ONFINEMENTs of “Xylitol is effective at preventing tooth decay.” Since it is impossible to handle all possible expressions that indicate C ON FINEMENT , we focus on covering as many as possible with three features: (1) the type of constraint, (2) the type of Premise, and (3) the type of Consequence. The features are defined in more detail below. IF-Constraint This feature indicates the type of logical constraint in the Text sentence. Its values can be “IF,” “ONLY-IF.” Premise This feature indicates the type of Premise in the Text sentence. The value “P+A” or “notP+A” means there is an Additional Condition on the Premise. The value “P” or “notP” means there is just a Premise. “not” represents the Premise have a negation. Consequence This feature indicates the type of Consequence. Its possible values are “C” (just a Consequence), “notC” (negated Consequence), “Cr ” or “notCr ” (certain partial Consequence). Semantic templates consist of a tuple of four feature values and a mapping to the confinement type they indicate. A full list of templates is given in Table 1. In the templates, a wildcard asterisk “*” indicates that any feature value can match in that slot of the template. The abbreviations E NT, C ONT and C ONF stand for E NTAILMENT, C ONFINEMENT and C ONFINEMENT respectively. Semantic templates are applied in turn from top pattern by determining the value of each feature and looking up the corresponding relation type in Table 1. We give a classification examples below. The user query is sentence S0 . Sentences S1 are Web texts. Query : S0 . Xylitol is effective at preventing tooth decay. Text [ONLY-IF P(x) ∧ AC(x) then C(x) ]: S1 . Xylitol is effective at preventing tooth decay when you take it every day without fail. In Example, IF-Constraint is “ONLY-IF”, Premise is “P+A”, and the type of Consequence is “C”. This instance has an additional condition and the Consequence matches the Query, so it is identified as an E XPLICIT C ONFINEMENT.
4 Verifying Semantic Templates In this section, we verify the effectiveness of semantic templates in recognizing C ONFINEMENT relations by testing them on real-world data in Japanese. To directly evaluate the quality of the templates, we construct a small data set of sentence pairs and manually annotate them with the correct values for each of the features defined in Section 3.2. 4.1 Data We constructed the Development set and the Open-test set of sample Japanese user queries and Internet text pairs following the methodology of Murakami et al. (2009). However, Murakami et al. (2009) annotated Query-Text pairs with coarse-grained AGREEMENT and C ONFLICT relations that subsume the
218
Table 1: Semantic templates for recognizing C ONFINEMENT Semantic features IF-constraint Premise Consequence
Relation
Number of positive example 8
Number of negative example 0
0
0
11
0
12
0
62
0
1
0
279
0
ONLY-IF
P+A
*
ONLY-IF
notP+A
*
*
*
Cr
*
*
notCr
IF
P+A
*
IF
notP+A
*
IF
P
C
E XPLICIT C ONF E XPLICIT C ONF E XPLICIT C ONF E XPLICIT C ONF I MPLICT C ONF I MPLICIT C ONF ENT
IF
notP
C
CONT
0
0
IF
P
notC
CONT
13
0
IF
notP
notC
ENT
0
0
ONLY-IF
P
C
ENT
3
0
ONLY-IF
notP
C
CONT
0
0
ONLY-IF
P
notC
CONT
0
0
ONLY-IF
notP
notC
ENT
0
0
Example S0 :キシリトールは虫歯予防に効果がある. Xylitol is effective at preventing tooth decay. S1 :キシリトールは 100%の時にだけ虫歯予防に効果があります. Xylitol is effective at preventing tooth decay only when it is 100%. S2 :キシリトールは 50%未満でない時にしか虫歯予防に効果がありません. Xylitol is effective at preventing tooth decay only when it is not under 50%. S3 :キシリトールは虫歯予防に僅かに効果があります. Xylitol is a bit effective at preventing tooth decay. S4 :キシリトールは虫歯予防にほとんど効果がありません. Xylitol is not almost of effective at preventing tooth decay. S5 :キシリトールは 100%ならば虫歯予防に効果があります. Xylitol is effective at preventing tooth decay if it is 100%. S6 :キシリトールは 100%でないならば虫歯予防に効果がありません. Xylitol is not effective at preventing tooth decay if it is not 100% S7 :キシリトールを食べると虫歯予防に効果があります. Xylitol is effective at preventing tooth decay if it is eaten. S8 :キシリトールを食べないと虫歯予防に効果があります. Xylitol is effective at preventing tooth decay if it is not eaten. S9 :キシリトールを食べると虫歯予防に効果がありません. Xylitol is not effective at preventing tooth decay if it is eaten. S10 :キシリトールを食べないと虫歯予防に効果がありせん. Xylitol is not effective at preventing tooth decay if it is not eaten. S11 :キシリトールを食べたときだけ虫歯予防に効果があります. Xylitol is effective at preventing tooth decay only when it is eaten. S12 :キシリトールを食べなかったときだけ虫歯予防に効果があります. Xylitol is effective at preventing tooth decay only when it is not eaten. S13 :キシリトールを食べたときだけ虫歯予防に効果がありせん. Xylitol is effective at preventing tooth decay only when it is eaten. S14 :キシリトールを食べなかったときだけ虫歯予防に効果がありせん. Xylitol is not effective at preventing tooth decay only when it is not eaten.
Table 2: Data set (Counts of sentences out of parenthesis and statements in parentheses) Development Open-test
Entailment 258 (282) 230
Contradiction 8 (13) 170
Confinement 79 (94) 200
345 600
All (389)
RTE relations of E NTAILMENT and C ONTRADICTION. As our task is to discriminate between C ON FINEMENT and RTE relations, we annotate each sentence pair or each statement1 pair with one of the following relations instead: E NTAILMENT, C ONTRADICTION, or C ONFINEMENT. In the case of C ON FINEMENT , we annotate Query-Text pairs which are not full E NTAILMENT or C ONTRADICTION but these Text partially agrees and disagrees with the Query. Annotations were checked by two native speakers of Japanese, and any sentence pair where annotation agreement is not reached was discarded. Table 2 shows that how many sentences or statements are in each data set. Annotated statements counts are written in parentheses. We use the Development set for evaluation of verifying semantic templates and develop list of lexical and syntactic patterns for semantic features extraction, and the Open-test set for evaluation in Section 6. 4.2 Verification Result After the data was prepared, we annotated it with the correct feature values for use with the semantic templates. This was done by manually checking for words or phrases in the sentences that indicated one of the features in Table 1. Once the features were set, we used them to classify each sentence pair. We give the numbers of instances that we could confirm for each pattern in the sixth column of Table 1 and the numbers of negative instances in the seventh column, which satisfy semantic template but does not agree Relation values in the fifth column. As a result we find that there were no statement pairs that could not be successfully classified. We grasp C ONFINEMENT relation with semantic templates for the most part. This verification data does not cover all combinations of patterns in our semantic templates, so we can not rule out the possibility of existence of an exception that cannot be classified by the semantic templates. However, we find these results to be an encouraging indication of the usefulness of semantic templates. Here are some example classifications found in the verification data. Coordinate clauses Combining multiple of I MPLICIT C ONFINEMENTs results in an E XPLICIT C ON FINEMENT relation (4)S0 . ステロイドは副作用がある. Steroid has side-effects. S1 . ステロイドの副作用はステロイド剤を長期に使用した場合におこることが多いですが短 1 Murakami et al. define a “statement” as the smallest unit that can convey a complete thought or viewpoint. In practice, this can be a sentence or something smaller such as a clause.
219
期間の使用では副作用の心配はありません. Long-term use of steroid causes side-effects, but there is no need to worry about side-effects in short-term usage.
In Example (4), S1 is an E XPLICIT C ONFINEMENT for S0 . This is derived from the combination of C ONFINEMENT of the two coordinate clauses of S1 : the former phrase “Long-term use of steroid causes side-effects” of S1 is an I MPLICIT C ONFINEMENT for S0 by our semantic templates and the latter phrase is an I MPLICIT C ONFINEMENT for S0 . Additional information for whole Query Combining of a C ONTRADICTION and an I MPLICIT C ON FINEMENT result in an E XPLICIT C ONFINEMENT (5)S0 . キシリトールは虫歯予防に効果的だ. Xylitol is effective at preventing tooth decay. S1 . 虫歯予防はキシリトールだけで済むわけではなく, 基本的には規則正しい食生活とキシリ トールを毎食後とることで虫歯の予防ができます. Tooth decay can not be prevented with Xylitol alone, but it can be fundamentally prevented with an appropriate diet and by taking Xylitol after every meal. The first clause before the comma in S1 of Example (5) corresponds to the entire sentence of S0 . The second clause after the comma helps us recognize that it is a C ONFINEMENT relation. This instance is also a combination of semantic templates, so we need to recognize negation of each statement and adversative conjunction but we do not need to add new features to Table 1.
5 Proposed System We propose a system which uses semantic templates for recognizing C ONFINEMENT consists of six steps: (I) linguistic analysis, (II) structural alignment, (III) Premise and Consequence identification, (IV) semantic feature extraction, (V) adversative conjunction identification, and (VI) semantic template application. Figure 1 shows the work flow of the system. This system takes as input corresponding to S0 and S1 , and return a semantic relation. 5.1 I. Linguistic Analysis In linguistic analysis, we conduct word segmentation, POS tagging, dependency parsing, and extended modality analysis. This linguistic analysis acts as the basis for alignment and semantic feature extraction. For syntactic analysis, we identify words and POS tags with the Japanese morphological analyser Mecab2 , and we use the Japanese dependency parser CaboCha (Kudo and Matsumoto (2002)) to produce dependency trees. We also conduct extended modality analysis using the resources provided by Matsuyoshi et al. (2010). 5.2 II. Structural Alignment To identify the consequence of S0 in S1 , we use Structural Alignment (Mizuno et al. (2010)). In Structural Alignment, dependency parent-child links are aligned across sentences using a variety of resources to ensure semantic relatedness. 5.3 III. Premise and Consequence identification In this step, we identify the Premise and the Consequence in S1 . When a sentence pair satisfies all items is satisfying, we can identify a focused chunk as the Consequence in S1 : 1. A chunk’s modality in S0 is assertion, this chunk is the Consequence in S0 2. A chunk in S1 align with the Consequence in S0 We identify the Premise in S1 when a sentence pair satisfies first, and either second or third item of the following conditions: 1. A case particle of chunks in S0 is either “が (agent marker)” or “は (topic marker)” and these chunks are children of the Consequence in S0 ’s dependency tree 2. The subject in S0 aligns with the subject of S1 3. All of the dependants of the expression “には (to, for)” have alignments in S0 dependency tree 2
http://chasen.org/taku/ software/mecab/.
220
Figure 1: An overview of a proposal system to recognize C ONFINEMENT 5.4 IV. Semantic Feature Extraction We extract features for the semantic templates using a list of lexical and syntactic patterns. These patterns were manually compiled using the development data set introduced in Section 4. Features for the semantic templates are then automatically extracted by applying these patterns to input sentence pairs. The following overviews our extraction approach for each feature. 5.4.1 IF-Constraint Feature Extraction Using CaboCha, we manually constructed lists of words and their POS that are indicators of the semantic condition under which a Premise occurs. We extract as features any words in the input sentences that appear in the list with the corresponding POS. The “IF” lexical type lists conjunctions that are the results of a conditional chunk or noun phrases that indicate a case or situation. The “ONLY-IF” lexical type is used to represent the most constraining situations. The following is our list of expressions. ⋆ IF: 場合 (in case), 時/とき/と (when), ば/なら/たら (if), で (with) ⋆ ONLY-IF: 限り/かぎり (for this time), だけ/しか/こそ (only), 初めて (for the first time), には (to, for) 5.4.2 Premise Feature Extraction We treat the words or phrases which are extracted from the constraint as conditions, and need to decide whether a given condition is the Premise or an additional condition for the Premise. The Premise is set to “P” when first step and either the second or third step of the following conditions are satisfied, and it is set to “P+A” otherwise: 1. ⋆ The condition have children in the S1 ’s dependency tree or the condition’s children are not aligned to chunks in S0 2. ⋆ The condition’s parent in S0 ’s dependency tree has any chunk with a child aligned with the Consequence in S0 , or the condition’s parent is not aligned with chunks in S0 3. ⋆ The condition’s parent does not have any expression with the meaning of “use” in the S0 ’s dependency tree When these step are satisfied and negation exists in conditional chunks, Premise is set to “notP+A,” if these step are not satisfied, Premise is set to “notP.” In the third step, we identify expressions with the meaning of “use” with our lexical list. For example 使う (use), 食べる (eat), 摂取 (take) and so on. If the condition’s parent has words in our lexical list, we identify that “Xylitol” and “eating Xylitol” and “using Xylitol” are equivalent. 5.4.3 Consequence Feature Extraction This feature is used to indicate the semantic relationship between Consequences of the sentences pair. Sentences with Consequences that share a certain amount of similarity in polarity and syntax are judged to have E NTAILMENT, otherwise they are in C ONTRADICTION. In order to be judged as E NTAILMENT, the following conditions must all be true: 1. The modality of the Consequences must be identical. 2. The polarity of the Consequences must be identical as indicated by the resources in (Sumida et al. (2008)) 3. The Premises of both sentences must align with each other
221
4. ⋆ The sentences must not contain expressions that limit range or degree such as “ほとんど (almost)” or “程度 (degree)” When all item are satisfied, the Consequence is set to “C”, otherwise it is set to “notC.” We identify whether the consequence has expressions which limit the degree or not. The Consequence is set to “Cr ” or “notCr ” when the following all conditions is satisfied: 1. Any of the children of the Consequence align with a chunk in S0 ’s dependency tree. 2. ⋆ There are expressions limiting the degree of the Consequence or the siblings in S1 ’s dependency tree When this two steps are satisfied and the all four steps to judge whether sentence pairs is E NTAILMENT or not are not satisfied, Consequence is set to “notCr .” 5.5 V. Adversative Conjunction Identification We manually compiled a list of target expressions including conjunctions such as “が (but).” When a S1 chunk containing an adversative conjunction that aligns with the Premise of S0 or the S0 ’s Premise depends on S1 chunk containing an adversative conjunction, we set each feature set in a chunk before an adversative conjunction and after an adversative conjunction to semantic templates. 5.6 VI. Semantic Template Application We apply semantic feature extracted in Step IV to semantic templates. If S1 matches multiple semantic templates with an adversative conjunction from Step V, we combine the semantic templates. We get a relation for a sentence pair in this step. 5.7 Example of Semantic Features Extraction Feature extraction is illustrated in greater detail in the examples S0 which is the query and S1 in Table 1. First, we identify words represented IF-Constraint is “ONLY-IF”: “時 (when)” is in S1 and the conditional chunk has a word “だけ (only).” Next, we evaluate each the type of Premise of each chunk to determine if it is a premise or an additional condition. The subject word “Xylitol” align between S0 and S1 , and the conditional chunk’s sibling in dependency tree of S1 is a chunk which has the subject. And the conditional chunk have a child which is not aligned any chunk in S0 , it is “100%の (100%).” And the conditional chunk has no negations. So, Premise is set to “P+A.” Finally, we check if the consequences to the conditions are aligned to the verbs and nouns indicating consequences in S0 : “prevent” and “is effective” are aligned, the modality and polarity of the Consequence are identical, these depended on by the condition, and the Consequence has no expressions which limited range or degree. Consequence is set to “C.” We set the semantic template features and get a result which the sentences relation is E XPLICIT C ONFINEMENT. Ideally patterns for setting semantic feature for semantic templates should be learned automatically, but this remains an area of future work. Nonetheless, our current experiment gives a good measure of the effectiveness of semantic templates in recognizing C ONFINEMENT relations.
6 Evaluation In Section 4, we verified that the semantic templates defined in Section 3.2 can successfully classify semantic relations as C ONFINEMENT given the correct feature values. In this Section, we present the results of an experiment in a more realistic setting by using semantic templates together with the features automatically extracted as described with our proposed system in Section 5 to determine whether or not a sentence pair has a C ONFINEMENT relation. 6.1 Setting up Evaluation While more research on recognizing E NTAILMENT or C ONTRADICTION between sentences pairs is necessary, it is important to recognize new relations that cannot be analysed in existing frameworks in order to provide Internet users with the information they need. Thus, We assume that unrelated sentence pairs will be discarded before classification, in this experiment we focus only on the recognition of C ONFINEMENT relations. So our goal in this experiment is to classify between C ONFINEMENT and N OT C ONFINEMENT. We will evaluate determining whether C ONFINEMENT sentence pairs are Explicit or Implicit in future. In our experiment, we used a gold data for structural alignment to evaluate semantic feature extraction.
222
Table 3: Results of recognizing confinement relations with our proposal system proposed system baseline system
Recall 0.65(129/200) 0.96(192/200)
Precision 0.57(129/225) 0.34(192/562)
F-Score 0.61 0.50
Table 4: Instances of incorrect classification
A False Negative
B
S0 イソフラボンで健康を回復できる. A person can regain their health with isoflavon. キシリトールは虫歯予防に効果がある. Xylitol has effects on preventing tooth decay.
C False Positive
D E
キシリトールは虫歯を予防することができる. Xylitol can prevent tooth decay. ステロイドで病気は改善できる. Steroids can cure illnesses. ステロイドは副作用が懸念される. Side effects are a worry for steroids.
S1 イソフラボンの健康への効果に期待しすぎての過剰摂取は禁物です. Excess intake of isoflavon to boost its health effects is prohibited. 歯を磨く・規則正しい食生活を送る等がきちんと行われている上でキシリトールを用いることが虫歯 予防に効果的となるのです. The use of xylitol is effective at preventing tooth decay when done while eating properly and brushing one’s teeth regularly. キシリトールを口にしていれば、虫歯を予防できると考えるのは大きな間違いです. It is a big mistake to think that one can prevent tooth decay if they put Xylitol in their mouth. アトピー性皮膚炎は、ステロイドの使用を止めれば完治する. Atrophic dermatitis will heal completely if steroid use is stopped. ステロイドの副作用は、どのくらいの量でどのくらいの期間使い続ければ現れるかは人それぞれです. The amount of steroids or period of time that causes side effects differs from person to person.
6.2 Baseline System We developed a baseline system that does not use our manually-compiled lexico-syntactic patterns in order to act as a point of comparison for the proposed system in evaluating their contribution to C ONFINEMENT recognition. The baseline system consists of performing all of the steps from of our proposed system that do not rely on manually compiled lexico-syntactic patterns. Step relying on these resources are marked with a ⋆ in Section 5 and are skipped in the baseline. Essentially, we conduct Steps I, II, and III, the parts of Step IV that can be done without manually-compiled patterns, and, finally, Step VI. In Step IV, we determine if there are any limitations on the Consequence in the Consequence Feature subset, but we do not judge whether the Consequence is E NTAILMENT or C ONTRADICTION in the baseline system. 6.3 Result and Error Analysis The results are given in Table 3. We find that our system has much higher precision than the baseline, improving by over 20%. In our system, the list of semantic patterns is effective at recognizing C ON FINEMENT . On the other hand recall has gone down compared to the baseline. The baseline judged that almost sentences are C ONFINEMENT, so the list of semantic patterns employed in our rule-based system is useful at eliminating false positives. Table 4 shows some instances of incorrect classification. Each instance is a pair (S0 , S1 ). Example A-S1 means “Excess intake of isoflavon can not boost one’s health” and “excess intake” is an additional condition for A-S1 . In this case “excess” is a lexical specifier of the specific condition and is indicated by the particle “は”. The particle “は (topic marker)” is not currently used as a feature in the semantic templates since it is very noisy, so this instance can not be detected. We need to expand our method of acquiring semantic patterns to better handle such cases. The additional condition phrase in Example B-S1 modifies “The use of Xylitol” instead of “is effective at preventing tooth decay”, preventing us from properly recognizing the limiting condition in this case. We need to conduct deeper scopal analysis to determine when the modifier of an embedded chunk should be considered as an additional condition. Example C-S1 is an instance where the system fails to recognize that “put in their mouth” is an expression meaning “use” since our lists of lexical words for features did not have it. We should increase our ability to recognize synonyms of “to use” by automatically mining data for paraphrases or approaching it as a machine learning task in order to handle examples like C-S1 . On the other hands “if steroid use is stopped” in example D-S1 is the premise which should indicate an IF condition and Negation exists, however we can not recognize it correctly since the phrase lacks negation. We will make a list of words and phrases that are antonyms of “use” in order to recognize such instances. The condition in example E-S1 is about how side-effects appear, and not a condition for the other sentence example E-S0 . This instance requires detailed semantic analysis and cannot be solved with alignment-based approaches. It represents a very difficult class of problems.
223
7 Conclusion On the Web, much of the information and opinions we encounter indicates the conditions or limitations under which a statement is true. This information is important to Internet users who are interested in determining the validity of a query of interest, but such information cannot be represented under the prevalent RTE framework containing only E NTAILMENT and C ONTRADICTION. In this paper, we provided a logical definition of the C ONFINEMENT relation and showed how it could be used to represent important information that is omitted under an RTE framework. We also proposed a set of semantic templates that use set of features extracted from sentences pairs to recognize C ONFINEMENT relations between two sentences. Preliminary investigations showed that given correct feature input, semantic templates could effectively recognize C ONFINEMENT relations. In addition, we presented empirical evaluation of the effectiveness of semantic templates and automatically-extracted features at recognizing C ONFINEMENT between user queries and Web text pairs, and conducted error analysis of the results. Currently, our system does not deal with unknown instances well since it extracts features for semantic template using manually constructed lexical patterns. In future work, we will learn features for the semantic templates directly from data to better handle unknown instances.
Acknowledgment This work is supported by the National Institute of Information and Communications Technology Japan.
References Abe, T. (1996). Restriction with‘ dake ’and modification with quantifier. Tsukuba Japanese Research 1, 4–20. in Japanese. Etoh, J. and M. Okumura (2005). Cross-document relationship between sentences corpus. In Proceedings of the 14th Annual Meeting of the Association for Natural Language Processing, pp. 482–485. (in Japanese). Giampiccolo, D., B. Magnini, I. Dagan, and B. Dolan (2007). The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, RTE ’07, Morristown, NJ, USA, pp. 1–9. Association for Computational Linguistics. Kudo, T. and Y. Matsumoto (2002). Japanese dependency analysis using cascaded chunking. In CoNLL 2002: In Proceedings of the 6th Conference on Natural Language Learning 2002 (COLING 2002 Post-Conference Workshops), pp. 63–69. Matsuyoshi, S., M. Eguchi, C. Sao, K. Murakami, K. Inui, and Y. Matsumoto (2010). Annotating event mentions in text with modality, focus, and source information. In Proceedings of the 7th International Language Resources and Evaluation (LREC’10). Mizuno, J., H. Goto, Y. Watanabe, K. Murakami, K. Inui, and Y. Matsumoto (2010). Local Structural Alignment for Recognizing Semantic Relations between Sentences. In Proceedings of IPSJ-NL196. (in Japanese). Murakami, K., S. Matsuyoshi, K. Inui, and Y. Matsumoto (2009). A corpus of statement pairs with semantic relations in Japanese. In Proceedings of the 15th Annual Meeting of the Association for Natural Language Processing. Murakami, K., E. Nichols, S. Matsuyoshi, A. Sumida, S. Masuda, K. Inui, and Y. Matsumoto (2009). Statement map: Assisting information credibility analysis by visualizing arguments. In Proceedings of the 3rd ACM Workshop on Information Credibility on the Web (WICOW 2009), pp. 43–50. Nakagawa, H. and T. Mori (1995). Pragmatic analysis of aspect morphemes in manual sentences in Japanese. The Association for Natural Language Processing 2(4), 19 – 36. in Japanese. Radev, D. R. (2000). Common theory of information fusion from multiple text sources step one: Cross-document structure. In Proceedings of the 1st SIGdial workshop on Discourse and dialogue, pp. 74–83. Sammons, M., V. G. V. Vydiswaran, T. Vieira, N. Johri, M.-W. Chang, D. Goldwasser, V. Srikumar, G. Kundu, Y. Tu, K. Small, J. Rule, Q. Do, and D. Roth (2009). Relation alignment for textual entailment recognition. In Proceedings of Recognizing Textual Entailment 2009. Sumida, A., N. Yoshinaga, and K. Torisawa (2008). Boosting precision and recall of hyponymy relation acquisition from hierarchical layouts in Wikipedia. In Proceedings of the 6th International Language Resources and Evaluation (LREC’08). Szpektor, I., E. Shnarch, and I. Dagan (2007). Instance-based evaluation of entailment rule acquisition. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pp. 456–463.
224
