Using Syntactic Distributional Patterns for Data ...

Using Syntactic Distributional Patterns for Data-Driven Answer Extraction from the Web Alejandro Figueroa1 and John Atkinson2, 1

2

Deutsches Forschungszentrum f¨ ur K¨ unstliche Intelligenz - DFKI, Stuhlsatzenhausweg 3, D - 66123, Saarbr¨ ucken, Germany Department of Computer Sciences, Universidad de Concepci´ on, Concepci´ on, Chile [email protected], [email protected]

Abstract. In this work, a data-driven approach for extracting answers from web-snippets is presented. Answers are identified by matching contextual distributional patterns of the expected answer type(EAT) and answer candidates. These distributional patterns are directly learnt from previously annotated tuples {question, sentence, answer}, and the learning mechanism is based on the principles language acquisition. Results shows that this linguistic motivated data-driven approach is encouraging. Keywords: Natural Language Processing, Question Answering.

1

Introduction

The increase of the amount of information on the Web has led search engines to deal with a huge amount of data as users have become retrievers of all sorts. Nowadays, search engines are not only focusing on retrieving relevant documents for a user’s particular request. They also provide other services (i.e., Group Search, News Search, Glossary), hence the complexity of the request of the users has addressed the research to Question Answering (QA) systems. These aim to answer natural language (NL) questions prompted by users, by searching the answer in a set of available documents on the Web. QA is a challenging task due to the ambiguity of language and the complexity of the linguistic phenomena that can be found in NL documents. Typical questions to answer are those that look for name entities as answers (i.e., locations, persons, dates, organizations). Nevertheless, QA systems are not restricted to these kinds of questions. They also try to deal with more complex ones that may require demanding reasoning tasks while the system is looking for the answer [11]. Usually, QA systems start by analyzing the query [4,7] in order to determine the EAT. The EAT allows the QA system to narrow the search space [8], while it is ranking documents, sentences or sequences of words in which the answer is 

This research is sponsored by FONDECYT, Chile under grant number 1040469 “Un Modelo Evolucionario de Descubrimiento de Conocimiento Explicativo desde Textos con Base Semantica con Implicaciones para el Analisis de Inteligencia.”

A. Gelbukh and C.A. Reyes-Garcia (Eds.): MICAI 2006, LNAI 4293, pp. 985–995, 2006. c Springer-Verlag Berlin Heidelberg 2006 

986

A. Figueroa and J. Atkinson

supposed to be. This set of likely answers is called answer candidates. In this last step of the zooming process, the QA system must decide which are the most suitable answers for the triggering query. This extraction and ranking of answer candidates is traditionally based on [6,7,8] frequency counting, pattern matching and detecting different orderings of query words, called paraphrases. Answer extraction modules attempt to take advantage of the redundancy provided by different information sources. This redundancy increases significantly the probability of finding a paraphrase, in which the answer can be readily identified. Normally, QA systems extract these paraphrases at the sentence level [10]. The rules for identifying paraphrases can manually be written or automatically learnt [6,10], and they can consist of pre-parsed trees [10], or simple string based manipulations [6]. In general, paraphrases are learnt by retrieving sentences that contain preciously annotated question-answer pairs. For example in [10], anchor terms (i.e., “Lennon 1980”) are sent to the web, in order to retrieve sentences that contain query and answer terms. Then, patterns are extracted from this set of sentences with their likelihood being proportional to their redundancy on the Web[7]. In most cases, the new set of retrieved sentences is matched with paraphrases in order to extract new answers. At the same time, a huge set of paraphrases [6] decreases considerably the need of deep linguistic processing like: anaphora or synonym resolution. In some cases, it reduces the extraction to a pattern matching by means of regular expressions[10]. As a result, strategies based on paraphrases tend to perform better when questions aim for a name entity as an answer: Locations, Names, Organizations. But, they perform poorly when they aim for Noun Phrases[10]. Due to the huge amount of paraphrases, statistical methods are also used for extracting answers. In [5], a strategy for answering questions is learnt directly from data. This strategy conceives the answer extraction problem as a binary classification problem in which text snippets are labelled as correct or incorrect. The classifier is based on a set of features from lexical n-grams to parse trees. The major problem of statistical-based approaches is that, frequently, they get inexact answers, which usually consist of substrings of the answer, the answer surrounded by some context words, or strings highly closed to answers. Nevertheless, it is still unclear how each different technique contributes to deal with the linguistic phenomena that QA systems face while searching for the answer. One solution for this may involve a trade-off between the implementation of rule-based and easy re-trainable data-driven systems. In [10], a strategy for combining the output of different kinds of answer extractors is introduced. This re-ranker is based on a Maximum Entropy Linear Classifier, which was trained on a set of 48 different types of features such as ranking in the answer extraction modules, redundancy, negative feedback, etc. Results show that a good strategy for combing answer extractors, based mainly on different strategies, can significantly improve the overall performance of QA systems [11]. Strategies based on paraphrases aim to find a re-writing of the query within the text where the answer is easily identified. Their main drawback is that whenever the answer is in an context, which do not match any re-writing rule, it will

Using Syntactic Distributional Patterns

987

not be identified. In this work, we take advantage of the redundancy of the Web in a different way. We claim that some answers can be readily identified by comparing their syntactic behaviour on snippets with the syntactic behavior of the EAT. The likelihood of answers to the EAT is supported by syntactical distributional patterns presented on their occurrences. In this data-driven strategy, the syntactic behavior of the EAT is directly learnt from previously annotated tuples {question, sentence, answer}. In contrast to another approaches, we do not use general-purpose classifiers, we base our learning process on Pinker’s language acquisition theory [3]. This states that innate knowledge and a learning strategy along with negative evidence are required for a child to successfully acquire many linguistics properties including syntactic categories. In our approach, the innate knowledge includes previously annotated tuples and the negative evidence is a set of words with an ambiguous syntactic behavior. Learning is carried out into two phases: a likelihood metric to identify how likely the syntactic behavior of the EAT is, and a mechanism to have new evidence added to the innate knowledge. Instead of combining the output of many answer extractors, the output is used for annotating pairs {sentence, answer} which are used for inferring the syntactic behavior of the EAT afterwards. Thus, any manual annotation or dependence on the context of the query is removed. The model takes advantage of the redundancy of the web so as to obtain words that behave best compared to the EAT, in contrast to other approaches, in which some metric according to their frequency is used for measuring the likelihood of strings as answers. The answer extraction strategy was assessed with 781 questions regarding locations. The model interprets this kind of question as one of the most complex and ambiguous scenarios for our strategy, because locations have an ambiguous syntactic behaviour in English. The annotator makes usage of a lexical database of locations for ranking answers to location-related questions and annotating pairs {sentence,answer}, the returned rank is only used as a baseline. Overall results suggest that our method is robust and can efficiently find right answers and even uncover more accurate answers in presence of ambiguity. This also manage to even find answers to questions that could not be distinguished in our baseline. This paper is organized as follows: section 2 describes the strategy for acquiring syntactic behavior patterns for the EAT, section 3 discusses our answer extraction strategy, in section 4 the experiments and main results are highlighted.

2

Automatic Acquisition of Syntactic Categories

The most commonly used document representation is known as the Vector Space Model. Here, a document D is represented as a vector in a space in which each dimension is associated with the frequency of one word wi in the dictionary W . D = (f req(w1 ), f req(w2 ), . . . , f req(wω )) ∈ ω In this representation, some grammatical information is lost as the order of words and punctuation is ignored leading to broken phrases. For example,

988

A. Figueroa and J. Atkinson

“Albert Einstein” is split into “Albert ” and “Einstein” without representing their syntactic relation. This model also does not take into account the role of words as modifiers in their local context, and/or as suppliers of the predicate or argument of the main proposition being expressed. The role of a word in a text is given by its syntactic category (i.e., noun, verb, adjective). From the statistical viewpoint, syntactic rules involve distributional patterns, whereas in linguistics, distributional analysis is referred to as the study of syntactic properties that are in essence distributional. Many efforts have been put in modelling the syntactic behavior of words using unsupervised mechanisms [1,2]. In these two approaches[1,2], each word wi ∈ W is represented by two vectors, called syntactic context vectors. The dimensions of the first vector φl (wi ) represent how often the other words in W appear immediately to the left of wi , whereas the second vector φr (wi ) follows a similar strategy for words that appear immediately to the right. To illustrate this, consider the next two sentences: “Galway is in Ireland ” and “Dublin is located in Ireland ”. The syntactic context vectors of these sentences are sketched in matrices of tables 1 and 2. Table 1. Left Syntactic Context Vectors

Dublin Galway in Ireland is located

Dublin Galway in Ireland is located 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 2 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0

Table 2. Right Syntactic Context Vectors

Dublin Galway in Ireland is located

Dublin Galway 0 0 0 0 0 0 0 0 0 0 0 0

in Ireland is located 0 0 1 0 0 0 1 0 0 2 0 0 0 0 0 0 1 0 0 1 1 0 0 0

In table 1, we read that “in” appears two times to the left of “Ireland ”, and in table 2, “Ireland ” appears two times to the right of “in”. The main problem of the syntactic context vectors is that the degree of overlap can not be computed in the original vector space due to its sparseness. A simple cosine-based similarity measure may draw misleading classifications, even though the frequency of words is high. For example: “a” and “an” do not share any neighbors as “an” appears

Using Syntactic Distributional Patterns

989

whenever the sound of the next word starts with a vowel so consequently, the similarity is zero, although they have the same syntactic category [1]. The first approach is due to Goldsmith and Belkin [2], who constructed a nearest-neighbor graph (i.e., matrix) in which vertices represented words and edges were pairs of words whose distribution in the corpus was similar. For this graph, they used the top 500 and 1000 frequent words. For each pair of words, the cosine of the angle of its syntax context vector was computed, and the 5, 10, 20 and 50 closest neighbors were selected. From the constructed matrix, a canonical representation C is built, in which a value of zero was assigned to every element in the diagonal and wherever was a zero was in the original matrix, a value of one was assigned whenever a value was greater than zero in the original matrix. A diagonal matrix E is defined to contain the degree values for each vertex. Then, the normalized laplacian of E − C is computed to contain nonnegative eigenvalues. The first and the second eigenvectors -corresponding to the lowest eigenvalues- derived from each syntax context vector were used to build a graphic representation of the syntactic behavior of the words in the corpus. These vectors have a coordinate for each of the K most frequent words in the corpus. Using these lowest-valued eigenvectors was suggested to provide a good graphical representation of words, in the sense that words with similar left-hand neighbors will be close together in the graph. Even though this strategy does not lead to a sharp distinction of syntactic categories, it can distinguish syntactically heterogeneous set of words [2]. The strategy was evaluated for two languages French and English. For English, the syntax category of many constituents (i.e., non-infinitive verbs, infinite verbs, nouns, etc) were correctly inferred. For French, other categories such as female nouns, plural nouns, finite verbs, etc. were clustered. A similar model for the acquisition of syntactic categories from raw text in presence of ambiguity is also introduced by [1]. In this model (TAG SPACE), two matrices are built from the syntactic context vectors of the 250 most frequent words. The Singular Value Decomposition (SVD) was used for reducing the dimension of both matrices and so dealing with data sparseness. The dimension of the matrices in the reduced space was 50 and a group average agglomeration algorithm was applied for clustering. In both approaches, syntactic context vectors are represented in a specially designed space, in which different syntactical categories show distinctions. Consequently, syntactic context vectors of words were found to contain information about their syntactic behavior.

3

The Answer Extractor System

Once a Natural-Language query triggers our QA system (QA-SYSTEM), this is sent out to Google so to retrieve a small number of snippets (i.e., usually 30), which are then normalized and cleaned up of math symbols and html tags. Next, the system performs the query analysis in order to determine the EAT by a simple

990

A. Figueroa and J. Atkinson

Wh-keyword matching. If the EAT is a location, it triggers our answer extraction module based on the acquisition of distributional syntactic patterns (SCV-AE). 3.1

Answer Extraction by Acquiring Syntactic Patterns

First, the answer extractor (SCV-AE) extracts WEB ENTITIES from the retrieved snippets. A WEB ENTITY is a stream of words in which every word of the sequence starts with a capital letter, for instance: “Robbie Williams”, “London”, etc. Then, the negative evidence is used for filtering the WEB ENTITIES according to a list of banned WEB ENTITIES, we see as banned words, query terms and words that usually start with a capital letter on web snippets (i.e., page, home, link, etc). This rule for distinguishing WEB ENTITIES is considered as part of the innate knowledge. From here, the set of sentences (determined by punctuation signs) and WEB ENTITIES are passed on to our Automatic Annotator (AA) which returns at most three ranked answers and the sentences where they occur. The annotated sentences are used by SCV-AE for updating the syntactic context vectors, and for computing the value of the likelihood L for every WEB ENTITY. The learning strategy is seen as the synergy between the annotator and the answer extractor. The rank of answers returned by AA is only used as a baseline. AA is based on a strategy for extracting answers to questions which aim at a location as answer. This measures the similarity between the query and each sentence by aligning characters of the query in each sentence. In addition, AA validates each WEB ENTITY using a lexical database of locations (WordNet). This strategy can identify answers if and only if they are on the lexical database. Let Q be the set of all questions that triggered the QA-SYSTEM which aimed to the same EAT. A is the set of answers to the questions in Q. Each component φi of the syntactic context vectors of the EAT of Q is given by: φli (EAT ) = sum∀Aj ∈A f req(wi , Aj ) φri (EAT ) = sum∀Aj ∈A f req(Aj , wi ) Where f req(wi , Aj ) is the frequency in which wi occurs immediately to the left of Aj , the sum over all Aj ∈ A gives the frequency of wi to the left of the EAT, and f req(Aj , wi ) is the homologous to the right. Next, φl (EAT ) and φr (EAT ) provide the information of the role of the EAT in the local context. For the simplicity’s sake, φl and φr refer to syntactic context vectors φl (EAT ) and φr (EAT ) respectively. If we consider our illustrative example, φl (LOCAT ION ) and φr (LOCAT ION ) are shown in table 3. Note that φr represents the null vector as there is no word occuring to the right of the EAT LOCATION. Then, the Syntactic Likelihood of an answer  A is computed as follows: 





L(A ) = φl φl (A ) + φr φr (A )

(1)

Using Syntactic Distributional Patterns

991

Table 3. Syntactic Context Vectors for EAT LOCATION φl φr

Dublin Galway in Ireland is located 0 0 2 0 0 0 0 0 0 0 0 0



Where φl φl (A ) is the sum of the product of each component of the left syntactic context vector of the EAT, whereas the left syntactic context vector of the answer   A , φr φr (A ) is the homologous to the right. Every answer is measured according to the amount of its context words in the snippets that match the context words of the EAT and the strength of this matching is according to their frequencies. At this point, the redundancy of the web is taken into account. The context words are assumed to occur more often in the context of the EAT have a stronger relationship with the EAT, and therefore, are stronger indicators for scoring a new answer. Consider a document consisting of the following sentence: “Saar is in Saarbr¨ ucken.”. Tables 4 and 5 illustrate the obtained syntactic context vectors. Table 4. Left Syntactic Context Vectors of the Document

Saar in is Saarbr¨ ucken

Saar 0 1 0 0

in 0 0 1 0

is Saarbr¨ ucken 0 0 0 0 0 0 1 0

Table 5. Right Ssyntactic Context Vectors of the Document

Saar in is Saarbr¨ ucken

Saar 0 1 0 0

in 0 0 1 0

is Saarbr¨ ucken 0 0 0 0 0 0 1 0

Computation of the likelihood of each word to the EAT can be seen in table 6. Note that the only word that contributes to the likelihood is “in”-when it is to the left to EAT-, then the only match occurs with the occurrence of “in” to the left of “Saarbr¨ ucken”. As a result, this is the only word with likelihood greater than zero. Experiments suggest that this likelihood is strongly affected by the data sparseness. However, the aim of the approach is not to cluster words to uncover their syntactic categories. The model assumes that every Aj ∈ A has the same syntactic behavior in the local context of the answer. Thus the main in terest is in the likelihood of A .

992

A. Figueroa and J. Atkinson Table 6. Syntactic Context Vectors for EAT LOCATION 

φl φl (A )  φr φr (A ) Total

4

Saar 0 0 0

in 0 0 0

is Saarbr¨ ucken 0 2 0 0 0 2

Experiments and Results

SCV-AE was assessed with three kinds of questions: miscellaneous where-typed questions from the CLEF and TREC corpus, an example question is as follows: Where did the 1992 Olympic Games take place? Next, a set of questions concerning capitals and cities around the world were tested. These have the following template: where is < city >? Where < city > is replaced with other cities (i.e., Berlin). We consider as a correct answer the name of the country, for example, Germany. For the third kind of questions, a list of monuments around the world was tested with a template similar to that of the cities: where is Fontana della Barcaccia? We accepted either a city or a country as a correct answer (i.e., Rome or Italy). An alternative answer is another city or country that has also a monument or a city with the same name as the requested place. For example, for “where is Santo Domingo? ”, the system answered “Venezuela” instead of “Dominican Republic” (there is a place named “Santo domingo” in Venezuela). We considered three set of questions, due to the fact that the CLEF/TREC questions are oriented to their respectively corpus, that is, the provided answer patterns correspond their corpus. Our system answers questions using the web as a target. Accordingly, it is unknown in advance whether one of occurrences of the answer will match the provided answer patterns[7]. Considering that there is no much variations on names of countries and cities, the SCV-AE was assessed with an additional set of questions. Table 7 and 8 show the results for a set of 781 questions. The MRR (Mean Reciprocal Rank) values are 0.833 and 0.830 for AA and SCV-AE respectively considering all the correct answers. Our strategy can also be compared with the approach by [9] which scored MRR values below 0.55 for a given set of locations. Note that this system also uses WordNet and other linguistic resources such as gazetteers. The performance of both the annotator and SCV-AE is worst for the CLEF/TREC dataset. Here, SCV-AE could not answer 60 (out of 229) questions (20.20%), whereas AA did it with 58 out of 229 (25.33%). If we consider the set of monuments/capitals, AA could not find an answer for 40 out of 552 (7.25%) questions, SCV-AE (6.16%). Our SCV-AE could not answer either because

Using Syntactic Distributional Patterns

993

Table 7. AA Results for Location Questions (Baseline) Settings Set Total Capitals 186 World Heritage Monuments 209 Biggest monuments 51 Greece 58 Stotland/Ireland/Wales 58 TREC-2002 39 TREC-2001 27 TREC-2000 70 TREC-1999 22 CLEF-2004 61 Total 781 % -

MRR No Answer 0.88 14 0.91 7 0.84 6 0.78 10 0.94 3 0.64 13 0.73 6 0.66 20 0.57 9 0.78 12 0.833 100 12.8

Exact Answers 1st 2nd 3rd 155 11 2 177 14 5 40 3 43 5 54 21 2 18 2 41 6 1 12 1 42 3 603 45 10 77.21 5.76 1.28

Alt. Answers 1st 2nd 3rd 4 6 1 1 1 3 1 2 4 21 2 2.69 0.26 -

the answer was a term on the query (10 questions) or because there was no answer at all. Note that the difference can also be due to some questions SCV-AE was able to answer but AA did not annotate, which may be due to certain dependency on the lexical database. There are a few cases where AA was able to annotate pairs {answer,sentence} with the right answer, whereas SCV-AE did not find enough evidence to identify the answer as a location (i.e., answering the query “Where is Sanna? ”). On the other hand, SCV-AE was capable of providing answers even when it was not possible to find them on the lexical database (i.e., capitals of Kyrgyzstan (Bishkek) and East Timor (Dili)). In addition, SVC-AE succeeds to identify answers when they were spelt in their original language and surrounded by text in English. However, this could not detect answers while they were surrounded by text written in another language. Note that there are some questions in this dataset that the strategy could not identify as locations (i.e., Where did ’N Sync get their name? - the initial letter of their member names) and so the strategy was unable to answer. On the other hand, the SCV-AE module assisted us to identify answers to questions such as Where is bile produced? (liver),Where is the Sea of Tranquility? (moon), Where is the volcano Olympus Mons located? (mars), Where does chocolate come from? (Cacao). SCV-AE was able to determine more precise locations (i.e., where is Gateway Arch? Jefferson National Expansion Memorial). Some examples of questions that could not be answered or annotated: Where did Bill Gates go to college?,Where did bocci originate?,Where do lobsters like to live?. In most cases, the right answer was ranked at the first position in almost 80% of the questions and at the second place at about 7% of the times. It is also important to highlight that the influence of the answers passed on from the AA module to the SCV-AE module biased the answers obtained by the SCV-AE module in the first answered questions.

994

A. Figueroa and J. Atkinson Table 8. SCV-AE Results for Location Questions

Settings Set Total Capitals 186 World Heritage Monuments 209 Biggest monuments 51 Greece 58 Stotland/Ireland/Wales 58 TREC-2002 39 TREC-2001 27 TREC-2000 70 TREC-1999 22 CLEF-2004 61 Total 781 % -

5

MRR No Answer 0.88 11 0.91 4 0.84 7 0.82 8 0.92 4 0.67 11 0.75 6 0.65 20 0.59 8 0.69 13 0.830 92 11.78

Exact Answers 1st 2nd 3rd 154 11 6 154 20 3 37 1 39 5 53 1 24 1 1 18 1 41 4 3 12 2 36 3 5 568 47 20 72.73 6.02 2.56

Alt. Answers 1st 2nd 3rd 4 25 2 1 6 6 1 1 2 2 2 2 48 5 1 6.15 0.64 0.13

Conclusions

This paper discussed a new data-driven approach which takes advantage of the large volume of documents on the web. Unlike related approaches using generalpurpose classifiers or intensive linguistics tasks or knowledge sources (i.e. parses, stemmers, lexical databases, formalisms), our model exploits a purpose-built learning approach based on Pinker’s theory of learning distributional syntactic categories. In our approach, distributional patterns are seen as syntactic context vectors, the innate knowledge is represented as previously annotated tuples {question, sentence, answer} and negative evidence is interpreted as ambiguous words. In contrast with statistically motivated approaches, our strategy was still capable of distinguishing precise answers strings. The model takes advantage of the redundancy of the web as a source of multiple paraphrases and answer occurrences. Hence it can learn the syntactic behavior of the EAT from annotated tuples {question, sentence, answer}, and compare it with the syntactic behaviour of occurrences of words on snippets. The designed strategy sharply identifies answers on the web due to the localized context of web snippets. Here, occurrences of the answer have a similar syntactical behavior so that they can readily be identified. Accordingly, for some kind of questions, parsing or tagging is not necessary for detecting which words match the syntactic category of the EAT. Moreover, the strategy does not need to match the context of the query for identifying answers and so using query re-writing rules for some kinds of questions is not mandatory. For the actual experiments, location questions were only considered as they involve fewer candidate answers. Thus, the assessment becomes less ambiguous and allows ambiguous syntactical patterns to easily be identified by our learning mechanism. Note that dates and locations as well as some prepositional phrases

Using Syntactic Distributional Patterns

995

syntactically behave similarly in English. Assessing the strategy by using different kinds of questions (i.e., persons, organizations) is also being planned in a future research.

References 1. Sch¨ utze, H.Ambiguity Resolution in Language Learning, Computational and Cognitive Models. CSLI Lecture Notes, number 71, 1997. 2. Belkin, M., Goldsmith, J.Using eigenvectors of the bi-gram graph to infer grammatical features and categories, Proceedings of the Morphology/Phonology Learning Workshop of ACL-02, 2002. 3. Pinker, S. Language Learnability and Language Development, Cambridge MA: Hardvard University Press, 1984. 4. Rijke, M., Monz, C. Tequesta: The University of Amsterdam’s Textual Question Answering System, NIST Special Publication SP, 2002. 5. Lita, L., V., Carbonell, J. Instance-based question answering: a data driven approach, Proceedings of EMNLP, 2004. 6. Dumais, S., Banko, M., Brill, E., Lin, J., Ng, A. Web question answering: is more always better?, Proceedings of SIGIR-2002, 2002. 7. Dumais, S., Banko, M., Brill, E., Lin, J., Ng, A. Data-Intensive question answering, In proceedings of the tenth Text REtrieval Conference (TREC 2001), November 2001, Gaithersburg, Maryland. 8. De Chalendar, G., Dalmas, T., Elkateb-Gara, F., Ferret, O., Grau, B., HuraultPlanet, M., Illouz, G., Monceaux, L., Robba I., Vilnat A. The question answering system QALC at LIMSI: experiments in using Web and WordNet, NIST Special Publication SP, 2003. 9. Monz, C. From Document Retrieval to Question Answering, IILC Dissertation Series DS-2003-4, Institute for Logic, Language and Computation, University of Amsterdam, 2003. 10. Echihabi, A., Hermjakob, U., Hovy, E., Marcu, D., Melz, E., Ravichadran, D. How to select an answer String?, Advances in Textual Question Answering, Kluwer, 2004 11. Moldovan, D., Harabagui, S., Clark, C., Bowden, M., Lehmann, J., Williams, J. Experiments and Analysis of LCC’s two QA Systems over TREC 2004, TREC 2004, 2004.