Evaluating the Use of Project Glossaries in Automated ...

Evaluating the Use of Project Glossaries in Automated Trace Retrieval Xuchang Zou, Raffaella Settimi, and Jane Cleland-Huang College of Computing and Digital Media, DePaul University, Chicago, IL, USA Abstract - Automated traceability methods use information retrieval techniques to dynamically generate traceability links, however they suffer from precision problems. This paper extends our previous work in using a project glossary to improve trace results and presents criteria for evaluating whether an existing project glossary can be used to enhance results in a given project. A new approach for automatically extracting a set of important terms and phrases is also described. Our experimental results suggest that these terms and phrases can be used effectively in lieu of a project glossary to help improve precision of the retrieved traces. Keywords: Requirement traceability, automated trace retrieval, information retrieval models.

1

Introduction

Requirements traceability supports a number of critical software engineering tasks such as requirements validation and verification, change management and software maintenance. It has been widely recognized as an essential task in an increasing number of process improvement standards and regulatory requirements [5]. However, traditional traceability methods require extensive human effort to implement effectively. Several researchers have therefore developed dynamic traceability algorithms which use Information Retrieval (IR) techniques to dynamically generate traceability links on an “as-needed” basis [1,2,6,9]. Because software artifacts such as requirements, design documents and source code contain large amounts of textual information, their dependencies can often be discovered using IR techniques such as the Vector Space Model [6], Latent Semantic Indexing model [1,9], and Probabilistic Network model [2]. The effectiveness of such IR-based requirements tracing tools are typically assessed using two standard IR metrics: recall defined as the proportion of true links that are retrieved by the tool out of all the true links, and precision defined as the proportion of retrieved links which are true links [4]. In the context of requirements traceability, a tracing tool must obtain high recall, i.e. it must retrieve as many true links as possible, as it is easier for the analyst to examine the subset of retrieved links to filter out the

unwanted ones, than to search through the entire document collection to find missing true links. Furthermore, we have found that practitioners are not willing to use an automated tracing tool unless they are confident that it can find all of the true links. Previous empirical studies in dynamic tracing retrieval indicate that at high recall levels of 90%, precision is usually below 40% and sometimes even less than 10% [1,2,6,9]. Although using an automated IR approach significantly reduces the effort required to manually perform tracing, the low precision of the results means that analysts still have to evaluate a large number of candidate links in order to find the true ones. The low precision of the trace retrieval results may affect the analyst’s trust in the accuracy of the tool and could limit the adoption of IR-based automated traceability methods in industry. In an effort to improve the precision of the results, our previous work [16] proposed a trace retrieval approach that utilized the information in the glossary of a software project. Because terms and phrases defined in the project glossary tend to capture many of the critical concepts of a project, links between two artifacts in which project terms or phrases co-occur are assigned higher relevance scores by the retrieval algorithm. The preliminary results have suggested that this approach can improve precision especially among the top retrieved links. However, the effectiveness of the glossary approach relies on the availability of a meaningful glossary which has been consistently utilized to construct all traceable artifacts. This paper addresses the problem that occurs when either no project glossary is available, or when the glossary has not been used consistently.

2 Using the project glossary Our previous work [2] described a Probabilistic Network (PN) model [14] for dynamically generating and retrieving links between requirements and software artifacts. Relevance scores are computed between two sets of artifacts, known as queries and documents. In this paper the queries are individual requirements. The probabilistic model assumes that a software artifact d containing a set of terms co-occurring in a requirement q is likely to be related to that requirement, and a trace should be established between d and q. The probability score p(d|q), representing the likelihood of a link between q and d, is defined in terms of

the frequency freq(d, ti) of terms ti co-occurring in both q and d, and is calculated as p(d | q) = ∑ p(d | t i ) p( q, t i ) / p(q ) . This expression is i

based on the tf-idf [13] standard weighting strategy. The first component p(d | t ) = freq(d , t ) / freq(d , t ) represents i

i

∑

i

i

the relative frequency of term ti in document d and increases with the number of occurrences of ti in d. The second component p(q, t i ) = freq(q, t i ) / ni , with ni being the number of traceable documents containing term ti, represents the inverse document frequency. This component assumes smaller values for frequently used terms that appear in many documents and therefore provide weak information about potential links. The last component represents the amount of textual information p(q)=∑ p(q,ti ) i

in the query. Potential traces to a requirement q are identified by retrieving all documents dj whose probability score p(dj|q) falls above a certain threshold. A high value of p(dj|q) represents strong evidence that a potential link between q and dj exists.

2.1

Retrieval algorithm utilizing the glossary

Previous studies analyzing the application of the basic PN algorithm to requirements tracing have shown that high recall levels are typically achieved at the expense of precision. For instance, recall levels of 80-90% correspond to low precision values ranging from 40% to levels as low as 10% [2]. To address the low precision problem, an approach was previously proposed to enhance the basic PN algorithm by considering the additional information included in the glossary of software projects. Key terms and phrases in the project glossary tend to capture the critical concepts of a project. Hence two documents that contain the same glossary items are likely to be related to the same specific concept. The enhanced retrieval algorithm therefore assigns heavier weights to glossary terms and phrases, and computes a higher probability score for a link between a requirement and a traceable document that share the same glossary items. The glossary approach is typically applied along with phrasing [16], an enhancement method that computes the probability score between two documents considering not only single terms but also phrases cooccurring in both documents. The reasons for incorporating the glossary approach into the phrasing method are as follows: 1) Phrases are detected in the phrasing method by using QTag, a freely available Part-of-Speech (POS) tagger [15] that uses a dictionary to identify the syntactic category of each token in the text. All phrases are assumed to be equally meaningful and contribute equally to strengthening the belief in a link. However, there are usually certain phrases or terms in a project that are more significant for capturing

its core concepts. As an example, in the IBS dataset [12], a system that describes the requirements and design of a public works department system for managing the de-icing of roads, four phrases were identified from one requirement by QTag: summary report, weather report, weather conditions and time period. Higher weights should be assigned to phrases that are related to core concepts for IBS such as weather report and weather conditions rather than to summary report and time period. The two more meaningful phrases appear in the glossary of IBS. Thus using the glossary enables the IR based traceability tool to identify more critical phrases and to weight them more heavily. 2) The project glossary may also contain additional phrases that the syntactic parser was unable to discover. These phrases may for example not fit into the prescribed grammatical template or may contain more than the standard number of terms. The probabilistic information retrieval using the project glossary is defined as follows. The set SPG={k1,k2,…,km} contains keywords defined in the project glossary, and SPH={t1,t2,…,tn} is the set of terms in phrases defined in the glossary. The new probability pPG(d|q) of a link between a document d and a query q is computed considering single terms and phrases co-occurring in both documents as well as additional information in the project glossary. The link probability score pPG(d|q) is defined as follows: p PG (d |q )∝ p (d |q )+ p f (d |q )+ p f (d |t i ) p f (q|t i ) (1) p (d |k i ) p (q|k i ) δ ∑ +δ ∑ p(q) p(q) k i ∈S PG t i ∈S PH where the component p(d|q) is the basic probability score defined in section 2, the value pf(d|q) represents the contribution of phrases to the link probability score and is computed similarly to p(d|q) but only considering the frequency of phrase terms co-occurring in both d and q. The remaining two components represent the use of project glossary terms and phrases in the new IR based tracing approach. The expression in (1) assigns heavier weights to project glossary keywords and phrases co-occurring in both d and q, and their contribution to the overall probability pPG(d|q) depends on the value of a parameter δ≥0 to be set by the analyst. In our experiments, the parameter δ is set equal to 0.5. A preliminary evaluation [16] of the results of the glossary approach application to trace retrieval indicates that a higher percentage of true links are found at the top of the retrieved links list. The results described in section 2.2.2 show that for the relatively large IBS dataset a significant increase of 17% (from 47% to 64%) in precision is achieved at recall level of 10%, indicating that more true links were among the top ranked retrieved links. With such improvement it is likely that important links will be found earlier in the analysis process and this might raise the confidence of the analyst on the tracing tool. However, the

analyst still needs to evaluate a long list of potential links if she wants to find all correct links.

2.2 Evaluating the project glossary Although some experiments show its effectiveness in improving precision for the top ranked retrieved links, the project glossary approach has some limitations. It assumes that a project glossary exists and furthermore, that it was used consistently throughout the design of software artifacts. In reality many software projects do not have a preconstructed glossary available or have a “weak” glossary that has not been consistently used. It would be helpful to predict the effect of the project glossary usage in link retrieval prior to running the automated trace retrieval technique. Thus, the following research problems were investigated: 1. What characteristics must an existing project glossary have in order to be potentially useful for improving the retrieval results? 2. For projects in which a glossary is unavailable, can a set of key terms and phrases be extracted and used in lieu of the project glossary to help improve the retrieval results? To answer the first question, an empirical study was conducted to analyze the retrieval results using existing project glossaries. The study identified a set of characteristics for the existing project glossaries that can predict whether the glossary approach is effective in improving the precision of tracing results.

2.2.1 Criteria for a useful project glossary Three project glossary characteristics have been investigated and the following criteria have been established: Criterion # 1: Project glossary items should be consistently used in the traced documents. A project glossary is usually provided as part of the requirements documentation to reflect the terminology used in a specific software project. It is strongly suggested to avoid using synonyms of the glossary items during the creation of the software artifacts as they can introduce ambiguity and could potentially lead to misinterpretation of those artifacts. However, existing project glossaries are frequently not consistently followed during the project development. As a result, using such project glossary information in the automated trace retrieval tool may have little or no impact on the retrieval results. One approach to evaluate whether or not the glossary terms have been followed is to detect the occurrence of the synonyms of individual glossary terms. The presence of synonyms of a given term, even a single occurrence in the traced documents, hints that the term may not have been consistently used. Criterion # 2: Glossary items should have high term specificity. Term specificity of individual glossary items is also a critical factor that can affect the effectiveness of the

glossary approach. Low term specificity indicates that the term may be associated to a range of possible concepts and consequently it may yield low precision in search results as a number of irrelevant documents containing the term will be retrieved. Various measures for term specificity have been extensively investigated in IR [7]. One commonly used measure for term specificity is the inverse document frequency for term t defined as idf(t)=ln(|D|/|Dt|), where |D| is the total number of documents in the collection, and |Dt| represents the number of documents containing t. In the context of this research, glossary terms specificity is computed within the collection of requirements, and represents the assumption that terms occurring in fewer requirements should be considered more specific. Criterion # 3: Glossary items should be domain specific. The third factor investigated in our study was domain specificity of the glossary items. A term is considered domain-specific when it occurs more frequently in domainspecific documents than in general documents. Some terms, such as “software”, are very common in the general technical documents and may deteriorate the retrieval precision when used in the glossary approach. Therefore, glossary items should be domain-specific in order to contribute effectively to the glossary approach in retrieving more true links. One simple measure of domain specificity DS(t) for a term t, is estimated as the term relative frequency within domain-specific documents versus the relative frequency computed in a general technical corpus [8], and can be calculated as follows:    freq(t , D ) freq(t , G )  DS (t ) = ln /   ∑ freq(t , D) ∑ freq(t , G )  t∈G  t∈D 

(2)

where freq(t,D) is the number of occurrences of term t in the domain-specific document collection D, and G represents the general corpus that contains documents from various domains. The overall domain specificity of a given project glossary can be evaluated as the average domain specificity of all items defined in the glossary. In our experiments, the domain-specific document collection D used to compute DS(t) in expression (2) for a specific project glossary consists of the requirements associated to that project glossary. The general corpus G contains all requirement specifications in the available software projects excluding the software project used in D. We are currently expanding the general corpus in order to obtain a more accurate measure of domain specificity. (Notice that in our experiments, when a term is unique to the given project, i.e. freq(t,G)=0, the domain-specificity value for that term is set by default equal to the high value of ln(10000)).

2.2.2 Criteria validation The effectiveness of the criteria proposed in section 2.2.1 that evaluate the existing project glossaries was validated for the following two datasets. The IBS dataset

project may have been caused by glossary entries, such as “road segment” and “vehicle” that were used in a large number of requirements and therefore had relatively low idf values. The average domain-specificity of the glossary items in SE450 was 5.03, about 53% lower than the score of 7.69 for the IBS glossary. The analysis reveals that the SE450 project glossary shows some weakness based on the proposed criteria. Synonyms of glossary terms were used in the traced documents of most fifteen projects, and for all projects both average term specificity and domain specificity scores were significantly lower than the corresponding values in IBS. This indicates that the project glossary approach applied to the SE450 fifteen projects is not expected to produce any significant improvement in the retrieval results precision when compared to the basic approach. To validate the criteria above, the enhanced trace retrieval algorithm using the project glossary information was applied to the IBS and the SE450 datasets. The results of these experiments are summarized in Figure 1 which displays the precision of the trace retrieval results at various recall levels using the basic PN, phrasing and the glossary approach. Figure 1a indicates that for the IBS dataset the glossary approach retrieves a higher percentage of correct links among the top ranked retrieved links, which correspond to low recall levels. In fact the highest increase of 17% in precision occurs at 10% recall level. The results also show that the precision of the three automated retrieval tools did not change significantly for high levels of recall. This means that the project glossary approach manages to push more true links to the top, but there are still true links that are scored relatively low and can be identified only if a larger list of links is retrieved. The application of the project glossary approach produced no significant improvement in the retrieval results for the SE450 project compared to the other two methods. Figure 1b shows that the precision at the top of the candidate links list decreased 16% when project glossary information was used in the phrasing algorithm. An analysis of the results revealed that several links between requirements and documents containing the glossary

1

1

0.8

0.8 Precision

Precision

[12] is a relatively large dataset with a complete and thoroughly validated trace matrix. It describes the requirements and design of a public-works department system for managing roads de-icing. The system activities include prediction of icy road conditions, work order management, truck management, and inventory control. The tracing task in IBS is focused on identifying links between 164 requirements and 71 UML class. The SE450 dataset contains 15 anonymous student term projects for a MS level Software Engineering class at DePaul University. The students used Java to implement a traffic simulation system. The dataset consists of 46 requirements that specify road maps, road segments, vehicles and stoplights, and 15 sets of Java source code which vary greatly in structure. The tracing task for this dataset is from the same set of requirements to the 15 sets of Java source code. Each student created a trace matrix for his/her own project to trace java classes to individual requirements. All the matrices were manually validated by two researchers to ensure the correctness of the matrices. The project glossary of IBS contains 6 keywords and 28 phrases. For SE450, the glossary contains only 10 entries consisting of 4 keywords and 6 phrases. The following analysis discusses in detail the application of the three criteria to the IBS dataset and to one of the fifteen SE450 projects. Criterion 1 evaluates the presence of synonyms of glossary items and was implemented using WordNet [1], a general purpose thesaurus which groups English words into sets of synonym and provides the semantic relationships among different synonym sets. No glossary synonyms were found in the traced UML classes of the IBS dataset. However, in the SE450 chosen project, synonyms of two glossary entries, “vehicle” and “obstacle”, were detected. The student did not follow the provided glossary terms but instead used “car” and “obstruction” throughout his/her code. The term specificity and the domain specificity values for the two glossaries were also evaluated. The average term specificity for the SE450 project was 0.78, less than 50% of the score of 1.63 for IBS. A lower score for the SE450

0.6 0.4 0.2

0.6 0.4 0.2

0

0

0.1

0.2 B as ic P N

0.3

0.4 0.5 0.6 Re call le v e ls P hras ing

0.7

0.8

P hras ing+ Glos s ary

(a) Recall-Precision in IBS

0.9

0.1

0.2 Basic P N

0.3

0.4 0.5 Re ca ll Le ve ls Phrasing

0.6

Phrasing+Glossary

(b) Recall-Precision in one project of SE450

Figure 1. Scatter plots of precision-recall values comparing retrieval results of 3 approaches.

0.7

keyword “vehicle” were incorrectly assigned higher scores and pushed towards the top of the retrieved links list. The keyword “vehicle” is frequently used in the SE450 project documents and has low a specificity value as it is used to describe different concepts. These results suggest that the criteria described above provide a simple way to detect if a project glossary is meaningful and can be used to improve the automated trace retrieval.

2.3

Keyword and phrase extraction method

3 Analysis of the extraction method The proposed keywords and phrases extraction method was analyzed for the SE450 dataset and the CM1 dataset. CM1 is a large dataset extracted from a NASA project for a science instrument and is available at the Promise Data Repository [11]. The dataset consists of 235 high-level, and 220 low-level requirements, with a traceability matrix containing 361 true links between these requirements. The matrix was constructed by NASA and manually verified and modified by Hayes et al. [6]. The extraction of important keywords and phrases in the two datasets follows the procedure described in section 2.3. First candidate keywords and phrases were extracted from the requirements by using QTag. The filtering step (A) based on the terms specificity is defined so that only keywords occurring in no more than 3 requirements in projects with less than 60 requirements or 5% of the total requirements for larger requirements sets are kept in the

100%

100%

80%

80% Precison change

Precison change

This section presents an automated technique for extracting a set of critical keywords and phrases which can be used in lieu of a project glossary to improve the precision of the retrieval algorithm when project glossaries are unavailable or “weak”. Keywords extraction methods have been extensively investigated in IR. Several algorithms have been proposed to automatically extract key terms from document collections in a variety of applications from text mining to web page retrieval. For examples, Matsuo et al [10] proposed a method to extract keywords using statistical information based on terms co-occurrence. This approach, like many other statistical keywords extraction algorithms, does not capture the syntactical information of the terms. Our proposed approach is based on a syntactical method that identifies critical keywords and phrases from the requirements collection. The categories of glossary items considered in this study are single nouns and noun phrases since they are the most common types in a glossary. The extraction of glossary keywords and phrases consists of the following two steps. Step 1: Generate candidate keywords and phrases. A Part-of-Speech (POS) tagger identifies single nouns and phrases. All single nouns and two-noun phrases are considered candidate glossary items. The experiments described in this paper consider only phrases consisting of two nouns. Step 2: Filtering. The candidate glossary items generated from step 1 will inevitably include some

unimportant terms. The following filters are applied to remove those terms: Filter A: Term specificity. Keywords and phrase with a term specificity value (idf(t)) below a certain threshold are filtered out as only high specificity terms are likely to help improve precision in retrieval results. The threshold varies and depends on the characteristics of the given document set. Filter B: Terms domain-specificity. Keywords and phrases with a domain specificity value (DS(t)) below a certain threshold are filtered out, as terms with high domain specificity are more likely to capture project core concepts. Filter C: Nouns filtering. The last noun (head) in the phrase is assumed to convey significant syntactical information about the phrase. As an example, in the phrase “road map”, the word “road” is used to modify the head noun “map”. As “road map” is more specific than single noun “map”, it is necessary to remove “map” from the keywords set. In general if a noun-noun phrase is kept as a key phrase in the extracted set, the head noun of this phrase cannot be a single keyword in the extracted set.

60% 40% 20% 0% -20%

60% 40% 20% 0%

1

2

3

4

5

6

7

8

9

1

10 11 12 13 14 15

2

3

4

5

6

7

8

9

10

11

12

13

14

-20%

Proje ct Recall 10%

Recall 20%

(a) compared to basic PN

P r o je c t Re c a ll 10 %

Re c a ll 2 0 %

(b) compared to the existing glossary usage

Figure 2: Precision change at 10% and 20% recall for algorithm using extracted key terms set for SE450

15

candidate set. In the domain-specificity based filtering step (B), all items that are unique to this project (i.e. freq(t,G)=0) were kept in the candidate glossary set. For the remaining candidate items, the top 50% items ranked in terms of domain-specificity were kept while the bottom 50% items were removed. It is worth noting that the thresholds for both term-specificity and domain-specificity were selected empirically. Methods to determine the optimal thresholds for different projects are still under investigation. The extraction method identified 5 keywords and 30 phrases in the SE450 project documents. Only two terms from the existing project glossary occur also in the extracted set, as they have relatively high term specificity and domain specificity. For CM1, the set contains 223 entries with 59 keywords and 164 phrases.

3.1

Trace retrieval with extracted set

The project glossary approach was then applied to the two datasets using the extracted key terms and phrases in lieu of the missing or weak project glossaries. The graph in Figure 2a displays changes in precision at 10% and 20% recall for the SE450 projects comparing results from the enhanced trace retrieval approach incorporating both phrasing and the extracted set and from the basic PN algorithm. A significant improvement in precision is observed for both recall levels, indicating a higher precision among the top retrieved links. The improvement was especially significant for project 14 for which the precision at 10% recall level increased by 43%. In this case, both the phrasing and the extraction method identified the key phrase “segment length” that is included in several true links. Those true links were consequently assigned higher scores by the enhanced retrieval algorithm and pushed to the top of the retrieved links list. Project 12 experienced a noticeable decrease in precision of about 4% at 10% recall level with respect to the basic algorithm. The decrease was in fact due to the presence of the phrase “road segment” occurring in several false links. Their probabilities were incorrectly 0.5

Precision

0.4 0.3 0.2 0.1 0 0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

Recall Levels basic PN

Phrasing

Glossary

Figure 3. Precision/Recall values of three retrieval algorithms for CM1 dataset

enhanced by phrasing, not by the glossary approach as this phrase has relatively low term specificity and was not included in the extracted key terms and phrases set. Similarly to IBS, the change in precision at high recall levels after the new algorithm was applied was also not noticeable in SE450. Neither phrasing nor the glossary method were able to enhance the probability scores for most of the missed true links as there were very few or even no shared terms and phrases among the query-document pairs. To demonstrate the improvement of using the extracted key terms set over the existing SE450 glossary, Figure 2b displays the change of precision at 10% and 20% recall levels after using the extracted set in the glossary approach compared to using the provided standard glossary for this dataset. As displayed in Figure 2b, the precision changes are mostly positive, indicating that the extracted key terms and phrases were more useful in identifying correct links than an existing weak project glossary. A significant 30% increase in precision was observed in project 4 at 10% recall level. Using the extracted key terms and phrases in the retrieval algorithm also increased the precision of the top ranked retrieved links for CM1. Figure 3 displays the scatter plot of the recall-precision values for three retrieval methods in this dataset. Compared to the basic PN algorithm, the glossary approach utilizing the extracted keywords and phrases increased the precision value from 22% to 31% at 10% recall level, with an additional increase of 4% with respect to the precision for the phrasing method only. In conclusion, these results show that the extracted key terms and phrases can be effectively used to improve precision in the top retrieved links.

4 Conclusions An extension to our previous project glossary retrieval algorithm is proposed to overcome some limitations of that approach. This paper suggests simple techniques to evaluate the quality of information in a project glossary and its usefulness in improving trace retrieval results. Occurrence of synonyms in the traceable documents and frequency of project glossary terms may provide information on the meaningfulness of a project glossary to describe critical concepts. A new procedure to extract critical keywords and phrases from requirements is proposed. Empirical results show that this set of keywords and phrases can be used in the project glossary trace retrieval algorithm when the glossary is unavailable or provides weak information about the project. The keywords and phrases extraction method introduced here is still at an early stage and will be evaluated against more extensive datasets in the future. An interesting question to be addressed in future work is whether extracted key terms and phrases provide additional relevant information about correct links even when a meaningful project glossary is available.

The new retrieval algorithm using either the project glossary information or the set of extracted key terms and phrases often achieves higher precision among the top ranked retrieved links. Although this improvement can increase the analyst’s trust on the tracing tool, some true links are still missed, and are hard to be retrieved using only textual content information. Alternative methods using additional sources of information have been proposed [2], but more work to improve precision and recall is needed.

Reference [1] G. Antoniol, G. Canfora, A. De Lucia, G. Casazza, “Information Retrieval Models for Recovering Traceability Links between Code and Documentation”, Proc. of the Intn’l Conf. on Software Maintenance, San Jose, CA, Oct. 2000, pp. 40-51. [2] J. Cleland-Huang, R. Settimi, C. Duan, X. Zou, “Utilizing supporting evidence to improve dynamic requirements traceability”, Proc. of the 13th IEEE Intn’l Requirements Eng. Conf., Paris, Aug. 2005. pp. 135-144. [3] C. Fellbaum, editor. Wordnet: An Electronic Lexical Database, MIT Press Books, 1998. [4] W. B. Frakes, R. Baeza-Yates, Information Retrieval: Data Structures and Algorithms, Englewood Cliffs, NJ: Prentice Hall, 1992. [5] O. Gotel, A. Finkelstein, “An analysis of the requirements traceability problem”, Technical Report TR93-41, Department of Computing, Imperial College of Science Technology and Medicine, London, UK, 1993. [6] J. Hayes, A. Dekhtyar, S. Sundaram, “Advancing Candidate Link Generation for Requirements Tracing: the Study of Methods”, IEEE Transactions on Software Engineering, 32(1), 2006, pp. 4–19. [7] K. Spärck-Jones, “A Statistical Interpretation of Term Specificity and its Application in Retrieval”. Journal of Documentation, 28 (1), 1972, pp. 11-21. [8] L. Kozakov, Y. Park, T. Fin, Y. Drissi, Y. Doganata, “Glossary extraction and utilization in the information search and delivery system for IBM Technical Support”, IBM SYSTEMS JOURNAL, 43(3), 2004, pp. 546-563. [9] A. Marcus, J. Maletic, “Recovering documentation-tosource-code traceability links using latent semantic indexing”, Proc. of the 25th IEEE Intn’l Conf. on Software Engineering, Portland, Oregon, 2003, pp. 125-135. [10] Y. Matsuo, M. Ishisuka, “Keyword Extraction from a Single Document using Word Co-occurrence Statistical Information”, Intn’l Journal on Artificial Intelligence Tools, 13(1), 2003, pp. 157-169. [11] PROMISE Software Engineering Repository, http://promise.site.uottawa.ca/SERepository [12] S. Robertson, J. Robertson, Mastering the Requirements Process, Reading, MA: Addison-Wesley, 1999.

[13] G. Salton, C. Buckley, “Term-weighting approaches in automatic text retrieval”, Information Processing and Management, 24(5), 1998, pp. 513-523. [14] S.K.M. Wong, Y.Y. Yao. “A Probabilistic Inference Model for Information Retrieval”, Information Systems, 16(3), 1991, pp. 301-321. [15] D. Tufis, O. Mason, “Tagging Romanian Texts: a Case Study for QTag, a Language Independent Probabilistic Tagger”, Proc. of the Intn’l Conf. on Language Resources & Evaluation (LREC), Granada, Spain, May 1998, pp. 589596. [16] X. Zou, R. Settimi, J. Cleland-Huang, "Phrasing in Dynamic Requirements Trace Retrieval", Proc. of the 30th Annual Intn’l Computer Software and Application Conf. (COMPSAC’06), Chicago, IL, Sept. 2006, pp. 265-272.