A Neural Network-Based Knowledge Retrieval ...

PROCEEDINGS OF THE 3RD INTERNATIONAL CONFERENCE ON APPLIED MATHEMATICS, SIMULATION, MODELLING (ASM'09) PROCEEDINGS OF THE 3RD INTERNATIONAL CONFERENCE ON CIRCUITS, SYSTEMS AND SIGNALS (CSS'09)

A Neural Network-Based Knowledge Retrieval System with Relevance Feedback Sassan Sheedvash (1) and Mahmood R. Azimi-Sadjadi (2) (1)

Hewellet Packard, IPG-IT

San Diego, CA 92127 USA [email protected] (2)

Department of Electrical and Computer Engineering Colorado State University Fort Collins CO, 80523 USA [email protected]

with a list of documents ordered by their relative scores. The user browses this results list to identify, implicitly or explicitly, the most relevant documents. If unsatisfied with the results, the user may submit a modified query. This process of query modification and results evaluation continues until the query refinements produce a list of documents that include the required concepts. This subjective query modification does not allow incorporation of expertise or feedback of other users to influence the results list. Moreover, identification of an optimal query carrying all of the required concepts is very difficult or sometimes impossible, even for domain experts.

Abstract This paper presents the results of a new neural network-based system for a large scale knowledge retrieval system. The system optimally map the original users’ queries using relevance feedback from multiple users. The learning can be implmented in either regression or classification modes using a simple three layer linear network. The first layer is an adaptable layer that maps from the query domain to the document domain. The second and third layers perform document-to-term mapping, search and scoring tasks. The proposed learning algorithms are successfully tested on a large text collection encompassing a wide range of HP products, and for a large number of commonlyused single and multi-term queries. The system was successfully deployed in April 2005, and is currently used by over 12,000 call center agents to resolve customer issues in a real time environment. The production data indicates a drastic improvement in resolution rates.

Systems that allow user contribution typically modify the original query using relevance feedback in order to improve retrieval efficiency. The modified query is expected to deliver a more refined list or documents than that delivered by the original query. Relevance feedback is required when a user identifies the most relevant document down in the results list, but does not know how to refine the query to elevate that document to the top of the list. The knowledge and text retrival systems that allow for user contribution may utilize relevance feedback. Relevance feedback is a mechanism through which the original query is modified based on the user-specified list of relevant and non-relevant documents in order to improve retrieval efficiency.

Keywords Knowledge management, text retrieval, relevance feedback, query mapping, connectionist neural networks, learing algorithms.

1. Introduction The main focus of most general purpose knowledge retrieval systems is to apply robust search and content matching effectively and consistently to a very large volume of information. In these systems, the user subjectively modifies the query to refine the search results. The modification process begins

ISBN: 978-960-474-147-2

Previous work [1] indicates that neural networks can be effective tools in adaptive information retrieval. The learning capabilities of neural networks provide an ideal basis for constructing a knowledge retrieval system. Recently, neural networks have been

202

ISSN: 1790-5117


applied to the text retrieval problem. Probabilistic document retrieval systems have been implemented using feed-forward networks in which the search results are ranked in the order of conditional probability that a given document is relevant.

and performance metrics that were analyzed based on field data measured when the system was deployed globally to about 12,000 users world-wide to the HP production environment.

2. Overview of MRTRS and Learning Phases

Current knowledge retrieval systems generally apply relevance feedback learning temporarily during a single session. Some of these systems rely on query modification or query expansion. Although these systems can deliver a refined list of documents, the information is not typically kept for future use nor shared with other users. In other words, these systems are not typically designed to continuously learn from users by modifying system parameters in response to relevance feedback. A thorough survey of the existing methods is provided in [6].

The MRTRS system is based shown in Figure 1 involves three learning phases: (1) initial modelreference learning; (2) model-reference following; and (3) relevance feedback learning from users. These three learning phases are implemented using a simple three-layer linear neural network driven by an ensemble of input-output relations from a reference text retrieval system (phases one and two) or from the user feedback (phase three) incorporating their characteristics, such as frequencies of feedback, expertise level of the users, or any additional rules or heuristic relevant to a particular domain. The reader is referred to [6] for more detail treatment.

The approach in [6] uses a new framework referred to as ``model-reference text retrieval system (MRTRS)'', which is inspired from the well-known model-reference adaptive control theory. In this paper, we first provide an overview of the system structure and learning phases along with the results

Figure 1 – Model Reference Text Retrieval

2.1 Phase One: Learning

initially train the system without the need of queries and their associated results lists. During this initial model-reference learning, the role of the query mapping subsystem is similar to that of a regulator in an adaptive control system in that it generates a

Initial Model-Reference

The goal of the initial model-reference learning phase is to capture the input-output mapping of a reference model for a set of queries and their corresponding listed documents. This can be done either in regression mode using score-based matching, or in classification mode [6] using the SVM framework. The reference model could be either a typical non-adaptive text retrieval system, or a simple document indexing system. In the latter case, only the document vectors dj’s are needed to

ISBN: 978-960-474-147-2

^

modified query analogous to a control signal,

q, −

which yields the desired response or the document ^

list,

203

r , for the original submitted query. Note that −

ISSN: 1790-5117


^

the dimension of the original query space

q is the

qi =

−

−

N

∑ j =1

wij d j = D w −

−

i

^

(2)

same as that of the mapped query q . The process −

^

Now, taking the derivative of J( q I, w i) wrt w

is shown using the broken lines in the upper loop of the block diagram in Figure 1. The desired

−

−

^

setting the result to zero yields D

r , for a submitted query is generated by

response,

−

^

T

−

the reference model. The static retrieval system, which plays a role similar to a plant in an adaptive control system, is a linear mapping system T

described by matrix D where D = [ d −

1

−

^

−

w i = (DT D)-1 r i −

−

(3) ^

d j is the jth document vector of size Mx1

collection,

which yields the desired result

−

as defined before. The document matrix is generated either by the indexing system within the text retrieval system or any other basic indexing process.

th

r i at the output of the −

^

retrieval system. Thus, LS solution for

q lies in the −

space spanned by the documents. This can be viewed as a generalization of the Rocchio’s formula [2] where all the documents are included and their associated weights are obtained using the learning mechanism described in this section.

The goal of this initial model-reference learning is to submit the original i query

−

w i,

In the document matrix for the entire

−

^

Combining with (2) gives the solution for the optimal

d2 … dj −

and

q i = r i.

−

… d N].

i

q i, and find the optimal −

^

mapped query,

q i, that yields the desired response,

2.2 Phase 2: Model Reference Following

−

Once the initial model-reference learning is completed and the system in the feed-forward path of Figure 1 captures the underlying input-output relationship of the reference model, it is crucial that the regulator is able to adapt itself to changes in the model or the content collection. These changes occur as a result of document re-indexing, adding new documents that may contain new terms, and deleting obsolete documents. The key requirement is that these changes must be incorporated into the system without impacting the performance or sacrificing the stability of the previously established learning.

^

r i. Since typically M >> N, this parameter estimation −

problem is under-determined. Thus, the problem can be cast as a minimum-norm least square (LS) [4] ^

where it is desirable to find a mapped query

q i with −

minimum distance from the origin (i.e. small number of terms), subject to constraint D

T

^

^

−

− ^

= [ r i1, −

^

^

−

−

^

q i = r i, where r i −

r i2, … r iN]T is the desired score vector for the

th

i submitted query. Accordingly, we can construct the Lagrangian function ^

^

^

J( q i, w i) = ½ −

−

^

q iT q i + −

−

N

∑

^

This model-reference following can be accomplished efficiently in the regulator of the proposed system using recursive (real-time or batch) learning. As in the initial learning, the upper loop in Figure 1, together with an appropriate adjustment mechanism, is used in this phase. In the proposed three layer network, these changes can be implemented easily via weight adaptation mechanisms, as well as, structural adaptation by node addition and deletion as presented in [6].

^

wij( r ij –

j =1

d jT q i) −

−

(1) where w −

T

i

= [wi1 .… wiN] and wij’s are Lagrangian ^

^

multipliers. Differentiating J( q i, w i) wrt −

−

q I and

2.3 Phase 3: Relevance Feedback Learning

−

setting the result to zero yields

ISBN: 978-960-474-147-2

204

ISSN: 1790-5117


tested based on predefined business rules together with the learning algorithms. The results show that the weights learned in the initial training could be used for document association. Finally, the system was implemented in the production environment and deployed.

Relevance feedback from expert users is implemented when users indicate whether or not the document viewed from the search results list actually resolves the issue. To collect this data, we post a simple, one-click survey at the beginning and at the end of the document view page. Upon completion of the problem-solving session, users ‘vote’ on whether or not the document opened from the results page actually provided an acceptable solution.

The data set used in Phase I for the initial training consisted of 5000 documents with a total of 30,000 distinct tokens. The network was trained in batch mode and the weights of the network are obtained. The ranks of the documents listed by the network were tested to determine the match with those of the underlying search engine. The results show that for the single-term queries 100 percent correct recall is achieved for all desired document ranks and demonstrated high recall in all ranges document scores. For multi-term queries, we observed that though almost perfect recall was achieved in terms of document positions, the network-generated ranks could deviate slightly from the search engine values. This is attributed to truncated rank values from underlying search and retrieval engine.

To meet the expert users' requirements and at the same time preserve the previous learning, the relevance feedback information is incorporated using two possible mechanisms, depending on the nature of the user feedback. This is accomplished by updating the parameters of the regulator or the weights of the first layer. The lower feedback loop (relevance feedback loop) of the system in Figure 1 provides expert users' votes on relevant and nonrelevant documents to the adjustment mechanism, which in turn updates the parameters of the regulator to meet the users' requirements by imposing relevance feedback. The user may provide relevance feedback to the adjustment mechanism either by assigning desired scores to the most relevant document(s) that he/she selects or simply by click-through selection. These relevance feedback types are implemented using either regression or classification-based learning. Clearly, this phase of learning captures certain user-based information and additional expertise that cannot be learnt from the reference model alone.

The knowledge base for the knowledge retrieval system consists of fifteen major collections of about 70,000 documents. The collections provide a wide range of information for support, diagnostics, and specifications for consumers and commercial HP products. The documents contain both unstructured and structured information. The majority of content is represented in text format, and some collections also contain mixed graphical and multimedia formats. The fifteen major collections are based on product types. Each collection contains approximately 5000 documents with about 30,000 distinct tokens while the entire corpus contains about 130,000 distinct tokens.

In Phase II, we applied incremental training to update the weights of the initial network trained in Phase I, and to formulate the new connection weights. The test results were evaluated based on the criterion of mean percent error between the document ranks generated by the underlying search engine and of the document ranks generated by the network. This measure is an indicator of the similarity between the ranks produced by the updated system and the ranks produced by the underlying search engine. The query terms for performance evaluation of the network were carefully chosen such that they represent a wide range of importance in the added document. This enables us to study how the mean percent error of the results varies depending on the importance of the query terms with respect to the document terms. The results show that the mean percent errors in the document ranks are very small, no more than 1.99% and furthermore, they are not dependent on the relation of the query terms to the document terms and their importance. This clearly indicates the usefulness of the proposed incremental training algorithm when new documents are to be added or deleted.

The algorithms performance was tested and evaluated in multiple phases. In Phase I, initial training; the network is tested against the results from the underlying search engine. In this phase, the goal was to capture the input/output behavior of the underlying “black-box” search engine. In Phase II, the weight update methodology was tested when new or modified documents were introduced or deleted. In Phase III, multiple user feedback is

In Phase III, thisl network-based relevance feedback scheme was deployed to HP Americas call center production environment on April 2005. We have monitored and summarized a selection of metrics as shown in Table 1. The number of feedback instances is a count of the number of votes cast by the users during a particular month. The number of first time find instances is a count of the number of votes stating that the solution was found on the first

3. Results

ISBN: 978-960-474-147-2

205

ISSN: 1790-5117


attempt. The number of solution find instances is a count of the number of votes stating that the solution was found, either on the first attempt or after two or more attempts. The number of no solution found instances is a count of votes stating that no solution was found. The first find rate is the percentage of first time find votes out of the total number of votes.

The solution find rate is the percentage of votes for solution find, either on the first attempt or after two or more attempts, out of the total number of votes. The no solution found rate is the percentage of votes indicating no solution was found out of the total number of votes.

Table 1: Relevance Feedback (RF) Production Metrics Jan 05

Feb 05

Mar 05

Apr 05

May 05

Jun 05

Jul 05

Aug 05

Sep 05

Number of feedback instances

1694

1142

1009

11953

16627

11439

8786

9196

7589

Number of first time find instances

1099

747

612

8026

12321

9042

7254

7144

5630

Total solution find instances

1300

843

724

9873

14478

10399

8146

8582

7076

Number of “no solution found” instances

394

299

285

2080

2149

1040

640

614

513

First-time find rate

64.8%

65.4%

60.6%

67.1%

74.1%

79.0%

82.5%

77.6%

74.1%

Solution find rate

76.7%

73.8%

71.7%

82.6%

87.0%

90.9%

92.7%

93.3%

93.2%

No solution found

23.2%

26.1%

28.2%

17.4%

12.9%

9.1%

7.2%

6.6%

6.7%

Figure 2 shows the changes and trend in the various “solution find” rates for January through September 2005. The data for January through March corresponds to real-time production measurements collected three months prior to deploying the relevance feedback system and is compared to the same metrics through the six months after the initial deployment in April. The dramatic increase in the number of votes cast in April and thereafter is partially a fortuitous side effect of posting the numbers of positive and negative votes on the results lists, in addition to using the votes in relevance feedback. The first-time find rate is based on users’ real-time confirmation that they have reached their desired solution document the very first time they have submitted their query, this is the most crucial and challenging indicator in identifying the search effectiveness. With deployment of the new learning system, the first-time find rate indicates a significant improvement of 22.3% and consistent trend in the following six months after deployment. Similarly, the solution-find rate is based on users’ real-time confirmation that they have reached their

ISBN: 978-960-474-147-2

desired solution document within two or more search queries. With deployment of the new learning system, the solution-find rate shows a drastic improvement of 29.9% as compared to results previously before the deployment. Complementary to the first two metrics is the no-solution found metric indicating users’ real-time confirmation that they could not find any desired solution within the results list presented. After deployment of the learning system, the results indicate a significant improvement of 76% with a continuing drop, indicating that many of the solutions that may have been lower in the results and hidden are now elevated for the specific queries with the relevance feedback learning capability of the system. Finally, the dip for the solution-find rate in July is due to and attributed that a collection of documents for a brand new product was introduced for which no users’ feedback data were available for learning, this indicates and enhances the fact that the learning capabilities of users’ behavior and feedback can indeed impact the overall performance of the retrieval system significantly.

206

ISSN: 1790-5117


100% 90% 80% 70% 60% 50% 40% 30% 20% 10%

First find rate

Solution find rate

Se p05

Au g05

Ju l-0 5

Ju n05

M ay -0 5

Ap r-0 5

M ar -0 5

Fe b05

Ja n05

0%

No solution find

Figure 2: Changes in solution find rates, January through September 2005 References

4. Conclusions

[1] D. Harman, Relevance feedback and other query modification techniques, pp. 241–263. Prentice-Hall, Inc., 1992.

It was shown that the MRTRS system presented in this paper captures the documents’ content not only from any underlying search and retrieval engine but also from expert users via relevance feedback. Additionally, requirements for document positions and ranks can be met almost perfectly. The incremental learning devised allows for on-line updating of the network parameters when new documents are added or deleted. These features make this network ideally suited for adaptive document retrieval applications. The system is extended to incorporate relevance feedback from multiple users of different expertise levels. The learning can be implemented, in regression or classification modes, for single and multi-term queries using the first layer of a three-layer linear network. The user feedback is employed to administer the user voting based upon frequency of votes, users’ expertise and other criteria. The second and third layers perform document-to-term mapping and search/retrieval tasks. The effectiveness of the proposed algorithms is demonstrated on a large domain-specific text database containing various HP products. Because of the general framework and flexibility of the Model Reference, Following, and Relevance Feedback Learning algorithms, it is feasible to extend the algorithms to any generic score-based system.

ISBN: 978-960-474-147-2

[2] J.J. Rocchio, Relevance feedback in information retrieval, In: G. Salton(ed), The Smart retrieval system: Experiments in Automatic Document Processing. Englewood Cliffs, 1971. [3]V. Vapnik, The Nature of Statistical Learning Theory. Springer, 1995. [4] L. L. Scharf, Statistical Signal Processing: Detection, Estimation, and time series analysis. Addison-Wesley, 1991. [5] G. H. Golub and C. F. Van Loan, Matrix Computations. The John Hopkins University Press. [6] M. R. Azimi-Sadjadi, J. Salazar, S. Srinivasan, and S. Sheedvash, An Adaptable Connectionist Text Retrieval System with Relevance Feedback,to appear, IEEE Trans. on NN, November 2007.

207

ISSN: 1790-5117