Proceedings of the 1999 ACM CIKM International Conference on Information and Knowledge Management, Kansas City, Missouri, USA, November 2-6, ACM Press, pages 399-404, 1999
Quality of Service transferred to Information Retrieval: The Adaptive Information Retrieval System Claudia Rolker
Ralf Kramer
Forschungszentrum Informatik (FZI) Haid-und-Neu-Str. 10-14 D-76131 Karlsruhe, Germany +49 721 9654 726
Forschungszentrum Informatik (FZI) Haid-und-Neu-Str. 10-14 D-76131 Karlsruhe, Germany +49 721 9654 702
[email protected]
[email protected]
ABSTRACT Users often quit an information retrieval system (IR system) very frustrated, because they cannot find the information matching their information needs. We have identified the following two main reasons: too high expectations and wrong use of the system. Our approach which addresses both issues is based on the transfer of the concept of Quality of Service to IR systems: The user first negotiates the retrieval success rates with the IR system, so he knows what to expect from the system in advance. Second, by dynamic adaptation to the retrieval context, the IR system tries to improve the user’s queries and thereby tries to exploit the underlying information source as best as possible.
Keywords Quality of Service (QoS), information retrieval, retrieval success rates, adaptation, query reformulation, recall, precision
1. INTRODUCTION 1.1 Motivation Usually, users search iteratively in an information retrieval system (IR system), e.g., a Web search engine, and try to find documents matching their information needs in a trial and error manner. Note that the term documents refers not only to documents, but to any sort of service or product indexed in a catalogue. There are several interactions between the user and the IR system, because the user reformulates his query until he is more or less satisfied with the retrieval result. An IR system is able to support a user during this iterative process. The IR system can try to find out the user’s information need from the reformulated queries and modify each query based LEAVE BLANK THE LAST 3.81 cm (1.5”) OF THE LEFT COLUMN ON THE FIRST PAGE FOR THE COPYRIGHT NOTICE
on this knowledge learnt and on the knowledge about the underlying data source. This support is much more promising, when the user gives relevance feedback per iteration result so that the IR system not only learns from the queries, but also from the subjective evaluation of the query results. If the user negotiates his expectations of the search quality with the IR system in advance, the method of query expansion within the IR system based on relevance feedback is even more guided through these quality agreements. Furthermore, at the end the user is more content with the system, because he knows what to expect from the IR system and because any illusions are wiped out from the very beginning [13]. The basic idea of our approach that supports the behavior described above is to transfer the concept of Quality of Service (QoS) to information retrieval.
1.2 Outline The rest of the paper is organized as follows: In Section 2, the basics are presented which comprise the concept of Quality of Service, the general QoS Model, the differences and similarities of the four main information retrieval techniques, and the definitions of relevance feedback and query expansion. Section 3 describes our approach and especially the transfer of QoS to information retrieval by providing the QoS Model for IR systems. Furthermore, the architecture realizing our approach is presented and the functionality of its components is explained. Among other things the QoS Model for IR systems defines the quality parameters which are negotiated between user and system. In contrast to all other quality parameters, the recall of a query cannot be computed, it has to be estimated. In Section 4, we describe and evaluate different ways to estimate the recall. In Section 5, we present our approach to meet the quality of service agreed upon between user and IR system adaptively. The approach takes advantage of the differences and similarities of information retrieval techniques. Related work is discussed in Section 6. The paper ends with a conclusion and an outlook in Section 7.
2. BASICS 2.1 Quality of Service The concept of Quality of Service (QoS) is well-known in the area of computer communication and multimedia systems. Although this concept is fairly well established, there is neither a common definition (see, e.g., [10] and [11]) nor a general accepted specification of a framework for QoS supporting systems (see, e.g., [2] and [11]). However, all QoS approaches have in common a client server architecture added by a QoS Management party. These three parties interact as follows: Before a service is carried out by the server, the guaranteed Quality of Service is negotiated between the client and the QoS Management. During the execution of the service, the QoS Management monitors the server with respect to meeting the negotiated QoS values. If the service is suddenly disturbed and the negotiated QoS values will not be achieved, the QoS Management tries to adapt the service by reconfiguring it. If there is no way to meet the negotiated QoS values, the QoS Management invokes an action to inform the client. We identified that each system which supports QoS can be described declaratively by a QoS Model which comprises the following components: • service description: All services for which QoS requirements can be specified are described. • list of QoS parameters: This list defines the possible parameters for each service on which clients can specify QoS requirements. • costs: Costs are the counterpart to QoS parameters. A counterpart is needed so that the user does not demand the highest/best values for the QoS parameters. • QoS Management policy: It captures a high level description about the scaling actions to be taken by the QoS Management in the event of violations of the QoS agreed upon. • list of supported service levels: One or more of the following three levels have to be offered. • The Best efforts level is the weakest. There is no assurance that the QoS agreed upon will in fact be provided (no QoS monitoring, no remedial actions). • At the Compulsory level, the achieved QoS will be monitored by the QoS Management and the service will be aborted, if it degrades below the compulsory level. • At the Guaranteed level, the desired QoS must be guaranteed so that the requested level will be met, barring rare events such as equipment failure. A QoS requirement demanded by a client on a service is either a single or a composite one. A single QoS requirement comprises the following components [11]: • name of the QoS parameter, • one or more values of the QoS parameter,
• role that the value plays: e.g., upper or lower limit, upper or lower threshold, • actions to be taken as a result of reaching a limit or threshold (e.g., store the value on disk, send a signal to the client, abort the service), and • level of guarantee for this QoS requirement (best effort, compulsory, or guaranteed). A composite QoS requirement is a composition of several single QoS requirements combined by Boolean operators.
2.2 Information Retrieval Techniques There are four main information retrieval techniques: Boolean retrieval, Probabilistic retrieval, Cluster-based retrieval, and Vector-space retrieval [19]. Each of these retrieval techniques is based • on vectors representing the documents or document clusters, respectively, • on vectors representing queries, and • on a relevance indicator algorithm (similarity algorithm) which computes the relevance of a document/cluster to a query and which puts out a numeric result. The difference between these four retrieval techniques is the meaning of the vector bases which leads to very different relevance indicator algorithms.
2.3 Adaptation Techniques in Information Retrieval Before we present our general approach in Section 3, we define two information retrieval terms in the way we use them in the rest of this paper. In the literature, these terms are often used with different meanings. We define Relevance Feedback as the document-based relevance judgement on a retrieval result. For all documents of the query result the user judges whether it is relevant to him or not. This judgement is immediately fed back to the IR system. We define Query Expansion as the automatic modification of a query based on additional knowledge (thesaurus, relevance feedback, statistics, etc.) with the goal to have a better retrieval result. Of course, there is the risk of query drift, i.e., the danger that the expanded query no longer exactly corresponds to the original one. Rocchio´s Algorithm [16] is an example for a query expansion algorithm.
3. GENERAL APPROACH 3.1 The QoS Model for IR Systems We transfer the concept of QoS to information retrieval by applying the QoS Model, which we introduced in Section 2.1, to IR systems as follows: The monitored service is the user´s search satisfying his information needs. It comprises several iterative queries.
The QoS parameters the users can negotiate with the IR system is the retrieval success rate after a specific number of iterations. Recall and precision define the retrieval success rate of a search. Recall specifies the ability to retrieve all relevant documents, whereas precision specifies the ability to retrieve only relevant documents and no irrelevant documents [15]. These two measures are usually taken to compare IR systems. The precise values of recall and precision can be hidden at the graphical user interface of the IR system by sliders because, e.g., a quantitative preference of recall to precision gives enough QoS hints to the system. The costs the user has to accept is the number of iterations. This indirectly specifies the amount of time the user is willing to spend on the complete search. This QoS parameter plays the role of a counterpart to recall and precision: The more iterations, the better the retrieval result [7]. A counterpart is needed so that the user cannot demand the highest values for recall and precision (tradeoff between quality and time). Our QoS Management policy is based on the assumption that the IR system offers several query expansion algorithms and several retrieval algorithms and that the user gives relevance feedback on each query result. The QoS Management tries to meet the agreed QoS requirements by selecting the right query expansion algorithm and the right retrieval algorithm depending on the given relevance feedback and the agreed QoS. The policy is described in more detail in Section 5.
relevance feedback and controls and influences the IR system so that the QoS requirements will be achieved. If not, it informs the user. The architecture of our QoS supporting system is also shown in Figure 1. The QoS Management can be split into: • QoS Monitoring: The QoS Monitoring monitors the IR system, especially the user´s relevance feedback. • QoS Adaptation: This component has to decide about the query expansion algorithm and the retrieval algorithm the next iteration cycle has to be carried out with. Furthermore, it has to define their configuration. It sets the algorithms and their configuration via the according interface of the IR system. • QoS Degradation: The QoS Degradation is called when the QoS requirements will not be met. It carries out the action which was specified in the QoS requirement (e.g., store the value on disk, send a message to the user, abort the service). • QoS Maintenance: This component is the control component. It receives the QoS requirements from the user and gets the relevance feedback via the QoS Monitoring component. It decides whether the agreed retrieval success rates will be met or not. If yes, it calls the QoS Adaptation, otherwise, it calls the QoS Degradation.
The level of guarantee is compulsory. The best efforts level is too weak, because no assurance with respect to the quality is given. The guaranteed level cannot be met, because there are too many uncertainties: unconscious reorientation of the user regarding the search goal, query drift, uncertainties in the estimation of recall, and wrong indexing of documents. Therefore, only the compulsory level of guarantee is reasonable.
3.2 The Architecture As introduced in Section 2.1, three parties interact in a QoS supporting system. In our adaptive IR system these parties are: • The user (client) formulates his query, sets up his QoS requirements on the retrieval success rates and sends both to the QoS Management. In each iteration he gives relevance feedback on the retrieved results. He receives a failure message if it is found out during the iterative retrieval process that the QoS requirement will not be met. • The IR system (server) offers several query expansion algorithms and several retrieval algorithms. Which expansion algorithm and which retrieval algorithm is used by this IR system can be defined via an interface. Moreover, the configuration parameters for the selected algorithms can be set via this interface, too. Any retrieval takes place on the same document set (having several representations). • The QoS-Management receives the QoS requirements on the retrieval success rates and the user's query. It monitors the
Figure 1: Architecture Figure 1 also illustrates two interlocked loops: • There is the retrieval loop consisting of the user feedback and the retrieval results. This loop can be also described as the iterative search from a user point of view. • The QoS loop takes place internally in our QoS system, as long as the agreed QoS requirements are met. It is interlocked with the retrieval loop.
for the number of relevant documents in the whole document set.
4. RECALL AND ITS ESTIMATON The important functionality of the QoS Maintenance is the computation of the current values for recall and precision, which together form the current retrieval success rate. Only on the basis of these values it can decide whether to call the QoS Adaptation or the QoS Degradation. Whereas the precision of a query result is computable based on the user´s relevance feedback, the computation of recall values is more problematic due to its reference to the whole number of relevant documents in the document set which is unknown (if it were known, all relevant documents would have been retrieved). In the following, we first present several methods for the estimation of recall and then evaluate them.
4.1 Methods to Estimate Recall Five methods estimating the recall value of a query are: 1.
Sample and equal distribution: Before starting the query, the user having his query in mind has to define the relevance of each document of an arbitrary document subset (sample set). The number of relevant documents in this set is used to compute the number of relevant documents in the whole document set under the assumption of equal distribution. Then the query is executed and the user gives relevance feedback on this result. Based on the estimated number of relevant documents in the whole set and on the relevance feedback, the recall value can be computed.
5.
Similar queries and their recall: A set of sample queries with recall and precision values is given. For a new query the most similar queries of the query set are selected. Based on their recall values the recall of the new query is computed.
4.2 Evaluation and Selection The big problem of Method 1 is the sample subset. First, users do not want to give relevance feedback to a sample set of documents. Second, the sample can mislead to a too high or too low number of relevant documents in the whole set. Third, in order to get a relevant document out of a set of documents, this set must usually be fairly large. Method 2 and 3 are based on general retrieval success rates of the IR system. Whereas Method 2 uses the relevance feedback for the estimation of the recall value, Method 3 takes into account the control variable of the query. Method 4 has been evaluated as unreasonable by its developer [6]. This method does not consider the subjectivity of results (relevance). Method 5 is the only one using the content of the query to estimate recall and seems to be the most promising method.
5. ADAPTATION Our approach supporting retrieval success rates as QoS parameters is based on the assumption that the IR system offers several query expansion algorithms and several retrieval algorithms. Only because of certain similarities and differences of these algorithms (see Section 2.2) the sequential execution of different algorithms is reasonable and promising. The QoS Adaptation decides about the order of these algorithms in order to ensure the agreed quality.
2.
Average recall-precision graph: The recall-precision graph for the IR system is created once (e.g., with Salton’s [15] or Robertson’s Algorithm [12]) based on a number of queries with their respective recall and precision. In order to estimate the recall of a new query, precision is first computed based on relevance feedback and then recall can be read from the curve based on the computed precision.
3.
Sweets normal distribution over a control variable: Swet [15] developed an algorithm in order to determine the recall value of a query based upon a normal distribution over a control variable of the retrieval algorithm (e.g., the number of terms a document has in common with a query). For the computation of the expected value and the standard deviation for this normal distribution a number of sample queries with values for recall and for the control variable is needed. In order to estimate the recall value for a new query, the area under the normal distributed limited on the left by the value of the control variable used for the query is computed.
The QoS Adaptation is able to choose the query expansion algorithm and the retrieval algorithm of the next iteration cycle, because of additional information (metadata) about them.
Expectation value for the number of relevant documents: Gövert [6] computes the expected value for the number of relevant documents based on the terms occurring in the query. His estimation is based the frequency of these terms used for indexing within the whole document set. Gövert developed three different algorithms. The recall of a query is the quotient of the number of relevant documents indicated in the relevance feedback and of the computed expected value
• relationships between query expansion algorithms.
4.
Recall and precision tendencies are given in form of metadata about • retrieval algorithms, • query expansion algorithms, • relationships between specific query expansion algorithm and specific retrieval algorithm, • relationships between retrieval algorithms, and
Metadata about the relationships between the query expansion algorithms and the retrieval algorithms is important, because not each retrieval algorithm can be combined with each expansion algorithm. It is also useful to know about the relationships among the query expansion algorithms. This information can be interpreted in two
ways: On the one hand, changing from one query expansion algorithm to a very similar one may mean to be able to transfer the information about the former queries belonging to the same search. On the other hand, the execution of a too similar query expansion algorithm will not lead to new relevant results. The same is true for the relationships between the retrieval algorithms. The QoS Adaptation solves an optimization problem based on the metadata described above and on the information about the current retrieval success rates and the agreed retrieval success rates. The metadata values are set in the way to support the following strategy: If the current retrieval success rates are far from the agreed values or these values have stayed the same during the last iterations with the same algorithms, the retrieval algorithm is changed. If the difference between the retrieval success rates is small, another query expansion algorithm is used. This strategy is reasonable for several reasons. Changing the retrieval algorithm implies a new relevance indicator algorithm or new vector bases for the query vector and the documents or both. Changing the query expansion algorithm means to stay with the current vector bases of the query and the documents, but to have a new query vector.
6. RELATED WORK The concept of Quality of Service is completely new in the area of information retrieval. Especially, the change of the retrieval algorithm and the query expansion algorithm for adaptation reasons is special about our approach. A related concept to the one presented in this paper are user agents [17]. They try to find out the user's vocabulary over a number of searches by relevance feedback. This knowledge is not used by the agents to change the retrieval algorithm, but to expand/modify the query the user has entered. Many query expansion algorithms have been proposed (e.g., [4] and [22]). However, their evaluation in the respective papers strongly indicates that they have been developed with the goal to be applied only once, although they could be used iteratively as well: The result of the expanded query is only compared to the result of the original query. No results of iterative tests are presented. Traders [3] and mediators [1] on information sources are also related concepts to our adaptive information retrieval because they also do a selection for the user. An information source consists of a document set and a retrieval algorithm. All information sources, which traders and mediators work on, usually differ in the document set. Traders and mediators select a group of information sources the query is sent to with respect to the underlying document set or with respect to the needed execution time of the retrieval algorithm. Whereas our QoS Adaptation selects only the retrieval algorithm because all retrieval algorithms are executed on the same document set. Furthermore, the selection is done by the QoS Adaptation with respect to the expected retrieval success
rates. Our method is intended to be iterative, whereas mediators and traders usually are not. Another group of related concepts is characterized by the parallel execution of several retrieval algorithms on the same document set. Well-known examples are Web meta search engines like MetaCrawler [18]. Internally, the results of the search engines are combined to a single search result. The major differences to our approach are that there is no selection of the retrieval algorithm and that there is no iteration.
7. CONCLUSIONS AND OUTLOOK In this paper, we presented our approach to transfer the concept of Quality of Service to information retrieval systems. By negotiating the retrieval success rates in advance, we achieve a balance between user expectations on the one hand and system capabilities, i.e., trade-off between retrieval quality and time (costs), on the other hand. By dynamically selecting the most adequate expansion and retrieval algorithms with respect to the retrieval context, we realize high quality retrieval results and avoid unsuccessful retrievals. The next steps will be to further elaborate the overall approach presented in this paper. Special attention will be paid to the adaptation (optimization process). We plan to use environmental catalogue systems we are involved in (WWW-UDK [9,21], the German Environmental Catalogue System; WebCDS [8,20], the Catalogue of Data Sources of the European Environment Agency; the European Environmental Information Services EEIS [5,14], which take into account specific earth observation data as well) to validate the approach. This incorporates two aspects, namely the technical integration and user involvement.
REFERENCES [1] Arens, Y., Hull, R. and King R. Reference Architecture for the Intelligent Integration of Information. Version 2.0, August 22, 1995, http://mole.dc.isx.com/I3/html/briefs/I3brief.html [2] Aurrecoechea, C., Campbell, A.T. and Hauw, L. A Survey of QoS Architectures. Multimedia Systems, 6(3), 1998, 138-151. [3] Bearman, M. Tutorial on Trading Function. University of Canberra. Revised February 1997, http://www.dstc.edu.au/AU/research_news/odp/trader/ trader.html [4] Buckley, C., Mandra, M., Singhal, A., Improving Automatic Query Expansion, International Conference on Research and Development in Information Retrieval (SIGIR) 98, 206-214, 1998 [5] European Environment Information Services (EEIS) Project Web page, http://eeis.ceo.sai.jrc.it/, June 1999 [6] Gövert, N., Evaluierung eines entscheidungstheoretischen Modells zur Datenbankselektion. Editor Tochtermann, K.: Hypertext
– Information Retrieval – Multimedia, Synergieeffekte elektronischer Informationssysteme, Schriften zur Informationswissenschaft. Universitätsverlag Konstanz, Konstanz, Germany, 1997 [7] Harman, D., Relevance Feedback Revisited. International Conference on Research and Development in Information Retrieval (SIGIR) 92, 1-10 [8] Kazakos, W., Kramer K., Nikolai, R., Rolker, C., Bjarnason, S. and Jensen, S, WebCDS - A Java-based Catalogue System for European Environment Data, in: Dognac, A., Özsu, M.T., Ulusoy O. (Ed.), Current Trends in Data Management, Technology Idea Group Publisher, 1999, 234-249 [9] Kramer, R., Nikolai, R., Koschel, A., Rolker, C., Lockemann, P., Keitel, A., Legat, R. and Zirm, K., WWW-UDK: A Web-based Environmental Metainformation System, ACM SIGMOD 26(1), 1997, 16-21 [10] ITU/ISO. Reference Model of Open Distributed Processing - Part 2: Descriptive Model. Draft International Standard 10746-2, Draft ITU-T Recommendation X.902, 1994
Data Catalogues via CIP. Proceedings of the Earth Observation and Geo-Spatial Web and Internet Workshop (EOGEO) '99, Washington D.C., USA, February 1999. http://webtech.ceos.org/eogeo99/Papers/rolker/rolker_ eogeo99.html [15] VanRijsbergen, C. J. Information Retrieval, London, Butterworths, 1997 [16] Rocchio, J.J., Relevance Feedback in Information Retrieval, in: G. Salton (Ed.), The SMART Retrieval System. Prentice-Hall, Inc., Englewood Cliffs, NJ, 313323. [17] Sanchez, J.A., Azevedo, F.S., and Leggett, J.J. Paragente: Exploring the Issues in Agent-based User Interfaces. In Proceedings of the First International Conference on Multi-Agent Systems (ICMAS-95), 320327, Menlo Park, California, June 1995. AAAI Press. [18] Selberg, E. and O. Etzioni, Multi-Service Search and Comparison Using the MetaCrawler, Fourth World WideWeb Conference, Boston, MA, December 11-14, 1995.
[11] ISO, IEC JTC1/SC21 N9680. Information TechnologyQuality of Service-Framework-Final CD, 1995
[19] Turtle, H.R. Inference Networks for Document Retrieval, Dissertation, Graduate School of the University of Massachusetts, February 1991
[12] Robertson, S.E. and Teather, D.A. A Statistical Analysis of Retrieval Tests: A Bayesian Approach, Journal of Documentation, 30(3), 1974, 273-282.
[20] WebCDS, start page of software http://www.mu.niedersachsen.de/system/cds/, August 1999
[13] Rolker, C. and Faber, J. Zusicherung der Dienstgüte bei der Suche nach Daten, 3. Workshop Föderierte Datenbanken, Magdeburg, Editors Schmitt, I., Türker, C., Hildebrand, E. and Höding, M., Shaker Verlag, 1998, 20-32
[21] WWW-UDK, start page of software, http://www.mu.niedersachsen.de/udkservlets/UDKServ let, August 1999
[14] Rolker, C., Kramer, K. and Kazakos, W. Interoperability among Earth Observation and General Environmental
[22] Yonggang, Qu., Automatic Query Expansion Based on a Similarity Thesaurus, PhD thesis, Swiss Federal Institute of Technology, ETH Zürich, 1995
