Expectation as a Mediator of User Satisfaction - Semantic Scholar

Expectation as a Mediator of User Satisfaction Anthony Cox and Maryanne Fisher


Faculty of Computer Science Dalhousie University Halifax, Nova Scotia, Canada




Department of Psychology York University Toronto, Ontario, Canada

Abstract An important issue when evaluating an information retrieval tool is the satisfaction of the tool’s users. While there are many factors that affect satisfaction, we advocate that a user’s expectation, with respect to a query’s response, is a key mediator of their satisfaction. We validate this perspective by measuring individuals’ expectation, judgment of response quality and satisfaction level for a set of four queries. Our experiment indicates that the difference between response quality and expectation significantly correlates with user satisfaction for all four queries. As well, the experiment demonstrates the effectiveness of a within-subjects measure to account for a variety of extraneous factors resulting from the characteristics and preferences of individual tool users.

1 Introduction Information retrieval is a complex process with many elements, such as query formation, query refinement, relevance judgment and solution identification. While each element of the retrieval process can be evaluated independently, such evaluation does not capture the interactions among the elements. Furthermore, there are numerous external user and contextual factors that confound the evaluation of a specific element of the retrieval process. Consequently, accurate evaluation of retrieval tasks must encompass the entire retrieval process without being sensitive to extraneous factors such as the user’s experience level and personal search strategies. Experimental design methodology, as practiced in the social sciences, utilizes statistically sound and scientifically robust techniques for managing the diverse characteristics displayed by experimental subjects. The use of a withinsubjects experimental design controls for extraneous factors by examining the effect of an experiment on each subject individually. Subjects are not compared against each other so that differences introduced by each subject’s individual characteristics are avoided. Each subject, therefore, serves as their own ‘control’, validating comparisons within a single subject as opposed to among subjects. We believe that accurate evaluation of the information retrieval process must use measurements made on a within-subjects basis. Perhaps the single most important issue when examining an information retrieval tool is the satisfaction exhibited by the tool’s users [3]. A user that is highly satisfied is likely to use the tool for future retrieval tasks, to use it more frequently and to adopt it as their primary tool. Consequently, it is necessary to examine the factors that affect a user’s satisfaction level so that composite measures can be developed. When searching for a lost object, casual observation suggests that we are most satisfied when we find the object when we least expect to do so. That is, our satisfaction is mediated by our expectation of success. Information retrieval is no different in that a tool user’s satisfaction is also significantly impacted by their expectations. Using this observation as a basis, it is possible to define a user’s satisfaction as follows:      

! "# %$&'() '+*

Given a value for the quality of the response and for the expectation of receiving an appropriate response, the user’s satisfaction can be defined as the difference of these values. ,

Moving to St. Mary’s University, Halifax, Nova Scotia, Canada as of 1 July 2004.

1

Factor Relevance Clarity Accuracy Completeness Reliability Quality of Presentation Format Timeliness

Description Is the response relevant with respect to needs? Is the desired information identifiable and locatable? Does the response meet predetermined expectations? Is the entire need for information met? Does the response come from a reliable source? Is the presentation acceptable? Is the response in a suitable format? Is the response stale or out of date?

Figure 1: Factors Affecting Response Evaluation A review of the literature supports our hypothesis. In a study on the effects of expectation on perceptions of satisfaction, Cherry et al. [2] found that the difference between actual and expected exam grades has an effect on, but does not significantly influence, satisfaction ratings. This subtractive approach for measuring disconfirmation was previously advocated by Pitt et al. [7]. They suggest that dissatisfaction, as evidenced by the difference between expected and perceived performance, is a factor affecting one’s satisfaction with a task outcome. Subtractive disconfirmation, the use of the difference between actual and expected performance, was first introduced by Swan and Trawick [10] using comparison theory. Similarly Collins et al. [4], when studying software satisfaction, found that expectancy was a predictor of satisfaction, but not a measure of task performance. Disconfirmation has also been examined by McKinney et al. [6] as an element of a model for information quality. However, their model uses a direct measurement of disconfirmation, rather than a subtractive measure, as it is considered more reliable. These previous studies suggest that a subtractive measurement is highly likely to provide a predictor for user satisfaction. The use of a differential measure can be viewed as a within subjects measure since the factors that affect the generation of expectation will similarly affect the evaluation of the response. For example, if a user demonstrates a ‘perfectionist’ personality style, this style will affect both their expectations and their response evaluation. We do not use the term response quality to describe a composite measure for the relevance of the documents identified in the response. It has been proposed that response relevance is a simple binary ‘yes or no’ decision [5]. Although later research (e.g. [9]) relaxes this definition, there are many other factors that affect a user’s perception of response quality. We informally polled colleagues at our respective institutions to identify some of these factors. Figure 1 provides a summary of our findings. While certainly not complete, this figure provides evidence of the many factors that can influence a user’s evaluation of document and response quality. For example, a user may not be satisfied with a relevant response because it is out of date, comes from an unreliable source, is in an unacceptable format, or it contains only part of the information they are seeking. In the next section, we experimentally verify the relationship between expectation, response quality and satisfaction. Then, we examine the results of the experiment and suggest some directions for future research before concluding the paper.

2 Experimental Exploration of Satisfaction To explore the relationship between satisfaction, response quality and expectation, we used a paper-based survey. The hypothesis was that satisfaction is predicted using the difference between a user’s judgments of the actual response quality and the expected response quality for a particular query. Participants were asked to express their expectation for a specific retrieval task and Google query before being shown the ‘response’ Google returned for the query. A sequence of questions then determined the participant’s evaluation of the response and their satisfaction with the outcome.

2.1 Participants There was a total of 36 participants in the final sample; 19 men (age, in years, - /.01* 243 , -65 /."7 , 8:9 /;1*