The Psychological Effects of Web-Based Search Result

Media Psychology, 12:295–319, 2009 Copyright © Taylor & Francis Group, LLC ISSN: 1521-3269 print/1532-785X online DOI: 10.1080/15213260903052232

Enhanced Information Scent, Selective Discounting, or Consummate Breakdown: The Psychological Effects of Web-Based Search Results SRIRAM KALYANARAMAN School of Journalism and Mass Communication, University of North Carolina, Chapel Hill, North Carolina, USA

JAMES D. IVORY Department of Communication, Virginia Polytechnic Institute and State University, Blacksburg, Virginia, USA

We report results from three experiments that broadly examined Web users’ psychological responses to search results featured on a mock search engine. Study 1 examined the interplay between search result relevance and ad relevance and showed that the former is the critical variable in affecting user attitudes toward both the search engine and the ad. Study 2 offered further evidence regarding the overwhelming influence of search relevance, with study results suggesting that neither individual motivations nor ad relevance played a significant role in affecting user attitudes. Finally, Study 3 supported the proposition that the persuasiveness of a relevant text-based ad appearing with relevant search results can be enhanced by adding a visual image to the ad. We point out the implications of the findings and recommend future directions for media effects research in the domain of search engines.

In the complex and constantly evolving world of the Web, few online venues have had as much of an enduring presence as search engines. This resilience can, at least in part, be attributed to a fundamental human need for information seeking and the opportunities afforded by search engines to fulfill such needs. In an environment where users are often saturated with an Address correspondence to Sriram Kalyanaraman, CB #3365, Carroll Hall, School of Journalism and Mass Communication, University of North Carolina, Chapel Hill, NC 275993365. E-mail: [email protected] 295

296

S. Kalyanaraman and J. D. Ivory

abundance of choices, the success (or failure) of a search engine has been largely predicated by the ability to balance technological innovation with user-centered value. The value that users can expect to derive from an online search task is primarily dictated by the strategies and techniques available to reduce feelings of uncertainty that often crop up during an information-seeking experience and affect subsequent decision making (see Wilson, Ford, Ellis, Foster, & Spink, 2002). In online search environments, if sufficient resources are available to reduce user incertitude, the search experience will be perceived as successful. Unsurprisingly, scholars in information retrieval (IR) and information science (IS) have identified uncertainty as a critical component in information-seeking environments and several conceptual frameworks have examined strategies to combat it (Wilson et al., 2002). At the heart of it all, there is widespread agreement that the concept of relevance— the degree to which the searched-for information is perceived as consonant with user expectations—is central to the process of successful electronic search and information retrieval (Mizzaro, 1997; Tombros, Ruthven, & Jose, 2005). While several studies in IR and IS (e.g., Crystal & Greenberg, 2006; Olston & Chi, 2003; Tombros et al., 2005) indicate the prominence of relevance criteria in Web-based search, current scholarship has largely ignored the psychological dimensions of the search experience. As every Web user has experienced, results based on a search query can be irrelevant and discrepant with user expectations and goals. Presumably, exposure to such unexpected search results can induce uncertainty and confusion, and the user may judge the provider of such results (i.e., the search engine) negatively. Additionally, although the primary purpose of search engines is to provide information, they also serve as potent persuasive environments. For instance, consumer analysts have recognized the emergence of search engine advertising as the key impetus for the resurgence of online advertising and the edifice upon which behemoths like Google have built their fortunes (see Graham, 2006). While the implicit assumption here is that users’ eyeballs will be captured by search ads, these assumptions remain largely unverified in Web-based communication research. Further, existing empirical research has yet to shed light on how search results, in conjunction with ads that are featured on the search engine, can influence user perceptions toward both the search engine as well as the ads. We address the above issues and attempt to showcase the importance of search engines as a fertile venue in examining user perceptions and attitudes. We report results from three experiments that broadly examined the effects of search relevance and ad relevance in search engines. In the following sections, we review pertinent literature from varied interdisciplinary frameworks and propose hypotheses for study.

Psychological Effects in Search Engines

297

LITERATURE REVIEW Theoretical Foundations of Information in Search Engines In the context of the current investigation, the most salient conceptual perspective on search comes from our understanding of information foraging theory (Pirolli & Card, 1999; see also Spink & Cole, 2006). Pirolli and Card proposed this framework based on optimal foraging theory in evolutionary ecology (Stephens & Krebs, 1986), which describes organisms’ quest for food and prey and their necessary adaptation to the environment during the course of such ‘‘foraging.’’ The essence of optimal foraging theory rests on the belief that organisms face a variety of challenges as they search for food or prey, and that their decisions are based on a consideration of costs accrued versus perceived rewards. Successful forage will result when organisms maximize value by embarking on a strategy that will allow them to choose the most relevant alternative among several options. The selection of such a choice is premised on opting for the one that is deemed to be most efficient in reducing uncertainty. Pirolli and Card (1999) adopted these principles and suggested that information seeking and behavior in IR environments could be explained by drawing similar analogues. When individuals seek information, they are faced with several alternatives and will choose the one that offers maximum perceived utility and, hence, satisfies their goals and aspirations. In information-rich scenarios where users are especially vulnerable to information overload, they may decrease resultant uncertainty by basing their choice on simple heuristics (‘‘if it is relevant to what I am looking for, it must be good’’). Rather than exhibiting complete rationality and evaluating all available information, humans are typically conditioned to operate within the parameters of ‘‘bounded rationality’’ and make judgments on ‘‘satisficing’’ (see Gigerenzer & Goldstein, 1996) by employing elementary rules of thumb. Central to information foraging theory is the notion of information patches. Just as organisms hunt for prey in certain ‘‘patches’’ or territories, the information-seeking environment is also characterized by ‘‘patchy’’ configuration (Pirolli & Card, 1999). The success of an organism’s search in a patch is dictated by several factors. For example, an animal in search of food may decide that a certain patch is worthwhile if it has enough prey to hunt. However, during the course of the forage, prey resources may become depleted, forcing the animal to make a cost–benefit analysis—whether to continue foraging in the patch or search for a new patch altogether. Similarly, as Pirolli and Card suggest, a Web page that comes up as a result of a search query can be considered an ‘‘information patch.’’ If users searching for information are satisfied by the salience of search results, then they will perceive the patch as ‘‘sticky.’’ Alternatively, users are likely to locate a new

298


patch (by clicking on hyperlinks or refining the search) if search results are discrepant with user expectations. The user’s decision to stay, or move in search of a new patch, is dictated by the discernment of the ‘‘value, cost, or access path of information sources obtained from browsing cues’’ (Olston & Chi, 2003, p. 181) and is labeled as ‘‘information scent.’’ According to Pirolli and Card (1999), bibliographic information, Web hyperlinks, and images can serve as proximal cues or heuristics and aid ‘‘scent.’’ Information scent is crucial in foraging because a user ‘‘preying’’ for information aims to select the one that offers the most value, while taking processing costs into account. Thus, users will navigate toward the most relevant information while ignoring information that is not perceived as relevant to their expectations (see Pirolli & Card, 1999). If the scent is strong enough, the search will be successful because users can voice preference for the most relevant information (see Spink & Cole, 2006). However, in the absence of scent, users are likely to be constrained by irrelevance of search results and the search process is likely to break down. The importance of information relevance and scent is also apparent from an inspection of the sensemaking literature, which examines the use of various technological features to make ‘‘sense’’ of technology (Griffith, 1999). Griffith’s features-based theory of sensemaking (FBST) makes the distinction between ‘‘core’’ and ‘‘tangential’’ features. Core features are the defining elements of a technology, the absence or removal of which will either alter the nature of the technology or negate that technology altogether. Tangential features, on the other hand, may add value to a technology but are optional additions. For instance, the basic ‘‘send’’ feature in e-mail is a core element, whereas the ability to include a voicemail message in an email message is a tangential component. In search contexts, relevance is a core dimension of search engine output, whereas the ad featured on a search engine— irrespective of the degree of relevance to the search results—is a tangential component. If the search results are perceived as irrelevant, the technology (search engine) will be perceived as ineffective since it is perceived to be lacking a core element. Other research that has examined effects of relevance in Web-based communication also suggests that relevant information will be perceived more positively than irrelevant information. For example, Web sites that dispense relevant, targeted messages are likely to ‘‘promote lingering and capture user attention’’ (Little, 2001, p. 53), and, hence, lead to more positive perceptions toward the site (Kalyanaraman & Sundar, 2006). Also, there is considerable evidence from the social–psychological literature on matching that has shown that matching messages to some aspect of a recipient’s self (goals, motivations, cognitions) can increase persuasion (Petty, Barden, & Wheeler, 2002). Based on these various considerations, we propose the following hypotheses for study.


299

STUDY 1 Hypotheses Effects of search results relevance. We predict main effects for search relevance such that the search engine will be perceived more positively when results are relevant to search terms than when the results are irrelevant. Formally, H1: Attitudes toward a search engine will be more positive when the search engine displays relevant search results than when it displays irrelevant search results. We also predict a main effect for search relevance on attitude toward the ad. When users are exposed to relevant search results, information scent will be strong enough such that their resultant attitudes should translate to an overall evaluation of all items contained within the search engine. The discrepancy instantiated by exposure to irrelevant search results will prohibit the development of such positive attitudes. So, if an ad is included in the search engine, the perceptions of the ad will be significantly more positive in the presence of relevant search terms than in the presence of irrelevant search terms. This can be stated as follows: H2: Attitudes toward an ad featured on the search engine will be more positive when the search engine displays relevant search results than when it displays irrelevant search results. Effects of ad relevance. We do not expect main effects for ad relevance on attitude toward the search engine. This is premised on the belief that when users enter a search term in a search engine, their expectations are related to the quality of search-related information that the engine presents and, hence, user attitudes toward the search engine will be contingent on their perceptions of search results. Irrespective of whether the ad is relevant or irrelevant to the search results, it will not be strong enough to afford strong scent for evaluation of the search engine. Also, our earlier expostulation of the distinction between core and tangential elements suggests that even a relevant ad will, at best, be perceived as a tangential feature and will not significantly affect attitudes toward the search engine as would a core feature (search relevance). Consequently, the type of ad should not impact user perceptions of the search engine. Therefore, we propose: H3: Ad relevance will not affect attitudes toward the search engine. Although ad relevance is not decisive in determining attitudes toward the search engine, it may impact user perceptions toward the ad itself. This

300


expectation is fueled by recent experimental evidence on matching relevance of ad content to editorial content. Kalyanaraman, Ivory, and Maschmeyer (2005) showed that ads related to news or editorial content are perceived more positively than ads that are not related to such content. Also, Xia and Sudharshan (2002) have suggested that attributes like online ads may be evaluated favorably provided they are congruent with the primary information featured on the site. Accordingly, we propose: H4: Attitudes toward an ad featured on the search engine will be more positive when the search engine displays a relevant ad than when it displays an irrelevant ad. Interaction between search results relevance and ad relevance. For reasons previously outlined, it is likely that interaction effects will be subsumed by the main effects of search relevance, especially when search results are relevant. Under such circumstances, it is unlikely that ad relevance would exert any significant influence on attitudes. After being exposed to relevant search results, users would largely base their evaluations on the relevance of the search results and not on the relevance of the ad. Because users will be satisfied with the relevance of search results, they would discount the influence of the ad. This expectation would be consistent with the affective primacy hypothesis (Zajonc, 1980), according to which a simple classification of good or bad can be assigned to stimuli; that is, when users are exposed to relevant search results, they will simply classify the search engine as ‘‘good’’ and not likely be influenced by the ad. We should expect stronger support for this proposition if there is no difference on attention paid to the ad, regardless of ad relevance. However, even if users attend to the ad, we should still not expect attentional differences to carry over to attitudinal judgments. Because users will be strongly influenced by search relevance, even if they attend to an irrelevant ad, they are unlikely to examine stimulus-discrepant information closely and, instead, be likely to assimilate such discrepant information (see Geers & Lassiter, 1999). Or, the positive feelings from their satisfaction with the search results may invoke a misattribution principle (see Payne, Cheng, Govorun, & Stewart, 2005), wherein they may be positively disposed toward the (irrelevant) ad as a result of their feelings toward the search engine. Irrespective of whether this occurs as a result of assimilation or misattribution, we would expect to see no differences on attitudes as a function of ad relevance, although attention paid to the ad (with relevant search) would be more for the irrelevant ad than it would be for the relevant ad. When the search results are irrelevant, the predictions are not as straightforward. As we reasoned earlier, in the absence of a core feature and the resulting loss of information scent, it is conceivable that the search evaluation process will break down completely. Therefore, when search results are irrelevant, it will not matter whether the ad is relevant or not as users simply


301

will not attend to the ad at all. In this scenario (and considering our previous explanation of what may happen when search results are relevant), we should not expect a significant interaction between search relevance and ad relevance on user attitudes. However, when information scent is absent, users may indulge in a process of ‘‘random search’’ (Pirolli & Card, 1999) and continue searching for any potentially beneficial information. When they encounter the ad, they may attempt to extract meaning from it. Consistent with object-based theories of visual attention, different stimuli are perceived as different objects, with a certain amount of attention being allocated to each object (Kahneman, Treisman, & Gibbs, 1992). When a new object (the ad) is introduced into the user’s visual domain, a portion of the user’s attentional resources will be diverted from the primary task (examining search results) to create an ‘‘object file’’ for the newly introduced object. In a similar vein, the splitattention effect suggests that attention is distributed or ‘‘split’’ when users have to attend to multiple sources of information (Tindall-Ford, Chandler, & Sweller, 1997). Under these circumstances, the relevant ad may induce more positive attitudes than will the irrelevant ad. Then, we should observe an interaction effect between search relevance and ad relevance. The two competing interaction hypotheses can be framed as follows: H5a: There will be no significant interaction between search relevance and ad relevance on either of the attitude measures. H5b: When search results are relevant, ad relevance will not affect attitudes. When search results are irrelevant, attitudes will be more positive when the search engine displays a relevant ad than when it displays an irrelevant ad. Relationship between search relevance, search engine credibility, and attitude toward the search engine. Several studies have demonstrated that relevance of information content can influence perceptions of credibility. So, when a message is perceived as relevant to some aspect of a recipient’s value system, it is rated as more credible (see Wathen & Burkell, 2002). Message congruence was also seen to influence credibility in Wilson and Sherrell’s (1993) meta-analysis. Similarly, Slater and Rouner (1996) showed that message relevance can influence credibility assessments. In electronic environments, also, relevance of information has been identified as an important determinant of credibility (e.g., Fogg, 2003). In addition, credibility (or perceived credibility) has long been shown to affect attitudes and persuasion (see Metzger, Flanagin, Eyal, Lemus, & McCann, 2003). The impact of credibility on persuasion has also been extensively studied under dual-process theories of persuasion, which broadly suggest that information processing occurs through one of two routes: via effortful, conscious, systematic processing; or via effortless, automatic, heuristic

302


evaluation (see Chaiken & Trope, 1999). Findings from these studies have generally shown that a credible source is more persuasive than a noncredible source, particularly under low-involvement conditions (e.g., Eagly & Chaiken, 1993; Petty & Cacioppo, 1988). In the context of search engines, it is reasonable to expect relevant search results to foster increased perceptions of search engine credibility. In turn, perceived credibility should affect attitudes toward the search engine. Accordingly, we propose our final hypothesis: H6: The relationship between search relevance and attitude toward the search engine (Hypothesis 1) will be mediated by perceived credibility. In our experimental design, we used two different search topics (entertainment and health) to examine whether topic type moderated hypothesized effects.

Method All participants (N D 176) in a completely balanced, 2 (search results relevance: relevant or irrelevant) 2 (ad relevance: relevant or irrelevant) 2 (search topic: entertainment or health) between-subjects factorial experiment were randomly assigned to one of eight experimental conditions. Manipulations were accomplished by assigning participants to enter one of two search terms (‘‘cancer risk’’ or ‘‘greatest films’’ to reflect the topics of entertainment and health) into a dummy Web search engine site, which was manipulated to generate either relevant or irrelevant results (e.g., the relevant ‘‘Detailed Cancer Risk Calculator’’ or the irrelevant ‘‘Tropic of Cancer—Britannica Concise Encyclopedia’’ for the ‘‘cancer risk’’ topic) and ads (e.g., a relevant ad for movies or an irrelevant ad for adhesive tapes and protective films for the ‘‘greatest films’’ topic). After exposure, participants filled out an online questionnaire including the study’s dependent measures.

Participants One hundred seventy-six undergraduate students (135 females, 41 males, mean age D 19.46 years) enrolled in communication courses participated in the experiment for course credit. For all studies reported here, informed consent was obtained according to the university’s Institutional Review Board (IRB) stipulations.

Stimulus Materials A dummy Web search engine was constructed for use as the stimulus material in the experiment. This site, titled ‘‘MetaSearch’’ and presented to participants


303

as a tool still in beta-testing, employed a layout loosely patterned after that of the Google search engine. The site was set up so that participants first viewed a query page into which they could type a search term. For each participant, the ‘‘MetaSearch’’ query page was linked to one of eight predefined results pages manipulated to match the participant’s assigned topic, search results relevance, and ad relevance conditions. Search results were taken from existing search results found on the Web. These were then compiled to reflect experimental manipulations. A pilot study (N D 60) revealed that the intended manipulations were successful at the .01 level. On each results page, 10 sites were presented as ‘‘search results,’’ above which was presented faux information about the number of pages found, links to other apparent site functions (e.g., advanced search, images), a search query box and accompanying ‘‘search’’ button. Each result consisted of a hyperlinked page title above a brief page description and, finally, a full URL for the search result. To the right of the results, each page featured a single ad in a highlighted box with the header ‘‘Sponsored Link’’ and a hyperlinked title followed by a description and URL. Any words from the assigned search term (‘‘cancer risk’’ or ‘‘greatest films’’) appearing in the results was featured in bold type.

Dependent Measures Attitude toward the search engine. Attitude toward the search engine was measured using an index of 16 items asking participants to rate how well a series of adjectives (e.g., ‘‘appealing,’’ ‘‘favorable’’) described the site (1 D describes very poorly, 7 D describes very well). These items were adapted from Kalyanaraman and Sundar (2006), and were found to be highly reliable (Cronbach’s alpha D .94). Attitude toward the ad. Attitude toward the ad was measured using an index made up of 12 semantic differential measures (e.g., ‘‘appealing/ unappealing,’’ ‘‘good/bad’’) anchored on a 7-point scale, which was adapted from previous studies (e.g., Ivory & Kalyanaraman, 2007). This index was highly reliable (Cronbach’s alpha D .91). Search engine credibility. Search engine credibility was assessed using a 5-item index (e.g., Bucy, 2004) asking participants to rate how well a series of adjectives (e.g., ‘‘reliable,’’ ‘‘fair’’) described the search engine (1 D describes very poorly, 7 D describes very well). This index was highly reliable (Cronbach’s alpha D .94). Ad attention. Ad attention was operationalized by asking participants to rate their agreement (1 D strongly disagree, 7 D strongly agree) with the following statement: ‘‘I paid a great deal of attention to the ad on the search engine.’’

304


Control measures. Participants were asked how many hours a day they spent browsing the Web, as well as how many hours in a day they spent using a search engine, and demographic information such as age and gender.

Procedure The experiment was administered to groups of students in a campus computer laboratory. The ‘‘MetaSearch’’ query pages (each linked to different results pages according to condition) were opened in an Internet Explorer window in advance. Upon arrival, participants were told that they would be participating in a study to evaluate the function and features of a search engine currently in beta development. Participants were instructed to take a slip of paper from the experimenter, enter the term printed on it verbatim into the search engine, view the search results as they normally would when using a search engine, and raise their hands to be granted access to the questionnaire. Participants were told that all results links had been disabled for the test, and that they would not need to look beyond the first page of results. The experimenter then passed out slips of paper, which were arranged in advance so that the search terms (either ‘‘cancer risk’’ or ‘‘greatest films’’) matched each participant’s experimental condition. After participants entered search terms, viewed results, and raised their hand to indicate completion, the search window was closed and replaced with the study questionnaire. Upon completing the questionnaire, participants were debriefed, thanked, and dismissed.

Results Attitude toward the search engine. Hypothesis 1 predicted a significant main effect for search relevance, while Hypothesis 3 predicted that ad relevance would not affect perceptions of the search engine. A 2 2 2 factorial analysis of variance (ANOVA) on the dependent measure of attitude toward the search engine revealed a main effect for search relevance, such that participants exhibited a significantly more positive attitude toward the search engine when the search results were relevant (M D 4.52, SE D .11) than when they were irrelevant (M D 3.01, SE D .11), F (1, 168) D 84.03, p < .001, ! 2 D .33. The main effects for ad relevance and search topic as well as interaction effects between any of the three independent variables failed to attain statistical significance. These results are consistent with our expectations. Attitude toward the ad. Hypotheses 2 and 4 hypothesized main effects for both search relevance and ad relevance on attitude toward the ad. A 2 2 2 factorial ANOVA with attitude toward the ad as the dependent


305

measure revealed a main effect for search relevance, with relevant search results eliciting more positive attitudes toward the ad (M D 3.57, SE D .10) than irrelevant search results (M D 3.18, SE D .10), F (1, 168) D 5.87, p < .01, ! 2 D .04. This finding lends support to Hypothesis 2. The main effect for ad relevance failed to attain significance, thereby failing to support Hypothesis 4. Also, none of the interaction effects or main effect for topic attained significance. Recall that we proposed two competing hypotheses to explore the interplay between search relevance and ad relevance. Hypothesis 5a reasoned why we shouldn’t expect to discern significant interaction effects, while Hypothesis 5b specified a significant interaction between search relevance and ad relevance on the attitude measures. The lack of a significant two-way interaction between search relevance and ad relevance on both attitude measures augments our rationale for Hypothesis 5a. Having found such evidence for Hypothesis 5a, we also performed a subsequent factorial ANOVA on the ad attention measure to gain further possible insights into the conceptual explanation underlying the mechanisms predicted by Hypothesis 5a. Although none of the main effects were significant, the interaction effect between search relevance and ad relevance was statistically significant, F (1, 166) D 4.41, p < .05, ! 2 D .06. The data suggest that relevant search results with an irrelevant ad (M D 3.12, SE D .27) generated more attention toward the ad than did relevant results with a relevant ad (M D 2.05, SE D .27). However, when search results were irrelevant, the perception of ad relevance was not as important (M D 2.41, SE D .27 for relevant ad and M D 2.49, SE D .27 for irrelevant ad, respectively). Search engine credibility. The final hypothesis (Hypothesis 6) predicted that perceived credibility would mediate the relationship between search relevance and attitude toward the search engine. A 2 2 2 factorial ANOVA on the dependent measure of search engine credibility revealed a main effect for search relevance, such that participants perceived the search engine as significantly more credible when the search results were relevant (M D 5.11, SE D .14) than when they were irrelevant (M D 3.05, SE D .14), F (1, 168) D 104.73, p < .001, ! 2 D .38. None of the other main effects or the interaction effects attained significance. Given the significant relationship between search relevance and attitude toward the search engine, we performed follow-up analyses (e.g., Baron & Kenny, 1986) to determine whether search engine credibility could mediate the relationship between search relevance and attitude toward the search engine. First, a simple regression confirmed the presence of a statistically significant relationship between search engine credibility and attitude toward the search engine, F (1, 174) D 459.10, p < .001, adjusted R 2 D .72. Next, search engine credibility was included as a covariate to explore its influence on the relationship between search relevance and attitude toward the search engine. The overall analysis of covariance (ANCOVA) was significant, F (2,

306


173) D 233.01, p < .001, adjusted R 2 D .71, but the main effect for search relevance was no longer significant, F (1, 173) D 1.93, p D .18, suggesting that search engine credibility mediated the relationship between search relevance and attitude toward the search engine. This result lends strong support to Hypothesis 6. Follow-up analyses using the control measures did not appreciably alter the results reported here.

Discussion The findings offer conclusive support to the notion that search relevance is so powerful that it can subsume almost all other effects. In addition, perceived credibility of the search engine mediated the relationship between search relevance and attitude toward the search engine. Also, with the exception of Hypothesis 4, the other propositions received strong support. Finally, these findings were consistent across topic type. In terms of the competing hypotheses, we showed support for Hypothesis 5a. A closer examination of the interaction effect on the ad attention measure provides a glimpse into why this may have occurred. The fact that there were no attention differences between the relevant and irrelevant ad in the presence of irrelevant search results suggests that participants did not perceive the presence of any information scent and, hence, their processing appears to have broken down completely. As for the distinction between the relevant ad and irrelevant ad when search results were relevant, we had suggested that if participants paid more attention to the irrelevant ad (compared to the relevant ad), it could possibly indicate that these participants were likely to either assimilate the information from the ad or exhibit a misattribution effect. This explanation appears to be consistent since differences on ad attention did not carry over to attitudinal judgments. Perhaps, participants were satisfied with the relevance of the search results, so everything else was subsumed. The main effects for search relevance, as well as the robust effect size, impart further validity to this viewpoint. Of course, the means on some of the measures are quite revealing. It appears that participants did not really care about the ad, as long as the search results were relevant. But, given the low means, participants may not have been motivated enough to process all information closely. Our instructions to participants asked them to evaluate the search engine based on search results, thereby possibly indicating to them that they should concentrate primarily on the search results and not on any extraneous information, such as ads. This possibility is also supported by qualitative data obtained from several study participants, who mentioned that they barely noticed the ad and that they essentially concentrated on the search results. These observations served as the rationale for Study 2.


307

STUDY 2 Study 2 was conducted to explore whether greater user involvement could affect the way users respond to an ad that appears on a search engine. Based on the large body of empirical findings from the persuasion literature, in which users have been shown to scrutinize messages more closely when they are highly involved (Eagly & Chaiken, 1993; Petty & Cacioppo, 1988), we manipulated participants’ level of involvement in Study 2. Our essential prediction is that if the findings are consistent with Study 1, then we should not expect to find any of the effects to be significant in Study 2 (because we present Study 2 participants with only relevant search results). This would lend further support to the belief that search relevance overrides everything else. On the other hand, if main effects for either involvement or ad relevance (or an involvement ad relevance interaction effect) are observed, it would indicate the conditions under which search relevance is not the only factor dictating information processing in search engines.

Method All participants (N D 276) in a 2 (involvement: high or low) 2 (ad relevance: relevant or irrelevant) 2 (search topic: entertainment or health) betweensubjects factorial experiment were randomly assigned to one of eight experimental conditions. Search topic and ad relevance were both manipulated as they were in Study 1, and involvement was manipulated by varying the instructions given to participants so that they were either encouraged to scrutinize the results carefully or not.

Participants Given that we predicted the possibility of obtaining null findings in Study 2, we performed an a priori power analysis to determine the required sample size to ensure adequate power. Using Cohen’s (1988) effect size estimates, the required sample size to detect moderate effect sizes (f D .25) with power of 0.95 and p < .05 for this between-subjects design was estimated to be 210 (N D 210, D 13.12, critical F (1, 202) D 3.89). This analysis was conducted according to Faul and Erdfelder’s (1992) guidelines. In the actual experiment, 276 undergraduate students (175 females, 101 males, mean age D 19.80 years) enrolled in communication courses participated in the experiment for course credit.

Stimulus Materials The same ‘‘MetaSearch’’ dummy site was employed, using the same variations in search terms (‘‘cancer risk’’ and ‘‘greatest films’’) and ads (relevant

308


and irrelevant) from Study 1. Because search relevance was held constant, all participants viewed the results used in the relevant results conditions from Study 1. Involvement was manipulated through variation in written instructions (see Procedure section below).

Dependent Measures All measures were exactly the same as used in Study 1, except for the addition of measures employed to assess the efficacy of the involvement manipulation. In order to obtain convergent validity, we used two separate measures of perceived involvement, both of which have been validated in previous research employing the involvement construct. Involvement. Participants’ motivation to process search engine information was assessed using Zaichkowsky’s (1985) Modified Personal Involvement Inventory made up of ten 7-point semantic differential items (e.g., ‘‘means a lot to me/means nothing to me’’). The index was reliable (Cronbach’s alpha D .86). Thought listing. Also, a thought-listing technique (see e.g., Brinol & Petty, 2003; Eagly & Chaiken, 1993) was employed by asking participants to list any thoughts that went through their mind as they viewed the search engine results, using open-ended response items. The number of thoughts listed was recorded as the thought-listing measure and served as an additional verification for the involvement manipulation (see Cacioppo & Petty, 1981; Kreuter, Bull, Clark, & Oswald, 1999).

Procedure The experimental procedure was identical to that of Study 1, except for one difference in the instructions printed on the search terms. In the highinvolvement condition, participants were told that features from the search engine might be used in the university’s Web tools and that their responses and feedback could influence the possibility of the university adopting this search engine, so they should go through all the information featured on the search engine carefully. These instructions were excluded in the lowinvolvement condition.

Results Manipulation check. A 2 2 2 factorial ANOVA with topic type, ad relevance, and involvement condition as independent variables and the 10item involvement scale as the dependent measure revealed a main effect for involvement, F (1, 268) D 9.92, p < .01, ! 2 D .07. Specifically, participants in the high-involvement condition scored higher on the involvement scale (M D 4.10, SE D .07) than did participants in the low-involvement condition


309

(M D 3.60, SE D .07). None of the other main effects or interaction effects attained statistical significance. A similar factorial ANOVA with the thought-listing measure as the dependent measure revealed a main effect for involvement, F (1, 268) D 6.96, p < .05, ! 2 D .05. Specifically, participants in the high-involvement condition listed more thoughts (M D 5.40, SE D .16) than did participants in the lowinvolvement condition (M D 4.75, SE D .16).1 None of the other main effects or interaction effects reached significance. These results indicate the success of the involvement manipulation. Dependent measures. Factorial ANOVAs on attitude toward the search engine, attitude toward the ad, and search engine credibility failed to reveal significant main effects or interaction effects (all ps > .20). As in Study 1, the control measures did not have any significant influence on these findings.

Discussion Although the efficacy of the involvement manipulation was established, there were no differences between low- and high-involvement participants on any of the attitudinal measures. It appears that as long as search results are relevant, individuals are likely to ignore other informational components on the search engine. This prompts an obvious question: If search ads are perceived as ineffective (at least as reported by participants in our two studies), should message creators reconsider them as potent advertising tools? Or, should they rethink the format in which these ads are presented? Of course, a parsimonious recommendation is that if search results are relevant, then they may also induce positive attitudes toward ads featured alongside the search results. However, perhaps the ads themselves could exert greater persuasive influence if they were framed differently. Currently, search engine ads are predominantly text based and blend in with the rest of the content. Perhaps, including other elements to text may make these ads stand out and make them more effective. We examine this possibility in our final study and explore whether the addition of a visual element (image) could shed further light on how users process search engine information.

STUDY 3 Some recent research in human–computer interaction has explored the use of thumbnails to enable users to search Web information more effectively (see Woodruff, Rosenholtz, Morrison, Faulring, & Pirolli, 2002). Essentially, thumbnails combine both visual and textual information and have the potential to enhance the quality of search experience, rather than text-based information or visual information in isolation (Woodruff et al., 2002). This belief is consistent with research on modality effects, which has shown

310


that messages delivered via more than one mode or modality are more effectively processed than those messages that are delivered via a single mode, especially when information from multiple modalities is congruent or related to each other in content (Edell & Keller, 1989; Leigh, 1992). One framework that has been used meticulously to examine differences between different modes of processing, especially the differences between textual information and visual information, is Paivio’s (1986) dual coding theory (DCT). DCT assumes that there are two cognitive subsystems that are specialized for encoding, organizing, transforming, storing, and retrieving information. One of these, the image (or nonverbal) system specializes in dealing with information about nonverbal objects and events. The other (the verbal system) specializes in dealing with linguistic information. The idea is that the image system combines visual, auditory, and other sensory components of nonverbal information into integrated wholes. Information that is presented in two modalities is independently represented in each of the encoding modalities, and the item can be recalled from either modality. Therefore, the underlying principle of DCT is that presenting information in both verbal and nonverbal form enhances information effectiveness. In the context of this experiment, if an ad that appears on a search engine is accompanied by a related picture, we may expect it to wield some persuasive influence. It may increase message processing because information is being delivered in more than one mode. Additionally, elements such as icons or images can improve information scent (Pirolli & Card, 1999). Of course, we expect the influence of the picture to be qualified by ad relevance. Accordingly, we propose a broad hypothesis as follows: H7: Picture presence will moderate the effects of ad relevance. When the picture is absent, ad relevance will not affect attitudes, but when a picture is present, a relevant ad will evoke more positive attitudes than will an irrelevant ad.

Method All participants (N D 104) in a completely balanced, 2 (ad relevance: relevant or irrelevant) 2 (ad picture: present or absent) between-subjects factorial experiment were randomly assigned to one of four experimental conditions. Manipulations were accomplished by assigning participants to enter a search term into the same ‘‘MetaSearch’’ site used in Studies 1 and 2. In all conditions, participants entered the same search term and received the same relevant results, but the search results included either a relevant or irrelevant ad with the ad accompanied either by a picture or not. As with Studies 1 and 2, participants filled out an online questionnaire after using the search site.


311

Participants One hundred four undergraduate students (71 females, 33 males, mean age D 19.55 years) enrolled in communication courses participated in the experiment for course credit.

Stimulus Materials The same ‘‘MetaSearch’’ dummy site was used, and the results page featured either a relevant or irrelevant ad, with the ad either accompanied with a picture or not. (See Figure 1 for examples.) As in Study 2, search relevance was held constant across conditions. Also, given the consistency of results across topic type thus far, we restricted our examination of the picture to the topic that was considered more salient to our undergraduate student sample (entertainment).

Dependent Measures All measures were exactly the same as used in Study 1 (Cronbach’s alpha D .93, .91., and .94, for attitude toward search engine, attitude toward ad, and search engine credibility, respectively).

Procedure The experimental procedure was identical to that of Study 1, except that all the slips of paper distributed featured the same search term (‘‘greatest films’’).

Results Attitude toward the search engine. A 2 2 factorial ANOVA on the dependent measure of attitude toward the search engine failed to reveal main effects for either ad relevance or picture presence. However, the interaction effect between ad relevance and picture presence was statistically significant, F (1, 100) D 5.28, p < .05, ! 2 D .04. The data suggest that a picture with a relevant ad (M D 4.70, SE D .17) will generate more positive attitudes toward the search engine than will a picture with an irrelevant ad (M D 4.01, SE D .17). However, when the picture was absent, ad relevance was not a key determinant of attitudes (M D 4.41, SE D .17 for relevant ad and M D 4.49, SE D .17 for irrelevant ad, respectively). Attitude toward the ad. A similar 2 2 factorial ANOVA with attitude toward the ad as the dependent measure revealed a main effect for ad relevance, F (1, 100) D 13.01, p < .001, ! 2 D .09. Specifically, when the ad was relevant, it elicited more positive attitudes (M D 3.73, SE D .13) than when it was irrelevant (M D 3.04, SE D .13). In addition, the interaction effect

312

313 FIGURE 1 Study 3 ‘‘MetaSearch’’ results ads with pictures and ad details.

314


between ad relevance and picture presence was statistically significant, F (1, 100) D 7.83, p < .01, ! 2 D .06. The data suggest that a picture with a relevant ad (M D 4.12, SE D .18) will generate more positive attitudes toward the ad than will a picture with an irrelevant ad (M D 3.17, SE D .18). However, when the picture was absent, ad relevance was not a key determinant of attitude toward the ad (M D 3.2, SE D .18 for relevant ad and M D 2.94, SE D .18 for irrelevant ad, respectively). The interaction effects on both attitudinal measures are in the predicted directions and support Hypothesis 7. The factorial ANOVA on the search engine credibility and ad attention measures did not reveal significant main effects for either of the independent variables. Also, the interaction effects were not significant. Like in the previous two studies, the inclusion of control measures did not significantly change the above findings.

Discussion Study 3 findings confirm our speculation that text-based ads can enhance their effectiveness by adding pictures. As we also showed, the effectiveness of a picture hinges on whether the ad is perceived as relevant to the search results. Based on salient concepts from the information-foraging perspective that have been particularly instrumental in guiding our investigation here, it appears that the judicious placement of a picture for an ad appearing on a search engine can enhance information scent and increase persuasion.

GENERAL DISCUSSION The findings from the three studies reported here offer several insights into how users evaluate search engines (see Figure 2 for an integrated conceptual model illustrating the essential findings from all three studies). Study 1 laid the foundation and showed that search relevance is the most important criterion that users employ to make attitudinal judgments. This finding is reinforced by what we observed in Study 2, where neither the effects of ad relevance nor user involvement had a significant influence. As we specified when drawing on different conceptual frameworks, relevance is a core attribute of search technology and appears to be potent enough to dominate other features in promoting information scent. Indeed, it appears that the entire process of evaluating search results breaks down in the face of exposure to irrelevant search results. The prominence of search relevance can also be gauged by the fact that users seem somewhat willing to even (selectively) discount the relevance of an ad, and do not let it affect their perceptions of the search engine as long as the search results are perceived as relevant to the search query. Of


315

FIGURE 2 An integrated conceptual model of findings from Studies 1, 2, and 3.

course, the IR literature has devoted attention to identifying features that inform relevance (e.g., type of domain, timeliness of featured information). Here, we did not manipulate any of these elements but instead relied on participants’ ‘‘naïve’’ formulations of what constituted relevance. Although future research can disentangle the effects of various components that lead users to form relevance criteria, we believe that our reliance on participants’ ‘‘implicit theories’’—the tendency to be guided by elemental understanding of the world (see Bianco, Higgins, & Klem, 2003)—of relevance are more applicable in how most Web users respond to search engine results. By showing that search engine credibility mediates the relationship between search results relevance and attitude toward the search engine, we identified a process-centered mechanism and provided empirical demonstration for why search relevance is persuasive. It remains to be seen whether features other than relevance can also increase perceptions of search engine credibility. Besides message-based manipulations (as employed in the experiments reported here), the variable-centered approach (Nass & Mason, 1990)—which isolates those attributes of a technology or interface that are endemic to that technology, and explores how different manipulations affect online information processing—can be applied to examine whether certain technological variables influence perceptions of credibility and affect subsequent information processing. We also showed that even in the face of individual motivations, users’ impressions of a search engine are primarily dependent on the relevance of the search results. It should be noted that we did not manipulate motivation in terms of user tasks, but rather relied on a global manipulation of

316


involvement, albeit one that has been widely employed in several studies on attitude change and persuasion. In contrast, task-based user motivations have typically been examined in the IS literature by categorizing users as ‘‘browsers’’—those who have specific goals and employ established patterns of navigational strategies—or ‘‘searchers’’—those who generally do not have any set goals and do not follow any set navigational patterns (see Moe, 2003). It would be a worthy endeavor to explore whether the effects of search relevance and ad relevance are moderated by user classification. In terms of adding a picture to an ad, the findings were in line with previous modality research. The picture with a relevant ad appeared to increase information scent as can be gauged by the two-way interaction on attitude toward the search engine. A suggestion for future research is to examine whether addition of an image to a search result also increases information scent. In addition to pictures, other attributes, such as hyperlinks that have been postulated as aiding information scent (Pirolli & Card, 1999), could be employed as conceptually interesting independent variables in future experimental designs involving search engines. Based on the studies reported here, there are some shortcomings that have promise for future research. In terms of search relevance, our manipulation reflected either complete relevance or complete irrelevance. While this is a useful strategy to unambiguously examine the effects of a variable that is central to the domain of search engines, theoretically, it is important to detect the threshold when negative effects of irrelevance begin to unfold. In addition, we presented only a single ad on our mock search engine. In many cases, existing search engines display multiple ads. Again, threshold effects need to be investigated for this element. Quite possibly, if users are bombarded with several ads, then even if the search results are relevant, the presence of the ads may cause users to lose scent and contribute to a deleterious experience. In conclusion, we issue a call to other new media researchers, particularly those interested in studying technology from a media effects perspective, to join us in systematic forage and further explore the many nuances of the exciting world of search engines.

NOTE 1. To ensure that the thought-listing measure was valid as a measure of involvement in that it was recording thoughts about the search engine and its content rather than unrelated thoughts, we coded all of the 1,397 thoughts listed by the study’s 276 participants. Of these, all but one listed thought pertained to participants’ perceptions of either the search engine or the results it displayed. For further verification, we employed two independent coders who were blind to the study’s experimental conditions, provided them a subset of 600 listed thoughts from the study (43% of the total thoughts), and asked them to list whether each thought pertained to perceptions of the search engine, evaluation of the


317

search results, or any other extraneous matter. The coders’ analysis was consistent with our expectations, and agreement between the two coders was high (r D .90).

REFERENCES Baron, R. M., & Kenny, D. A. (1986). The moderator-mediator variable in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51, 1173–1182. Bianco, A. T., Higgins, E. T., & Klem, A. (2003). How ‘‘fun/importance’’ fit affects performance: Relating implicit theories to instructions. Personality and Social Psychology Bulletin, 29, 1091–1103. Brinol, P., & Petty, R. E. (2003). Overt head movements and persuasion: A selfvalidation analysis. Journal of Personality and Social Psychology, 84, 1123– 1139. Bucy, E. P. (2004). The interactivity paradox: Closer to the news but confused. In E. P. Bucy & J. E. Newhagen (Eds.), Media Access: Social and Psychological Dimensions of New Technology (pp. 47–72). Mahwah, NJ: Erlbaum. Cacioppo, J. T., & Petty, R. E. (1981). Social psychological procedures for cognitive response assessment: The thought listing technique. In T. Merluzzi, C. Glass, & M. Genest (Eds.), Cognitive assessment (pp. 309–342). New York: Guilford. Chaiken, S., & Trope, Y. (1999). Dual-process theories in social psychology. New York: Guilford. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum. Crystal, A., & Greenberg, J. (2006). Relevance criteria identified by health information users during Web searches: Research articles. Journal of the American Society for Information Science and Technology, 57, 1368–1382. Eagly, A. H., & Chaiken, S. (1993). The psychology of attitudes. Fort Worth, TX: Harcourt Brace Jovanovich. Edell, J. A., & Keller, K. L. (1989). The information processing of coordinated media campaigns. Journal of Marketing Research, 26, 149–163. Faul, F., & Erdfelder, E. (1992). GPOWER: A priori, post-hoc, and compromise power analysis for MS-DOS [computer program]. Bonn, Germany: Bonn University, Department of Psychology. Fogg, B. J. (2003). Persuasive technology: Using computers to change what we think and do. San Francisco: Morgan Kaufmann. Geers, A. L., & Lassiter, G. D. (1999). Affective expectations and information gain: Evidence for assimilation and contrast effects in affective experience. Journal of Experimental Social Psychology, 35, 394–413. Gigerenzer, G., & Goldstein, D. G. (1996). Reasoning the fast and frugal way: Models of bounded rationality. Psychological Review, 103, 650–669. Graham, J. (2006, August 14). Google search ads find momentum. USA Today, p. 3B. Griffith, T. L. (1999). Technology features as triggers for sensemaking. Academy of Management Review, 24, 472–488. Ivory, J. D., & Kalyanaraman, S. (2007). The effects of technological advancement and violent content in video games on players feelings of presence, involve-

318


ment, physiological arousal, and aggression. Journal of Communication, 57, 532–555. Kahneman, D., Treisman, A., & Gibbs., B. J. (1992). The reviewing of object files: Object-specific intergration of information. Cognitive Psychology, 24, 175– 219. Kalyanaraman, S., Ivory, J. D., & Maschmeyer, L. (2005, May). Interruptions and online information processing: The role of interruption type, interruption content, and interruption frequency. Paper presented at the 55th annual conference of the International Communication Association (ICA), New York. Kalyanaraman, S., & Sundar, S. S. (2006). The psychological appeal of personalized online content in Web portals: Does customization affect attitudes and behavior? Journal of Communication, 56, 110–132. Kreuter, M. W., Bull, F. C., Clark, E. M., & Oswald, D. L. (1999). Understanding how people process health information: A comparison of tailored and non tailored weight-loss materials. Health Psychology, 18, 487–494. Leigh, J. H. (1992). Modality congruence, multiple-resource theory and intermedia broadcast comparisons. Journal of Advertising, 21, 55–62. Little, J. R. ( 2001). A librarian’s perspective on portals. EDUCAUSE Quarterly, 36(2), 52–54. Metzger, M. J., Flanagin, A. J., Eyal, K., Lemus, D. R., & McCann, R. M. (2003). Credibility for the 21st century: Integrating perspectives on source message, and media credibility in the contemporary media environment. Communication Yearbook, 27, 293–335. Mizzaro, S. (1997). Relevance: The whole history. Journal of the American Society for Information Science, 48, 810–832. Moe, W. W. (2003). Buying, searching, or browsing: Differentiating between online shoppers using in-store navigational clickstream. Journal of Consumer Psychology, 13, 29–39. Nass, C., & Mason, L. (1990). On the study of technology and task: A variable-based approach. In J. Fulk & C. Steinfeld (Eds.), Organizations and communication technology (pp. 46–67). Newbury Park, CA: Sage. Olston, C., & Chi, E. H. (2003). ScentTrails: Intergrating browsing and searching on the Web. ACM Transactions on Computer-Human Interaction, 10, 177–197. Paivio, A. (1986). Mental representations: A dual-coding approach. New York: Oxford University Press. Payne, B. K., Cheng, C. M., Govorun, O., & Stewart, B. D. (2005). An inkblot for attitudes: Affect misattribution as implicit measurement. Journal of Personality and Social Psychology, 89, 277–293. Petty, R. E., Barden, J., & Wheeler, S. C. (2002). The Elaboration Likelihood Model of persuasion: Health promotions that yield substantial behavioral change. In R. J. DiClemente, R. A. Crosby, & M. Kegler (Eds.), Emerging theories in health promotion practice and research: Strategies for improving public health (pp. 71– 99). San Francisco: Jossey-Bass. Petty, R. E., & Cacioppo, J. (1988). The elaboration likelihood model of persuasion. Advances in Experimental Social Psychology, 19, 123–203. Pirolli, P., & Card., S. K. (1999). Information foraging. Psychological Review, 106, 643–675.


319

Slater, M. D., & Rouner, D. (1996). Value-affirmative and value-protective processing of alcohol education messages that include statistical evidence or anecdotes. Communication Research, 23, 210–235. Spink, A., & Cole, C. (2006). Human information behavior: Integrating diverse approaches and information use. Journal of the American Society for Information Science and Technology, 57, 25–35. Stephens, D. W., & Krebs, J. R. (1986). Foraging theory. Princeton, NJ: Princeton University Press. Tindall-Ford, S., Chandler, P., & Sweller, J. (1997). When two sensory modes are better than one. Journal of Experimental Psychology: Applied, 3, 257–287. Tombros, A., Ruthven, I., & Jose, J. M. (2005). How users assess Web pages for information seeking. Journal of the American Society for Information Science and Technology, 56, 327–344. Wathen, C. N., & Burkell, J. (2002). Believe it or not: Factors influencing credibility on the Web. Journal of the American Society for Information Science and Technology, 53, 134–144. Wilson, E. J., & Sherrel, D. L. (1993). Source effects in communication and persuasion research: A meta-analysis of effect size. Journal of the Academy of Marketing Science, 21, 101–112. Wilson, T. D., Ford, N. J., Foster, A. E., Spink, A. (2002). Information seeking and mediated searching: Part 2: Uncertainty and its correlates. Journal of the American Society for Information Science and Technology, 53, 704–715. Woodruff, A., Rosenholtz, R., Morrison, J. B., Faulring, A., & Pirolli, P. (2002). A comparison of the use of text summaries, plain thumbnails, and enhanced thumbnails for Web search tasks. Journal of the American Society for Information Science and Technology, 53, 172–185. Xia, L., & Sudharshan, D. (2002). Effects of interruptions on consumer online decision processes. Journal of Consumer Psychology, 12, 265–280. Zaichkowsky, J. L. (1985). Measuring the involvement construct. Journal of Consumer Research, 12, 341–352. Zajonc, R. B. (1980). Feeling and thinking: Preferences need no inferences. American Psychologist, 35, 151–175.