Rapport technique : 2009/02/N.VIB
Effects of Domain Knowledge on Reference Search With the PubMed Database: An Experimental Study
Nicolas Vibert, Christine Ros, Ludovic Le Bigot, Mélanie Ramond, Jérôme Gatefin, and Jean-François Rouet - Centre de Recherches sur la Cognition et l’Apprentissage (CeRCA) - CNRS UMR 6234 - Poitiers, France
A paraître dans / To appear in : Vibert, N., Ros, C ., Le Bigot, L., Ramond, M., Gatefin, J. & Rouet, J.-F.. Effects of domain knowledge on reference search with the PubMed database : An experimental study. Journal of the American Society for Information Science and Technology.
Correspondence concerning this article should be addressed to: Nicolas Vibert Université de Poitiers –CNRS Centre de Recherches sur la Cognition et l’Apprentissage (CeRCA) – CNRS UMR 6234 MSHS - Bâtiment A5 - 5 rue Théodore Lefebvre - 86000 Poitiers, France
[email protected] 1
ABSTRACT Many researchers in medical and life sciences commonly use the PubMed online search engine (http:// www.pubmed.gov) to access the MEDLINE bibliographic database. The researchers’ strategies were investigated as a function of their knowledge of the content area. Sixteen life science researchers with no experience in neuroscience and 16 neuroscience researchers of matched professional experience performed five bibliographic search tasks about neuroscience topics. Objective mea- sures and concomitant verbal protocols were used to assess behavior and performance. Whatever their knowledge of PubMed, neuroscientists could find adequate references within the allotted time period. Despite their lack of knowledge in neuroscience, life scientists could select adequate references with the same efficiency. However, differences were observed in the way neuroscientists and life scientists proceeded. For instance, life scientists took more time to read the task instructions and opened more abstracts while selecting their answers. These data suggest that regular use of online databases combined with graduate-level expertise in a broad scientific field like biology can compensate for the absence of knowledge in the specific domain in which references are sought. The large inter-individual variability in performance within both groups implies that beyond domain knowledge, individual cognitive abilities are the main determinants of bibliographic search performance.
INTRODUCTION The advent of microcomputers and the introduction of online journals and databases at the beginning of the 1990s led to considerable development of online bibliographic and abstract or full-text databases with search software (Downing, Moore, & Brown, 2005; Rouet, 2006; Tenopir et al., 2003). This caused a massive change in the information-seeking behaviors of scholars and scientists in many fields ofresearch, including biological and health sciences (De Groote & Dorsch, 2003; Tenopir et al.; Vibert, Rouet, Ros, Ramond, & Deshoullières, 2007). Scientists are turning their backs on printed documents, relying instead on online bibliographic resources, which they regard as more practical when it comes to retrieving references from the mass of publications available. The use of electronic journals and articles goes on increasing as search engines, abstract databases and full-text archives of the main journals are more and more integrated within single, easily available online resources (Bar-Ilan & Fink, 2005; Tenopir et al.). The overall goal of this study was to shed a light on the behavior and strategies used by high-level scientists when they search for references inside these new online bibliographic resources. The experiment focused on the impact of expert knowledge in a particular domain of life sciences, namely, neuroscience, on users’ information-seeking behavior. No quantitative experimental study has addressed how scientists at PhD-level and above perform non-mediated online searching (see Spink, Wilson, Ford, Foster, & Ellis, 2002 for mediated searching) or looked at the effect of highlevel domain-specific knowledge on the efficiency of online reference retrieval (for review, see Ondrusek, 2004). A few studies tackled the behavior of clinicians searching the MEDLINE bibliographic database (Hersh & Hickam, 1998; McKibbon et al., 1990). Marchionini and co-workers studied how small populations of computer scientists, business specialists, and professional lawyers searched online reference databases relevant for their fields, but quantitative analysis of the data was limited (Marchionini, Dwiggins, Katz, & Lin, 1993). Bates (1996) studied the search behavior of history of art and humanities scholars, but used mainly a descriptive approach.
CONSEQUENCES OF THE DEVELOPMENT OF ONLINE BIBLIOGRAPHIC RESOURCES ON THE INFORMATION -SEEKING BEHAVIOR OFLIFE SCIENTISTS AND HEALTH PROFESSIONALS According to De Groote and Dorsch (2003) and Tenopir et al. (2003), the PubMed search engine (http://www. pubmed.gov), which provides access to the MEDLINE database operated by the United States National Library of Medicine, has become the most popular online bibliographic resource for life scientists and clinicians. They perceive PubMed as both easy to use and very efficient, and they acces s it directly, i.e. without having librarians or information specialists do the searching for them. Because most scientists and clinicians have learned to use PubMed on their own, 2
the way they use it is very basic and strikingly similar to the way laypeople use Web-based general search engines (Aula & Nordhausen, 2006; for review, see Markey, 2007). Markey states that “for the vast majority of people’s information needs, doing one’s own searching is convenient, immediate and instantaneous—connect to the Internet, launch a Web browser, type a query into a search engine’s dialog box, browse ranked retrievals, and link to one or more full-length retrieved documents” (p. 1079). Vibert et al. (2007) used individual questionnaires and interviews to demonstrate that as other life scientists, French neuroscientists make a massive use of online bibliographic resources. Their preferred online resources for work-related information were PubMed and the Google search engine. Neuroscientists used them with a variety of objectives, but the swiftness of searches was a decisive factor to determine the usefulness of an online resource. The neuroscientists who were interviewed were asked to give examples of online bibliographic searches they had recently done (unpublished data). PubMed was used in 19 out of the 22 examples and three keywords in average were included in each query. The goal of two thirds of the searches was to find “a few references” or “the most recent references” on a topic, and 75% to 80% of the searches led to the selection of “a few” or just one or two relevant references. The participants estimated that only about 10% of the references returned by PubMed were relevant to their goal, which suggests that the bibliographic searches of neuroscientists have a low “precision” (proportion of relevant citations retrieved in each search). These data confirm what was observed in previous studies of health professionals’ behavior. The average number of three keywords in each query is similar to the median value reported by Herskovic, Tanaka, Hersh, and Bernstam (2007) for a large sample of queries issued on PubMed. The low precision of the searches is a typical feature, which was noted as soon as MEDLINE became accessible to end users (Hersh & Hickam, 1998; Hersh et al., 2002; McKibbon et al., 1990; Pao, Grefsheim, Barclay, Woolliscroft, McQuillan, & Shipman, 1993). Despite the low precision, end user satisfaction is reported as high. Several authors state that because of strong time constraints, busy clinicians retrieve only a low number of relevant references when searching on a clinical topic. In summary, the way health professionals and life science researchers use online bibliographic resources has dramatically changed over the last 20 years. When access to online resources was difficult, the most frequent goal of online searches was to obtain a complete, exhaustive bibliography abouta given topic in a single, long session. Nowadays, online bibliographic searches are performed daily or weekly by researchers and clinicians with different, new objectives (Hersh, 1994; Hersh et al., 2002; Markey, 2007; McKibbon et al., 1990). End users’ most frequent goal is to find quickly a few references to answer a precise question, or to monitor regularly new publications in a field or by an author. Apart from the few instances where they tackle a new or unfamiliar field of research (like when preparing a proposal), researchers tend to no longer engage into broad, exhaustive searches.
IMPACT OF THE DEVELOPMENT OF ONLINE BIBLIOGRAPHIC RESOURCES FOR LIBRARY AND INFORMATION SCIENCE The notion of relevance, which is central to information retrieval, must be reconsidered in the light of the ongoing evolution of searchers’ behavior. In most retrieval evaluation studies, relevance was assumed to mean topicality. Expert topical relevance judgments were used to measure the success ofbibliographic searches by computing ratios of relevant and non-relevant documents retrieved and not retrieved (Harter, 1992; Swanson, 1988). The most common ones were “recall” (the ratio of the number of relevant documents retrieved by a given search to the total number of relevant documents in the database) and “precision” (the proportion of retrieved documents that are relevant to the search question). However, several authors questioned whether the efficiency of a search can be assessed only by counting how many references that relate topically to the query are obtained (Harter; Hersh, 1994; Swanson). These authors promoted an alternative view where relevance is defined “not as merely on the topic of the user’s search, but rather as providing information that can be used by the searcher” (Hersh, p. 201). In line with the theory of psychological relevance (Sperber & Wilson, 1986), the relevance of a reference is viewed as a subjective or “situational” notion, which 3
depends on the goal and context of the search and the knowledge of the user (Shaw, 1995; White, 2007; Hsieh-Yee, 1993). When monitoring new publications in their field with PubMed, neuroscientists will get long lists that include old references, which they already know, together with the newer ones. Even if the older references are topically relevant, they will not be relevant for the neuroscientist in that particular context. Although topical relevance must obviously be considered, a fully relevant reference must bring useful information that will modify the user’s knowledge and answer his or her information need (Hersh, 1994; Serola & Vakkari, 2005; White). Besides, a topically irrelevant reference may contain useful information, for instance, from a methodological point of view. In the context of a daily use of online bibliographic resources by scientists and scholars, the relevance of a reference will depend on its topic, but also its recency and authors, the journal of publication, whether the full text ofthe reference is accessible, etc. (Harter, 1992; Shaw, 1995; Vibert et al., 2007). Because neuroscientists use PubMed with many different objectives, the relevance of any reference may vary according to the searcher’s specific purposes, making it an even more subjective concept. Consequently, recall and precision do not really represent any more realistic assessments of the success of a bibliographic search or of user’s satisfaction. Most of the daily bibliographic searches performed by neuroscientists are not meant to be exhaustive. Very often, PubMed users just want to find a few references to answer a question or fulfill a precise, timely information need. Perfectly adequate references may not exist in the database and the recall ratio becomes useless. The main indicators of the success of the search will be whether adequate references are found, but also how well they fulfill the end user’s information need (Aula & Nordhausen, 2006). The information- seeking behavior of scientists may reflect a tradeoff between information-seeking costs and utility of information, as advocated by the information foraging theory (Pirolli & Card, 1999, Spink & Cole, 2006). In this frame, information search/retrieval is a problem-solving task, which can be categorized as well-structured or ill-structured. In a well-structured problem, the goal is clear: There is only one correct answer and all the information needed to solve the problem is accessible (Chi & Glaser, 1985). When one of these conditions is not met, the problem is ill-structured. In most cases, information-seeking tasks are ill-structured because there may be either no or several answers possible and many different routes can be taken to get the right information (Tabatabai & Shore, 2005). In that context, people try to maximize their rate of gaining valuable information, i.e., information that will answer their needs. The fact that each information- seeking action takes time and energy often makes exhaustive information seeking too costly to be justified. Thus, people will stop seeking information when they estimate that the utility of the information that can be gained does not match the cost of further information seeking (Fu & Gray, 2006; Gray & Fu, 2004). The decision to stop is achieved by local decision rules, which limit exploration of the environment and may lead to suboptimal performance. In the context of bibliographic search, this would explain why scientists and health professionals stop searching online resources as soon as they have obtained a few references that satisfy part of their needs. Because of the ill-structuredness of the task, nobody can tell for sure whether the database includes more adequate references, and people stop searching because they are afraid of losing time in seeking, inexistent information.
INTER-INDIVIDUAL FACTORS THAT PREDICT ONLINE INFORMATION -SEEKING PERFORMANCE EXPERIENCE OF ONLINE
INFORMATION SEEKING AND OF THE SEARCH TOOL OR RESOURCE. According to Marchionini (1995), performance of individuals in online information seeking depends on the four major components listed below (see also Downing et al., 2005). The first two components would be the overall experience of online information seeking and the knowledge or experience of the online tool or resource that is used, which may be equated with “procedural” knowledge. Experiments using either the World Wide Web or limited hypertexts or file collections have consistently shown that searching experience modified how individuals perform their searches and enhanced performance (for review, see Aula & Nordhausen, 2006; Jenkins, Corritore, & Wiedenbeck, 2003; Ju, 2007; Marchionini et al., 1993; Tabatabai & Shore, 2005). The same was found for people seeking information through online bibliographic resources, including PubMed or MEDLINE (Hersh et al., 2002; Pao et al., 1993). However, when the experiment involved
4
difficult, domain-specific search tasks (Hölscher & Strube, 2000; Jenkins et al.), people with searching experience got better performance only if they had sufficient knowledge of the field.
DOMAIN
KNOWLEDGE. The third major determinant of performance according to Marchionini would be the
individual’s knowledge of the domain in which information is sought. Better domain knowledge should lead to improved informationseeking performance, but cognitive explanations of why this should happen were rarely made explicit. Studies of expertise demonstrated that experts not only possess a large body of domain knowledge, but also that it is structured in a particularly efficient way (for review, see Brailey, Vasterling, & Franks, 2001; Haerem & Rau, 2007; Wiley, 1998). Experts’ domain knowledge is easily accessible, flexible, and with strong connections between big, meaningful chunks of data that can be retrieved without much effort. An expert’s knowledge usually contributes to better, more conceptual problem representations. In the context of online information seeking, the known connectedness and flexibility of experts’ domain knowledge should help them reformulate and remember the search tasks as lexical encodings of abstract concepts. Indeed, Hsieh-Yee (1993) and Hölscher and Strube (2000) showed that people with domain knowledge used more diverse keywords in their queries than control participants. In addition, domain knowledge should help online searchers to form situation models when reading the titles or abstracts of the references returned by the search engine. This may provide them with a richer basis to evaluate search results quickly in terms of one’s search goals, as suggested by some data (Bhavnani & Bates, 2002; Hölscher & Strube; Jenkins et al., 2003). Because in addition domain experts should better remember the search task and maintain their goals more easily in working memory, more of their cognitive capacities might be left for rejecting irrelevant information as well as monitoring the search process (Aula & Nordhausen, 2006; Bhavnani & Bates; Hsieh-Yee; Hung, Johnson, Kaufman, & Mendonça, 2008). Yet another explanation of domain experts’ greater efficiency in document-related tasks is that they possess more knowledge of the sources of information typical of their discipline (Rouet, Favart, Britt, & Perfetti, 1997). This would help domain experts evaluate the usefulness and trustworthiness of documents regardless of how much they know about the particular problem or topic at hand. In accordance with common expectations, individuals with domain-specific knowledge perform informationseeking tasks differently than novices (for review, see Downing et al., 2005; Vibert et al., 2007). This was demonstrated for Web search tasks (Hembrooke, Granka, Gay, & Liddy, 2005; Hölscher & Strube, 2000; Jenkins et al., 2003), for people searching limited hypertexts (Ju, 2007; Marchionini et al., 1993; Patel, Drury, & Shalin, 1998) and for end users searching for references within online databases (Hsieh-Yee, 1993; Marchionini et al.; Wildemuth, de Bliek, Friedman, & File, 1995) including PubMed or MEDLINE (Hersh & Hickam, 1998; McKibbon et al., 1990). In contrast, domain knowledge would not always increase search effectiveness or performance measured either by the recall and precision ratios, the number and adequacy of relevant results obtained (independently of recall and precision), or the time spent on the task. Although authors like Downing et al. (2005), and Hölscher and Strube state that domain “experts” take less time and retrieve more information than people with less domain knowledge, other studies reported no significant impact of domain knowledge on search performance (for review, see Ju). This discrepancy may result from the various definitions of domain knowledge or the different tasks and experimental environments used, and the next paragraphs examine the relevant literature in more detail in an attempt to explain apparent inconsistencies. Information-seeking tasks are considered ill-structured because the goal of the problem and path to reach it cannot be unambiguously defined or characterized from the initial state (Tabatabai & Shore, 2005). This is true in general but not for very specific tasks such as practice task 1 of this experiment (i.e., “Find three articles signed by Daniel Zytnicki in 1993”). At the time of the experiment indeed, there were only three articles in PubMed fulfilling these requirements, and they could be found right away by typing “Zytnicki D” within the query bar and limiting the year of publication to 1993. The five experimental tasks of this study (Appendix B), despite being rather specific and involving precise topical requirements, are ill-structured in the sense that several solutions exist and that different queries may lead to adequate references. However, they are certainly less ill-structured than more open search tasks such as “What is the state-of5
theart in the controversy of treating depression with medication versus with cognitive-behavioral therapy?” In that example, neither the relevant topics nor the number of references to find are defined. This suggests that informationseeking tasks are better described as ranging on a continuum from low to high ill-structuredness than as just being illstructured, and that the influence of prior domain knowledge on search success may depend on the tasks used in experiments. In their experimental studies using the Web, both Hölscher and Strube (2000) and Jenkins et al. (2003) used quite illstructured search tasks that were rather general and openended, and did not specify how many Web sites or how much information was needed. They found that the domain expertise of students and professional nurses, respectively, led to better search performance but only if the participants had a sufficient amount of Web information-seeking experience. Twenty years ago, Egan (1988) was already pointing out that domain specific knowledge begins to predict performance only after users have acquired some experience with the information retrieving system they use. More experiments were actually performed using limited hypertexts or file collections than the World Wide Web. Most of them found that domain expertise had a significant or near significant effect on either the adequacy of retrieved information or the time taken to find it. This was true for populations of undergraduate or graduate students (Jacobson & Fusani, 1992; Kiestra, Stokmans, & Kamphuis, 1994; Linde, 1989; Patel et al., 1998; Rouet, 2003; Salmeron, Canas, & Fajardo, 2005), graduate students and young researchers/scholars (Mack, Manoff, Miller, & Smith, 2004), high-level computer scientists (Marchionini et al., 1993), but also laypeople (Laberge & Scialfa, 2005). The search tasks were very variable, ranging from simple, fact-finding questions to openended, very ill-structured questions. Domain expertise had significant effects on almost all tasks, even if the effect tended to be stronger for more ill-structured tasks (Mack et al.; Marchionini et al.). Only one study (Wildemuth et al., 1995) did not find any evidence for a relationship between the domain knowledge of graduate students and search efficiency. We found it interesting that this was the only work where efficiency was estimated using the recall and precision ratios. In addition, the search tasks were done within limited toxicology, bacteriology, or pharmacology databases and were rather precisely defined, with a lot of vocabulary and potential keywords included in the instructions. The situation is more contrasted for information seeking within online bibliographic resources. On one hand, some experiments reported clear effects of domain knowledge on the efficiency of reference finding within a law full-text CDROM database (Marchionini et al., 1993), the PsychLit database (Maidenberg, 1991), or the FirstSearch archival tool (Downing et al., 2005). The participants were either professional lawyers compared with professional search intermediaries in Marchionini’s study, or graduate students in the two other cases. The ill-structuredness level of the search tasks was variable, but Downing et al. explicitly stated that “the questions were formed in a way that avoided keywords” (p. 199). Search efficiency was assessed by the duration of searches and by evaluating the adequacy of retrieved references, except for the Maidenberg study where the recall and precision ratios were used and where only recall was positively related with the self-reported domain knowledge of the searchers. On the other hand, several other studies did not find any evidence of effects of domain knowledge on the efficiency ofreference searches performed with the DIALOG database (Saracevic & Kantor, 1988), a local online library catalog (Allen, 1991), or MEDLINE (Hersh & Hickam, 1994; McKibbon et al., 1990; Pao et al., 1993). The participants were undergraduate students (Allen, 1991), professional search intermediaries (Saracevic & Kantor), or in the case of MEDLINE searches either medical students (Pao et al.) or clinicians with information- seeking experience compared with professional search intermediaries (Hersh & Hickam, 1994; McKibbon et al.). In general, the search tasks were rather ill-structured, but potential keywords were included, such as “Is there any evidence in the literature to substantiate an interaction between cyclosporine and erythromycin (i.e., two distinct antibiotics)?” The only common characteristic of these five studies, which may be the reason for the reported absence of effect of domain knowledge, was the exclusive use of recall and precision ratios to evaluate search efficiency. Altogether, the literature review suggests that domain knowledge does increase the efficiency of information seeking on the Web, within Web sites and within bibliographic databases if people have some experience with the search tool or resource they use. Domain knowledge is useless if the participants do not have the procedural knowledge to perform 6
the search. The beneficial effect of domain knowledge seems to hold for all kinds of participants and most search tasks but would be larger for more ill-structured tasks. For this effect to be visible, however, search performance must be assessed either by the number and adequacy of relevant Web sites, Web pages, references that were retrieved, or by the amount oftime spent on the search. When search efficiency was evaluated only by computing recall and precision ratios, almost no effect of domain knowledge was obtained. This confirms the assumption made above that with the advent of easy-touse online resources, recall and precision do not represent any more realistic assessments of the success of a bibliographic search. The positive effect of domain knowledge on informationseeking efficiency must be contrasted with data obtained by authors working on decision making, creative problem solving, resource planning, or information display within complex environments. Experiments performed on these other types of ill-structured problems have questioned the view that people with better domain knowledge always show better performance (Bilalic, McLeod, & Gobet, 2008; Ericsson & Lehmann, 1996; Ju, 2007). In some situations, domain knowledge may even lead to inferior performance (Brailey et al., 2001; Devine & Kozlowski, 1995; Haerem & Rau, 2007; Wiley, 1998). Suboptimal performance of domain experts would result from excessive specialization, reliance on informal knowledge instead of real data, or time pressure and stressful conditions (Hashem, Chi, & Friedman, 2003; Lonka, Joram, & Bryson, 1996; Patel, Kaufman, & Arocha, 2002). No experimental evidence exists for similar adverse effects of domain knowledge on informationseeking tasks, but their occurrence in some situations can not be excluded.
GENERAL
COGNITIVE ABILITIES . The fourth and last major determinant of individuals’ performance in online
information seeking would be what Marchionini called “general cognitive abilities.” Few studies explored in more details what abilities would be involved (Downing et al., 2005). Spatial visualization, vocabulary level, and speed of processing were the main predictors of performance for both the Web and online bibliographic resources (Hersh et al., 2002). Others argued that information-seeking abilities depend on cognitive styles such as perceptual field dependence and independence or holist and serialist biases (Palmquist & Kim, 2000). Retrieval effectiveness was also linked to male gender and an imager (as opposed to verbalizer) cognitive style, but no conclusive results were obtained (Ford, Miller, & Moss, 2001; Ford, Wilson, Foster, Ellis, & Spink 2002).
EXPERIMENTAL DESIGN AND HYPOTHESES The experiment presented here focuses on the impact of knowledge in neuroscience on online bibliographic information seeking via the PubMed resource. Its main purpose was to assess the effects of expert domain knowledge on experienced users’ bibliographic search strategies and performance. A group of 16 neuroscientists at a doctorate level or above were asked to perform five bibliographic search tasks dealing with neuroscience topics. Their strategies and retrieval performance were compared with those of a group of 16 life scientists of matched age, gender, and professional experience working in various life science research fields outside neuroscience. Hence, both groups of participants shared a common undergraduate-level knowledge in biology and a similar understanding of the basic physiological proces ses that characterize living beings. Their research experience also implied that they all had good knowledge of the scientific methods of investigation and online bibliographic search tools. Provided both groups of participants have similar experience of online information seeking and PubMed, any difference in their search strategies or performance must result from the high-level neuroscience knowledge specific to neuroscientists. Knowledge of the PubMed online tool was assessed for each participant to check whether it related to his or her performance and was indeed similar for both groups of researchers. Both neuroscientists and life scientists were expected to have a low knowledge of the PubMed resource because, as stated above, most of them have learned to use PubMed on their own and in a very basic way. Concomitant verbal protocols were used to observe the search strategies of participants, as recommended by Van den Haak, de Jong, and Schellens (2004). Care was taken to ensure maximum ecological validity of the experiment. The five experimental reference-seeking tasks were derived from the real examples of online bibliographic searches obtained by Vibert et al. (2007). Because the goal 7
of two thirds of these examples was just to find “a few references” or “the most recent references” on a topic (see above), the experimental tasks were designed to be rather specific, with precise requirements, and the participants were asked to find only one or two references. Hence, all five experimental tasks were moderately ill-structured compared with more open reference search tasks. In addition, a 15-minute time limit was imposed for each task to take into account the time constraints that impinge on busy scientists. Asking participants to find only one or two references according to well-defined criteria helped eliminate part of the subjectivity that can affect estimation of the relevance and adequacy of selected references for the tasks. The mere fact that numerical ratings were used to evaluate search success in that experiment may appear inconsistent with what was said above, namely, that relevance is a situational notion that depends on the goal and context of the search. However, the adequacy of references retrieved by participants can still be rated quite objectively within the context of an experiment where the end user’s information need is controlled. Many authors have used experimental data to elaborate models of how people conduct online information seeking on the Web or in bibliographic databases (for reviews, see Aula & Nordhausen, 2006; Borgman, 1999; Hung et al., 2008; MacPherson, 2004; Rouet, 2006; Spink et al., 2002). All models include iterative processes in which phases of query formulation or refinement alternate with phases of evaluation of the references returned by the search system. The verbal protocols were used to elaborate within this framework a step-by-step model of how high-level scientists search for references with PubMed, as was done for other search systems by Hölscher and Strube (2000), MacPherson or Rouet (2006). The experiment tested two main hypotheses. According to the literature on the effects of domain expertise on information seeking (see above), domain-specific knowledge should have a beneficial effect on even moderately ill-structured online reference search tasks, provided performance on the tasks is measured by how well the retrieved references fulfill task requirements (i.e., not by computing recall and precision ratios) and all participants have a minimal experience of the search tool they use. Because both conditions were met, the first hypothesis was that neuroscientists, because of their domain-specific knowledge, should get more adequate references than life scientists as answers to the tasks. Neuroscientists were also expected to return adequate references in a shorter period of time compared with non-specialists. Second, the way participants search for bibliographic references was expected to depend on their knowledge of neuroscience, namely, the life scientists were expected not to perform the search tasks in the same way as neuroscientists. For instance, the life scientists were expected to take more time to memorize the instructions for the task and have more difficulties generating keywords for their queries to PubMed.
MATERIALS AND METHOD PARTICIPANTS Two groups of people participated in the study. A first group of 16 participants were high-level researchers working in the field of neuroscience (“neuroscientists”), while a second group of 16 participants were high-level life science researchers working outside the field of neuroscience (“life scientists”). The research of life scientists covered a wide range of topics, including genetics, molecular toxicology or pharmacology, cellular biology, muscle physiology, plant physiology, and biochemistry. All 32 participants were working in state-funded research laboratories located in either Paris (21 participants), Bordeaux (5 participants), or Poitiers (6 participants). Most French neuroscientists and life scientists follow quite similar undergraduate university programs for 4 years, after which they specialize within a specific field like neuroscience, genetics, plant physiology, or others. Depending on the universities, the undergraduate biology programs may include basic neuroscience classes that do not exceed about 50 hours. Some of the life scientists tested followed some of these programs. However, the basic neuroscience notions presented in these classes are negligible compared with the knowledge acquired by even the youngest of the neuroscientists after at least 3 years of full-time, post-graduate training, and research in neuroscience. No relationship 8
between the performance of life scientists in the neuroscience search tasks and their eventual attendance to undergraduate neuroscience classes was seen. All participants filled in a questionnaire that contained several items covering their individual characteristics including age, gender, and level of formal training in neuroscience. They were asked if they had received any formal training to PubMed and had to estimate how often they used this tool, using a 5-points Likert scale ranging from daily use to no use. In addition, they had to indicate the number of years of experience they had using computers for their work, bibliographic and documentary information-seeking tasks, and PubMed. The two groups of participants were matched for the type of positions they held and professional experience. Each group included five full-time researchers (four researchers and one senior “research director”), six lecturer/researchers (four assistant professors and two professors), and five PhD or postdoctoral students (“young researchers”). The two groups were also similar in terms of age and sex ratio (Table 1). The two groups of researchers did not have significantly different experience using computers for their work (t(30) = − .49, ns), computers for bibliographic and documentary search tasks (t(30) = − .84, ns), and PubMed (t(30) = .34, ns). Six of the 16 neuroscientists and 12 of the 16 life scientists stated that they used PubMed on a daily basis, while the 14 other researchers said they did so at least once a week. Non-parametric statistical analysis showed that a higher proportion of life scientists reported using PubMed daily (χ2(1) = 4.57, p
