Using evidence in practice - Wiley Online Library

DOI: 10.1111/j.1471-1842.2007.00739.x

Using evidence in practice Blackwell Publishing Ltd

Blogs, wikis and podcasts: the ‘evaluation bypass’ in action? Andrew Booth

Introduction Pick up any recent issue from our professional press, of which the September 2007 Library + Information Update is my current exemplar, and you are assailed by enthusiasm for the ‘Web 2.0 technologies’. Clearly these technologies, recently examined in an excellent overview within this journal,1 have tremendous potential to transform our library and information services from a physical locus to the hub of a virtual social network. I cannot help but be reminded, however, of a formative staging post on my personal road to evidence-based library and information practice (EBLIP)—namely reading an article in 1990 on the fad or phenomenon of that time, the CDROM database.2 Labelling CD-ROM a ‘transient technology’, those two agents provocateurs, Tony McSean and Derek Law, accurately captured the lemming-like tendency for librarians to rush into adoption of this new, but essentially unproven, delivery mechanism. Since that time I have frequently reflected on our personal and professional gravitation towards novel technologies—even recording my thoughts on my Psion Organizer! In the context of EBLIP, I have recently become absorbed in the concept of the ‘evaluation bypass’, first encountered within the context of innovative surgical procedures.3 Similar concerns within health information practice occasioned me to ask: ‘What questions do you and your colleagues ask when contemplating the introduction of some new technology or some innovative service? It is true that, on occasions, lack of time or other logistic constraints will conspire to cause what we might

298

label an “evaluation bypass”. On such occasions ... we may move to uncritical adoption of a new technology or procedure. Hopefully, such instances are the exception rather than the rule.’4 In truth, that final sentence is more vain aspiration than firm conviction. Our adoption of information technologies is largely uncritical—as Muir Gray is reported to have asked—‘if information was a drug would it ever get licensed?’

Scenario At a team Strategic Away Day you discuss whether it is feasible to replace the once-successful monthly paper library newsletter with a library blog. You suspect that there has not been a formal evaluation of the comparative merits of these two alternatives. This gets you wondering what evidence base exists for the clutch of Web 2.0 technologies, namely blogs, wikis and podcasts. Is this an example of the ‘evaluation bypass’ in action?

Searching for evidence A rapid search was conducted of the four most relevant easily accessible databases—lisa, the free lista service from EBSCO, eric and the Web of Knowledge (isi). For inclusion in this cursory literature survey, articles should report either research or evaluation and examine an Intervention that is a constituent Web 2.0 technology. Evaluation could employ either quantitative or qualitative measures and could adopt one or more of the following Perspectives; library management, information professional or service user. Articles reporting effectiveness, as opposed to other outcomes of interest, preferably contain a Comparison (either between competing Web 2.0 technologies, e.g. blog versus wiki or between a Web 2.0 technology

© 2007 The author Journal compilation © Health Libraries Group 2007 Health Information and Libraries Journal, 24, pp.298– 302

Using evidence in practice

and either paper-based or digital precursor). For inclusion, research should be conducted in a library or related (in the widest possible interpretation) setting. Articles evaluating use of Web 2.0 technologies for increasing democratic participation or evaluating Wikipedia were specifically excluded. Keywords included blog*, podcast* and wiki* and the words ‘research’ and ‘evaluat*’ were particularly looked for as markers of evaluative studies.5 A more sensitive approach to retrieval would include ‘Web 2.0’ and the names of other less common constituent technologies, and would itemize a list of specific research designs (e.g. focus group, survey, questionnaire). Nevertheless, it was concluded that the risk of a relevant and useful abstract, not including one of the three most common Web 2.0 technologies, was minimal.

Retrieval results As anticipated, there are several hundred references on the three principal Web 2.0 technologies. These are typically either descriptive articles or editorials. For example, of almost 200 references on the eric database, only one had been tagged as either a research or evaluative article. This is not to deny that descriptive accounts may be useful within the context of an overall decision. For example, Withers reports using a wiki among information desk staff at Miami University,6 while the article most immediately relevant to our scenario, Preparing for the Birth of Our Library blog,7 outlines ten considerations when implementing a library blog as a way of communicating with library patrons (Table 1). Blog* was the most commonly cited of the three technologies, as one might expect for a word that has been in existence since 1996 and which was

voted the Merriam-Webster’s Dictionary Word of the Year for 2004. On the isi Web of Science database, many evaluative works examined technical functionality of certain features and did not examine impact of the technology. These correspond to laboratory studies examining efficacy when our principal interest is how it works in practice—its effectiveness.

Examining the evidence Upon examining the characteristics of identified studies, we can confirm the paucity of the evidence base. An article that, from its title, sounds evaluative—‘The library blog: innovative idea or wasted words’8—merely reports a survey that showed that 75 per cent of students were satisfied with blogs that were posted and wished them to continue. Of course, within EBLIP such a survey constitutes a very low level of evidence. Key when examining such evidence is how the sample was identified and also what the comparator is— asking the famed ‘eight out of 10 cat owners’ whether they would eat Whiskas cat food would probably elicit difference responses depending on whether they had access to their usual diet or whether cat food was the only available source of sustenance! This 75% satisfaction rate is further perpetuated in an earlier article which, it transpires, reports exactly the same survey from the Paul J. Gutman Library, Philadelphia University.9 Also of relevance to our scenario, but of similarly low level evidence, is a report from Georgia State University on a blog, Science News, developed by their faculty librarians.10 This reports that ‘preliminary user statistics’ offer ‘encouragement that patrons are reading it’.

Table 1 Considerations when implementing a library blog 1 Scope/vision—decide on focal point for blog 2 Identify primary and secondary audience—this helps set a tone and clearly speak to its interests 3 Determine high-priority topics—topics most important to communicate to audience 4 Sources for new content 5 Time frame to launch of blog 6 Ideas for advertising and launching—consider schedule of intended audience 7 Evaluation criteria—monitor number of visitors at regular intervals 8 Publication guidelines—establish guidelines for posting regularly 9 Editorial guidelines—be professional and consistent; 10 Measuring success—consider whether time investment has paid off in improved dialogue with patrons and colleagues

© 2007 The author Journal compilation © Health Libraries Group 2007 Health Information and Libraries Journal, 24, pp.298–302

299

300


In a more formal observational study, a set of blogs was monitored during September and October 2003 and descriptive statistics and content analysis were used for evaluation.11 The author found that ‘most of these blogs are excellent information hubs, i.e. provide short summaries (sometimes with a personal touch) with links to further information on major issues related to the topic of the blog’. Of course most library services are not exclusively concerned with delivery and support. Many librarians participate in information literacy training and in this context one of the more recently published evaluations looks potentially useful.12 Examining blogs, in terms of peer learning and peer support, on a module of an accredited library and information science (LIS) degree programme, again using content analysis, the authors found that they are ‘learning tools in their own right and not simply an option for providing information online’. In a similar context, but in one of few studies evaluating multiple technologies, an Open University study13 examined student feedback on each of four technologies (blogging, audio conferencing, instant messaging and Harvard’s Rotisserie system) and, perhaps unsurprisingly given the nature of such case studies, this was overwhelmingly positive. Brief consideration of articles on Web 2.0 technologies leads to realization that a wider evaluation framework is required than that we typically use. In addition to the Content and Technical aspects typically evaluated for web resources, there is an important Social dimension. In a previously published review,14 I have observed that information resources that possess a ‘community of practice’ dimension require such an evaluative framework and, propitiously, this is reflected in the Web 2.0 literature. In line with the above, some evaluations examine the impact of blogs in preventing feelings of isolation and alienation, particularly within a distance learning environment, and such findings from teaching have obvious implications for virtual library services which often provide similar outreach at a distance.15 Podcasting has similarly been examined as a mechanism for reducing isolation and promoting inclusivity among distance learners.16 Student listeners saw podcasts as

especially effective in clarifying and enhancing their understanding, reinforcing what they had learnt, and guiding the direction of their study efforts. Factors that are likely to determine choice and frequency of media use are immediacy of feedback, personal focus, transmission of cues, functionality, usability, and ease of use.17 Although these feature in the context of podcasting, they likely transfer to any Web 2.0 technology under consideration.

Applying the evidence The variable nature and quality of the evidence makes it unsuited to formal appraisal and conclusions from this brief overview can only be tentative and exploratory. Starting from the broadest principle of ‘first do no harm’, then, assuming that in maintaining the blog library staff do not neglect their current service responsibilities, instituting a library blog fulfils first principles.18 Certainly, the blog does carry inherent advantages over its virtual library predecessor, the static web page, a point substantiated by a comparative experiment which found that participants assigned to blogs perceived an organization’s ‘conversational human voice’ to be greater than participants who were assigned to read traditional web pages.19 This led to a corresponding increase in trust, satisfaction and commitment. Given the temptation, not least among librarians, to identify a technological solution and then to look for a problem to solve with it, we do well to step back from our scenario and ask—is investing time in maintaining a newsletter-type blog the best possible use of this time? For example, our library service delivers a funding support service and so maintenance of a funding RSS feed and a swicki (search wiki—tailored search engine) might be a legitimate alternative activity. It is useful to step back and address a prerequisite background question—is this the best use for a blog? What are the alternatives? In this context, an overview of possible applications may prove enlightening.20 Additional food for thought is provided by a comparative article that concludes that, although wikis are currently the least popular conversational technology (from which podcasts are naturally excluded), they hold greatest potential for



facilitating knowledge management needs.21 Consideration must also be given to how the eventual tool, whether it be blog or wiki, is to be structured.22

Conclusion and reflection The above is not necessarily typical of the EBLIP process. I started from the assumption that there is little evaluative material for Web 2.0 technologies and was gratifyingly able to demonstrate this ‘evaluation bypass’ from my retrieved results. In defence of my approach, this process was initiated by a real-life scenario with which we, and many others, wrestle at the moment and I did not pick an obscure and acknowledged transient technology as an exemplar. I should also acknowledge that I identified several potentially useful items for implementation of our Library blog although none establishes that our approach might be effective (or even if it will be more successful than if I had not spent an afternoon reviewing the evidence base and had pushed blindly ahead!) Clearly blogs and wikis currently command significant planning time and effort—stepping for a moment outside the health sector indicative figures from the US school library sector in 2006 suggest that 29 per cent of respondents are currently using or plan to use blogs within the next year, with 18 per cent indicating similar plans for wikis.23 While such volume of projected use makes it likely that uptake will forge ahead regardless of the existence of, or even the findings from, evaluation, it does conclusively demonstrate that Web 2.0 technologies are currently being fasttracked along an evaluation bypass. As Sharples and colleagues point out within the context of surgical interventions, the evaluation window for new technologies is very limited: ‘The ideal moment is between standardization of the new technique and its wide dissemination, but this is often a very brief moment given the understandable enthusiasm for the newest methods’.3 Such observations lead me to return to a plea, voiced previously in this feature, namely of evaluation alongside innovation.24 Only by putting in place

mechanisms for capturing evaluative data as we introduce new technologies will we provide others with the opportunity to build upon what we are currently learning—and, after all, isn’t knowledge transfer the business in which we are all inextricably involved?

References 1 Kamel Boulos, M. N. & Wheeler, S. The emerging Web 2.0 social software: an enabling suite of sociable technologies in health and health-care education. Health Information and Libraries Journal 2007, 24, 2–23. 2 McSean, T. & Law, D. Is CD-ROM a transient technology? Library Association Record 1990, 92, 837–41. 3 Sharples, L. D., Caine, N., Schofield, P. M., Shapiro, L. M., Dunning, J. & Wallwork, J. Randomised trials of new surgical procedures are necessary. Heart 1999, 81, 100–1. 4 Booth, A. Clear and present questions: formulating questions for evidence-based practice. Library Hi Tech 2006, 24, 355–68. 5 Beverley, C. Searching the library and information science literature. In: Booth, A., & Brice, A. (eds). Evidence-Based Practice for Information Professionals. London: Facet, 2004: 89–103. 6 Withers, R. Something wiki this way comes. College and Research Libraries News 2005, 66, 775–7. 7 Blair, J. & Cranston, C. Preparing for the birth of our library blog. Computers in Libraries 2006, 26, 10–3, 54. 8 Bell, S. The library blog: innovative idea or wasted words. Library Issues: Briefings for Faculty and Administrators 2006, 26, 1–4. 9 Bell, S. Where the readers are. Library Journal 2005, 130(Suppl.), 8, 10, 12, 14. 10 Vogel, T. M. & Goans, D. Delivering the news with blogs: the Georgia State University library experience. Internet Reference Services Quarterly 2005, 10, 5–27. 11 Bar-Ilan, J. Information hub blogs. Journal of Information Science 2005, 31, 297–307. 12 Hall, H. & Davison, B. Social software as support in hybrid learning environments: the value of the blog as a tool for reflective learning and peer support. Library and Information Science Research 2007, 29, 163–87. 13 Weller, M., Pegler, C. & Mason, R. Use of innovative technologies on an e-learning course. Internet and Higher Education 2005, 8, 61–71. 14 Booth, A. In pursuit of e-quality: the role of ‘communities of practice’ when evaluating electronic information services. Journal of Electronic Resources in Medical Libraries 2004, 1, 25–42. 15 Dickey, M. D. The impact of web-logs (blogs) on student perceptions of isolation and alienation in a web-based distance-learning environment. Open Learning: the Journal of Open and Distance Learning 2004, 19, 279–91. 16 Lee, M. J. W. & Chan, A. Reducing the effects of isolation and promoting inclusivity for distance learners through

© 2007 The author Journal compilation © Health Libraries Group 2007 Health Information and Libraries Journal, 24, pp.298–302

301

302


17

18

19

20

podcasting. Turkish Online Journal of Distance Education—TOJDE 2007, 8, 85 –105. Shim, J. P., Shropshire, J., Park, S., Harris, H. & Campbell, N. Podcasting for e-learning, communication, and delivery. Industrial Management and Data Systems 2007, 107, 587– 600. Booth, A. Where’s the harm in EBLIP? Current perspectives, future developments. Journal of the European Association for Health Information and Libraries 2006, 2, 34–7. Kelleher, T. & Miller, B. M. Organizational blogs and the human voice: relational strategies and relational outcomes. Journal of Computer-Mediated Communication 2006, 11: 2, 395– 414. Available at: http://jcmc.indiana.edu/vol11/issue2/kelleher.html (accessed 18 September 2007). Zhuo, F. Blogs in American academic libraries: an overview of their present status and possible future use.

21

22

23

24

Web Information Systems—Wise 2006 Workshops, Proceedings Lecture Notes in Computer Science 2006, 4256, 145–52. Wagner, C. Supporting knowledge management in organizations with conversational technologies: discussion forums, weblogs, and wikis. Journal of Database Management 2005, 16, I–VIII. Nicol, D., Littlejohn, A. & Grierson, H. The importance of structuring information and resources within shared workspaces during collaborative design learning. Open Learning: the Journal of Open and Distance Learning 2005, 20, 15–49. Brewer, S. & Milam, P. SLJ’s Technology Survey 2006: New technologies—like Blogs and Wikis—are taking their place in the School Media Center. School Library Journal 2006, 52, 46. Booth, A. Research. Health Information and Libraries Journal 2001, 18, 60–3.
