Information skills training: a systematic review of ... - Wiley Online Library

Information skills training: a systematic review of the literature* Blackwell Publishing Ltd.

Alison Brettle, Health Care Practice R&D Unit, University of Salford, Salford, UK

Abstract The objectives of this study were to undertake a systematic review to determine the effectiveness of information skills training, to identify effective methods of training and to determine whether information skills training affects patient care. A systematic review, using an iterative approach to searching, was employed. Studies selected for inclusion in the review were critically appraised using a tool used in previous reviews. A tabular approach was used to provide a summary of each paper allowing synthesis of results. One thousand, three hundred and fifty-seven potentially relevant papers were located. On the basis of titles and abstracts, 41 potentially relevant studies were identified for potential inclusion. Further reading and application of the inclusion criteria left 24 studies for critical appraisal and inclusion in the review. Study designs included randomised controlled trials, cohort designs and qualitative studies. The majority of studies took place in US medical schools. Wide variations were found in course content and training methods. Eight studies used objective methods to test skills, two compared training methods and two examined the effects on patient care. There was limited evidence to show that training improves skills, insufficient evidence to determine the most effective methods of training and limited evidence to show that training improves patient care. Further research is needed in a number of areas.

Introduction The ability to search the literature effectively is essential if practising and future clinicians are to implement evidence-based practice. Librarians and information professionals are well placed to teach these skills and spend large amounts of time teaching searching skills to students and qualified clinicians. Training often focuses on searching electronic resources, in particular . Librarians are rarely qualified teachers and there are many different methods of training. However, is Correspondence: Alison Brettle, Research Fellow (Information), Health Care Practice R&D Unit, University of Salford, Statham Building, Salford M5 4PW, UK. E-mail: [email protected] *A full report of this review is being published as part of the HCPRDU reports series. Further details are available from the HCPRDU website: http://www.fhsc.salford.ac.uk/hcprdu

there any evidence to show that training in literature searching skills is effective? What aspects of literature searching should be taught and what are the best methods of doing this? Finally, do information skills make a difference to patient care? The author began to consider these questions after having provided a large number of training sessions, using a variety of methods in a range of settings to both student and practising clinicians. Whilst subjective evaluations of training sessions showed that users valued and benefited from training, informal comments showed that they continued to experience problems undertaking literature searches. With the evidence-based librarianship movement beginning to gain momentum, as signalled by a conference in Sheffield in September 2001, it seemed timely to undertake a systematic review to answer these questions.

© Health Libraries Group 2003 Health Information and Libraries Journal, 20 (Suppl. 1), pp.3–9

3

4

Information skills training, Alison Brettle

The aims of the review were to: • examine whether training improves searching skills; • identify effective methods of training; • examine whether training effects patient care. Methods An iterative approach was adopted to ensure that all relevant papers examining information skills training and its effects were identified. Three databases were selected for searching to represent different areas of interest:  (education),  (information) and  (clinical). Following the search spiral approach1 (an iterative approach to searching and refining based on search results), the investigator undertook a scoping search using subject headings and free text searching on these databases covering the period 1995 to September 2002. This time frame was selected as this was the period when evidence-based practice was developing within the UK and end-user searching of electronic databases was becoming more widespread. The scoping search identified a number of relevant studies available on this topic area, highlighted further relevant search terms and helped to clarify the inclusion and exclusion criteria (Appendix 1). The search strategies were refined and a more comprehensive search was developed and rerun for each database (Appendix 2). This included a wide range of terms (subject headings and free text) to describe training, education and literature searching. A pragmatic approach to the review meant that no restrictions on types of study (qualitative and quantitative) were made during the search. However, only papers in English were included. A total of 1345 potentially relevant papers were located from these searches (545 from , 288 from  and 512 from ). Titles and abstracts of these papers were scanned and full papers were obtained for those meeting the inclusion criteria (n = 29). To supplement the database searches, hand searches of two key journals were undertaken covering the period of the review: the Journal of the Medical Library Association (formerly Bulletin of the Medical Library Association) and Health Information and Libraries Journal (formerly Health Libraries Review). These journals were selected as they offered US and UK

perspectives, respectively, of the health information and library field. Hand searches identified a further 12 potentially relevant studies. Their citations were checked for further studies, but none were identified. Sixteen studies were excluded from the review after further reading as they did not meet the inclusion criteria. One paper could not be obtained and was also excluded. A total of 24 studies were selected for review and were critically appraised using a tool developed by the Health Care Practice R&D Unit (HCPRDU).2 This tool covers quantitative, qualitative and multi-method studies. It was developed for, and extensively used in, systematic review work undertaken by the HCPRDU. As the studies were significantly heterogeneous, the investigator used a pragmatic approach to synthesis in extracting similar themes from which useful conclusions could be drawn. Results Study design Quantitative and qualitative designs were included in the review. A wide range of study designs was employed. These are summarized in Table 1. Setting Studies took place in many different settings. The majority took place in North America (16 in the United States and one in Canada). Six of the remaining studies were undertaken in the UK and one in the Netherlands. Twenty of the studies took place in an academic setting. The majority were in medical schools (13), nursing schools (4), pharmaceutical courses (2) and dental schools (1). Four studies involved practising clinicians. Three of these took place in the UK involving general practice staff and one took place with clinicians operating in rural areas of the United States. Methods of training Numerous different training methods were utilized. These included didactic sessions, demonstration of techniques, hands-on sessions, one-to-one sessions, small groups, large groups, interactive web packages,

© Health Libraries Group 2003 Health Information and Libraries Journal, 20 (Suppl. 1), pp.3 – 9

Information skills training, Alison Brettle Table 1 Summary of study design and relevance of study to review. Study design

Paper

Relevance to review

Randomised controlled trial (2)

Rosenberg et al., 1998 Erickson and Warner, 1998

Effectiveness of information skills course6 Impact of a MEDLINE course3

Experimental (1)

Verhoeven et al., 2000

Evaluation of different methods of literature retrieval10

Quasi-experimental (1)

Ghali et al., 2000

Evaluate mini EBM course15

Cohort (10)

Dorsch et al., 1990 Verhey, 1999 Fox, N. J. et al., 1999 Grant, 1996 Sowinski et al., 2000 Burrows et al., 1999 Fox, L. M. et al., 1996 Gibson et al., 2000 Vogel et al., 2002 Poyner et al., 2002

Integrate EBM course into curriculum21 Evaluate information literacy programme11 Evaluate information programme delivered via virtual classroom16 Evaluate impact of literature searching course7 Evaluate literature searching module delivered via Internet12 Evaluate adequacy of MEDLINE instruction22 Effectiveness of information literacy programme8 Evaluate computer literacy course17 Evaluate effectiveness of MEDLINE workshop9 Evaluate intensive information skills training5

Observational (1)

Brandt and Lehman, 1995

Evaluate traditional versus web-based literature searching session 4

Qualitative (4)

Martin, 1998 Martinale, 1995 Lambert-Lanning, A., 2000 Richwine and McGowan, 2001

Evaluate new style user education13 Investigate teaching search process versus practical skills23 Evaluation of literature searching workshop14 Impact of access and training in virtual health library18

Questionnaire only (5)

Earl and Neutens, 1999 Schwartz and Schwartz, 1995 Hicks, A., 1998 Hook, S. A., 1996 Butros, A., 1997

To determine whether EBM course improves patient care19 Impact of MEDLINE training24 Evaluation of information skills programme25 Evaluate Internet searching workshop26 Evaluate e-mail delivery MEDLINE course27

sessions delivered via e-mail and various combinations of the above methods. The number of different methods used makes comparisons of effectiveness across studies highly problematic. Two studies involved a direct comparison of different types of training methods.3,4 Erickson and Warner compared a one-hour user-driven tutorial with either a one-hour instructor-led tutorial or no intervention, but found little differences between groups who had received training. Brandt and Lehman compared a ‘traditional’ practical with a facilitated web-based session. Based on student perceptions the authors conclude that both methods were equally effective. Training content The content of the training programmes varied widely, again compromising any comparison across studies. The majority of sessions included instruction on question formulation, controlled vocabulary (thesaurus searching/MESH) and

Boolean operators. Length of sessions also varied greatly with a range of 1–3 h. One study provided intensive one-to-one training including four individual 2–3 h sessions.5 Effectiveness of training Outcome measures. Most studies used user questionnaires to assess the value and nature of the training given. Eight of these also included questions regarding users’ perceived improvement in skills following training. Objective tests of skills were used as outcome measures in a number of studies. Seven of these undertook post-skills testing. Only five,5,6,7 − 9 carried out pre- and postskills testing. Two studies3,10 used recall and precision as outcome measures and four studies linked assessment to student assignments as part of their curriculum.11 − 14 Training versus no training. A number of studies compared training with no provision of training.

© Health Libraries Group 2003 Health Information and Libraries Journal, 20 (Suppl. 1), pp.3 –9

5

6


One of these,6 showed that training was effective when compared with no training, whereas others3,15 also indicated that training improved medical students’ skills, based on perceived improvements rated by the students. It is difficult to make more direct comparisons between these studies as both their approach to teaching and their focus was quite different. Within a nursing setting, one study16 demonstrated that training was effective within a broader information literacy programme whilst another11 concluded the opposite. Improvement in skills. Fifteen studies considered improvement in skills. Five of these used an objective measure and assessed skills pre- and post-training.6 − 9 These provided limited evidence of the effectiveness of training. Three studies indicate that training can improve skills.6 − 8 One showed a non-significant improvement in skills3 and another demonstrated that training can provide a basic level of competence.17 Improvements in patient care. Two studies considered the effect of providing literature search skills on patient care, using user perceptions as an outcome measure.18,19 Earl and Neutens, in a small exploratory study, reported that participants believed training would improve patient care. Richwine and McGowan, in a study of the effectiveness of providing health information services and training on patient care in a virtual library setting, found that the majority of users reported that use of the library made them handle a clinical situation differently. Discussion The quality of studies included in this review has been problematic. Many studies were flawed, either in their design, execution or reporting. Common flaws included use of insufficient or inappropriate outcome measures and small sample sizes. Inadequate reporting presented difficulties in understanding the nature of the interventions, how results had been obtained and how conclusions had been drawn. Furthermore, the studies were highly heterogeneous, including differences in settings, training methods and outcome measures. This is to be

expected, as such a wide range of health professionals and trainee health professionals require information skills and relationships between library and information professionals and those responsible for course development are likely to differ at every institution. However, such factors will likely impact on the results of training and need to be taken into account when assimilating the results. The content, timing and method of delivery may be more appropriate in some situations rather than others. For example, students may benefit from receiving basic instruction at the beginning of a course and then more advanced instruction at a later stage. This remains to be investigated. Although user questionnaires are appropriate for measuring the outcome of training—particularly in ascertaining the value of training or the nature and format of the session—more objective methods are likely to be appropriate when determining the effectiveness of training in terms of improvement in skills. Most studies failed to include user questionnaires in their report, with each study seemingly having developed its own questionnaire rather than using a pre-validated tool or validating the questionnaire. Asking users to rate their perceived change in skills following training may be useful, but not as accurate as testing the skills of participants using a more objective method. A small-scale evaluation20 has indicated that users under-report their skills post training in comparison to the results obtained from an objective test taken at the same time. Although the findings of this review are limited by the poor quality of included studies, the following factors should also be taken into consideration when interpreting its results. The review covered a relatively short time period, 1995–2002. Some studies undertaken before this time may have an impact on the conclusions drawn. Sources used to locate studies for inclusion in the review were selected purposively to reflect subject areas within which studies on health information skills training might be reported. No attempt was made to locate grey literature or unpublished studies. A more comprehensive search of the literature would likely affect the numbers of studies retrieved and the results obtained. Furthermore, use of more free text terms and similar strategies across databases



may improve the number of studies retrieved. An alternative or complementary approach would be to widen the search to non-health related studies which might provide more evidence in relation to methods of training. Conclusions This paper has described a systematic review undertaken to ascertain whether providing information skills training is effective, whether some training methods are more effective than others and, finally, whether providing information skills training improves patient care. Taking a pragmatic approach to analysing the studies and interpreting the results revealed very limited evidence to show that training does improve search skills. If more studies had used objective, rather than subjective, outcome measures this evidence would have been much stronger. Comparatively few studies compared different training methods and the heterogeneity of studies meant that it was not possible to compare methods across studies. Therefore no conclusions can be drawn regarding effective training methods. In relation to the effect of information skills training on patient care, two studies (using subjective methods) indicated a positive effect on patient care.

research, by editors and peer reviewers and by those who provide training in research methods. Implications for practice One conclusion that can be drawn from all the studies is that users value training and, on the whole, believe that their skills have improved following training. This is encouraging for those library and information professionals who provide training. It should also encourage others to undertake more research to provide more objective evidence regarding what works, in what situations, and why. However, there appears to be substantial duplication of effort with many information professionals choosing to develop their own courses and methods of evaluation. Better access to research evidence and mechanisms to reduce duplication, or to encourage sharing of resources and ideas, could prove to be invaluable. Finally, the results of this review are also encouraging for evidence-based library practice. Teaching of library skills is obviously valued and there are some indications that training is effective. It is believed that over the next few years the number and quality of studies undertaken will continue to improve to allow the evidence base to yield a more definitive answer.

Implications for research Further research is therefore needed in a number of areas. This includes testing skills pre- and posttraining, comparing training methods and conducting research to determine whether there is an actual, rather than perceived, effect on patient care. It would be particularly beneficial to conduct larger studies with longer follow-up using objective outcome measures, where this is appropriate. There is a shortage of validated measures available. Research could be undertaken to develop and validate measures to enable library and information professionals to evaluate the effects of their training more easily. Better reporting of studies would significantly improve the quality of the systematic review. Although there appear to be sufficient numbers of studies being undertaken to conduct a review, their quality and reporting could be improved. This issue could be addressed by those doing the

Acknowledgements The author would like to thank Andrew Long for his encouragement and support in undertaking the study and for the constructive feedback on the draft article. References 1 Grant, M. J., Brettle, A. & Long, A. F. Developing a review question: a spiral approach to literature searching. Poster presentation. Beyond the basics of systematic reviews, Oxford, July, 2000. 2 Health Care Practice R&D Unit. Assessment tools. 2001. http://www.fhsc.salford.ac.uk / hcprdu / assessment.htm (accessed 1 November 2002). 3 Erickson, S. & Warner, E. R. The impact of an individual tutorial session on  use among obstetrics and gynaecology residents in an academic training programme: a randomized trial. Medical Education 1998, 32, 269–73. 4 Brandt, K. A., Lehmann, H. P. & Welch, W. H. Teaching literature searching in the context of the WWW. Proceedings of the Annual Symposium on Computer Applications in Medical Care 1995, 888– 92.


7

8

Information skills training, Alison Brettle 5 Poyner, A., Wood, A. & Herzberg, J. Distance learning project— information skills training: supporting flexible trainees in psychiatry. Health Information and Libraries Journal 2002, 19, 84–9. 6 Rosenberg, W. M. C., Deeks, J., Lusher, A., Snowball, R., Dooley, G. & Sackett, D. Improving searching skills and evidence retrieval. Journal of the Royal College of Physicians of London 1998, 32, 557 – 63. 7 Grant, K. L., Herrier, R. N. & Armstrong, E. P. Teaching a systematic search strategy improves literature retrieval skills of pharmacy students, American Journal of Pharmaceutical Education 1996, 60, 281 – 6. 8 Fox, L. M., Richter, J. M. & White, N. E. A multidimensional evaluation of a nursing information-literacy program. Bulletin of the Medical Library Association 1996, 84, 182 – 90. 9 Vogel, E. W., Block, K. R. & Wallingford, K. T. Finding the evidence: teaching medical residents to search . Journal of the Medical Library Association 2002, 90, 327 – 30. 10 Verhoeven, A., Boerma, E. J. & Meyboom de Jong, B. Which literature retrieval method is most effective for GPs? Family Practice 2000, 17(1), 30 – 5. 11 Verhey, M. P. Information literacy in an undergraduate nursing curriculum: development, implementation and evaluation. Journal of Nursing Education 1999, 38, 252 – 9. 12 Sowinski, K. M., Scott, S. A. & Carlstedt, B. C. An internet training module for traditional and non traditional doctor of pharmacy students: content and evaluation. American Journal of Pharmaceutical Education 2000, 64, 15 – 9. 13 Martin, S. Reflections on a user education session with nursing students. Health Libraries Review 1998, 15, 111 – 6. 14 Lambert-Lanning, A., Watson, L. & Evans, M. F. Integrating medical informatics into the undergraduate curriculum. Bibliotheca Medica Canadiana 2000, 21(3), 16 – 9. 15 Ghali, W. A., Saitz, R., Eskew, A. H., Gupta, M., Quan, H. & Hershman, W. Y. Successful teaching in evidence-based medicine. Medical Education 2000, 34, 18 – 22. 16 Fox, N. J., Dolman, E. A. & Lane, P. The WISDOM project: training primary care professionals in informatics in a collaborative virtual classroom. Medical Education 1999, 33, 365–70.

17 Gibson, K. E. & Silverberg, M. A two-year experience teaching computer literacy to first-year medical students using skill based cohorts. Bulletin of the Medical Library Association 2000, 88, 157–64. 18 Richwine, M. & McGowan, J. J. A rural virtual health sciences library project: research findings with implications for next generation library services. Bulletin of the Medical Library Association 2001, 89, 37–44. 19 Earl, M. F. & Neutens, J. A. Evidence-based medicine training for residents and students at a teaching hospital: the library’s role in turning evidence into action. Bulletin of the Medical Library Association 1999, 87, 211–4. 20 Grant, M. J., Brettle, A. & Long, A. F. Development of Information Skills within the Masters in Research [MRes (Health and Social Care)], University of Salford, Final Report to TLQIS Programme. Salford: HCPRDU, University of Salford, 2002. 21 Dorsch, J. L., Frasca, M. A., Wilson, M. L. & Tomsic, M. L. A multidisciplinary approach to information and critical appraisal. Bulletin of the Medical Library Association 1990, 78, 38–44. 22 Burrows, S. C. & Tylman, V. Evaluating medical student searches of  for evidence based information: process and application of results. Bulletin of the Medical Library Association 1999, 87, 471–6. 23 Martindale, K. Teaching information skills on CD_ROM: a conceptual approach. Learning Resources Journal 1995, 11(2), 37–40. 24 Schwartz, D. G. & Schwartz, S. A.  training for medical students integrated into the clinical curriculum. Medical Education 1995, 29, 133–8. 25 Hicks, A. Developing information skills training for National Health Service Personnel: experiences at the Trent Institute for Health Services Research. Program, 1998, 32(2), 123–6. 26 Hook, S. A. Learning to use the Internet: an interactive team taught workshop for dental faculty and staff. Bulletin of the Medical Library Association 1996, 83, 409–11. 27 Butros, A. Using electronic mail to teach  . Medical Reference Services Quarterly 1997, 16(1), 69–75.



Appendix 1 Inclusion criteria • Studies undertaken between 1995 and 2001. • Studies that took place in a health related setting or studies that took place with student clinicians (any profession), in an academic or clinical setting. • Studies that involved any information skills related training (whether part of a course, library induction, etc.) or training that involved at least one of the following: literature searching related skills, health related sources, or studies examining methods of training in information skills or studies examining the effect of information skills training on patient care. • Studies that involved at least one evaluation component (user evaluation or other kind of evaluation or testing) of course or skills obtained.

Exclusion criteria • Information skills training that was not related to health. • Basic IT training (e.g. use of windows and e-mail) (unless part of a course that included literature searching skills). • Papers that described training or the development of courses without an evaluation component. • Papers evaluating training methods that were not related to health and information skills. • Papers evaluating whether any training (other than information skills related) affects patient care.

Appendix 2—search strategies ERIC

1 su = ‘information retrieval’ 2 su = ‘online searching’ 3 su = databases 4 su = ‘information literacy’ 5 su = ‘information seeking’ 6 su = ‘computer literacy’ 7 su = ‘search strategies’ 8 su = ‘library skills’ 9 su = ‘bibliographic databases’ 10 su = ‘information skills’ 11 1 or 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9 or 10 12 su = ‘library instruction’ 13 su = ‘skill development’ 14 su = ‘library education’ 15 su = ‘instructional effectiveness’ 16 su = ‘computer assisted instruction’ 17 su = ‘training’

18 19 20 21

12 or 13 or 14 or 15 or 16 or 17 11 and 18 year = 1995–2002 In English

Key OCLC interface Su = search in subject field ‘xx’ = searches as phrase

LISA

1 su = online information retrieval 2 su = searching 3 su = CD ROMS 4 su = information literacy 5 su = MEDLINE 6 1 or 2 or 3 or 4 or 5 7 su = user training 8 training program* 9 instruction 10 ‘virtual classroom’ 11 teaching 12 7 or 8 or 9 or 10 or 11 13 6 and 12 14 Year of publication = 1995–2002

Key Su = search subject field * = truncation symbol

MEDLINE

1 exp information storage and retrieval/ 2 computer literacy/ 3 medical informatics/ 4 medline/ 5 CD-ROM/ 6 Literature search$.tw. 7 Or/1–6 8 Computer user training/ 9 Education/ 10 Computer-assisted instruction/ 11 Training.tw. 12 Or/8–11 13 7 and 12 14 limit 13 to English and abstracts and 1995 –2002

Key Ovid Biomed software interface / = MESH term $ = truncation symbol tw—searches title, abstract and subject heading fields


9