2011; 33: 719–723
Using bibliographic software to appraise and code data in educational systematic review research ROBIN KING, BARBARA HOOPER & WENDY WOOD Colorado State University, USA
Abstract Background: Bibliographic database software is often recommended as a tool that can assist researchers in managing the large numbers of references produced in early stages of a systematic review. The uses of such software in systematic review research are often represented solely in terms of bibliographic functions, while the uses that extend beyond simple bibliographic functions have not been explored. Aims: This article provides a guide on how to use extended functions of bibliographic software to systematically complete the steps of appraising search results and coding references for inclusion in or exclusion from the systematic review. Methods: The process is illustrated using an ongoing systematic review as a case description and using screenshots from the bibliographic database, EndNoteÕ , the authors’ preferred software program for the study. Results: The case description illustrates how bibliographic software serves not only to organize and store search results, but also to appraise and code search results and to explicitly track researchers’ decisions across the systematic review. Conclusion: Bibliographic tools can contribute to make the methods that researchers adopt for the phase following the initial literature search more transparent and systematic. The illustrative case description involves a systematic review of academic education in occupational therapy, but easily generalizes to systematic reviews in other health science professions.
Introduction
Practice points
Systematic review research plays an increasingly important role in developing and implementing health sciences education (Hammick et al. 2010). Systematic reviews provide evidence for effective teaching and learning on behalf of future health care providers. When evidence is lacking, systematic reviews also provide maps of educational topics that need further elaboration and investigation (Hammick 2005). Because of their important role in advancing health science education, systematic reviews require a rigorous and trustworthy data management plan at each phase of the process. Published guides now exist to aid researchers in managing most phases of a systematic review. Published guides include, for example, Haig and Dozier’s (2003a, 2003b) guides to conducting database searches and the Best Evidence Medical Education (BEME 2003) guide to conducting multiple phases of a review. To the best of our knowledge, however, no guide exists for the phase following the literature search during which researchers review the large number of references generated by the search and determine their fit with the study. BEME (2003) referred to this phase as the ‘‘initial appraisal of the literature search’’ ( p. 5). This phase has also been referred to as ‘‘managing citations, abstracts and primary studies’’ (Hammick et al. 2010, p. 8) and ‘‘the handling of search results’’ (Dornan et al. 2006, p. 6). Researchers tend to report
. Researchers conducting systematic reviews appraise the retrieved references and code them for inclusion and exclusion from the study. . The specific methods used are often not as transparent and clear as steps taken in other phases of the review. . Bibliographic software programs can be used to not only compile the results of a search but also to appraise references, code for inclusion and exclusion across multiple researchers, and address discrepancies within the research team. . Utilizing the features of bibliographic software programs that extend beyond common bibliographic functions and can help make the process following the literature search more transparent and systematic.
the steps taken during this phase quite generally. Steps are often reported as: reviewing or screening the titles and abstracts of all references, obtaining full text of all references deemed eligible by at least one researcher, assigning each article to two independent researchers who further review for inclusion or eligibility, and addressing discrepancies through discussion (Hamdy et al. 2006; Hammick et al. 2007; Colthart et al. 2008; Buckly et al. 2009; Akl et al. 2010). Thus, the specific methods used during this phase often remain veiled,
Correspondence: B. Hooper, Center for Occupational Therapy Education, Department of Occupational Therapy, Colorado State University, 1573 Campus Delivery, Fort Collins, CO 80523-1573, USA. Tel: 1 970 4911325; fax: 1 970 4916290; email:
[email protected] ISSN 0142–159X print/ISSN 1466–187X online/11/090719–5 ß 2011 Informa UK Ltd. DOI: 10.3109/0142159X.2011.558138
719
R. King et al.
however inadvertently. As a result, scholars have called for transparency, rigor, and quality control during this phase of the systematic review (Reeves et al. 2002; Hammick et al. 2010). Additionally, many guides to systematic review research recommend bibliographic database software as potential tools for the phase of managing the large amount of reference data created from the search (Egger et al. 2001; Reeves et al. 2002; Best Evidence Medical Education 2003; Haig & Dozier 2003a, 2003b; Dornan et al. 2006; Buckley et al. 2009; Hammick et al. 2010). Yet these guides generally limit the use of bibliographic tools to the obvious functions of importing, storing and retrieving references, deleting duplicate references, and creating citations when writing. In actuality, most bibliographic programs can be used not only to accomplish importing and storing but also to appraise references and code them for inclusion and exclusion. The aim of this article is to describe how to use bibliographic software to: (1) conduct the initial appraisal of search results and (2) allow multiple researchers to code references for inclusion in or exclusion from the study and to address discrepancies. This guide begins after the systematic review phases of generating a research question, creating a study protocol, determining an initial search strategy, conducting the search, and exporting references into a bibliographic program.
Choice of bibliographic software Each bibliographic database program has unique features that can be matched to the protocol established by the research team. Published comparisons of various programs are readily available (http://www.adeptscience.co.uk/products/refman/ reference/chart.html). For the systematic review presented here, the team’s protocol involved not only sorting and appraising the results of individual search strategies, but also categorizing retrieved references into the subsets of education they represented. The team found that EndNoteÕ ’s Custom Groups feature made this sorting process easy. Additionally, the research protocol involved two researchers coding each reference for inclusion and exclusion, and also providing justifications for coding choices when necessary. EndNoteÕ Custom Fields feature supported this process by providing a place directly within the bibliographic software itself to store this data. Therefore, the bibliographic software, EndNoteÕ was a clear choice for the needs of this study group because the steps of sorting, appraising, and co-coding references for inclusion and exclusion could all be conducted, stored, and tracked within a single tool.
produced by a particular search. Second, the complete bibliographic record of references of varying levels of relevance is appraised, including journal type, subject headings, and abstract. These elements of the record can reveal additional search terms. Both levels of the appraisal can be conducted using EndNote’sÕ Custom Groups feature. With Custom Groups, researchers create subsets of references from all retrieved references, such as a group that contains the results of each search. For example, all the results from our first search in Cumulative Index for Nursing and Allied Health (CINAHL) were placed in a Custom Group named CINAHL 1. The results from our second search in CINAHL were placed in a Custom Group named CINAHL 2. Each search could then easily be appraised for the types of references it elicited and its effectiveness for eliciting references that were well-matched to the study. While appraising the results of each search, researchers can simultaneously examine the full bibliographic record of references that seem highly relevant, looking for new terms to add to the search strategy. In our systematic review, we also wanted to appraise the educational topics that were most prevalent in the search results. Therefore, based on the references we retrieved, we created additional Custom Groups for retrieved topics, such as ‘‘approaches to learning,’’ ‘‘degree types,’’ ‘‘directors and faculty,’’ and ‘‘transition to practice,’’ among others. Using Custom Groups, we easily compared the relative prevalence of these topics in the search results, providing a glimpse into the education literature overall (Figure 1).
Initial appraisal of references: Set up Custom Groups The initial appraisal of references serves to clarify search terms and syntax that will generate the most inclusive results (Best Evidence Medical Education 2003). The appraisal proceeds on two levels. One, each search is appraised to see if the results are ‘‘the type of results you were looking for, and how you might refine your search to make it more specific or more sensitive, or both’’ (Haig & Dozier 2003a, p. 475). This appraisal involves a quick inspection of the references 720
Figure 1.
Sample Custom Groups.
Bibliographic software in systematic reviews
To create Custom Groups, a researcher can right-click on ‘‘Custom Groups’’ in the left hand panel and choose ‘‘Create Group’’ from the resulting pop-up menu. EndnoteÕ then prompts for the new group’s name and a container for the new group appears in the Group’s pane. To add references to the new Custom Group, a researcher can select one or more references and drag them into the correct Custom Group. EndnoteÕ automatically tallies the numbers of references in each group.
Coding references for inclusion and exclusion: Set up Custom Fields and Smart Groups Custom Fields The initial appraisal of search results helps refine criteria for references that will and will not be included in the study. The criteria that researchers use to determine which references are in and which are out are translated into codes. Each article is then tagged with one or more codes that justify why it was included or excluded. For example, in our study we decided to limit the scope by excluding articles pertaining to fieldwork education. These articles were tagged with the code Ex-FWE. Similarly, we decided to include articles pertaining to student perspectives of learning experiences. These articles were coded In-SP. For our study, we had 36 codes for inclusion and 14 codes for exclusion. Researchers can assign inclusion and exclusion codes directly in EndnoteÕ ’s Custom Fields. Each reference imported into EndnoteÕ has several Custom Fields which can be seen by opening a reference and scrolling down to the first Custom Field (Figure 2). Custom Fields can be configured to store researchers’ codes, making it easy to compare coding across researchers. To prepare the Custom Fields, choose ‘‘Edit 4 Preferences’’ from the top menu. Select ‘‘Reference Types’’ from the left-hand panel of the pop-up window, and click on ‘‘Modify Reference Types’’ in the right-hand panel of the window (Figure 3). Scroll down to the Custom Fields and
Figure 2. Custom Fields on an opened reference, before configuration of field names.
assign a field to each researcher by entering a name in the box to the right of the Custom Field (Figure 4). To ensure that the new Custom Field names show up in all references in the library, click ‘‘Apply to all Ref Types.’’ For our field names, we used each researcher’s initials – BH_Status, WW_Status, and RK_Status – to indicate Custom Fields for specific researchers and ‘‘Final_Status’’ to store the consensus code(s). When one researcher configures names for the Custom Fields, the updated names appear in EndNoteÕ only for that researcher. That researcher can share the changes with the team using EndNoteÕ ’s export and import feature seen in Figure 3. Alternatively, each researcher can repeat the above process on his or her computer. At times, Custom Fields may not appear when opening individual references. In this case, look to the upper right hand corner of the EndNoteÕ display. If a ‘‘Show Empty Fields’’ message appears, clicking the message allows the Custom Fields to display (Figure 5).
Figure 3. Configuring reference types.
Figure 4.
Configuring Custom Fields with names.
721
R. King et al.
Figure 6. Configuring a Smart Group for a researcher to code a subset of articles.
Figure 5.
Displaying Custom Fields in references.
Figure 7. Reference list pane showing Custom Field headings.
Smart Groups Once the Custom Fields are configured, researchers can divide references among team members using EndNoteÕ ’s Smart Groups feature. Smart Groups differ from Custom Groups in one way: for Custom Groups, you create empty groups and add references to them manually; for Smart Groups you rely on formulas to automatically include specific references in the Smart Group. For example, in order to have two reviewers code each article, we decided to have two of our three reviewers code half of the references and one reviewer code all the references in the library. A Smart Group was created for each reviewer by choosing ‘‘Groups 4 Create Smart Group’’ from EndNoteÕ ’s top menu. The pop-up window provides fields for assigning a formula which then determines what references from the library appear in the group. To create the Smart Group for the reviewer who needed to review half of the library, in the first row we selected the Custom Field that was assigned to that researcher, chose the comparison operator ‘‘Is’’, and left the search term blank (indicating that the researcher had entered no codes). In the second row, we chose the field ‘‘Title’’ and the comparison operator ‘‘Is Less than,’’ and we entered the search term ‘‘H’’ (Figure 6). This Smart Group was then automatically populated with copies of all titles A–H. One final step is required to display the Custom Fields in the library’s Reference List pane (Figure 7). Choose ‘‘Edit 4 Preferences’’ from EndNoteÕ ’s top menu. Click on ‘‘Display Fields’’ in the left-hand panel, and in the right hand panel choose the reviewer’s Custom Field (Figure 8). Once each researcher has a Smart Group populated with the articles he or she should code, EndNoteÕ can be used to complete the inclusion–exclusion coding. To code, a researcher opens his or her assigned Smart Group. Articles that the researcher needs to code appear in the Reference List pane. The researcher opens each reference (by double clicking on the reference) and carefully reviews it in terms of 722
Figure 8. pane.
Displaying Custom Field headings in reference list
the content in the title, abstract, keywords, source, and type of publication. One or more inclusion–exclusion codes are assigned to the reference by entering the code in the Custom Field assigned to that researcher. For example, in Figure 9, BH entered the code In-CD and RK entered In-CD. The final status was therefore In-CD and the article was included in the study. Each coded reference is then saved. Once saved, the coded article is automatically dropped from the researcher’s Smart Group and the Smart Group’s tally for the articles the researcher needs to code decreases by one. After the second researcher has coded the same article, he or she compares the identified codes with those of the first reviewer. If there is agreement, the agreed upon code(s) are entered in the Custom Field, ‘‘Final_Status.’’ If there is a discrepancy in coding, then ‘‘COMPARE’’ is entered into the final status for later review. To identify all references where there was discrepancy in the two reviewers’ codes, we created an additional Smart
Bibliographic software in systematic reviews
because of the close proximity of Custom Fields in EndNoteÕ , it did not appear possible to ensure that the second review of a reference was a blind review. This can be addressed by assigning the last Custom Field to the first reviewer. This way, the second reviewer does not scroll far enough to see the code assigned by the first reviewer. Declaration of interest: There are no person, financial, or organizational conflicts of interest associated with this publication.
Notes on contributors Figure 9. Sample coding process using Custom Fields. Group to show references with conflicting assignments; that is, all references for which COMPARE had been entered into the Custom Field for final status. In the first row of this Smart Group’s formula, we chose Custom Field 5 (the field we were using for final status) and ‘‘Field Begins with’’ from the drop list and we entered ‘‘COMPARE’’ as the search term. This group was automatically populated with the references that had been tagged with ‘‘COMPARE,’’ allowing for discussion and consensus. This ‘‘Compare’’ Smart Group optimized our time by providing a means to have asynchronous online discussions about references. Often, researchers read each others’ notes in the Custom Fields and came to agreement without further discussion, streamlining the coding process. In the systematic review of occupational therapy education, we also used EndNoteÕ to track the frequency distribution of inclusion and exclusion codes, the status of full text acquisitions, and the codes that changed in the data abstraction phase of the review. Altogether, therefore, EndNoteÕ , selected because its extended features were well suited to the research protocol, is a valuable tool that serves not only to organize and store search results, but also to appraise and code search results and to track researchers’ decisions across the systematic review. Thus depending on the features of the specific software, bibliographic tools can contribute to making more transparent and systematic the methods that researchers adopt for the phase following the initial literature search.
Limitations This article described how to set up key features of one bibliographic software program specifically as they relate to and support the systematic review process. It is important to note that one limitation of EndNoteÕ published in product reviews is that the database does not support multi-user access, meaning simultaneous access. To work around this limitation, we placed EndNoteÕ on a shared drive and each team member used the database during separate time blocks. If a team member inadvertently accessed the database while another was using it, EndNoteÕ reported the error via a popup message. The second person could read references but not make updates. We did not choose to use EndNote WebÕ to resolve the access issue because the Web version does not support Smart Grouping. Another potential limitation was that,
ROBIN KING is a graduate student in occupational therapy at Colorado State University and graduate research assistant to the Center for Occupational Therapy Education, Colorado State University. BARBARA HOOPER, PhD, OTR, FAOTA, is an assistant professor in the Department of Occupational Therapy at Colorado State University. She is the director of the Center for Occupational Therapy Education, Colorado State University. WENDY WOOD, PhD, OTR, FAOTA, is a professor and head of occupational therapy at Colorado State University.
References Akl EA, Pretorius RW, Sackett K, Erdley WS, Bhoopathi PS, Alfarah Z, Schunemann HJ. 2010. The effect of educational games on medical students’ learning outcomes: A systematic review: BEME Guide no 14. Med Teach 32:16–27. Best Evidence Medical Education (BEME) 2003. Guide for topic review groups on carrying out BEME systematic reviews. [Retrieved 2009 September 4]. Available from: www.bemecollaboration.org Buckley S, Coleman J, Davison I, Khan KS, Zamora J, Malick S, Morley D, Pollard D, Ashcroft T, Popovic C, et al. 2009. The educational effects of portfolios on undergraduate student learning: A Best Evidence Medical Education (BEME) systematic review. BEME Guide no. 11. Med Teach 31:340–355. Colthart I, Bagnall G, Evans A, Allbutt H, Haig A, Illing J, McKinstry B. 2008. The effectiveness of self-assessment on the identification of learner needs, learner activity, and impact on clinical practice: BEME Guide no. 10. Med Teach 30:124–145. Dornan T, Littlewood S, Margolis SA, Scherpbier A, Spencer J, Ypinazar V. 2006. How can experience in clinical and community settings contribute to early medical education? A BEME systematic review. Med Teach 28:3–18. Egger M, Smith GD, Altman DG. 2001. Systematic reviews in health care. 1st ed. London: BMJ Publishing Group. Haig A, Dozier M. 2003a. BEME Guide no 3: Systematic searching for evidence in medical education – Part 1: Sources of information. Med Teach 25(4):352–363. Haig A, Dozier M. 2003b. BEME Guide no. 3: Systematic searching for evidence in medical education – Part 2: Constructing searches. Med Teach 25(5):463–484. Hamdy H, Prasad K, Brownell M, Anderson A, Williams R, Zweirstra R, et al. 2006. Beme systematic review: Predictive values of measurements obtained in medical schools and future performance in medical practice. Med Teach 28:103–116. Hammick M, Dornan T, Steinert Y. 2010. Conducting a best evidence systematic review. Part 1: From idea to data coding. BEME Guide no. 13. Med Teach 32:3–15. Hammick M, Freeth D, Koppel I, Reeves S, Barr H. 2007. A best evidence systematic review of interprofessional education: BEME Guide no. 9. Med Teach 29:735–751. Hammick MR. 2005. Evidence-informed education in the health care science professions. J Vet Med Educ 32:399–403. Reeves S, Koppel I, Barr H, Freeth D, Hammick M. 2002. Twelve tips for undertaking a systematic review. Med Teach 24:358–363.
723
