based medicine teaching requirements in the ... - Wiley Online Library

Journal of Evidence-Based Medicine ISSN 1756-5391

ARTICLE

Evidence-based medicine teaching requirements in the USA: taxonomy and themes Allen F. Shaughnessy1 , John R. Torro2 , Kara A. Frame2 and Munish Bakshi3 1

Family Medicine, Tufts University School of Medicine, Boston, MA, USA Greater Lawrence Family Medicine Residency, Lawrence, MA, USA 3 Family Medicine, Boston University, Boston, MA, USA 2

Keywords Accreditation/standards; clinical competence; education; evidence-based medicine; internship and residency/standards. Correspondence Allen F. Shaughnessy, Tufts University Family Medicine Residency at Cambridge Health Alliance, 195 Canal Street, Malden, MA 02176, USA. Tel: +1-781-338-0507; Fax: +1-781-338-0150; Email: [email protected] Received 26 June 2014; accepted for publication 23 August 2015. doi: 10.1111/jebm.12186

Abstract Background: In the USA, recent changes to oversight of residency training codify the requirements for teaching evidence-based medicine and information mastery (lifelong learning) knowledge, skills, and attitudes. Objective: The goal of this project is to determine current requirements for teaching of evidence-based medicine (EBM) and information mastery (IM) in specialty residency education in the USA. Methods: The project was a qualitative thematic analysis using content analysis. The source of the requirements for EBM and IM were the “milestone” statements for all 28 major specialties and transition year programs. Milestone descriptors related to EBM/IM were extracted and codes were developed and applied to each descriptor by four researchers. The resulting codes were coalesced into themes and tested against the milestone descriptors. Results: The coding process identified 15 content areas comprising five themes. Two themes related to the knowledge and skills of EBM and three themes related to the knowledge, skills, and attitudes of IM. EBM themes encompassed basic critical appraisal skills and knowledge of clinical epidemiology principles and statistics. IM themes centered on identifying one’s information needs for patientspecific information, using information sources, and using current awareness services to remain abreast of changes in medicine. In general, they align well with the Sicily Statement on Evidence-Based Practice. No specialty required competence in all areas. Conclusion: New training requirements for specialties in the USA require the development of both classic EBM skills as well as skills for managing information. However, there is marked variation in the requirements among specific specialties.

Introduction Training in residency programs in the USA lays the groundwork for what will be the state of clinical practice among its graduates. In an attempt to standardize training and push programs to develop a competency-based approach to education, the Accreditation Council for Graduate Medical Education (ACGME) introduced the Outcome Project in 1999, outlining six domains of clinical competency (1). Evidence-based medicine (EBM) as a general requirement is included as part of the “Practice-Based

Learning and Improvement” (PBLI) competency domain (Table 1) under the general goal of being able to “appraise and use scientific evidence.” In addition, the PBLI competency domain also includes skills in managing information, broadly characterized as, “use technology to optimize learning” (1), “Information Mastery” (2), an outgrowth of evidence-based medicine, focuses on identifying information needs related to the care of specific patients and using evidence sources to find this information, as well as developing systems to be alerted to new, relevant, and valid information.

C 2016 Chinese Cochrane Center, West China Hospital of Sichuan University and John Wiley & Sons Australia, Ltd JEBM 9 (2016) 53–58

53

Evidence-based medicine taxonomy

A.F. Shaughnessy et al.

Table 1 Alignment between milestones and Sicily Statement Taxonomy from milestones

Sicily Statement educational outcomes (8)

EBM 1: Validity of research evidence

r r r r

EBM 2: Critical appraisal of specific study types IM 1: Identifying need for information

r r

IM 2: Answering patient-specific questions

r r r

IM3: Keeping up with and implementing changes in medicine (current awareness)

The learner understands the strengths and weaknesses of the different sources of evidence The learner can assess the relevance of the appraised evidence to the need that prompted the question The learner can appraise the validity of a study The learner identifies knowledge gaps during the course of practice and asks foreground questions to fill these gaps The learner should ask focused questions that lead to effective search and appraisal strategies The learner can design and conduct a search strategy to answer questions The student can appraise the importance of the outcomes and translate them into clinically meaningful summary statistics The student reflects on how well EBM activities are performed The student can explore the patient’s values and the acceptability of the answer

PBLI-1 Locates, appraises, and assimilates evidence from scientific studies related to the patients’ health problems Level 1

A milestone descriptor

Describes basic Concepts in clinical epidemiology, biostatistics, and clinical reasoning Categorizes the design of a research study

Level 2 Identifies prosandcons of Various study designs, associated types of bias, and patient-centered outcomes Formulates a searchable question from a clinical question Evaluates evidence-based point-of-care resources

Level 3

Level 4

Applies a set of critical Appraisal criteria to different types of research, including synopses of original research findings, systematic reviews and meta-analyses, and clinical practice guidelines

Incorporates principles of evidence-based care and information mastery into clinical practice

Level 5 Independently teaches And assesses evidencebased medicine and information mastery techniques

Critically evaluates information from others, including colleagues, experts, and pharmaceutical representatives, as well as patient-delivered information

Figure 1 Example of a milestone statement.

In 2009, the ACGME began to develop a new process, the “Next Accreditation System,” which requires the regular tracking and reporting of resident progress regarding competency development. This reporting is based on defined milestones, which are, “developmentally based, specialty specific achievements that residents are expected to demonstrate at established intervals as they progress through training” (Figure 1) (3). This new system of accreditation required each specialty to develop milestones in a close collaboration among the American Board of Medical Specialties certifying boards, their ACGME Residency Review Committee, specialty organizations, program director associations, and residents. These milestones are anchored to each description of clinical competency originally described in the Outcome Project but each specialty was given the freedom to adapt the milestone descriptors to their unique clinical environment and scope of practice.

54

The advent of the Next Accreditation System, with the increased specificity of the milestones, for the first time codifies the requirements for evidence-based medicine teaching in residency education. The goal of this project was to determine the themes and topics represented across all the milestones for all specialties.

Methods The project was a qualitative content examination using thematic analysis. We conducted the project with interpretivism theoretical perspective and deductive approach using directed content analysis (4, 5). The investigators included an expert in evidence-based medicine and three practicing physicians who were educators in residency training programs. All investigators had additional training in medical education and evidence-based medicine.



Box 1. Description of the practice-based learning and improvement competency domain (1)

r r r r r r r r

Identify strengths, deficiencies, and limits in one’s knowledge and experience Set learning and improvement goals Identify and perform appropriate learning activities Incorporate formative evaluative feedback into daily practice Systematically analyze practice and implement changes to improve practice Appraise and use scientific evidence Use technology to optimize learning Participate in the education of patients, families and other health professionals

The sources for our analysis were the milestone documents published on the ACGME website for 28 specialties and the transition year residency (6). We did not include subspecialty residency milestones or fellowships. Each document consists of separate milestone statements describing the progression of competency that should be documented. Each statement consists of five levels of competency (Figure 1), ranging from the level of competence expected at the start of that residency through the level expected of an experienced clinician following several years of practice. These levels, in turn, contain one or more descriptors of competence. Only descriptors for the first four levels of competence were included in this analysis since residents were only expected to meet this fourth level by the end of training, with the fifth level intended to be aspirational. A single author (AFS) extracted all applicable milestone descriptors for all specialties. A second author checked a sample of five milestones to determine the completeness of this extraction. Since agreement was 100%, additional milestone statements were not checked. A single researcher (AFS) performed open coding for each competency descriptor to develop preliminary start codes. Using a constant comparative approach, all four researchers reviewed all descriptors separately, coding each descriptor with one or more codes or adding new codes as warranted. We compared the coding performed by all four researchers, resolving discrepancies by consensus. The codes were reworked in an iterative process until all descriptors were represented by one or more codes and no new codes were identified (ie, saturation was reached). At this point, consolidation was performed to minimize overlap. We coalesced these codes into three distinct themes that represented the individual codes in a logical manner. The codes were continually reworked until we achieved the best fit under each theme. This step


was performed by AFS and KAF and then sequentially refined by the group. The milestones were then checked against these codes to assure the codes’ applicability and completeness.

Results Our initial analysis resulted in the generation of 37 codes for 273 separate descriptors in the milestones. These were reduced to 15 content areas (Box 2). These content areas were further categorized as relating to “evidence-based medicine” or “information mastery.” Most of the milestone descriptors were located in PBLI milestone statements, although some specialties included applicable descriptors under Patient Care and Medical Knowledge rubrics.

Evidence-based medicine content areas These themes encompass critical appraisal skills and knowledge of clinical epidemiology principles and statistics (7). The first theme, Validity of Research Evidence, focuses on recognizing study design and quality, sources of bias, and issues of relevance, applicability, and generalizability. A typical milestone descriptor, usually designated as an initial level of competence, for example, level 1 or 2, is “Describes basic concepts in clinical epidemiology, biostatistics, and clinical reasoning, and can categorize the study design of a research study” (level 1, Dermatology). The second theme, Critical Appraisal of Specific Study Types, describes the critical appraisal of specific types of literature, including original research, reviews and metaanalyses, and clinical practice guidelines. Some milestone descriptors are general, such as those from emergency medicine: “Demonstrates the ability to critically appraise scientific literature and apply evidence-based medicine to improve one’s individual performance” (level 3). Other milestones are very specific: “Applies a set of critical appraisal criteria to different types of research, including synopses of original research findings, systematic reviews and meta-analyses, and clinical practice guidelines” (level 3, Dermatology, Orthopedics, Plastic Surgery, and Thoracic Surgery). Generally, these skills in critical appraisal were considered by be higher level, milestone level 3 or 4. Urology milestones, however, consider some critical appraisal skills to be beyond the level achievable during residency training, with “Appraises systematic reviews, clinical practice guidelines, and cost-effectiveness studies for validity, impact, and applicability” assigned as level 5.

Information mastery themes Milestone descriptions emphasizing these themes ask residents to develop competence in the use of information to


55


Box 2. Evidence-based medicine and information mastery themes Evidence-Based Medicine Themes EBM 1: Validity of Research Evidence

r r r

Knowledge and application of indicators of study quality: Identifying study design and knowing the hierarchy of evidence, including levels of evidence Identification of bias in medical research Understanding of relevance, applicability, generalizability of research data

EBM 2: Critical Appraisal of Specific Study Types

r r r

Original research, including studies of therapy, diagnosis, prognosis, and harm Reviews, including systematic reviews and metaanalysis Clinical practice guidelines

Information Mastery Themes IM 1: Identifying the Need for Information

r r r

Identification of a specific need for information or learning (in contrast to a learning plan or general selfdevelopment) Formulation of a searchable question Demonstration the use of knowledge gained via formal evidence-based presentation

IM 2: Answering Questions

r r r

Selecting and using appropriate databases to conduct a literature search, including demonstration of basic computer and search skills Using and critiquing point-of-care information sources Evaluation of information from other, nondatabase sources such as colleagues, experts, the pharmaceutical industry, or patient provided information

IM3: Keeping Up and Implementing Changes in Medicine (Current Awareness)

r r r

Demonstrating the linkage between new evidence and change in practice or practice improvement (in contrast to general practice improvement) Understanding the limitations of using evidence to guide decisions Demonstration of the use of a system for keeping up with relevant changes in medicine

improve their decision-making and the subsequent care of patients (2). The first theme, Identifying Need for Information, centers on identifying one’s need for patient-specific information (in contrast to general knowledge enhancement) and converting this information need into a question for search-

56


ing (eg, using the patient-intervention-comparison-outcome model). Some milestones required residents to demonstrate their abilities via formal evidence-based presentations. The second theme, Answering Patient-Specific Questions, concerns the answering of questions using a variety of point of care information resources. For example, Allergy and Immunology milestone descriptors progress from “Conducts a limited literature search and use databases” (level 1) to “Presents a critical appraisal and synthesis of the literature either orally or in writing” (level 4). While most milestone descriptors focused on obtaining information from computerbased information resources, some specialties specified the use of other information resources, for example, “Critically evaluates information from others, including colleagues, experts, and pharmaceutical representatives, as well as patient-delivered information” (level 3, Family Medicine). The third theme, Keeping Up with Changes in Medicine, includes the demonstration of a current awareness system, for example, “Utilizes system or process for staying abreast of relevant changes in clinical practice” (level 4, Ophthalmology) as well as demonstration of practice change or practice improvement based on current evidence, for example, “Assimilates evidence from scientific studies into practice” (level 4, Preventive Medicine: Occupational Medicine). A single specialty, neurology, specifies the questioning of the epistemology embodied in evidence-based medicine, requiring that a resident, “understands the limits of evidencebased medicine in patient care” (level 4). All but one specialty (colorectal surgery) had milestones related to at least one of these themes. No specialty had all aspects of all five themes.

Discussion The requirements for EBM competence set forth across the milestones encompass the classic EBM steps of developing a question and finding and evaluating the research for validity, impact, and applicability, decision-making based on evidence, and assessing one’s own performance in incorporating new evidence into practice. These milestones statements align well with the further explication of these steps in the “Sicily Statement on Evidence-Based Practice” (8) (Box 1). The concept of shared decision-making is represented in this taxonomy, obliquely, as “recognizing the limitations of evidence-based medicine.” The milestone statements also reflect information mastery concepts. Some milestone statements specified the ability to distinguish relevant research outcomes (patient-oriented evidence that matters) (9) from other types of evidence. The milestones incorporate the concepts of using a variety of sources to answer questions, using point-of-care resources to answer specific clinical questions (2), using resources



such as practice guidelines and other expert-based resources (10), and developing lifelong learning skills and systems for keeping up with changes in medicine (11). Missing from the taxonomy is any mention of evaluating information received from the pharmaceutical industry through pharmaceutical representatives (12) or other sources of this type of information (13). In addition, we anticipated that cognitive errors and decision-making skills would be addressed in the milestones. Recent literature in the evidence-based realm has focused on the use of evidence and the ways in which decision-making can be improved (14–17). However, other than general statements, no milestone descriptors required documentation of the appropriate processing of evidence. A limitation to the NAS is that the milestones, while more descriptive than the previous six domains of clinical competency (1) are not specific enough to be specifically measured. ten Cate et al (18) have described the need for the identification of specific “entrustable professional activities.” The activities are the clinical activities that trainees can be trusted to perform with minimal or no supervision. Training programs have to identify specific behaviors of information use by residents that reflect the development of competence embodied in the milestones descriptions. The requirements for EBM and IM competency differ substantially across milestones. We describe these differences in a separate paper. The milestone statements relating to EBM and IM represent a small proportion of the requirements for each specialty. Although we did not specifically analyze other areas, arguably more important aspects of medical practice, such as competence in patient care and interpersonal communication skills, are represented in the milestones statements for each specialty. Our use of an interpretive research paradigm has both strengths and limitations. This approach assumes reality to be subjective and changing, with no ultimate truth (19). As a result, there are multiple ways to develop a taxonomy. Our coding frame was undoubtedly colored by our previous experience with teaching and practicing evidence-based medicine, and researchers from different backgrounds may have other interpretations. A second limitation is that our content analysis, which focused only on the literal interpretation of the milestone statements. Since we were not part of the milestone writing process for most specialty groups, we could only infer meaning from our interpretation of the milestone descriptors. We have limited ability, therefore to decipher the signs implanted in the milestone descriptions (ie, semiotics). There may be additional or different messages embedded in the descriptors that have specific meaning to the members of the specialty that are not accessible to those outside the specialty.


Conclusion EBM training in residency sets the trajectory of evidencebased implementation during subsequent clinical practice. Milestones set the standard for what will be taught and therefore they will influence the use of evidence in practice for the future. Although EBM and IM competence is well represented across the new milestones that will guide preparation of physicians for practice in the USA, much work remains to assure that all physicians reach a suitable level of preparation. These milestones will likely require changes to the curriculum of many residencies and their faculty may not be adequately prepared to meet this demand. In addition, tools to assess resident development and implementation of these competencies are limited and incomplete (20). However, it is reassuring to see that evidence-based medicine and information mastery skills have been codified as requirements for training programs.

References 1. Swing SR. The ACGME outcome project: retrospective and prospective. Medical Teacher 2007; 29(7): 648–54. 2. Slawson DC, Shaughnessy AF. Evidence-based medicine: should we be teaching information management instead? Academic Medicine 2005; 80(7): 685–9. 3. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system-rationale and benefits. New England Journal of Medicine 2012; 366(11): 1051–6. 4. Crotty M. The foundations of social research: meaning and perspective in the research process. Thousand Oaks, CA: SAGE Publications, 2005. 5. Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qualitative Health Research 2005; 15(9): 1277–88. 6. Accreditation Council for Graduate Medical Education. Milestones. Available at: https://www.acgme.org/acgmeweb/ tabid/430/ProgramandInstitutionalAccreditation/NextAccreditationSystem/Milestones.aspx. Accessed on March 11, 2014. 7. Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB. Evidence-based medicine. How to practice and teach EBM. New York: Churchill Livingstone, 2000:3–6. 8. Dawes M, Summerskill W, Glasziou P, Cartabellotta A, Martin J, Hopayian K, et al. Sicily Statement on evidence-based practice. BMC Medical Education 2005; 5(1): 1–7. 9. Slawson DC, Shaughnessy AF, Bennett JH. Becoming a medical information master: feeling good about not knowing everything. Journal of Family Practice 1994; 38(5): 505–13. 10. Slawson DC, Shaughnessy AF. Obtaining useful information from expert-based sources. British Medical Journal 1997; 314(7085): 947–9. 11. Shaughnessy AF, Slawson DC. Are we providing doctors with the training and tools for lifelong learning? British Medical Journal 1999; 319(7220): 1–3.


57


12. Shaughnessy AF, Slawson DC, Bennett JH. Teaching family practice residents to be information masters: a curriculum for the evaluation of pharmaceutical representatives. Family Medicine 1995; 27(9): 581–5. 13. Shaughnessy AF, Slawson DC, Bennett JH. Identifying fallacies in pharmaceutical advertising: separating the wheat from the chaff. Journal of General Internal Medicine 1994; 9(10): 563–8. 14. Croskerry P. Achieving quality in clinical decision making: cognitive strategies and detection of bias. Academic Emergency Medicine 2002; 9(11): 1184–204. 15. Schmidt HG, Norman GR, Boshuizen HP. A cognitive perspective on medical expertise: theory and implication. Academic Medicine 1990; 65(10): 611–21.

58


16. Gambrill E. Critical thinking in clinical practice: improving the quality of judgments and decisions, 3rd edn. Hoboken, NJ: John Wiley & Sons, 2012. 17. Bate L, Hutchinson A, Underhill J, Masrkey N. How clinical decisions are made. British Journal of Clinical Pharmacology 2012; 74(4): 614–20. 18. ten Cate O. Entrustability of professional activities and competency-based training. Medical Education 2005; 39(12): 1176–7. 19. Bunniss S, Kelly DR. Research paradigms in medical education research. Medical Education 2010; 44(4): 358–66. 20. Ilic D. Assessing competency in evidence based practice: strengths and limitations of current tools in practice. BMC Medical Education 2009; 9: 53–58.
