EDUCATIONAL ADVANCE
The Emergency Medicine Milestones: A Validation Study Robert C. Korte, PhD, Michael S. Beeson, MD, MBA, Chad M. Russ, MS, Wallace A. Carter, MD, the Emergency Medicine Milestones Working Group, and Earl J. Reisdorff, MD
Abstract Objectives: The Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Medical Specialties sought to define milestones for skill and knowledge acquisition during residency training. Milestones are significant objective observable events. The milestones are listed within a structure that is derived from the ACGME general competencies. Major groups of milestones are called “subcompetencies.” The original 24 subcompetencies containing 255 milestones for emergency medicine (EM) were developed through a multiorganizational group representing most EM stakeholder groups. To assure that the milestones reflected EM resident progress throughout training, the EM Milestones Working Group (EM MWG) sought to validate the individual milestones. Methods: A computer-based survey was sent to all EM residency programs. The survey period began on April 30, 2012, and concluded on May 15, 2012. Respondents were asked to assign each milestone to a specific level of skill or knowledge acquisition. These levels ranged from a beginning resident to an accomplished clinician. There were two different forms that divided the milestones into two groups of 12 subcompetencies each. Surveys were randomly assigned to programs. Results: There were five respondents (the program director and four key faculty) requested from each of the 159 residences. There were responses from 96 programs (60.4%). Of the 795 survey recipients, 28 were excluded due to prior exposure to the EM milestones. Of the remaining 767 potential respondents, 281 completed the survey (36.6%) within a 16-day period. Based on the survey results, the working group adjusted the milestones in the following ways: one entire subcompetency (teaching) was eliminated, six new milestones were created, 34 milestones were eliminated, 26 milestones were reassigned to a lower level score, and 20 were reassigned to a higher level. Nineteen milestones were edited to provide greater clarity. The final result was 227 discrete milestones among 23 subcompetencies. Conclusions: The EM milestones were validated through a milestone assignment process using a computer-based survey completed by program directors and key faculty. Milestones were revised in accordance with the results to better align assignment within each performance level. ACADEMIC EMERGENCY MEDICINE 2013; 20:730–736 © 2013 by the Society for Academic Emergency Medicine
A
s part of the Next Accreditation System, the Accreditation Council for Graduate Medical Education (ACGME) has proposed a new
method for tracking resident acquisition of knowledge, skills, abilities, attitudes, and experiences.1 The ACGME and the American Board of Medical Specialties sought
From the American Board of Emergency Medicine (RCK, MSB, CMR, EJR), East Lansing MI; and the Residency Review Committee-Emergency Medicine (WAC), Chicago IL. Received December 14, 2012; revision received February 13, 2013; accepted March 20, 2013. The Emergency Medicine Milestones Working Group consists of (alphabetical): Theodore A. Christopher, MD (Association of Academic Chairs in Emergency Medicine); Jonathan W. Heidt, MD (Emergency Medicine Residents’ Association); James H. Jones, MD (American Board of Emergency Medicine); Lynne E. Meyer, PhD, MPH (Accreditation Council for Graduate Medical Education); Susan B. Promes, MD (Society for Academic Emergency Medicine); Kevin G. Rodgers, MD (American Academy of Emergency Medicine); Philip H. Shayne, MD (Council of Emergency Medicine Residency Directors); Susan R. Swing, PhD (Accreditation Council for Graduate Medical Education); Mary Jo Wagner, MD (American College of Emergency Physicians); and Earl J. Reisdorff, MD (American Board of Emergency Medicine). The authors have no relevant financial information or potential conflicts of interest to disclose. Supervising Editor: John Burton, MD. Address for correspondence and reprints: Michael S. Beeson, MD, MBA; e-mail:
[email protected].
730
ISSN 1069-6563 PII ISSN 1069-6563583
© 2013 by the Society for Academic Emergency Medicine doi: 10.1111/acem.12166
ACADEMIC EMERGENCY MEDICINE • July 2013, Vol. 20, No. 7 • www.aemj.org
to define objective milestones for skill and knowledge acquisition as a physician progresses in residency training. Categories that contained significant objective observable events (“milestones”) that define competency acquisition are called “subcompetencies.” In late 2011, the ACGME, through the Residency Review Committee for Emergency Medicine (RRC-EM), and the American Board of Emergency Medicine (ABEM) jointly convened the Emergency Medicine Milestones Working Group (EM MWG). This group was composed of representatives from major stakeholder organizations within the specialty. By using the newly developed ABEM certification standards and material developed by ACGME expert panel groups for common competencies, the EM MWG identified the original 24 subcompetency groups and developed the 255 milestones within these groups. Resident acquisition of milestones will be assessed for every resident every 6 months. Ideally, a progressive pattern of achieving increasingly more complex and sophisticated milestones will characterize a resident’s experience in training. Performance on subcompetency milestone acquisition will be a data set that is annually reviewed by the RRC-EM for each residency program. Given the prominent role of milestone assessment for the individual resident, as well as its use as a residency measurement, it is important to have a valid set of milestones. This is achieved through expert consensus with subsequent validation by the academic EM community, especially program directors and faculty who will use the milestones as an evaluation device. ABEM undertook this study on behalf of the EM MWG. The goal of this activity was to validate the milestones and additionally provide information that could be used to modify the milestones. METHODS Study Design and Population A computer-based survey was developed from the original draft of the milestones developed by the EM MWG. Surveys were sent to all 159 residency programs that were ACGME-accredited at the time of the survey. The two survey forms were randomly assigned to programs. Program directors and key faculty were asked to complete the survey. Once the program director received the survey, he or she selected four key faculty to also complete the survey. Beyond asking that the participating faculty be key faculty, no other directions were provided. It was anticipated that program directors would select faculty who had experience in evaluating residents. This study was considered exempt by the institutional research review board of Akron General Medical Center. Survey Content and Administration Each subcompetency had a separate survey page. Markers were extracted from the level categories and listed alphabetically. The respondent was asked to designate into which of the five levels the marker should be placed. Given the size of the survey (24 subcompetencies and 255 milestones), the milestones were separated into two forms (Forms A and B), each of which had 12
731
subcompetencies to complete. Form A contained even numbered subcompetencies; Form B contained oddnumbered subcompetencies. In this way, the subcompetencies were nearly evenly distributed. The milestones validation survey used a computerbased platform Vovici EFM Continuum (Herndon, VA). Form A and Form B were randomly sent to all 159 EM residency programs on April 30, 2012. Randomization was achieved by using a Bernoulli trial assignment through SAS (version 9.2, SAS Institute, Cary, NC) using an equal probability of 0.5. Despite a delay of 2 days for one of the forms due to a programming issue, the survey was concluded on May 15, 2012 (16 days). There was no follow-up survey sent to garner additional responses. The short duration of the study was due to a desire to release the milestones at a national EM meeting. There was considerable interest on the part of the EM community to see the milestones, which had been embargoed prior to the completion of the survey. Given the study design, the milestones could not be shared prior to the completion of the survey. ABEM was assisted by the Council of Emergency Medicine Residency Directors in encouraging program directors and faculty to complete the survey. The survey results were also directly deposited in Vovici EFM Continuum. The frequency of milestone assignment into a specific level designation was the primary outcome. In addition, the cumulative responses for each level were determined. The number of residencies that had any respondents was noted, as well as the overall response rate of all potential respondents. The survey results were then used by the EM MWG to amend the assignment of milestones within a level using a set of predefined decision rules (see Data Supplement S1, available as supporting information in the online version of this paper). Data Analysis The final number of subcompetencies and milestones was determined. The number of milestones within a given level was also defined. Data analysis was performed using SAS (version 9.2, SAS Institute). Frequencies, cumulative frequencies, and chi-square analysis were used where appropriate. Significance for chisquare was predetermined to be a < 0.05. RESULTS Of the 795 survey recipients, 28 were excluded due to prior exposure to the milestones. Program directors and faculty who were involved in the development of the milestones, as well as the development of the ABEM certification materials, were excluded. This was a prospective exclusion criterion. Respondents were asked to voluntarily exclude themselves in the survey instructions if this exclusion applied. Of the remaining 767 potential respondents, 281 completed the survey (36.6%). There were 390 copies (49.1%) of Form A sent and 405 copies (50.9%) of Form B sent. Of the 281 respondents, 126 (44.8%) completed Form A and 155 (55.2%) completed Form B. There was no significant difference in the overall response rate between the two forms (p = 0.08).
732
Korte et al. • EM MILESTONES VALIDATION
There were responses from 96 programs (60.4%). Likewise, there was no significant difference in response rates between the two forms for residencies (p = 0.32). Of all responses, 36.7% came from program directors. Of the original 255 milestones, the survey results supported keeping 155 (60.8%) milestones at the same level and without any editing. An example of results for a milestone and how the milestone assignments were revised is shown in Tables 1, 2, and 3. The survey results for the teaching subcompetency showed a nearly random pattern of responses, suggesting that the assessment of the progressive acquisition of teaching skills within a residency was problematic. The final number of milestones that were selected was 227, which was an 11.0% reduction in milestones from the original milestone set.2 There were 20 of the 255 milestones (7.8%) that were moved to a higher level and 26 milestones (10.2%) that were moved to a lower level. There were 34 milestones (13.3%) that were eliminated. There were 19 (7.5%) milestones that required editing. Every level category for every subcompetency required a milestone. The result of moving and eliminated milestones was that some levels no longer contained a milestone. Thus, there were six milestones that were created to fill “empty” levels. In the final set of milestones, there were 38 (of 227) in Level 1 (16.7%), 50 in Level 2 (22.0%), 61 in Level 3 (26.9%), 47 in Level 4 (20.7%), and 31 (13.7%) in Level 5. DISCUSSION Pursuant to the ACGME mandate, every medical specialty must develop a unique set of milestones.1 These milestones will be used as an evaluation platform for every resident every 6 months. There are three tiers to each milestone set. The highest tier is the general competency assignment. The second tier is the subcompetency within that general competency. Each subcompetency contains progressively complex and sophisticated observable objective milestones. The ability range of these milestones spans from a recently
graduated physician from medical school (Level 1), to an experienced attending physician (Level 5). As the resident acquires greater skill, the performance of various milestones should reflect a higher-level rating within that subcompetency. Ideally, all residents should acquire all markers in the Level 4 category by the time of residency graduation. The development of the milestones used a substantial foundation of data in their construction. The milestones were developed from three key sources: ABEM initial certification standards (unpublished, internal document); the Model of the Clinical Practice of Emergency Medicine (EM Model)3; and ACGME common competency expert panels (unpublished, internal document). The ABEM initial certification standards were developed through multiple advisory panels, an ABEM internal task force, and a national survey of 26,000 ABEM-certified physicians. This input was reviewed by the ABEM board of directors and refined into a final set of certification standards and items that reflected essential knowledge, skills, and abilities (KSAs) for emergency physicians. These were largely anchored on physician tasks listed in the EM Model.3 The EM Model further provided a list of clinical procedures germane to practicing emergency physicians. Finally, the ACGME expert panels created milestones for the common competencies: professionalism, interpersonal and communication skills, systems-based practice, and practice-based learning and improvement. The common competency materials were amended to be more directly applicable to EM. The resulting draft EM Milestones had 24 subcompetencies with 255 associated milestones that were placed in five different performance levels. Level 1 was to represent the competency of a graduating medical student, Level 2 was an early intermediate level, Level 3 was an advanced intermediate level, Level 4 was a graduating resident level, and Level 5 was an experienced attending physician level. By placing the ABEM initial certification standards (and associated KSAs) in the Level 4 category,
Table 1 Original Matrix: Emergency Stabilization Level 1 Describes a primary assessment on a critically ill or injured patient Recognizes abnormal vital signs
Level 2
Level 3
Level 4
Level 5
Recognizes when a patient is unstable requiring immediate intervention Prioritizes vital critical initial stabilization action in the resuscitation of a critically ill or injured patient Performs a primary assessment on a critically ill or injured patient
Discerns relevant data to formulate a diagnostic impression and plan Reassesses after implementing a stabilizing intervention
Manages and prioritizes critically ill or injured patients Recognizes in a timely fashion when further clinical intervention is futile Evaluates the validity of a DNAR order Integrates hospital support services into a management strategy for a problematic stabilization situation
Develops policies and protocols for the management and/or transfer of critically ill or injured patients
Emergency stabilization prioritizes critical initial stabilization actions and mobilizes hospital support services in the resuscitation of a critically ill or injured patient and reassesses after stabilizing intervention.* DNAR = do not attempt resuscitation. *This is also the new ABEM standard for initial certification.
ACADEMIC EMERGENCY MEDICINE • July 2013, Vol. 20, No. 7 • www.aemj.org
733
Table 2 Study Responses: Emergency Stabilization Milestone
Level 1
Level 2
Level 3
Level 4
Describes a primary assessment on a critically ill or injured patient (L1) Cell frequency 34 56* 10 1 Cumulative 34 90* 99 100 Recognizes abnormal vital signs (L1) Cell frequency 90 7 2 0 Cumulative 90 98 99 99 Recognizes when a patient is unstable requiring immediate intervention (L2) Cell frequency 29 54 17 0 Cumulative 29 83 99 99 Prioritizes vital critical initial stabilization actions in the resuscitation of a critically ill or injured patient (L2) Cell frequency 4 30 52 14 Cumulative 4 34 86 100 Performs a primary assessment on a critically ill patient (L2) Cell frequency 30 54 16 1 Cumulative 30 83 99 100 Discerns relevant data to formulate a diagnostic plan (L3) Cell frequency 25 54 20 1 Cumulative 25 79 99 100 Reassesses after implementing a stabilizing intervention (L3) Cell frequency 11 54 32 2 Cumulative 11 65 97 99 Manages and prioritizes critically ill or injured patients (L4) Cell frequency 3 15 59 23 Cumulative 3 18 77 100 Recognizes in a timely fashion when further clinical intervention is futile (L4) Cell frequency 0 13 50 35 Cumulative 0 13 63 98 Evaluates the validity of a DNAR order (L4) Cell frequency 10 34 38 17 Cumulative 10 45 82 99 Integrates hospital support services into a management strategy for a problematic stabilization situation (L4) Cell frequency 0 6 33 39 Cumulative 0 6 38 78 Develops policies and protocols for the management and/or transfer of critically ill or injured patients (L5) Cell frequency 0 3 17 27 Cumulative 0 3 21 48
Level 5 0 100 1 100 1 100 0 100 0 100 0 100 1 100 0 100 2 100 1 100 22 100 52 100
Data are reported as percentages. L1 connotes that the original level assignment was in Level 1; L2 connotes that the original level assignment was in Level 2; etc. ABEM = American Board of Emergency Medicine; DNAR = do not attempt resuscitation. *Numbers ≥ 50% of assignment to an individual marker level, or ≥ 50% for the accumulative frequency, triggered level reconsideration by the working group.
there would be alignment of graduating resident competency and the ABEM initial certifying standards. There were nine patient care subcompetencies built on the physician tasks found in the EM Model. Of these, eight used the ABEM certification standards to define the desired milestone performance for residency completion. There were five additional patient care subcompetencies that were procedure-based. For procedures-based subcompetencies, there were no available ABEM initial certification standards. The procedure-based patient care subcompetencies and milestones were developed through expert panel deliberation and consensus by the EM MWG. The level assignments of the procedure-based milestones by the EM MWG were closely aligned with the survey results. There was a subcompetency for medical knowledge that also did not incorporate an ABEM certification standard and was developed by the EM MWG. Of the remaining eight subcompetencies that were based on the common competencies (e.g., systems-based practice), three used ABEM-developed certification standards. In general, the
subcompetencies and milestones related to the common competencies relied substantially on the ACGME expert panel material, yet integrated ABEM data as well as EM MWG expert opinion. To our knowledge, no other medical specialty has surveyed all residencies to validate the milestones. Moreover, no other specialty started with such robust information and certifying board input as the EM Milestones. The study reported here provided data that resulted in refining the original milestones. These refinements used a set of heuristics that were approved by the EM MWG prior to adjusting the milestones (Data Supplement S1). One such decision rule was that for milestones where 50% or more of survey respondents chose a level different from the EM MWG assignment, the respondents’ results should be considered, particularly if it is at a higher level than the original assignment. The manner in which milestones were modified can be understood by examining the modification process for the subcompetency of emergency stabilization. The
734
Korte et al. • EM MILESTONES VALIDATION
Table 3 Revised Matrix: Emergency Stabilization Level 1 Recognizes abnormal vital signs (1)
Level 2 Recognizes when a patient is unstable requiring immediate intervention Performs a primary assessment on a critically ill or injured patient Discerns relevant data to formulate a diagnostic impression and plan (2)
Level 3 Manages and prioritizes critically ill or injured patients (3) Prioritizes vital critical initial stabilization action in the resuscitation of a critically ill or injured patient (4) Reassesses after implementing a stabilizing intervention Evaluates the validity of a DNAR order (5)
Level 4 Recognizes in a timely fashion when further clinical intervention is futile Integrates hospital support services into a management strategy for a problematic stabilization situation
Level 5 Develops policies and protocols for the management and/or transfer of critically ill or injured patients
1. The marker “Describes a primary assessment on a critically ill or injured patient” was eliminated. It was moved to Level 2 where an existing marker was “Performs a primary assessment on a critically ill or injured patient.” 2. This marker was originally at Level 3 and was moved down. 3. This marker was originally at Level 4 and was moved down. 4. This marker was originally at Level 2 and was moved up. 5. This marker was originally at Level 4 and was moved down. DNAR = do not attempt resuscitation.
original scoring matrix had 12 milestones distributed across the five levels (Table 1). Of note, the description of emergency stabilization (“prioritizes critical initial stabilization actions and mobilizes hospital support services in the resuscitation of a critically ill or injured patient and reassesses after stabilization”) is the actual ABEM testing standard for initial certification. The study results contained the frequency of level assignment as well as the cumulative frequencies for each milestone (Table 2). Applying the aforementioned heuristic rules, the subcompetency for emergency stabilization was revised to a final form (Table 3). For this subcompetency, one milestone was eliminated, one milestone was moved to a higher level, and three milestones were moved to lower levels. The remaining 22 subcompetencies were also adjusted by the EM MWG in a similar fashion. Once the revisions were completed, the final version was submitted to the RRC-EM for approval. On September 7, 2012, the RRC-EM approved the validated EM Milestones,3 making EM the first medical specialty to approve a final set of milestones. LIMITATIONS The respondents were asked to assign each milestone to a particular time or sequence in resident training. They were blinded to the original placement by the MWG. They did not know the content of the discussions or the rationale of the MWG members. This might have influenced marker assignment. Ideally, 100% of respondents would have participated. A greater response rate could have altered results, but it is unknown if this would have been a significant change. Still, acquiring responses from 60% of EM residencies is considerable, given the 16-day survey window without a subsequent resending of the survey. The overall response rate of 37% was less robust. Obviously,
the strength and confidence in validity would be greater with increased numbers of respondents. The intention of this study was to assure that the milestones were assigned to the correct levels. It did not ask if the subcompetencies themselves were valid categories, nor did it have respondents comment on the milestones as measurements. Given the broad input into the EM Model that is revised every 2 years, as well as the survey of over 26,000 emergency physicians in the foundational ABEM work,2 there is substantial support for the milestone subcompetency categories. In addition, the link of the common competencies to the ACGME general competencies and the use of expert panels strengthen the support for the selected milestone subcompetency categories. The assignment of a milestone into a specific level could vary among residents of the same year level. Where a milestone was assigned could be determined by when a resident is assigned to a certain clinical rotation. For example, when an anesthesia rotation occurred might influence when a particular airway-related milestone was achieved. This would not be a function of competency or program quality, but merely a scheduling artifact. There were six new milestones created that did not undergo the same validation process. Likewise, those items that received editing did not undergo resurvey. CONCLUSIONS The Emergency Medicine Milestones were developed with previously developed material from American Board of Emergency Medicine that closely aligns the Level 4 milestone expectations with the American Board of Emergency Medicine certifying standards. Using an initial set of emergency medicine milestones developed by the EM Milestone Working Group, the American Board of Emergency Medicine conducted a study
ACADEMIC EMERGENCY MEDICINE • July 2013, Vol. 20, No. 7 • www.aemj.org
whereby program directors and key faculty assigned objective, observable actions into progressive levels of competency acquisition. The study was used to affirm or alter the milestones or their assignments to levels. In less than 2 years, a final set of EM milestones was produced, validated, and approved by the Accreditation Council for Graduate Medical Education.
735
The Emergency Medicine Milestone Project. Available at: https://www.abem.org/PUBLIC/_Rainbow/ Documents/EMMilestonesMeeting4_Final1092012.pdf. Accessed Apr 27, 2013. 3. Perina DG, Brunett CP, Caro DA, et al.; 2011 EM Model Review Task Force. The 2011 model of the clinical practice of emergency medicine. Acad Emerg Med. 2012;19:e19–40.
References 1. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system–rationale and benefits. N Engl J Med. 2012;366:1051–6. 2. Accreditation Council for Graduate Medical Education and American Board of Emergency Medicine.
Supporting Information The following supporting information is available in the online version of this paper: Data S1. Heuristics for milestone modification.
