cohort study involving 14 residency programs. A 60- ... ultrasound education best prepare residents for an ... Statistic
180
Costantino et al.
d
PREDICTORS OF SUCCESS IN US EDUCATION
Predictors of Success in Emergency Medicine Ultrasound Education Thomas G. Costantino, MD, Wayne A. Satz, MD, Sarah A. Stahmer, MD, Anthony J. Dean, MD Abstract Objectives: To compare emergency medicine resident performance on an ultrasound-oriented, American Board of Emergency Medicine–styled written examination with the following variables in resident education: number of ultrasound scans performed, presence of a formal, structured ultrasound rotation, presence of a mandatory ultrasound rotation, number of hours of didactic ultrasound education, and percentage of ultrasound education taught by emergency physicians. Methods: This was a prospective cohort study involving 14 residency programs. A 60question multiple-choice test was completed by individual residents and returned for scoring. Results: 262 residents completed the study. Average score was 39.1/60 6 6.5 (65%). Scores improved as residency year increased (year 1: 36.6, year 2: 39.3, year 3: 42.6) (p , 0.005). Scores improved as number of scans performed increased from 34.3 (57%) for those residents who had performed 0–10 scans to 45.4 (76%) for those with .150 scans (p , 0.005). The presence of an
ultrasound rotation at an emergency medicine residency program also produced a statistically significant increase in test score (OR 1.82; 95% CI ¼ 1.29 to 2.55). Residents at programs spending the least time (6 to 15 hours) on didactic education throughout the residency predicted examination failure (OR 0.60; 95% CI ¼ 0.39 to 0.93). Increasing the amount of resident ultrasound teaching by emergency physicians improved the score, but this did not reach significance (p ¼ 0.357). Conclusions: Improved resident performance on an ultrasound written examination was associated with increasing resident year, number of scans performed, and the presence of an ultrasound rotation at the residency program. Increasing the number of didactic hours spent on ultrasound each year beyond 15 hours showed no improvement in resident performance. Key words: ultrasound; education; emergency medicine; residency. ACADEMIC EMERGENCY MEDICINE 2003; 10:180–183.
The use of ultrasound by emergency physicians has become increasingly widespread in recent years. The American Board of Emergency Medicine (ABEM), the American College of Emergency Physicians (ACEP), and the Society for Academic Emergency Medicine (SAEM) have included bedside ultrasonography as part of the core content for emergency medicine (EM).1 Ultrasound training is now a requirement for EM residents, and ABEM now includes ultrasound images on its written board certification examination. Prior to becoming a requirement, emergency medicine bedside ultrasound (EMBU) had been incorporated into the curricula of more than 70% of EM residency programs over the past several years.2 Published recommendations for training of EM residents range from a four-hour course to two weeks.3,4 SAEM
guidelines developed by a special task force in 1994 recommended .150 approved scans and 40 hours of didactic instruction,5 though few programs currently meet those requirements.6 Current ACEP guidelines suggest the equivalent of at least 16 hours of didactics and a minimum of 25 ultrasound scans for each type of scan performed.2 There is considerable variability among programs that offer ultrasound education as to the best way to meet these training guidelines. Although many studies have shown the clinical utility of ultrasound in the hands of emergency physicians,7–10 or compared training methods to see which has the most clinical effectiveness,11 to the best of our knowledge, no study has compared which aspects of ultrasound education best prepare residents for an ABEM-styled written examination.
From Drexel University College of Medicine, Department of Emergency Medicine, Philadelphia, PA (TGC, WAS, AJD); and Cooper Hospital/University Medical Center, Department of Emergency Medicine, Camden, NJ (SAS). Dr. Satz is currently at Temple University Hospital, Philadelphia, PA; Dr. Dean is currently at the University of Pennsylvania, Philadelphia, PA. Received June 4, 2002; revision received September 2, 2002; accepted September 3, 2002. Address for correspondence and reprints: Thomas Costantino, MD, MCP Hospital, Department of Emergency Medicine, 3300 Henry Avenue, Philadelphia, PA 19128.
METHODS Study Design. This was a cohort study of EM residents to determine the effect of several variables on resident knowledge of ultrasound as measured by a multiple-choice examination. Five key variables determined by the authors were included: 1.
Number of ultrasound scans performed (Numscan)
ACAD EMERG MED
d
February 2003, Vol. 10, No. 2
d
181
www.aemj.org
2.
Presence of a formal, structured ultrasound rotation (USrot) 3. In the presence of a structured ultrasound rotation, whether it was mandatory or elective (Mand) 4. Number of hours of didactic training (Didhrs) 5. Percentage of ultrasound training taught by emergency physicians (EMtaught) Study Setting and Population. Residents from 14 regionally diverse EM residency programs participated in the study. The examinations were distributed during June 2000 and data collection was completed by March 2001. Study Protocol and Measurements. We created a 60-question multiple-choice examination, consisting of ultrasound images and clinical scenarios (40 questions) and theoretical problems (20 questions) (TGC, AJD, SAS). The questions and images were taken from cases from our experience and based on typical clinical scenarios for EM ultrasound. The examination was reviewed for consistency and accuracy by EM sonographers not involved in the research protocol. Questions were designed to test residents on sonographic principles, physics, technique, artifacts, anatomy, and the six primary applications of EMBU (aorta, cardiac, trauma, gallbladder, kidney, and firsttrimester pregnancy). Data Analysis. Statistical analysis was conducted with the Systat 8.0 software package (SPSS, Chicago, IL) The data were entered into an Excel spreadsheet (Microsoft, Inc., Redmond, WA) and imported into Systat 7.0 for statistical analysis. All tests were twosided, and significance was accepted at the 5% level. Statistical analysis of the matching criteria was done with chi-square tests (pass–fail vs Resyear, Numscan, USrot, Mand, Didhrs, EMtaught). Statistical analysis of test scores was done with Kruskal-Wallis tests. We used post-hoc Mann-Whitney U tests with Bonferroni adjustment to determine within-group differences. Bonferroni corrections were applied to adjust for multiple testing.
TABLE 1. Variables among the 14 Emergency Medicine (EM) Residency Programs (N 5 262 resident respondents) Variable
n (%)
Mean Score
Standard Deviation
EM year 1 EM year 2 EM year 3 0–10 scans 11–25 scans 26–50 scans 51–100 scans 101–150 scans .150 scans EM rotation Mandatory No EM rotation 6–15 didactic hours 15–30 didactic hours 30–45 didactic hours 80% EM faculty taught 90% EM faculty taught 95% EM faculty taught 100% EM faculty taught
101 (38.5) 92 (35.1) 69 (26.3) 67 (26.0) 46 (17.8) 47 (18.2) 42 (16.3) 22 (8.5) 34 (13.2) 185 (70.6) 159 (85.9) 77 (29.4) 107 (40.8) 69 (26.3) 86 (32.8) 45 (17.2) 62 (23.7) 23 (8.8) 132 (50.4)
36.6 39.3 42.6 34.3 37.2 39.4 41.1 43.9 45.4 40.1 40.2 36.8 38.4 39.9 39.3 37.8 38.9 39.0 39.7
6.4 5.8 5.9 5.8 5.4 5.3 4.4 4.7 6.0 6.5 6.7 6.0 5.9 6.9 6.9 5.1 6.4 5.8 7.1
significant increase in test score (p , 0.0005). Residents with .150 scans had a much higher pass rate (OR 3.19; 95% CI ¼ 1.34 to 7.57) than residents who completed ,10 scans (OR 0.14; 95% CI ¼ 0.06 to 0.31) (Figure 1). The presence of an ultrasound rotation at an EM residency program also produced a statistically significant increase in test score (OR 1.82; 95% CI ¼ 1.29 to 2.55). There was no statistically significant increase in test score among residencies with different amounts of time spent in didactic teaching (p ¼ 0.281). Residents at programs spending the least time (6 to 15 hours) on didactic education throughout the residency had greater examination failure than those in programs with more didactic time (OR 0.60; 95% CI ¼ 0.39 to 0.93). The range of ultrasound education taught by emergency physicians (versus sonologists/sonographers from other specialties) was 80% to 100%. No
RESULTS Two hundred sixty-two of 412 eligible residents at the 14 EM residency programs involved in the study completed the test. The mean score was 39.1/60 6 6.5 (65%). Results are summarized in Table 1. There was a statistically significant increase in score as resident level increased (p , 0.0005). There was a statistically significant pass rate (score . 70%) for third-year residents versus first-year residents (OR 2.25; 95% CI ¼ 1.45 to 3.51) An increased number of ultrasound scans previously performed was related to a statistically
Figure 1. Examination score by number of ultrasound scans performed (values are mean 6 standard deviation).
182
Costantino et al.
significant difference between this parameter and improved test score was found (p ¼ 0.357).
DISCUSSION A number of recommended formats have been advanced for developing a training program for ultrasound education.2,3,5 These recommendations are often loosely based on anecdotal experience of the authors, and there are only limited data supporting any one format. This study was the first to describe the relationship between resident performance on a written examination with specific aspects of ultrasound education. The findings of this study suggest that increasing resident experience with ultrasound and clinical experience in the emergency department (Postgraduate year) is associated with improved performance on a written examination. This reflects a central feature of EMBU, which is that sonographic image acquisition and interpretation must be integrated with the clinical setting. Both the SAEM model curriculum and the newer ACEP guidelines for US training recommend a minimum of 150 ultrasound scans to achieve competency.2,5 Although this study did not measure competency with performance of emergency ultrasound, those residents with greater numbers of US examinations did perform better on this examination. However, we did not continue to track performance beyond 150 scans to see whether improvement continued beyond this point. The presence of an ultrasound rotation also resulted in a significant improvement in examination score. Although the exact structure of such a rotation was not included in the survey, it had to be more comprehensive than a two-day introductory course. An increase in didactic hours above 15 hours did not yield a significant increase in test score. This may be due to a plateau effect in education that occurs above that level. However, having fewer than 15 didactic hours during residency was associated with an increased examination failure rate among thirdyear residents. Several studies have shown that emergency physicians and other non-radiologists can perform limited ultrasound such as focused abdominal sonography for trauma (FAST) with as little as 4 – 10 didactic hours.11,12 The model curriculum endorsed by SAEM originally intended that residencies should have greater than 40 hours of didactic lectures before residents would be considered trained in emergency ultrasound.5 However, our study did not show a benefit in meeting that standard. The current ACEP guidelines suggest 16 hours of didactic education during residency training.2 The percentage of ultrasound education originating from emergency physicians ranged from 80% to 100% in our study population. No significant improvement in test score was demonstrated relating to the
d
PREDICTORS OF SUCCESS IN US EDUCATION
percentage of teaching from emergency physicians. This finding was not surprising in view of the very narrow range of this variable. The finding that resident sonography training was almost universally carried out by emergency physicians suggests that most programs are fulfilling the mandate of the American Medical Association House of Delegates Resolution 802, which recommends that ‘‘training and education standards [be] developed by each physician’s respective specialty.’’13 It also coheres with ACEP’s guidelines recommending that ‘‘committed faculty given the responsibilities of emergency ultrasound coordinator or director should organize the training of emergency medicine residents.’’2
LIMITATIONS This study attempted to identify which approach to resident education in EMBU would best prepare residents for an ABEM type of written examination. However, the questions were created by the authors, who are not examination writers for ABEM, and may not be representative of actual ABEM questions. The questions, subject matter, and topic allocation were not prospectively validated. The survey was not randomized, and as subject residencies were enrolled based upon their appearance at the 2000 SAEM annual meeting ultrasound interest group, a potential for selection bias exists. Not all residency programs that were sent the examination returned it, and not all residents from programs that did return the examination completed it, leading to a potential reporting bias. Although a recommended format for the test was promulgated, the examinations were not proctored, and testing conditions varied somewhat between sites. The data regarding the structure of EMBU training from different residency programs were self-reported, leading to another potential reporting bias. As EMBU is increasingly incorporated into residency programs, studies are needed to further define optimal training techniques. It is hoped that prospectively validated testing tools will be developed. In addition to evaluation of the variables investigated in the present study, future studies will consider the effect of other educational variables, including objective structured clinical examinations (OSCEs), use of models, electronic and computerized teaching aids, and experience beyond 150 ultrasound examinations.
CONCLUSIONS In an attempt to elucidate key components of an EM residency ultrasound curriculum, the following criteria were found to significantly improve resident performance on an ABEM-styled written examination: increasing level of training, increasing number of ultrasound scans performed, and having an ultrasound
ACAD EMERG MED
d
February 2003, Vol. 10, No. 2
d
183
www.aemj.org
rotation offered during residency. No significant increase in resident performance was observed when didactic hours spent on ultrasound education was increased above 15 hours during the residency.
5. Mateer JR, Plummer D, Heller M, et al. Model curriculum for physicians training in emergency ultrasonography. Ann Emerg Med. 1994; 23:95–102. 6. Witting MD, Euerle BD, Butler KH. A comparison of emergency medicine ultrasound training with guidelines of the Society for Academic Emergency Medicine. Ann Emerg Med. 1999; 34:604–9. 7. Ma OJ, Mateer JR, Ogata M, Kefer MP, Wittmann D, Aprahamian C. Prospective analysis of a rapid trauma ultrasound examination performed by emergency physicians. J Trauma. 1995; 38:879–85. 8. Mateer JR, Aiman EJ, Brown MH, Olsen DW. Ultrasonographic examination by emergency physicians of patients at risk for ectopic pregnancy. Acad Emerg Med. 1995; 2: 867–73. 9. Kuhn M, Bonnin RL, Davey MJ, Rowland JL, Langlois SL. Emergency department ultrasound scanning for abdominal aortic aneurysm: accessible, accurate, and advantageous. Ann Emerg Med. 2000; 36:219–23. 10. Rosen CL, Brown DF, Chang Y, et al. Ultrasonography by emergency physicians in patients with suspected cholecystitis. Am J Emerg Med. 2001; 19:32–6. 11. Salen P, Meanson S, Heller M. The focused abdominal sonography for trauma (FAST) examination: considerations and recommendations for training physicians in the use of a new clinical tool. Acad Emerg Med. 2000; 7:162–8. 12. Lanoix R, Baker WE, Mele JM, Dharmarajan L. Evaluation of an instructional model for emergency ultrasonography. Acad Emerg Med. 1998; 5:58–63. 13. American Medical Association House of Delegates. H-230.960 Privileging for Ultrasound Imaging. 802. 99. 2001.
The authors acknowledge the following emergency medicine residency programs for their participation in this study: State University of New York at Buffalo, Palmetto Richland Memorial Hospital, Carolinas Medical Center, Cooper Hospital/University Medical Center, New York Methodist Hospital, Highland General Hospital, Grand Rapids MERC/MSU Program in Emergency Medicine, Maricopa Medical Center, San Antonio Uniformed Services Health Education Consortium, St. Luke’s-Roosevelt Hospital Center, State University of New York at Stony Brook, University of California San Francisco – Fresno, and University of Mississippi Medical Center.
References 1. Hockberger RS, Core Content Task Force II. The Model of the Clinical Practice of Emergency Medicine. Acad Emerg Med. 2001; 8:660–81. 2. American College of Emergency Physicians. ACEP emergency ultrasound guidelines—2001. Ann Emerg Med. 2001; 38:470–81. 3. Mandavia DP, Aragona J, Chan L, Chan D, Henderson SO. Ultrasound training for emergency physicians—a prospective study. Acad Emerg Med. 2000; 7:1008–14. 4. Jehle D, Guarino J, Karamanoukian H. Emergency department ultrasound in the evaluation of blunt abdominal trauma. Am J Emerg Med. 1993; 11:342–6.
d
Where to Find AEM Instructions for Authors For complete instructions for authors, see the January or July issue of Academic Emergency Medicine; visit the SAEM web site at www.saem.org/inform/ autinstr.htm; or contact SAEM via e-mail at
[email protected], via phone at 517485-5484, or via fax at 517-485-0801.
