Evaluating Externship Programs: Impact of Program Length on Clinical Productivity Ana Karina Mascarenhas, B.D.S., M.P.H., Dr.P.H.; Sarah R. Freilich, B.S.; Michelle M. Henshaw, D.D.S., M.P.H.; Judith A. Jones, D.D.S., M.P.H., D.Sc.D.; Madalyn L. Mann, R.D.H., M.S.; Spencer N. Frankl, D.D.S., M.S.D. Abstract: Community-based dental education programs such as externship programs have become an integral component of dental school education. Qualitative evaluation of externship programs has captured increased confidence, efficiency, skill, and independence among students. To better understand these findings, quantitative evaluation of the externship experience is necessary. The current study investigated the clinical care (defined as dental procedures) provided by Boston University School of Dental Medicine students who completed six-week general dentistry externships compared to those completing ten-week general dentistry externships. Results indicate that the scope of services provided in the ten-week externship differed from the six-week externship. The longer externships allowed for students to perform more complex procedures toward the latter part of their externships. Students participating in the ten-week externship also provided more procedures per week than students in the six-week externship. Because scheduling of patient appointments is generally a function of the site, number of patient visits and procedures performed per patient did not differ between the two groups. Our findings provide insight into the reason why longer externships increase clinical confidence, efficiency, and skill. In the continuing effort to improve the externship experience in dental schools, these findings should be taken into consideration. Dr. Mascarenhas is Associate Professor and Director, Division of Dental Public Health; Ms. Freilich is Research Assistant, Division of Health Policy and Health Services Research; Dr. Henshaw is Associate Professor and Assistant Dean, Division of Community Health Programs; Dr. Jones is Professor and Chair, Department of General Dentistry; Prof. Mann is Associate Professor and Director, Office of Extramural Programs; and Dr. Frankl is Professor and Dean—all at Boston University School of Dental Medicine. Address correspondence and requests for reprints to Dr. Ana Karina Mascarenhas, 560 Harrison Avenue, Boston, MA 02118; 617-638-4456 phone; 617-638-6381 fax;
[email protected]. This study was supported by the Robert Wood Johnson Foundation through its Pipeline, Profession, and Practice: CommunityBased Dental Education program. Key words: externship, evaluation, dental, clinical productivity, student, extramural Submitted for publication 2/16/06; accepted 1/4/07
F
or many years, graduate schools across the country have utilized externship programs to provide enhanced educational and clinical experiences for students. In dental schools, externship programs provide students with a unique learning experience that renders them better prepared for their future as dentists. Students have the opportunity to learn about the needs of the community, treat a diverse group of patients, improve their critical thinking skills, and enhance their technical abilities. Moreover, students learn practice, patient, and time management.1-3 When students who participated in an externship program were compared to those who did not, DeCastro et al. found that externship students treated 2.5 times more patients, provided twice as much care, and felt better prepared to treat patients with special needs, work with other staff members, and use their time effectively.3 In addition, students in the DeCastro et al. study who completed extern-
516
ships reported a greater sense of confidence and felt more efficient, skilled, and better prepared to enter the dental profession. The development of extramural programs in dental schools began in the late 1960s; however, these programs varied in size and duration, were optional and open to students of any class, and lacked educational goals.4 The development of externship programs peaked in the 1970s as a result of available funding and concern over a shortage of dental professionals. In the 1980s, funding decreased, which eliminated some programs; however, extramural programs continued to be and are currently an integral part of dental education.5 In a recent survey,6 which defined an extramural program as a program in which undergraduate dental students provide any aspect of dental care to individuals in settings outside the main dental school clinical facility, fifty-four U.S. and ten Canadian dental schools were asked about their
Journal of Dental Education ■ Volume 71, Number 4
involvement in extramural programs. Of the fifty-one schools that responded to this survey, all except two U.S. dental schools offered extramural programs, in which senior students spent on average 5.3 weeks. The types of extramural or community sites used in these externships included community health centers, hospitals, nursing homes, correctional facilities, schools and day care centers, mobile clinics, and private dental offices. Although many dental schools currently offer externship programs, these programs vary considerably in design, objectives, rotation length, scope of services, and the extent of student participation.6 As a result of these differences, some programs are more successful than others. In an analysis of the 2003 American Dental Education Association (ADEA) senior survey data, which assesses graduating dental students’ perceptions about their externship experience, Thind et al. reported that students who spent one to two weeks on an externship were less likely to characterize their experience as positive compared to students in rotations of three to five weeks.7 Students who spent less time on their externships were also less likely to report that they had an improved ability to care for a diverse population of patients and said that their time was inadequate. As a result of such fundamental differences, it is imperative to conduct meaningful evaluations of these programs to determine which type of design is most effective. Researchers have been evaluating externship programs since their initial inclusion in the dental school curriculum in the 1960s. However, essentially all past studies used solely qualitative assessments to evaluate programs. In the 1970s, interviews of preceptors and students, site visits, and written evaluations completed by externs and preceptors were used to determine the effectiveness of externship programs.2,8-10 In recent years, similar methods have been used to assess the outcomes of these programs.1,3,7 While these techniques have led to substantial and important results reflecting the benefits of externships to students and the differences across externships programs, to date, there has been almost no assessment of the clinical care and the scope of services provided by students on externship. Nor has there been any quantitative assessment of length of the clinical rotation on student experiences. To address this gap in the available data, Boston University’s School of Dental Medicine (BUSDM) implemented a quantitative assessment of its students’ clinical production during externships in order to investigate one of the main differences among
April 2007 ■ Journal of Dental Education
programs: length of externship rotations. To analyze this aspect of externship design, the clinical care (defined as dental procedures) provided by students who completed six-week general dentistry externships was compared to those completing ten-week general dentistry externships.
Methods Community-based education and service programs play a major role in Boston University School of Dental Medicine’s mission. Over the past twenty-five years, in keeping with our “School without Walls” philosophy, the school initiated several innovative community-based programs, which have successfully been integrated into the curriculum. In 1978, BUSDM received a federal grant for dental schools interested in creating extramural experiences for their students. As a result, the externship program, a six-week community-based experience in general dentistry for all senior dental students, was developed. In this program, senior students provide general dental services at external or community sites to diverse patient populations including the homeless, the elderly, indigent children and families, and persons with special needs, all under the guidance of community-based faculty. Students are assigned to a single site and provide clinical care at the site for forty hours per week. Students do not have any other academic responsibilities at the dental school while on externship. The majority of our externship sites are at community health centers located in underserved regions in and around the Greater Boston area. Other sites include Veterans Administration hospitals, U.S. Coast Guard facilities, and correctional institutions. Like other externship programs, our program affords students the opportunity to enhance their clinical skills and appreciate the cultural and socioeconomic differences of their patients, while providing needed dental services to underserved populations. Additional details about the program have been previously published.11 In 2002, BUSDM was one of eleven schools to receive a Robert Wood Johnson Foundation (RWJF)funded Pipeline, Profession, and Practice: Community-Based Dental Education grant to develop the New England Dental Access Project. As part of this project, we proposed expanding our long-standing six-week general dentistry externship program to a ten-week program for all senior students. In the 2003-04 academic year, we pilot-tested this change
517
on a group of senior students who participated in a ten-week externship. The rest of the senior class participated in the six-week externship program. However, we were able to collect clinical productivity data for only one cycle of rotations (n=16) between February 17, 2004, and March 26, 2004. At BUSDM, students are assigned to externships based on academic standing, readiness, completing third-year competencies, and having successfully completed the Applied Professional Experience Program (APEX). A profile is developed that describes students’ skills, knowledge level, and ability to work independently. Therefore, when a student is assigned to an externship, he or she is deemed competent to be successful at the externship. For the ten-week externship, the profile was developed, and students ready to go on externship were identified. The new ten-week externship was explained to these students, and volunteers were solicited. The students on the ten-week externship were on rotation simultaneously between July 7, 2003, and September 12, 2003, and could be considered rising seniors. The six-week and ten-week externships were identical in structure and evaluation and were at some of the same sites only at different times during the academic year. Additionally, the ten-week externship has two additional responsibilities or assignments: a case presentation and public health project. Students were expected to spend about half a day each week on the public health project. Evaluation of a student’s clinical ability at BUSDM is not based on procedural quantity using points or a requirements system. It is competency-based, using formative and summative evaluations. Therefore, every procedure that a student does at the dental school or while at an externship prepares students for their summative evaluations. Other evaluations of the externship that are performed are qualitative evaluations. Students self-assess their cognitive skills, patient management skills, professional attitudes and habits, student-faculty relationship, and clinical skills using a survey instrument. Using the same instrument, they are assessed by the externship preceptor. In the ten-week externship, the case presention and public health project are graded. As part of the RWJF-funded Pipeline program, BUSDM developed a custom-designed, interactive, web-based application to collect data on the clinical care or dental procedures performed by the students while on their externships. At the end of each day, students are required to enter productivity data using the web-based application. In addition to
518
demographic patient data such as age, gender, race, and type of insurance, clinical data were collected using American Dental Association (ADA) procedure codes. Other data collected were date of service and whether the procedure and treatment plan were completed or in progress. Data from the web-based program were downloadable to an Excel file. The data were first cleaned in Excel and then exported to SPSS for analysis. New variables were created to indicate week of service and type of service—diagnostic, preventative, restorative, oral surgery (exodontia), endodontic, fixed and removal prosthodontic, periodontal procedures, and general adjunctive services—by grouping the relevant ADA procedure codes. Descriptive statistics were performed with means and standard deviations calculated. The main analyses performed were comparisons of clinical productivity between the six-week and ten-week externships. Comparisons were made over the entire externship period and over weeks one through six, comparing clinical productivity week by week. Differences in mean number of procedures or patient visits between the six-week and ten-week programs were tested using t-tests. To correct for differences among sites, all of the above analyses were repeated for students who completed the six-week and ten-week externships at the same externship sites. There were six sites that were common between the two rotations.
Results Results presented here are for the data collected between February 17, 2004, and March 26, 2004, for the six-week externship and between July 7, 2003, and September 12, 2003, for the ten-week externship. Sixteen students participated in the six-week externship, and ten students participated in the ten-week externship. Services provided by both groups included diagnostic, preventative, restorative, oral surgery (exodontia), fixed and removable prosthodontic, endodontic, periodontal procedures, and general adjunctive services. The services provided most frequently in the six-week externship in weeks one through six included diagnostic, preventative, restorative, and oral surgery procedures. During the ten-week externship, similar procedures were performed in the first six weeks. Additionally, in weeks seven through ten, prosthodontic rather than preventative procedures were performed more frequently (Table 1). Table 1 also reports the means and standard deviations for
Journal of Dental Education ■ Volume 71, Number 4
Table 1. Top five procedures including mean ±sd provided by senior students during the six-week and ten-week externships Rank
Six-Week Externship
Weeks 1-6
Mean ±sd
1 2 3 4 5
Oral exam 20 ±22 Two-surface amalgam 9 ±8 Prophylaxis 8 ±8 Extraction 8 ±9 Composite: 6 ±7 One-surface posterior
Ten-Week Externship Weeks 1-6
Mean ±sd
Oral exam 36 ±21 Intraoral x-ray 25 ±38 Extraction 13 ±10 Prophylaxis 12 ±7 Two-surface amalgam 11 ±5
each of the five most commonly performed procedures. As seen in this table, students in the ten-week externships over the same period did substantially more of the same procedures (oral examinations, two-surface amalgams, extractions, and prophylaxis) than students in the six-week externship. Over the entire externship period, the sixteen seniors in the six-week externship performed 1,898 procedures, while the ten seniors in the ten-week externship completed 2,644 procedures (Table 2). Each of the ten-week externs provided a greater number of procedures compared to the six-week externs due to the longer duration of their externships (p=0.0002). To account for this difference in length of the externships, the mean number of procedures performed each week by six- and ten-week externs was compared. As seen in Table 2, the mean number of procedures performed per week by ten-week externs (27.1 ±10.7) was substantially higher than for the six-week externs (20.3 ±10.5) although the difference was not statistically significant (p=0.12). The clinical productivity was further explored by comparing weeks one through six for both groups of externs (Table 2). The mean number of procedures
Weeks 7-10
Mean ±sd
Oral exam 15 ±9 Intraoral x-ray 11 ±18 Extraction 6 ±5 Two-surface amalgam 6 ±5 5999 (denture- 5 ±7 related procedure)
provided by the ten-week externs was 178 ±74 and statistically significantly more than that of the sixweek externs (119 ±64) over the first six weeks of the externship (p=0.04). During this six-week period, the seniors participating in the ten-week externships provided 46 percent more procedures per week than the six-week externs, with a mean of 29.7 ±12.4 for the ten-week externs compared to seniors in the six-week externship (20.3 ±10.5). Figure 1 shows the mean clinical productivity each week over the externship. As seen in the figure, for weeks one through six, the mean number of procedures for the ten-week externs was consistently higher than that of the six-week externs, although the clinical productivity of six- and ten-week externs was not statistically significantly different in weeks one, two, four, and six. In weeks three and five, ten-week externs provided a statistically significantly greater number of procedures per week compared to six-week externs (p=0.01 and p=0.03, respectively). The total number of patient visits for six- and ten-week externs over the entire externship period was 1,505 and 1,675, respectively. As seen in Table 3, six- and ten-week externs had a similar number of
Table 2. Clinical productivity of senior students during the six-week and ten-week externships Total number of procedures completed over entire externship Mean number of procedures provided by each extern over entire externship Mean number of procedures per week over entire externship period Total number of procedures completed over weeks 1-6 Mean number of procedures provided by each extern over weeks 1-6 Mean number of procedures per week during weeks 1-6
April 2007 ■ Journal of Dental Education
Six-Week (N=16)
Ten-Week (N=10)
P-value
1,898 119 ±64 20.3 ±10.5 1,898 119 ±64 20.3 ±10.5
2,644 264 ±106 27.1 ±10.7 1,779 178 ±74 29.7 ±12.4
— 0.0002 0.12 — 0.04 0.049
519
Mean Number of Procedures
40 35 30 25 6 week externs 10 week externs
20 15 10 5 0 1
2
3
4
5
6
7
8
9
10
Weeks Figure 1. Clinical productivity of senior students during the six-week and ten-week externships
patient visits per week; however, the ten-week externs completed a greater number of procedures during each patient visit (p=0.05), with a mean of 1.6 ±0.7 compared to 1.2 ±0.3 by the six-week externs. Figure 1 also shows the clinical productivity of ten-week externs over the entire ten weeks. As seen in the figure, the mean productivity in weeks one through six was higher when compared to weeks seven through ten, with an average of 29.7 ±12.4 procedures performed in weeks one through six and 22.5 ±9.1 in weeks seven through ten. This difference was statistically significant (p=0.02). To correct for inherent differences between sites such as patient mix and clinical support staff available, all of the above analyses were repeated for students in the six-week and ten-week extern-
ships attending the same externship sites. There were six sites that participated in both the six-week and ten-week externships. These six sites hosted eight students from the six-week externship and eight students from the ten-week externship. Table 4 and Figure 2 illustrate these results. The findings reported above, that ten-week externs provided a greater number of procedures per week than the six-week externs, were confirmed (p=0.0001). Although not statistically significant, ten-week externs saw more patients per week than the six-week externs (Table 5). In contrast to the results using all students, analyses for students attending the same sites reflected no difference between the two groups with respect to number of procedures completed per patient (Table 5).
Table 3. Patient visits for six-week and ten-week externships
520
Six-Week (N=16)
Ten-Week (N=10)
P-value
Total number of patient visits over entire externship Mean number of patient visits per week Mean number of procedures per patient visit
1,505 16.1 ±6.2 1.2 ±0.3
1,675 17.1 ±3.9 1.6 ±0.7
— 0.65 0.05
Journal of Dental Education ■ Volume 71, Number 4
Table 4. Clinical productivity of senior students assigned to the same sites during the six-week and ten-week externships Total number of procedures completed over entire externship Mean number of procedures provided by each extern over entire externship Mean number of procedures per week over entire externship period Total number of procedures completed during weeks 1-6 Mean number of procedures provided by each extern during weeks 1-6 Mean number of procedures provided per week during weeks 1-6
Six-Week (N=8)
Ten-Week (N=8)
P-value
837 104.6 ±58.3 18.2 ±9.6 837 104.6 ±58.3 18.2 ±9.6
1,755 219.4 ±52.5 22.7 ±6.2 1,189 148.6 ±46.4 24.8 ±7.7
— 0.0001 0.20 — 0.05 0.08
35
30
Number of Procedures
25
20 6 week 10 week
15
10
5
0 1
2
3
4
5
6
W eeks
Figure 2. Clinical productivity of senior students during the six-week and ten-week externship at the same sites during weeks one through six
Table 5. Patient visits for six-week and ten-week externships at the same sites
Six-Week (N=8)
Ten-Week (N=8)
P-value
Total number of patient visits over entire externship Mean number of patient visits per week Number of procedures per patient visit
634 13.8 ±4.1 1.3 ±0.3
1,286 16.5 ±2.5 1.4 ±0.3
— 0.07 0.42
April 2007 ■ Journal of Dental Education
521
Discussion In previous studies that employed interviews and journals (reflections) to collect data, dental students have reported that they had a greater sense of confidence and felt more efficient, skilled, and better prepared to enter the dental profession as a result of their externship experience.3 These student selfreports, while invaluable, have not been tested, nor have they been verified by objective clinical data. Our comparison of six-week and ten-week externships using quantitative measures strives to substantiate these qualitative findings. After careful analysis of the procedures performed during the six-week and ten-week externships, it was evident that longer externships result in greater clinical productivity and therefore more clinical experience. The mean number of procedures performed each week by ten-week externs (27.1 ±10.7) was much higher than that of six-week externs (20.3 ±10.5). On average, students in the ten-week externship performed 35 percent more procedures than those in the six-week program over the entire externship. This finding did not change even when comparing weeks one through six for both groups of externs. Ten-week externs provided 46 percent more procedures per week than the six-week externs for weeks one through six. For each of the six weeks, tenweek externs provided consistently more procedures than did six-week externs (Figure 1). The reasons that the ten-week students completed more procedures in the first six weeks was because they did more procedures per patient (Table 3). Although not statistically significant, students in the ten-week externship saw a few more patients each week than those in the sixweek externship (Table 3 and 5). Additionally, sites prefer students to stay longer and possibly scheduled the ten-week students differently than they do students on shorter duration externships. In addition to greater productivity, longer externships allowed students to perform more involved and/or complex treatments, requiring multiple appointments. Although, in the first six weeks, there were no differences seen between the six-week and ten-week externships in the type of procedures performed, in weeks seven through ten more removable prosthodontic procedures were performed as evidenced by a higher number of ADA code 5999, which was used to indicate a continuing removable prosthodontic procedure (Table 1). Additionally, the ten-week externship allowed students to perform
522
more removable prosthodontic procedures such as complete dentures and cast metal partial dentures. Although the clinical productivity was greater in the ten-week externships, a decline in productivity was seen in the latter part of the externship. Moreover, the maximum clinical productivity in the ten-week externship was seen at week five, with declining productivity until week ten, particularly in weeks seven and nine (Figure 1). Some decline in productivity is expected as students prepare to return to the dental school. However, a higher decline was seen than expected. This finding is perhaps due to students performing procedures that required more time to complete and is substantiated by the finding that in weeks seven through ten more prosthodontic procedures were performed. In addition, it is possible that students were not being assigned as many new patients in the latter part of their externship because of their impending departure so as to allow time to complete the patients or cases they had started. Furthermore, students on the ten-week externship had additional responsibilities or assignments, including a case presentation and public health project that needed to be completed by the end of the externship, which would have taken time away from seeing patients and may account for the decline in productivity. When patient visits were analyzed, we found that ten-week and six-week externs had similar numbers of patient visits per week; however, ten-week externs completed a greater number of procedures during each patient visit. Scheduling of patient visits is generally the responsibility of the site, not the student. Therefore, little difference is seen between ten-week and six-week externships (Table 3). This was further confirmed when students in ten-week and six-week externships at the same sites were evaluated (Table 4 and 5 and Figure 2). Ten-week and six-week externs at the same sites did not show any differences with respect to number of procedures completed per patient. There are several limitations with our study or other factors that could explain results that need to be considered. There are variations in student ability and motivation, or the students in the ten-week externship could have been stronger students with higher academic or clinical grade point averages. The results presented here could not account for these student factors, which are difficult to account for in any study. However, higher grade point averages do not necessarily mean higher clinical productivity, which is the main outcome measured in this study. Further, since the students were from the same graduating cohort and an externship rotation is a requirement for all
Journal of Dental Education ■ Volume 71, Number 4
students, who are assigned to rotations only if faculty deem them to be competent to go on rotation, there is no reason to believe that the two groups of students were different. Sample size is another limitation in the current study as there were just ten and sixteen students in each group. This could have accounted for some of the clinically significant differences seen between groups not being statistically significant. Additionally, this could affect the generalizability of our results. The other limitation is reporting bias. Students were required to enter productivity data at the end of each day. Students were told that the completeness and accuracy of the data would be monitored and verified; however, we were not able to verify this data both for completeness and accuracy because it would be very time-consuming and an inordinately difficult task given the available manpower. From student focus groups, debriefing sessions, and feedback, we learned that not all data may have been entered each day, as students were kept very busy and were pressed for time during both types of externships. Additionally, students are not evaluated by the number of procedures performed while on externship. Therefore, we believe the results for productivity presented here are potentially an underestimate of the true productivity. If all data had been entered, then the productivity for both groups would certainly be higher. Because the productivity of both groups would increase, it is unlikely that the relationships between six-week and ten-week externships would have changed. Although less likely from our perspective, it is conceivable that students overestimated their productivity, but again there is no reason to believe that only one group would have done so. Therefore, the relationships seen here would remain.
Conclusion The conclusions of this study are as follows: • Our results demonstrate that more overall procedures and procedures per week were performed
April 2007 ■ Journal of Dental Education
in the ten-week externship than the six-week externship. • With an additional four weeks on externship, students were able to perform more complex procedures. • Scheduling of patients and number of procedures performed per patient were generally determined by the policies and management of each site and were not under the control of the students.
REFERENCES 1. Skelton J, Mullins MR, Kaplan AL, West KP, Smith TA. University of Kentucky community-based field experience: program description. J Dent Educ 2001;65(11):1238-42. 2. Heise AL, Lewis GP, Bader JD, Harris JH. Experiential education for dental students at the University of Kentucky. J Dent Educ 1976;40(5):272-5. 3. DeCastro JE, Matheson PB, Panagakos FS, Stewart DC, Feldman CA. Alumni perspectives on communitybased and traditional curricula. J Dent Educ 2003;67(4): 418-25. 4. Born DO, DiAngelis AJ. Extramural education programs in the ‘80s. J Dent Educ 1986;50(12):731-3. 5. Shreve WB, Clark LL, McNeal DR. An extramural dental education program in a rural setting in Florida. J Community Health 1989;14(1):53-60. 6. Ayers CS, Abrams R, Robinson M. U.S. and Canadian dental school involvement in extramural programming. J Dent Educ 2001;65(11):1272-7. 7. Thind A, Atchison K, Andersen R. What determines positive student perceptions of extramural clinical rotations? An analysis using 2003 ADEA senior survey data. J Dent Educ 2005;69(3):355-62. 8. McMillan WB, Foglesong HD. An evaluation of the University of Minnesota summer rural dental externship program. J Public Health Dent 1975;35(4):260-5. 9. Caine RL. The dental externship: an educational methodology for extramural experiences. J Dent Educ 1975;39(11):737-9. 10. Pride JR, Chambers DW. An expanded model for extramural dental clinics. J Dent Educ 1977;41(4):191-5. 11. Henshaw MM, Frankl CS, Bolden AJ, Mann ML, Kranz SM, Hughes BL. Community-based dental education at Boston University Goldman School of Dental Medicine. J Dent Educ 1999;63(12):933-7.
523
