AAIM is the largest academically focused specialty organization representing departments of internal medicine at medical schools and teaching hospitals in the United States and Canada. As a consortium of five organizations, AAIM represents department chairs and chiefs; clerkship, residency, and fellowship program directors; division chiefs; and academic and business administrators as well as other faculty and staff in departments of internal medicine and their divisions.
Faculty Characteristics Affect Interview Scores During Residency Recruitment
Julie Oyler, MD,a Katherine Thompson, MD,a Vineet M. Arora, MD, MPP,a Jerry A. Krishnan, MD, PhD,b James Woodruff, MDa a
Department of Medicine, University of Chicago, Chicago; bDepartment of Medicine, University of Illinois, Chicago.
Faculty interviews are a key component of the residency selection process. There has been much debate about how to use faculty interview scores during the ranking process.1-8 Most studies have focused on gender bias,9 diversity issues,10 interview date bias,11 or the decision to blind faculty to residency candidates’ characteristics to achieve an unbiased evaluation.3-12 There has also been controversy evaluating whether a structured interview can better predict candidate competency.13,14 Extensive literature has addressed whether residency interviews, United States Medical Licensing Examination (USMLE) scores, election to Alpha Omega Alpha (AOA), medical school reputation, clinical grades, and publications can predict future success in residency.15-28 Finally, strong comparative statements in candidates’ letters of recommendation were associated with higher professionalism scores during internship.29 Funding: None. Conflict of Interest: None. Authorship: All authors had access to the data and played a role in writing this manuscript. University of Chicago Institutional Review Board approval was obtained. Presented at: the National Society of General Internal Medicine meeting, May 6, 2011, Phoenix, AZ; the Society of General Internal Medicine Midwest Regional meeting, September 16, 2011, Chicago, IL, where it won an award as the Best Medical Education Innovation Poster; and the National Association of Program Directors in Internal Medicine meeting, April 23, 2012, Atlanta, Georgia. Requests for reprints should be addressed to Julie Oyler, MD, 5841 S. Maryland Ave., MC 3051, Chicago, IL 60637. E-mail address:
[email protected]
However, few studies have focused on how faculty characteristics might affect the candidates’ interview score. A survey of all general surgery programs found the candidate interview to be the most important factor in determining final selection.30 Another study described the “Bob effect,” in which substituting 2 faculty members with the same name, one with universally positive candidate interview scores and one with negative scores, could shift the applicant 30 places on a rank list of 150. The authors of this study proposed that faculty could be assigned a “bias” factor in a new ranking process.31 Finally, a meta-analysis of the relationship between residency selection strategies and subsequent trainee performance found that faculty interview ratings had lower than expected association with performance given the high value often assigned to them during resident selection.32 Because of the lack of data on faculty characteristics and their effect on candidate interview scores, we collected candidate and faculty data over 5 years. The aim of this study was to identify candidate and faculty characteristics that lead to variation in residency candidate interview scores to improve the residency selection process.
MATERIALS AND METHODS Study Setting and Population We conducted a retrospective cohort study using interview data from candidates interviewed at the University of Chicago Internal Medicine Program from
0002-9343/$ -see front matter Ó 2015 Alliance for Academic Internal Medicine. All rights reserved. http://dx.doi.org/10.1016/j.amjmed.2015.01.025
AAIM Perspectives
AAIM Perspectives
546
The American Journal of Medicine, Vol 128, No 5, May 2015
September 2004 to March 2009. A total of 15,485 apFaculty Measures. Faculty characteristics were obplications were received through the Electronic Resitained from a publically available department website. dency Application Service (ERAS) for preliminary, Faculty sex and medical school/PhD program location categoric, and MD/PhD candidate positions. Of these, were recorded. Estimated ranking for faculty medical 2277 candidates (15%) were invited to interview, and school/PhD programs were used according to the U.S. 1921 invited candidates (84%) completed the interview News & World Report “Best Medical Schools day. Each candidate was interResearch” category rankings.34 viewed by 1 to 2 faculty memMedical schools/PhD ProPERSPECTIVES VIEWPOINTS bers and evaluated by an grams were grouped into 4 Internship Selection Committee Both candidate and faculty characteris- categories: US Top 25, US Top (ISC) blind ERAS packet retics contribute to scores used to evaluate 26 to 50, US Below 50, and view later during the interview foreign medical schools. Facinternal medicine residency candidates. season. ISC members are core ulty were categorized on the Certain faculty characteristics (age/ basis of duration of MD or PhD faculty members who perform more than 20 interviews per experience, fellowship, and PhD training) status as follows: 10 years, 11 year and receive 2 hours of are associated with higher interview to 20 years, and 21 years. training regarding program Faculty completion of fellowscores for residency candidates. goals, updates, and guidelines ship and PhD programs was Selection committees should consider recorded. Faculty title was regarding program values and these findings when assigning value to recorded every year to reflect candidate scoring. MD/PhD faculty interview scores during the rank changes in title due to promocandidates received up to 7 inprocess. person interviews, of which 1 tion. Finally, heavy involveto 3 were with PhD faculty. ment in the admission process Faculty interviewers were assigned to interview candiwas recorded from a list of faculty participating in the dates according to faculty availability and candidates’ ISC at any time during the 5-year study period research interests. Interviewers received an electronic (Table 2). copy of the candidate’s ERAS application before the interview. After the interview, faculty electronically rated applicants on a 1 (least qualified) to 10 (most Table 1 Characteristics of the 1921 Internal Medicine qualified) scale and gave general comments. During Residency Candidates blind ERAS packet reviews, ISC members electronically receive the candidates’ ERAS application and do not Residency Candidate Characteristics meet the candidate in person, but they use a similar (N ¼ 1921) N (%) scoring system as described earlier. Institutional review Female 916 (48%) board approval was obtained, and data were de-identified PhD 179 (9%) before analysis. Other Graduate Degree 215 (11%)
Study Measures Candidate Measures. Candidate characteristics were electronically abstracted from the ERAS database. Candidate sex, year applied to internal medicine residency, PhD degree, and presence of a higher degree including JD, MBA, MPH, or other Masters degree were recorded. Candidate race was not included because it was self-reported and not available for all applicants. Candidate medical school was recorded and ranked according to the U.S. News & World Report “Best Medical Schools Research” category rankings.33 Medical schools were grouped into 4 categories: US Top 25, US Top 26 to 50, US Below 50, and foreign medical schools. ERAS fields documenting election to AOA, USMLE Step 1 scores, and publications (any poster, paper, or presentation listed in the ERAS publication field was counted as publications) were recorded (Table 1).
(eg, MBA, JD, MPH) Medical School Location and Rank US Top 25 US 26 to 50 US Below 50 Foreign AOA Member Have Publications* USMLE 1 Score† Min, Max Mean (SD) Median (Q1, Q3) Year Applied 2005 2006 2007 2008 2009
963 370 565 23 652 1383
(50%) (19%) (30%) (1%) (34%) (72%)
174, 273 233 (17) 235 (221, 246) 328 326 456 392 419
(17%) (17%) (24%) (20%) (22%)
AOA ¼ Alpha Omega Alpha; SD ¼ standard deviation; USMLE ¼ United States Medical Licensing Examination. *Data on publications missing in 2/1921 ¼ 0.1%. †USMLE 1 score missing in 349/1921 ¼ 18.2%.
Oyler et al
Faculty Interview Scores During Residency Recruitment
Table 2 Characteristics of the 314 Faculty Interviewers in Their First Year of Interviewing Faculty Characteristics (N ¼ 314 Faculty)
N (%)
Female PhD Fellowship Medical School or PhD Program Location and Rank US Top 25 US 26 to 50 US Below 50 Foreign Years as an MD or PhD Category 10 11-20 21 Years as MD or PhD Min, Max Mean (SD) Median (Q1, Q3) Title Fellow Clinical Associate Clinical Instructor Assistant Professor Associate Professor Professor Heavy Involvement in Admissions Process*
128 (41%) 53 (17%) 196 (62%)
145 33 89 47
(46%) (11%) (28%) (15%)
126 (40%) 105 (33%) 83 (27%) 2, 58 15.3 (10.6) 12 (7, 21) 14 23 60 101 57 59 34
(5%) (7%) (19%) (32%) (18%) (19%) (11%)
SD ¼ standard deviation. *Indicates faculty participating in the ISC during the 5-year study period.
Statistical Analysis Sample ranges, means, standard deviations, quantiles, and proportions were used to describe the distributions of faculty and candidate characteristics. A multivariate mixed model was fit to the data using individual faculty evaluations of candidates as the response variable and the predictor variables listed in Table 3. Model assumptions were assessed using conventional residual plots, and no evidence of problematic collinearity among the model predictors was found. All hypothesis tests were 2-sided at the a ¼ 0.05 level. SAS 9.2 (SAS Institute Inc, Cary, NC) was used for all analyses.
RESULTS There were 5651 individual interview scores of the 1921 candidates from 314 faculty members. Of the 5651 interviews performed by individual faculty, 3812 (67%) were in-person interviews, 236 (5%) were MD/ PhD interviews, and 1603 (28%) were blind ERAS packet reviews. Some 80% of the 1921 candidates had 3 interviews each, typically including 2 in-person faculty interviews and 1 ISC blind ERAS packet review. Some 15% of candidates had 2 interviews, 1.5% had 1
547
interview, and 3.6% had >3 interviews. Only 2 candidates (0.1%) had missing publication data, and 349 candidates (18.2%) had missing USMLE data.
Candidate Characteristics Of the 1921 medical school candidates interviewed, 916 (48%) were female, 179 (9%) had a PhD, and 215 (11%) had another graduate degree (Table 1). Many candidates were from US top 25 medical schools (50%), and few (1%) were from foreign medical schools. Although 16% of each medical school class can be elected to the AOA,34 34% of the candidates in this study had been elected to the AOA. The national USMLE average for US medical students is 225,35 and the mean USMLE 1 score for candidates in this study was 233 (standard deviation, 17.5).
Faculty Characteristics A total of 314 faculty interviewed the 1921 candidates over the 5-year study period. Some 41% of the faculty were female, 17% had a PhD, and 62% had completed fellowship (Table 2). Many faculty were from US Top 25 medical schools (46%) and 15% were from foreign medical schools. Faculty had completed medical school a mean of 15 years before (range, 2-58 years). Many faculty (40%) were within their first decade in practice, although 33% of faculty were in their second decade of practice and 26% were in or beyond their third decade in practice. Many of the faculty interviewers (32%) were Assistant Professors, and 11% of the faculty had heavy involvement in the admission process, meaning they had been on the ISC at some point during the 5-year study period.
Multivariate Analysis Finally, the effect of candidate and faculty characteristics on individual faculty scores of candidates (Table 3) was evaluated using multivariate mixed model. Multivariate analysis showed that on a scale from 1 to 10, AOA membership (þ0.48, P < .01) and USMLE score (þ0.10 for every 10 points above the USMLE mean of 220, P < .01) had the most significant positive estimated effect, and medical school rank (US Medical Schools 26 to 50, 0.26, P < .01; US Medical Schools Below 50, 0.55, P < .01; Foreign Medical School 0.22, P ¼ .26) had the most negative estimated effect on the candidate score given by individual faculty. Individual faculty scores also reflected a favorable disposition toward female candidates (þ0.10, P ¼ .02), PhDs (þ0.48, P < .01), publications (þ0.41, P < .01), and an in-person interview (þ0.10, P < .02). For example, an MD/PhD and AOA candidate with multiple publications, from a US Top 25 medical school with an USMLE score of 260 would receive 1.77 points
548
The American Journal of Medicine, Vol 128, No 5, May 2015
Table 3 Effects of Residency Candidate and Faculty Characteristics on Individual Faculty Scores of Candidates in Multivariate Mixed Model Residency Candidate Characteristic
Estimated Effect on Score (95% CI)
P Value
Female candidate No other education PhD Other graduate degree (eg, MBA, JD, MPH) Candidate Medical School or PhD Program Location and Rank US Top 25 US 26 to 50 US Below 50 Foreign AOA member Has publications* USMLE 1 score Centered at 220, scaled by 10† In-person interview
0.10 (0.02-0.19) (Reference) 0.48 (0.33-0.63) 0.12 (0.01 to 0.25)
.02 e
