Assessing Competence of Residents to Discuss End-of-Life Issues

JOURNAL OF PALLIATIVE MEDICINE Volume 8, Number 2, 2005 © Mary Ann Liebert, Inc.

Assessing Competence of Residents to Discuss End-of-Life Issues MARY KATHLEEN BUSS, M.D.,1 G. CALEB ALEXANDER, M.D.,2 GALEN E. SWITZER, Ph.D.,3 and ROBERT M. ARNOLD, M.D.4

ABSTRACT Background: Residents are often responsible for eliciting patients’ treatment preferences at the end of life (EOL), yet we have a limited understanding of their competence in this task. Objective: To assess the competence of medical residents to discuss advance directives (AD) with patients using two measures: self-assessment (perceived competence) and self-reported behaviors (behavioral competence). To examine the relationship between educational experiences and these two measures of competence. Design: Cross-sectional self-report questionnaire. Subjects: Internal medicine residents from two university- and one community-based program. Results: The 282 respondents (84% response) had an average of 6.2 EOL discussions per month. Few residents reported having received useful feedback from a resident (8%) or an attending (7%) about their ability to discuss ADs. Even fewer reported that work rounds (4%) or attending rounds (5%) were frequently forums for learning about EOL care. Mean perceived competence was 3.8 (range, 1–5). In multivariable analyses, greater perceived competence was significantly associated with higher postgraduate year (p 0.001), having residents demonstrate exemplary AD discussions (p 0.001), and less formal education (p 0.01). Behavioral competence was significantly associated with reporting that work rounds were useful for learning about EOL care (p 0.002), less formal education (p 0.02) and a greater number of EOL discussions per ward month (p 0.009). The correlation between perceived and behavioral competence (r 0.25, p 0.001) was modest but statistically significant.

1Divisions

of Pain and Palliative Care and Thoracic Oncology, Department of Medical Oncology, Dana Farber Cancer Institute, Harvard Medical School, Boston, Massachusetts. 2Robert Wood Johnson Clinical Scholars Program, MacLean Center for Clinical Medical Ethics, and the Department of Internal Medicine, University of Chicago Hospitals, Chicago, Illinois. 3Department of Medicine Measurement Core Co-Chief, Center for Health Equity Research and Promotion, Veteran’s Affairs Pittsburgh Healthcare System, Division of Internal Medicine, University of Pittsburgh, Pittsburgh, Pennsylvania. 4Department of Medicine, Leo H. Criep Chair in Patient Care, Division of General Internal Medicine, Section of Palliative Care and Medical Ethics, Center for Bioethics and Health Law, Institute for Performance Improvement, University of Pittsburgh School of Medicine, Pittsburgh, Pennsylvania. Some material in this article was presented in abstract form at the 2001 meeting of the American Society of Clinical Oncologists and can be found under the following reference: Buss MK, Switzer GE, Alexander GC, Brufsky A, Arnold RM. Housestaff education on end-of-life issues. Poster presentation. Proceedings of American Society of Clinical Oncology. 2001;20:1608.

363

364

BUSS ET AL.

Conclusions: Many residents view themselves as competent to discuss ADs with patients but fail to engage in recommended behaviors for such discussions. Increasing experiential learning may be the most promising means of enhancing residents’ abilities to discuss EOL issues with patients. INTRODUCTION

I

NTERNAL MEDICINE housestaff feature prominently in the care of dying patients and are often responsible for the difficult task of eliciting patients’ treatment preferences at the end of life (EOL). Studies reveal that, upon entering residency, many trainees feel unprepared to discuss EOL issues with patients.1 Additionally, many training programs fail to meet the requirements for EOL training outlined by the Accreditation Council for Graduate Medical Education.2 Efforts to improve palliative care education in medicine residencies are in progress.3 Assessing competence to provide EOL care is a challenge. The most common mode of measuring residents’ competence is through self-evaluation of one’s ability. Educational interventions have been shown to improve residents competence based on this self-assessment.4–6 However, by relying on residents’ self-assessment, these studies potentially equate effectiveness of training with personal confidence or satisfaction with the training. Other literature argues that self-reported confidence or competence does not reflect true ability or performance and that a more accurate measure of competence may be actual (self-reported or observed) behaviors discussing EOL care.7,8 Residents may report feeling “confident” or “adequately trained” in obtaining do-not-resuscitate (DNR) orders, yet may omit important components from their advance directives (AD) discussions.9 Experts recommend that AD discussions include: a query about possible surrogates, focus on positive goals or values,10 provide information about the success of different therapies and elicit treatment preferences in a variety of scenarios to ascertain “states worse than death.”11,12 However, Tulsky et al.13 showed that residents frequently failed to include many of these components in discussions with patients about cardiopulmonary resuscitation (CPR) preferences.13 In this study, we examine residents’ competence using two different measures: self-assessment (perceived competence) and self-reported

behaviors (behavioral competence). In addition, we examine the relationship between these two measures as well as each measure’s association with residents’ sociodemographic characteristics and educational experiences.

METHODS Subjects: We included all internal medicine housestaff at the University of Pittsburgh Medical Center Shadyside Hospital, University of Pittsburgh Medical Center Presbyterian Hospital, and the Hospital of the University of Pennsylvania. We excluded the investigators (M.K.B. and G.C.A.). Each subject received a survey with a $2 cash incentive in the mail. Nonrespondents received a second mailing (all programs) and a third mailing (University of Pittsburgh programs) at 3week intervals. At all sites publicity at housestaff conferences and through e-mail encouraged voluntary participation. Institutional Review Boards at participating institutions deemed the study exempt from further review.

Measures We designed a questionnaire to solicit internal medicine residents’ attitudes and experiences regarding AD discussions with patients. The questionnaire defined such a discussion broadly, “any discussion with patients or families regarding medical treatment decisions at the end of life including DNR/DNI (do not resuscitate/do not intubate) status as well as the less tangible issues surrounding death.” The questionnaire included domains examining: (1) perceived competence, (2) residents’ EOL educational experiences, (3) specific content of AD discussions, and (4) sociodemographic characteristics of respondents. Items on perceived competence included self-rated competence, comfort, and knowledge. Questions concerning educational experiences included the frequency of formal, didactic lectures as well as the frequency of informal curricular activities, such as observing others, being observed, and getting

ASSESSING COMPETENCE OF RESIDENTS TO DISCUSS END-OF-LIFE ISSUES

feedback on AD discussions. Questions on specific content included the frequency of addressing possible interventions, such as resuscitation, defibrillation, and intubation, as well as quality of life issues, such as states considered worse than death, comfort with taking risks and fears about dying. Items regarding residents’ experiences and the specific content of AD discussions used five-point Likert scales ranging from almost never to almost always or strongly disagree to strongly agree. The questionnaire (available upon request) was developed in consultation with a literature review and a palliative care expert who assessed content validity. A psychometrician evaluated individual items for clarity. Field pretesting with debriefing guided further revision of the final instrument.

DATA ANALYSIS Defining perceived competence: To construct the perceived competence variable, we conducted a principal components factor analysis with 14 questionnaire items. The analysis revealed a single stable factor representing perceived competence. This factor comprised three items: (1) “I feel competent to discuss EOL issues with patients,” (2) “I have adequate knowledge to discuss EOL issues with patients,” and (3) “I feel comfortable discussing EOL issues with patients,” which were averaged to form a composite scale with good internal reliability (Cronbach 0.84).

Defining behavioral competence Based on the literature cited in the introduction, as well as discussions with other experts in EOL care, we identified key elements for a competent AD discussion that fell broadly into following four different domains: 1. The patient’s desired surrogate if he/she becomes incompetent (one item); 2. The kind or extent of risks (chance of success of an intervention) the patient is comfortable taking (one item); 3. The patient’s preferences regarding at least one of the following four interventions (CPR, chest compressions, defibrillation, intubation or mechanical ventilation) (highest rating of these four items); 4. The patient’s beliefs regarding at least one of the three following values (states worse than death, most feared outcomes of possible treat-

365

ment, or what was most feared about death and dying) (highest rating of these three items). For each item in the above four domains, we asked residents how often they discussed the subject. Because the last two domains (numbers 3 and 4 above) had multiple items, we took the most frequently reported item on the Likert scale response. For example, looking at domain 4 above, if a resident reported “almost never” (Likert scale 1) asking a patient her thoughts on “states worse than death” and “what she fears most about dying,” but “nearly always” (Likert scale 5) inquiring about the “outcomes of treatment most feared,” then the resident would receive a score of 5 for this domain. In order to develop a continuous scale of behavioral competence, we summed the Likert scale responses for the two domains with one item with the most frequently reported item in each domain with multiple items. Thus, we derived a scale from 4–20, with a higher value representing greater behavioral competence.

Bivariate and multivariate analyses For the purposes of bivariate analyses, responses “most of the time” and “nearly always” on the Likert scale were collapsed and subsequently reported as “frequent.” We examined the distribution of our two outcome variables—perceived competence and behavioral competence. Then, we used t tests and nonparametric correlation coefficients to examine the bivariate relationships between respondent sociodemographic and educational variables and our primary outcome variables. Finally, we used multiple linear regression to examine the multivariate relationships. In our model, we included predictors that were of at least borderline significance on bivariate analysis (p 0.10). We excluded some educational variables (Tables 1 and 2) due to collinearity with other predictor variables. Analyses were conducted using SPSS (SPSS, Inc., Chicago, IL and JMP version 4.04 (Cary, NC).

RESULTS Respondent characteristics Of the 336 surveys mailed, 282 were returned for an overall response rate of 84%. Of the 282 respondents, the mean age was 29, 59% were male, and 63% were white (Table 3). Seventy-six per-

366

BUSS ET AL. TABLE 1. ASSOCIATIONS BETWEEN HOUSE OFFICERS CHARACTERISTICS AND PERCEIVED COMPETENCE DISCUSSING EOL CARE (n 282)*

Respondent characteristics Age 20–27 28–29 30 Sex Male Female Race White Asian Other Postgraduate year PGY-1 PGY-2 PGY-3 Training track Categorical Primary care Preliminary or transitional Career plans Subspecialty Primary care Other

Total, na

Perceived competence

83 115 84

3.72 3.84 3.82

166 116

3.84 3.74

178 83 21

3.86 3.66 3.81

113 91 74

3.51 3.87 4.15

211 23 45

3.80 3.54 3.90

139 102 34

3.80 3.80 3.73

Educational experiences Work rounds have been a forum for teaching and learning about EOL discussions. Attending rounds have been a forum for learning about EOL discussions. Discussions about EOL issues on rounds have been useful in guiding my subsequent EOL discussions. Attendings are good examples in conducting EOL discussions with patients. Residents are good examples in conducting EOL discussions. I have received useful feedback from attendings regarding my EOL discussions.b I have received useful feedback from residents regarding my EOL discussions.b Formal EOL educationc Number of EOL discussions per ward month

Bivariate p value

Regression coefficient (reference group)

p value

0.55.

—

—

0.31.

—

—

0.18.

—

—

0.0001

0.29 (per one year increase)

0.0001

0.21.

—

—

0.88.

—

—

Correlation coefficient

Bivariate p value

0.16

0.008

0.07


p value 0.23

0.25

0.06 (per increasing agreement) —

0.04

0.53

—

—

0.004

0.95

—

—

0.29

0.0001

0.22

0.001

0.21

0.001

0.18 0.34

—

0.23 0.0001 (per increasing agreement) — — —

0.08 (per additional element) 0.0001 0.02 (per additional discussion) 0.003

— 0.006 0.06

totals may be less than 282 because of missing data; model R2 0.24 excluded from multivariate model because of correlation with beliefs regarding work rounds as useful forum for teaching/learning about EOL care (r 0.39–0.42). c0 (least)–5 (most) scale based on exposure to EOL lectures or small/group discussions during medical school and residency. aColumn bItems

cent of respondents were categorical residents, while 15% were in primary care tracks and 9% were in other training programs (e.g., preliminary/transitional or medicine/pediatrics).

Educational experiences Housestaff reported a mean of 6.2 discussions with patients regarding EOL care during an av-

ASSESSING COMPETENCE OF RESIDENTS TO DISCUSS END-OF-LIFE ISSUES TABLE 2. ASSOCIATIONS BETWEEN HOUSE OFFICERS CHARACTERISTICS BEHAVIORAL COMPETENCE DISCUSSING EOL CARE (n 282)*

Respondent characteristics Age 20–27 28–29 30 Gender Male Female Race White Asian Other Postgraduate year PGY-1 PGY-2 PGY-3 Training trackb Categorical Primary care Preliminary or transitional Career plans Subspecialty Primary care Other

Total, na

Behavioral competence (range, 4–20)

83 115 83

13.63 13.35 13.77

166 115

13.36 13.84

178 83 20

13.61 13.65 12.65

113 91 74

13.04 13.51 14.31

211 23 45

13.61 14.11 11.83

139 102 34

13.45 13.96 12.56

Work rounds have been a forum for teaching and learning about EOL discussions. Attending rounds have been a forum for learning about EOL discussions. Discussions about EOL issues on rounds have been useful in guiding my subsequent EOL discussions. Attendings are good examples in conducting EOL discussions with patients. Residents are good examples in conducting EOL discussions. I have received useful feedback from attendings regarding my EOL discussions.c I have received useful feedback from residents regarding my EOL discussions.c Formal EOL educationd Number of EOL discussions per ward month

AND

Bivariate p value


p value

0.60

—

—

0.18

—

—

0.37

—

—

0.02

0.17 (per one year increase)

0.47

0.009

0.71 (primary care vs. categorical) 1.10 (primary care vs. preliminary) 0.25 (other vs. general medicine) 0.08 (other vs. subspecialty)

0.06

0.06

Educational experiences

367

Correlation coefficient

Bivariate p value

0.22

0.001

0.06


0.08 0.44 0.78

p value 0.002

0.35


0.10

0.11

—

—

0.09

0.18

—

—

0.11

0.06

0.410

0.20

0.001


0.14

0.02

—

—

0.20

0.001

0.020

0.24

0.0001

0.26 (per additional element) 0.08 (per additional discussion)

—

—

0.009

aColumn

totals may be less than 282 because of missing data. differences persisted when analysis limited to PGY-1 trainees, model R2 0.15. cItems excluded from multivariate model because of correlation with beliefs regarding work rounds as useful forum for teaching/learning about EOL care (r 0.39–0.42). d0 (least)–5 (most) scale based on exposure to EOL lectures or small/group discussions during medical school and residency. bSignificant

erage ward month. Approximately half of the respondents reported having lectures (51%) or small group discussions (45%) on EOL care during the preclinical years of medical school (data not

shown). Similar percentages of residents reported lectures or small group discussions during the clinical years of medical school and during residency. By comparison, informal or bedside train-

368

BUSS ET AL. TABLE 3.

RESPONDENT CHARACTERISTICS (n 282)

Age, mean (range), years Male, % Ethnicity, % White Asian Other Year in Training, % PGY-1 PGY-2 PGY-3 Site, % University of Pennsylvania University of Pittsburgh Shadyside Hospital Training track, % Categorical Primary care Preliminary or transitional Career plans, % Subspecialty Primary care Other

29 (20–46) 59 63 29 8 41 33 27 53 34 14 76 15 8 51 37 12

PGY, postgraduate year.

ing on EOL issues during residency training was much less prevalent. Few found work rounds (4%) or attending rounds (5%) a frequent forum for learning about EOL issues. Approximately one fourth of respondents reported that residents (23%) or attendings (23%) frequently served as role models for conducting EOL discussions. Even fewer reported having frequently received consistent, useful feedback from a resident (8%) or an attending (7%) about their ability to discuss EOL issues. Over half had never been observed or received useful feedback from an attending (63%) or a resident (57%) after an EOL discussion.

Perceived competence Most residents reported comfort (81%), competence (79%) and adequate knowledge (61%) to discuss EOL issues with patients. On a scale of 1 to 5 (5 highest competence), the mean perceived competence score was 3.8 (standard deviation 0.80). As illustrated in Table 1, on bivariate analysis greater perceived competence was associated with postgraduate year (PGY) but not with the other sociodemographic characteristics examined. In addition, there was a weak but statistically significant association between perceived competence and several educational exposures. Surprisingly, greater formal education was associated with lower rather than greater perceived competence to discuss EOL care. On

multivariate analysis, greater perceived competence remained statistically significantly associated with postgraduate year, the belief that residents are good examples in conducting EOL discussions, and less formal EOL education (Table 1).

Association between perceived and behavioral competence On bivariate examination, there was a statistically significant, but modest correlation between perceived and behavioral competence (r 0.25, p .001). Among respondents in the highest quartile of behavioral competence, approximately 41% were in one of the lowest two quartiles of perceived competence. Conversely, among respondents in the highest quartile of perceived competence, 44% were in one of the lowest two quartiles of behavioral competence.

Behavioral competence Content of EOL Discussions. Figures 1 and 2 illustrate the frequency that residents reported addressing different subjects during their EOL discussions. For example, most residents reported that they frequently addressed CPR (86%), chest compressions (83%), defibrillation (83%), and intubation or mechanical ventilation (90%). By contrast, few residents reported frequently asking about patients’ values and goals in their EOL discussions such as states that the patient considered worse than death (19%), what (the patient) fears most about possible outcomes of treatment (16%), or how relationships with or obligations to family or friends weigh in their decision-making process (14%). Bivariate and Multivariate Analysis. On bivariate examination, several respondent characteristics and educational exposures were associated with behavioral competence (Table 2). For example, there was increasing behavioral competence among residents in training longer, ranging from a mean score of 13.04 among PGY-1 residents to a mean score of 14.31 among PGY-3 residents (p 0.02). Similarly, residents who reported being in a primary care track or who anticipated entering a primary care field were statistically more likely than their counterparts to exhibit greater behavioral competence. Finally, as with perceived competence, behavioral competence correlated significantly with several educational exposures. On multivariate analysis, greater be-


FIG. 1.

Content of end-of-life (EOL) discussions regarding interventions.

havioral competence remained statistically significantly associated with more frequently finding work rounds useful for learning about EOL care, less formal EOL education, and a greater number of EOL discussions per ward month (Table 2).

DISCUSSION To the best of our knowledge, this is the first study to examine and directly compare measures of residents’ perceived and behavioral competence to discuss EOL care across a broad population of internal medicine residents. In addition,

FIG. 2.

369

we examined predictors of competence including an extensive list of residents’ educational exposures in discussing EOL care. We found that although residents viewed themselves as competent, their reported behaviors did not conform to expert recommendations for EOL discussions. Furthermore, we found numerous missed opportunities for education regarding EOL care. Our study has a number of important implications. First, previous work has concluded that learning skills to discuss EOL care requires an awareness about one’s communication behaviors.14 However, the modest, although statistically significant association between perceived competence and reported behaviors in our study

Content of end-of-life (EOL) discussions eliciting goals and values.

370

implies that many residents lack this awareness. Residents whose perceived competence exceeds their reported behaviors may also lack motivation to learn these skills. Learning theory posits that intrinsic motivation is required for learners’ to relate concepts to everyday practice. Empirical evidence corroborates this with respect to communication skills in medical education. Marteau et al.8 found that medical students with greater confidence in their communication skills had poorer nonverbal facilitation and reduced likelihood of picking up leads from a patient. They concluded that inability to judge communication skills was a barrier to improving them. Thus, the prevalence of high perceived competence among the residents in our study may mean they are less aware of their deficiencies and less motivated to improve their skills. Perhaps, identifying the discrepancy between what residents say they talk about and what experts recommend AD discussions include would increase residents’ motivation to learn how to conduct them better. One potential way to assist residents in identifying their proficiency gaps is suggested by our finding that behavioral competence was correlated with experiential learning opportunities during residency, such as conducting more EOL discussions on an average ward month and getting useful feedback from attendings. Behaviorial learning theory, upon which competencybased education is founded, requires the creation of an environment that elicits and reinforces desired behaviors. Experiential learning in other studies has been shown to improve competency. Despite rather extensive efforts to increase awareness and develop curriculum for EOL training in residency,3,15 most residents in our sample found few examples of how to conduct AD discussions and have little to no observation or feedback on the discussions they have. Given that residents are regularly discussing ADs and CPR preferences with patients, there are many missed opportunities for education on this important topic. Surpisingly, we found that increased formal education correlated with worse behavioral and perceived competence on bivariate analyses and this remained statistically significant on multivariate analyses. The failure of formal education to enhance competence to discuss ADs corroborates other literature that shows lectures do not impact skills training16 and suggests that more lectures on AD discussions may have a negative impact on residents’ skills in EOL communica-

BUSS ET AL.

tion. The negative correlation between formal education and competence, either perceived or behavioral, was unexpected and deserves further study. One potential explanation is that didactics may stress the importance of having these discussions, but fail to provide practical pointers for conducting these discussions well. In short, didactics may tell residents what to do, but not how to do it. In contrast, experiential learning, which is positively correlated with both behavioral and perceived competence, has the potential to provide both the “what” and the “how” of conducting AD discussions. Our findings also have implications for evaluating curricula for teaching EOL communication skills. Currently, many interventions designed to improve EOL care skills rely on participants’ evaluations to measure the efficacy of the intervention.5,6 Our study, in conjunction with other literature,8,9,13,17 argue strongly against using self-rated competence as the sole or primary means of assessing residents’ skills in discussing EOL issues. Educational research that relies solely on perceived competence may draw misleading conclusions. At the least, residents should be asked to report their behaviors (as in our study) and it should be compared to recommended behaviors. Direct assessment of behaviors, either with real or standardized patients18 may be a better, although much more costly, standard for such studies. Further research examining whether self-reported behaviors correlate with actual behaviors are needed to determine how well reported behaviors (which we call behavioral competence) function as a surrogate for direct observation of behaviors. Our study has a number of limitations. First, we examined a limited number of training programs and our results may not be generalizable across all geographic regions or fields. Second, we do not have data based on direct observation of AD discussions and our results are subject to recall and social desirability bias. Third, our results may not reflect curricular changes and secular trends that have occurred since our data collection.

CONCLUSION Many medicine residents view themselves as competent to discuss ADs and CPR preferences with patients, but they fail to report behaviors recommended for such discussions. Thus, perceived competence should not be used as the sole


means of evaluating success of interventions designed to improve EOL communication abilities. Didactics, which are common in medical education, have less, and even a potentially negative impact, on EOL skills. In contrast, experiential learning, which remains rare in medical residency training, correlates with behavioral and perceived competence.

ACKNOWLEDGMENT Funded by the University of Pittsburgh Veteran Administration General Medicine Research Fund Grant.

REFERENCES 1. Ury WA, Berkman CS, Weber CM, Pignotti MG, Leipzig RM: Assessing medical students’ training in end-of-life communication: A survey of interns at one urban teaching hospital. Acad Med 2003;78:530–537. 2. Weissman DE, Block SD: ACGME requirements for end-of-life training in selected residency and fellowship programs: a status report. Acad Med 2002;77: 299–304. 3. Weissman DE, Mullan PB, Ambuel B, von Gutten C: End-of-life curriculum reform: Outcomes and impact in a follow-up study of internal medicine residency programs. J Palliat Med 2002;5:497–506. 4. Weissman DE, Ambuel B, Norton AJ, Wang-Cheng R, Schiedermayer D: A survey of competencies and concerns in end-of-life care for physician trainees. J Pain Symptom Manage 1998;15:82–90. 5. Blank LL: Defining and evaluating physician competence in end-of-life patient care. A matter of awareness and emphasis. West J Med 1995;163:297–301. 6. Bishop MC, Gores FA, Stempel J, Torrington P, Tynan C, Jaskar D, Garewal HS: A collaborative end-oflife care curriculum. Am J Hosp Palliat Care 2000; 17:137–140. 7. Stewart J, O’Halloran C, Barton JR, Singleton SJ, Harrigan P, Spencer J: Clarifying the concepts of confidence and competence to produce appropriate selfevaluation measurement scales. Med Educ 2000;34: 903–909. 8. Marteau TM, Humphrey C, Matoon G, Kidd J, Lloyd M, Horder J: Factors influencing the communication skills of first-year clinical medical students. Med Educ 1991;25:127–134.

371

9. Tulsky JA, Chesney MA, Lo B: See one, do one, teach one? House staff experience discussing do-not-resuscitate orders. Arch Intern Med 1996;156:1285–1289. 10. Fischer GS, Arnold RM, Tulsky JA: Talking to the older adult about advance directives. Clin Geriatr Med 2000;16:239–254. 11. Pearlman RA, Cain KC, Patrick DL, AppelbaumMaizel M, Starks HE, Jecker NS, Uhlmann RF: Insights pertaining to patient assessments of states worse than death. J Clin Ethics 1993;4:33–41. 12. Schonwetter RS, Walker RM, Kramer DR, Robinson BE: Resuscitation decision making in the elderly: The value of outcome data. J Gen Intern Med 1993;8: 295–300. 13. Tulsky JA, Chesney MA, Lo B: How do medical residents discuss resuscitation with patients? J Gen Intern Med 1995;10:436–442. 14. Novack DH, Suchman AL, Clark W, Epstein RM, Najberg E, Kaplan C: Calibrating the physician. Personal awareness and effective patient care. Working Group on Promoting Physician Personal Awareness, American Academy on Physician and Patient. JAMA 1997;278:502–509. 15. Weissman DE, Mullan P, Ambuel B, von Gunten CF, Hallenbeck J, Warm E: End-of-Life Graduate Education Curriculum Project. Project abstracts/progress reports—Year 2. J Palliat Med 2001;4:525–547. 16. Davis D, O’Brien MA, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisey A: Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes? JAMA 1999;282:867–874. 17. Prystowsky JB, Bordage G: An outcomes research perspective on medical education: The predominance of trainee assessment and satisfaction. Med Educ 2001;35:331–336. 18. Yedidia MJ, Gillespie CC, Kachur E, Schwartz MD, Ockene J, Chepaitis AE, Snyder CW, Lazare A, Lipkin M, Jr: Effect of communications training on medical student performance. Jama 2003;290:1157–1165.

Address reprint requests to: Mary Kathleen Buss, M.D. Division of Pain and Palliative Care, SW 312 Dana Farber Cancer Institute 44 Binney Street Boston, MA 02115 E-mail: [email protected]