Vol. 40, No. 8
579
Faculty Development
Long-term Outcomes of a Primary Care Faculty Development Program at the University of Wisconsin Craig L. Gjerde, PhD; Khin Mae Hla, MD, MHS; Patricia K. Kokotailo, MD, MPH; Barbara Anderson, MS Background and Objectives: To meet a need for primary care teachers, the Bureau of Health Professions funds faculty development programs for primary care preceptors. The purpose of this study was to determine how graduates of our faculty development program identified its long-term effect on professional outcomes. Methods: Our program was a year-long series of five weekend workshops focusing on the preparation of preceptors to teach curricular areas relatively new to medical education—evidence-based medicine, teaching skills, technology tools, doctor-patient communication, quality improvement, and advocacy. Participants included physicians in community-based practices and university-based physicians. We surveyed the first 100 graduates of our program about professional and academic outcomes they attributed to program participation. Outcomes were categorized using the Kirkpatrick evaluation model; open-ended comments were analyzed thematically. Results: Eighty responses were received (80% response rate). Ninety percent of respondents were teaching medical students and residents. Outcomes attributed to the program included improvement in teaching skills, improvement in clinical skills, intrapersonal growth and increased self-confidence, and increased interdisciplinary networking and mentoring. Ninety-one percent had recommended the program to others. Conclusions: Graduates identified positive outcomes and found the fellowship useful for developing the skills and self-confidence required of teachers. This training may be valuable for teachers in today’s learning environment. (Fam Med 2008;40(8):579-84.) Since 1978, the Bureau of Health Professions has provided grants to primary care departments for faculty development programs to increase the number and quality of primary care teachers. Academic institutions have also recognized the need for faculty development programs to provide teaching and leadership skills for their teachers.1,2 The effects of such programs on the participants’ clinical and academic skills, however, have not been well studied.3,4 Given the reductions in federal funding for Title VII, institutional cost constraints, and the production-driven physician workplace,4-8 it is appropriate to ask: “What are the long-term benefits to primary care physicians who give up hours of their professional time or family time to participate in faculty development opportunities?” While it is important to answer this question and to document the longer-term outcomes from faculty development programs, the task is challenging because (1) physician teachers work in a variety of settings From the Department of Family Medicine, University of Wisconsin.
and have different academic needs, (2) it is difficult to select appropriate nonparticipants for control groups, (3) comparisons with other programs are not always meaningful, since programs often differ substantially in content and target audiences, and (4) there is a poor response to surveys mailed long after training. Yet funding agencies and employers want evidence of program benefit to justify continued funding.5 One way to conceptualize the outcomes of a program is to use the four levels identified by Kirkpatrick:9 level 1 is the participants’ reaction to the program (eg, satisfaction); level 2 is the learning resulting from the program—changes in knowledge, attitudes, and skills; level 3 is behavior change—performance of learned skills back in the job setting; and level 4 is organizational impacts (eg, better patient care, job retention). Kirkpatrick points out value at each level but encourages evaluators to strive for higher levels. In their review of assessments in early faculty development programs, Reid and colleagues pointed out that many assessments might not be “adequate to convince potential funders and participants of program effective-
580
September 2008
Family Medicine
ness” and suggested development of a set of specific outcomes that could be used in different types of programs.3 Morzinski and Simpson10 used the Kirkpatrick model to assess outcomes in their 2-year academic fellowship program for full-time faculty. Given their context, their level-3 outcomes included administrative project outcomes, leadership positions, and academic productivity (publications and presentations), and their level-4 outcome was retention in academic medicine. The purpose of this study was to assess the longterm academic and professional outcomes of clinical preceptors who completed our part-time faculty development program. We used the Kirkpatrick model as a framework for the evaluation.
Program Evaluation We have conducted extensive program evaluations and have previously reported favorable program evaluations by graduates (Kirkpatrick level 1) and short-term improvement (level 2) in fellows’ teaching, EBM, and technology skills.12,13 To assess our level-3 and level-4 outcomes, we surveyed program graduates with known addresses (n=100 of the 105) from the seven groups graduating between 1996 and 2003. We wanted to assess their reflections about the program and determine any longer-term outcomes that they would attribute to their participation in the program. Two reminders were mailed, and one e-mail reminder was sent. The study was exempted by the Institutional Review Board.
Methods Our Primary Care Faculty Development Program (PCFDP) fellowship was a year-long series of five weekend workshops. All participants were primary care physicians from Wisconsin and bordering communities who served as preceptors for medical students or residents. Participants included physicians in community-based practices and university-based physicians. Physicians who cared for underserved populations and physicians underrepresented in medicine were highly recruited, but the program was open to all qualified applicants and was filled on a first-come, first-served basis. Enrollment was capped at 20 per year. A federal grant provided participants’ registration expenses but no stipends or release time. All participants were salaried from their practice.
Instrument We designed a survey instrument (available from the corresponding author) that would capture responses to the typical kinds of scholarly outcomes reported in the literature and outcomes that might be relevant for clinician-teachers (more focused on educational and clinical leadership). We included some exploratory outcomes that we thought might be accomplished by some fellows; it was not expected that all items on the survey would achieve a high level of outcome. We also asked two open-ended questions that might evoke additional unanticipated level 3 and 4 outcomes. One question asked “In what ways have you continued with the work you reported in that project or with a followup project?” A second question asked “What are the major impacts of the PCFDP on your career as a teacher and clinician?” The survey instrument inquired about demographics, current teaching, opinions about the fellowship, specific outcomes in each of the program-related skill areas, traditional scholarly accomplishments, and the open-ended questions. For the program-related outcomes listed on the instrument, respondents placed a checkmark by accomplishments that they attributed to program participation; lack of a checkmark was interpreted as a not having achieved that outcome as a result of program participation. For the eight opinion questions, a 5-point Likert scale was used (strongly agree, agree, neutral, disagree, strongly disagree).
Program Goals and Structure The goal of the program was to prepare preceptors to be better teachers by enhancing their skills in several new content areas and teaching approaches. A core group of teaching faculty from primary care departments and the medical library provided learning sessions of 2–4 hours on program areas including evidence-based medicine (EBM), technology tools (handson library searching, using personal digital assistants [PDAs]), doctor-patient communication, culturally sensitive communication, quality improvement (QI), teaching skills (curriculum design, giving feedback, competency-based education), and advocacy. Each fellow worked with faculty mentors to integrate program content into an applied project (educational, research, clinical, or QI) and presented it on the final weekend. While other components of the grant (distance education, short course, annual workshop, community-based workshops) have changed over the years studied, the core curriculum of the fellowship had remained essentially the same with content updates in information sources and adjustments for participants’ entry-level skills (especially in EBM and technologies.)
Data Analysis The checkmark and opinion responses were analyzed by frequency, and descriptive statistics were calculated. For each open-ended question, written responses were initially classified for content by the third author, and a list of themes was generated. All responses were then coded by the other authors using those themes. Discrepancies in the coding were discussed and reconciled. The response themes generated for each question are detailed in the Results section.
Faculty Development Results Responses were received from 80 respondents (80% response rate). Table 1 shows that the mean age at entry into the program was 39.4 (SD=7.9) and age ranged from 28 to 54. A majority of respondents (53%) were females. The majority of the fellows were white (81%). Most were trained in family medicine (61%), with 24% in general pediatrics and 15% in general internal medicine. Respondents did not differ significantly from nonrespondents in specialty or gender. Ninety percent of the graduates were teaching in some capacity at the time of the survey, with 86% teaching medical students, 75% teaching residents, and 35% teaching practicing physicians. Program-related topics that graduates were teaching included doctor-patient communication (48%), EBM (39%), presentation skills (28%) and giving feedback (26%), quality improvement (18%), and cultural competence (16%). Table 2 displays responses to the question, “What has happened to you in your professional and academic life that you attribute, at least partially, to your participation in the PCFDP?” The highest responses from the program topics were being a better teacher (81%), better at practicing and teaching EBM (73%), and better at using technological tools (68%). In the professional development and leadership area, the highest responses were being more confident as a teacher (68%) and having a more active role as an educational leader (60%). As for scholarly accomplishments, 32% of graduates had presented their project at grand rounds, and 25% had presented a poster at a meeting. The percentages of respondents who responded positively to the statement “Overall, I feel the PCFDP fellowship was a worthwhile learning experience” was 94% (64% of 80 strongly agreeing and 30% agreeing); it was 91% for “I have recommended the fellowship to others” (54% of 79 strongly agreeing and 37% agreeing) and 90% for “I am satisfied with the amount I learned from the fellowship” (36% of 80 strongly agreeing and 54% agreeing.) Forty-seven of the 80 program graduates provided a written response to the question, “In what ways have you continued with the work you reported in that project or with a follow-up project?” We identified four project themes in these responses: (1) continuation or extension of original project, (2) application to patient care and clinical practice, (3) leadership in clinic and academic settings, and (4) scholarship and teaching. The first of these included responses such as “Continued teaching and data collection with anticipation of manuscript submission” and “Assigned another curricular area to revamp, while finishing my current curricular assignment.” Theme 2 included “I still use the form I developed” and “I have used my project for clinical care.” Theme 3 included “I was able to help our financial people in procuring Medicare pass-through dollars for
Vol. 40, No. 8
581
Table 1 Characteristics of 80 Primary Care Fellowship Graduates of University of Wisconsin Primary Care Faculty Development Program Between 1996 and 2003 Characteristic Gender Female Male Specialty Family medicine General internal medicine General pediatrics Age group 33 and under 34–38 39–43 44–48 49 or older Practice site Rural Small town Suburban Urban Affiliation Community practice University Ethnicity White Asian Black/African American Hispanic/Latino American Indian
n
%
42 38
53 47
49 12 19
61 15 24
25 18 12 10 15
31 23 15 13 19
9 15 20 35
11 20 25 44
31 49
39 61
65 8 3 3 1
81 10 4 4 1
Percentages may not total 100% because of rounding.
our residency program,” “The project became an important link between public health and private medical practices” and “Not at all (successful)—have tried at a university clinic setting, but without cooperation from my colleagues and administrators.” Theme 4 included “Pilot work for successful K08 award” and “I have moved on to stage II of the project—I have $9,000 in grant funding and am conducting a clinical trial.” Forty-five of the 80 graduates responded to the fundamental question “What are the major impacts of the fellowship on your career as a teacher or clinician?” Since some responses had several components, we split them into 63 discrete elements. Our coding identified three themes for the outcomes: (1) improvement in teaching and clinical skills (with 33 elements reported), (2) intrapersonal growth and increased self confidence
582
September 2008
Family Medicine
Table 2
Table 3
Percentage of Graduates Attributing Specific Skills to Their Participation in the Primary Care Faculty Development Program*
Samples of Discrete Elements Reported by 45 Graduates in Response to the Fundamental Question “What are the Major Impacts of PCFDP on Your Career as a Teacher and Clinician?”*
Skills Related to Program Teaching Areas 81% Better teacher 73% Better at practicing and teaching EBM 68% Better at using technological tools 38% Better at quality improvement tasks 33% Better at communicating with my patients 23% More skilled in practicing and/or teaching about cultural competence Professional Development and Leadership 68% More confident as a teacher 60% More active role as educational leader 36% More active role as clinical leader 30% More active in community and patient advocacy 29% More respected as a teacher 23% Interact more effectively with colleagues in primary care disciplines other than my own 18% More respected as an education leader Academics 20% Joined an education committee in the department, medical school or hospital 15% Primary care colleagues consult me more frequently regarding skills covered in the fellowship 14% Became a course director or program coordinator Scholarly Accomplishments 32% Made a presentation concerning the project at grand rounds 25% Presented an academic poster 23% Developed teaching or clinical materials that are used by others 23% Submitted a grant 20% Made a scientific presentation at a meeting 20% Submitted a manuscript to a journal 13% Received a grant 13% Had a peer-reviewed paper accepted for publication 13% Received an award or other public recognition for work * This table displays the percentage of 80 graduates who checked each skill, with percentages rounded to nearest whole integer.)
Improvement in Teaching and Clinical Skills Teaching Skills “It improved my skills in curriculum development.” “Much better at feedback and acutely aware of importance of goal setting.” “Big help with teaching and evaluating students.” “Focus on . . . stimulating self-learning among medical students, residents, and nurse practitioner candidates.” Clinical Skills “Gave me greater . . . EBM/computer skills.” “Using Internet searches to help keep current in patient care.” “I use more technology on daily basis for information for direct patient care.” “Improved understanding of quality issues.” Intrapersonal Growth and Increased Self-Confidence “I changed my professional career profile from primary care to academic teaching.” “I feel more comfortable with important faculty tasks.” “More confidence in teaching and doing projects.” “Reaffirmation of my continued search for educational and personal excellence.” “Refocus my personal search for career satisfaction.” “Confidence builder.” Interdisciplinary Networking and Mentoring “Relationships developed with . . . faculty of medical school.” “Learned the importance of having mentors and of collaborative work.” “Collaborative work.” “Networks created.” “Meeting and networking with department colleagues from other campuses on curriculum development.” * The 63 elements are clustered into three themes. PCFDP— Primary Care Faculty Development Program EBM—evidence-based medicine
EBM—evidence-based medicine
(21 elements), and (3) interdisciplinary networking and mentoring (nine elements). Sample thematic elements are reported in Table 3. Discussion Our findings showed that graduates were satisfied with the program (Kirkpatrick level 1), felt that they acquired several program-taught skills (levels 2 and 3), achieved scholarly and academic outcomes (level 3), and had remained in teaching and were now more confident and more active as educational leaders (level 4). These long-term judgments are consistent with previous high evaluations of program quality and our documentation of learning reported by participants during their programs.13
Graduates attributed several teaching and clinical skills (level 3) to program participation. These were their self-assessments of how they had applied their training in their work sites—better teaching in classroom and clinical settings, better application and teaching of evidence-based medicine in patient care, better use of technology tools, as well as other forms of educational and clinic leadership. Graduates identified two level-4 outcomes that we had not anticipated in our instrument—interdisciplinary networking and mentoring, and intrapersonal growth and increased self-confidence. We classified these as level-4 outcomes, since faculty retention in teaching and in practice are functions of career satisfaction, networking, and self confidence. While these outcomes were not explicitly asked in our questionnaire, they suggest new level-4 outcomes that we and others can follow more explicitly in future studies.
Faculty Development The written comments about overall program outcomes revealed that graduates experienced interpersonal growth and increased self-confidence from their participation in the program. They commented on outcomes such as “I changed my professional career profile from primary care to academic teaching” and “Refocus my personal search for career satisfaction.” They also benefited from networking with colleagues from other campuses and departments and from being mentored by project faculty. Their comments included outcomes of “Meeting and networking with department colleagues from other campuses” and “Learned the importance of having mentors and of collaborative work.” In their written comments about their projects, graduates described continued project work, and several of their examples focused on clinical leadership and quality improvement. The comments suggested they had positive attitudes about applied research. We believe that much of the highly valued mentoring occurred in conjunction with completion of the PCFDP project. Faculty devoted many hours to mentoring fellows. The project also provides a major take-away benefit in terms of scholarship. The program prepared graduates to tackle a task that many of them had not been trained to do—teaching the next generations of physicians. Program participation gave the graduates the confidence and skills to go back to their institutions and be leaders and innovators. The Kirkpatrick model provided a useful framework for the study; the open-ended qualitative component yielded outcomes not previously identified. As noted by Wright and associates, studies focusing on assessing only objective markers of academic success may miss “detection of deeper and more sustained changes in participants’ professional or personal growth and relationships with others.”14 The four domains of program effects that Wright reported—intrapersonal development, interpersonal development, development as a teacher and career development—were consistent with the program impacts reported by graduates in our study.14 Limitations This study has certain limitations. First, this is a study of one program that had a specific agenda and format. The participants included a mixture of community-based and university-based physicians. It may not be possible to generalize our results to other programs. Second, participants were volunteers who were interested in program specifics. Since there was no control group, we cannot be sure that some of the outcomes attributed to the program were not attributable to volunteer bias and other factors such as gender, age, or concurrent events. The high response rate to
Vol. 40, No. 8
583
our survey mitigates, to some extent, the possibility of response bias. Third, self-report may be a limitation; yet, asking participants is perhaps the only way to gather such information. In fact, some level 3 and 4 outcomes may be best assessed by asking participants to reflect back on program impacts.6, 11 Conclusions Our assessment identified several positive long-term outcomes for our graduates. The Wisconsin area now has nearly 120 primary care clinicians whose teaching skills were “upgraded” to the level of contemporary skills being taught to residents and medical students (although we believe that early graduates already need a refresher).15 In addition, this cadre of clinicians has been trained in issues of quality, practice-based learning and improvement, doctor-patient communication, advocacy, technologies, practice guidelines, and EBM in patient care, all essential components for excellence in patient care. These skills are highly compatible with the competency-focused education mandated by the Accreditation Council for Graduate Medical Education (ACGME).16 Our open-ended questions evoked additional level 4 outcomes (intrapersonal growth, increased self-confidence, interpersonal networking and mentoring) that we and some other researchers should include in future studies. Wright and colleagues have also suggested studying such outcomes.14 We believe that future studies should be done to identify whether these benefits and outcomes apply equally to community-based and university-based participants. Federal agencies and institutions that fund faculty development programs should be satisfied to see that a program can have positive impacts on preceptors’ academic and professional skills.17-20. Johnson and Barratt recently demonstrated that many pediatrician preceptors felt incapable to teach and assess certain components of the new ACGME general competencies; they suggested that the ACGME should address this need via faculty development.21 Our findings underscore the justification for and important role of federal agencies and health care institutions in enhancing medical education by continuing to financially support faculty development programs. Acknowledgment: The program has been supported by a HRSA grant from the Collaborative Faculty Development Program D55-HP-05152 and earlier grants. Corresponding Author: Address correspondence to Dr Gjerde, University of Wisconsin, Department of Family Medicine, 4284 HSLC, 750 Highland Avenue, Madison, WI 53705-2221. 608-265-6125. Fax: 608-262-2327.
[email protected].
584
September 2008 References
1. Searle NS, Hatem CJ, Perkowski L, Wilkerson L. Why invest in an educational fellowship program? Acad Med 2006;81(11):936-40. 2. Gruppen LD, Simpson D, Searle NS, Robbins L, Irby DM, Mullan PB. Educational fellowship programs: common themes and overarching issues. Acad Med 2006;81(11):990-4. 3. Reid A, Stritter FT, Arndt JE. Assessment of faculty development program outcomes. Fam Med 1997;29(4):242-7. 4. Irby DM, Hekelman FP. Future directions on research on faculty development. Fam Med 1997:29(4):287-9. 5. Muerer LN, Morzinski JA. Published literature on faculty development programs. Fam Med 1997;29(4):248-50. 6. Hewson MG, Copeland HL, Fishleder AJ. What’s the use of faculty development? Program evaluation using retrospective selfassessments and independent performance ratings. Teach Learn Med 2001;13(3):153-60. 7. Pinheiro SO, Liechty DK, Busch KV, Johnson ES, Dora DL, Butler RM. Institutional impact of a part-time faculty development fellowship program for osteopathic community-based physicians. J Am Osteopath Assoc 2002;102(11):637-42. 8. Skeff KM, Stratos GA, Mygdal WK, et al. Clinical teaching improvement: past and future for faculty development. Fam Med 1997:29(4):252-7. 9. Kirkpatrick DL. Evaluating training programs. San Francisco: BerrettKoehler Publishers, Inc, 1994. 10. Morzinski JA, Simpson DE. Outcomes of a comprehensive faculty development program for local, full-time faculty. Fam Med 2003;35(6):434-9
Family Medicine 11. Anderson WA, Stritter FT, Mygdal WK, Arndt JE, Reid A. Outcomes of three part-time faculty development fellowship programs. Fam Med 1997;29(3):204-8. 12. Feldstein DA, Hla KM, Gjerde CL, et al. An objective assessment of the effectiveness of an EBM faculty development program on clinicians’ knowledge. J Gen Intern Med 2004;19(suppl 1):118. 13. Gjerde CL, Kokotailo P, Olson CA, Hla KM. A weekend program model for faculty development with primary care physicians. Fam Med 2004; 36 (Feb suppl) 14. Wright AM, Carrese JA, Wright SM. Qualitative assessment of the long-term impact of a faculty development programme in teaching skills. Med Educ 2007;41:592-600. 15. Cole KA, Barker LR, Kolodner K, Williamson P, Wright SM, Kern DE. Faculty development in teaching skills: an intensive longitudinal model. Acad Med 2004;79(5):469-80. 16. www.acgme.org/Outcome/. Accessed December 10, 2007. 17. Ullian JA, Shore WB, First LR. What did we learn about the impact on community-based faculty? Recommendations for recruitment, retention, and rewards. Acad Med 2001;76(4 suppl):S78-S85. 18. Bland CJ, Simpson D. Future faculty development in family medicine. Fam Med 1997;29(4):290-3. 19. Quirk M, Lasser D, Domino F, Chuman A, Devaney-O’Neil S. Family medicine educators’ perceptions of the future of faculty development. Fam Med 2002;34(10):755-60. 20. DeWitt TG. Faculty development for community practitioners. Pediatrics 1996;98(6 Pt 2):1273-6. 21. Johnson CE, Barratt MS. Continuity clinic preceptors and ACGME competencies. Med Teach 2005;27(5):463-7.
