Training and Evaluating Tobacco-specific ... - Semantic Scholar

12 downloads 0 Views 215KB Size Report
Jan 28, 2006 - medical students in tobacco cessation counseling.10. Using SPIs is a ... Results: SPIs highly rated the content, organization, and presenters ...... cans, American Indians and Alaska Natives, Asian Americans and Pacific.
28

January 2006

Family Medicine

Training and Evaluating Tobacco-specific Standardized Patient Instructors Kristie Long Foley, PhD; Geeta George, MPH; Sonia J. Crandall, PhD, MS; Kathy H. Walker; Gail S. Marion, PA-C, PhD; John G. Spangler, MD, MPH Background and Objectives: A comprehensive training program to develop tobacco-specific standardized patient instructors (SPIs) was implemented and evaluated at Wake Forest University. Methods: Descriptive statistics were used to assess SPIs’ experience with the training program and medical students’ perceptions of the SPI-student interaction. Two standardized scales, used to assess student performance on counseling (Tobacco Intervention Risk Factor Interview Scale [TIRFIS]) and cultural competency (Tobacco Beliefs Management Scale-Tobacco Cultural Concerns Scale [TBMS-TCCS]), were tested for internal and interrater reliability and sensitivity to varied student performance. Costs of the program were measured. Results: SPIs highly rated the content, organization, and presenters of the training program. Medical students positively evaluated their experience with the SPIs. The TIRFIS and TBMS-TCCS subscales demonstrated good internal reliability, and inconsistencies in ratings by different SPIs were minimal. In addition, a range of scores on both measures attest to the sensitivity of the instruments to assess variations in student performance. Significant start-up costs are associated with developing this training program, although costs decline when SPIs are retained long term. Conclusions: The SPI training program was effective in developing a cohort of knowledgeable and reliable SPIs to train medical students in ways to improve their tobacco intervention counseling skills. Retaining SPIs long term should be a primary goal of implementing a cost-effective, successful training program. (Fam Med 2006;38(1):28-37.) Tobacco use is the leading cause of preventable death in the United States. Despite the well-known health hazards of tobacco, about one fourth of the US population smokes cigarettes.1,2 In addition, cigarette and smokeless tobacco use may be increasing among certain demographic groups,3-7 and the national per capita consumption of smokeless tobacco has tripled in the past several decades.8 Most smokers would like their health care provider to help them quit smoking, but clinicians often feel inadequately trained to provide the help that addicted tobacco users need.9-12 In fact, research has documented that US medical schools are inadequately preparing graduates to deal with tobacco dependence.9,10

From the Department of Public Health Sciences (Dr Foley) and the Department of Family and Community Medicine (Drs Foley, Crandall, Marion, and Spangler, and Ms Walker), Wake Forest University; and the Physician Assistant Program, George Washington University (Ms Goerge).

Explicit national guidelines are available to train physicians in tobacco cessation.13 Various methods, such as the traditional didactic approach, patient-centered methods, role playing, and the use of standardized patient instructors (SPIs), have also been developed and used with varying degrees of success in training medical students in tobacco cessation counseling.10 Using SPIs is a method that has proven especially successful.14-18 SPIs are lay individuals trained to act as patients to teach medical students, residents, and physicians history-taking skills and interviewing and counseling techniques. Few resources are available, however, to train these lay individuals to become tobacco-specific SPIs who can reliably and effectively integrate into medical education. As a result, medical educators rely on their own knowledge and skills to provide such training and evaluation of their SPI program(s). This article describes the training, use, and evaluation of tobacco-intervention SPIs in undergraduate medical education at Wake Forest University School of Medicine (WFUSM).

Medical Student Education Methods WFUSM Program WFUSM’s Integrated Tobacco Dependence Curriculum (ITDC), described elsewhere,19 began in 1993 and relies heavily on SPIs for training medical students on how to counsel patients to quit tobacco. The SPI encounter has been a central feature of the ITDC. In 2003, WFUSM received a grant from the National Cancer Institute to expand the ITDC and develop a tobacco cessation curriculum that could be disseminated to all medical schools nationally. An important goal of this grant is to expand on the existing SPI program by recruiting a cadre of ethnically diverse SPIs and training the SPIs to reliably evaluate student performance regarding smoking and smokeless tobacco counseling skills, with explicit emphasis on culturally competent counseling skills. This study describes the implementation and evaluation of the SPI training program during its first year of implementation. Implementation of the SPI Training Program New SPIs (n=6) were recruited and trained, and existing SPIs (n=6) were retrained to (1) portray smoking or smokeless tobacco users, (2) improve the SPI cultural competency, specifically as it relates to tobacco, and (3) provide consistent, constructive feedback to medical students on culturally competent tobacco counseling. Recruitment was primarily conducted by word of mouth through employees at Wake Forest Family Medicine. Additional recruitment was attempted through the medical school’s coordinator for the SPI program and through a local agency that serves minority persons, but these efforts were less successful. The training was organized by a master’s-trained project manager, and content was delivered by four faculty with expertise in family medicine, tobacco, patient-physician communication, counseling, and public health. The new and existing SPIs were trained in two groups by the same faculty. Training occurred in 3 half days for the new SPIs and 2 half days for the existing SPIs to minimize the participant burden. Table 1 provides an overview of the three-part training program, including the goals, objectives, and methods to achieve the objectives. Newly recruited SPIs received a 9.5 hour training program (Sessions 1, 2, and 3), and existing SPIs received an abbreviated 6-hour training program (Sessions 2 and 3). All SPIs were provided reading materials that aided them in preparing to teach medical students on culturally competent tobacco cessation counseling. SPIs were encouraged to develop their own unique scenarios for their roles. They were provided assistance, as needed, to develop their scenarios. Also, a significant component of the training program was devoted to ensuring that SPIs provided consistent feedback to students using standardized instruments for tobacco cessation counseling. To improve the consistency of SPI ratings of student

Vol. 38, No. 1

29

performance (ie, interrater reliability), individual and group ratings of videotaped SPI-student interactions were conducted. On completion of the training program, the SPIs are prepared to interact with third-year medical students, playing the role of a tobacco-using patient. A 30-minute counseling session with the student involves the SPI, who starts out in the role of a precontemplator (a tobacco-using patient who has never considered quitting use), who moves through the “Stages of Change” to a contemplator (a tobacco-using patient who is considering quitting use), and then an active quitter (a tobaccousing patient who is ready to quit use). 20 The student in turn is expected to interview the patient, using counseling specific to the patient’s stage of change. Instruments Following the counseling session, the SPI breaks the role of the patient and takes on the role of the instructor, giving the student specific, constructive feedback about his/her tobacco knowledge and counseling styles, using the Tobacco Intervention Risk Factor Interview Scale (TIRFIS). This reliable instrument, based on the University of Rochester Risk Factor Interview Scale,14 includes feedback regarding both smoked and smokeless tobacco counseling.19,21 This instrument was originally developed to assess smoking counseling skills, but all smoking references were replaced in fall 2003 with generic tobacco language (to be applicable to smoking and smokeless tobacco). The wording of the items remained otherwise unchanged. The scale has four subscales of three items each: relationship, positive focus, instigating behavioral change, and general interviewing techniques. Each item is scored from 1–5, with a total possible score of 60. A higher score indicates better counseling skills. In an analysis of the Smoking Cessation Risk Factor Interview Scale (the precursor to the TIRFIS, adapted in 2003 to be tobacco generic), the scale demonstrated that the total scale demonstrated very good reliability (α=.85), and subscale reliabilities ranged from .58 to .71.21 The item-to-total correlations (ITC) ranged from 0.43 to 0.66, with one item having an ITC of 0.29. In addition, the SPIs rate the student’s cultural competency skills specific to tobacco cessation counseling using the Tobacco Beliefs Management Scale-Tobacco Cultural Concerns Scale (TBMS-TCCS). This scale was adapted from the Disease Beliefs and Measurement Scale (DBMS) and Cultural Concerns Scale (CCS).22 In its original validation, the DBMS identified the extent to which students explored the patients’ beliefs about diabetes, blood sugar control, and insulin therapy (Cronbach’s alpha=0.61). The CCS assessed the extent to which students acknowledged and attempted to accommodate the patient’s cultural beliefs about food preparation (Cronbach’s alpha=0.83).

30

January 2006

Family Medicine

Table 1 Goals, Objectives, and Methods Used in Sessions 1, 2, and 3 of the SPI Training Program Session 1 (3.5 hours) Goals: • To provide an overview of the epidemiology health effects of tobacco use • To discuss strategies for the treatment of tobacco use and dependence • To explore the role of the SPI in training and evaluating medical students in tobacco cessation counseling Objective • To briefly describe the epidemiology of tobacco use • To discuss health effects associated with smoked and smokeless tobacco • To describe the need for increased education of physicians for tobacco intervention • To have SPIs be able to articulate the Stages of Change

Methods to Meet Objective • PowerPoint presentation by WFUSM faculty member (project director) • Have SPIs read, as homework, appropriate materials that were mailed to them prior to training • PowerPoint presentation by WFUSM faculty member (project director) • Discussion and review of the internally developed algorithm on tobacco cessation counseling by WFUSM faculty member (project director) • Discussion by WFUSM faculty member (project director) • Handout provided and in-class discussion on addiction and the appropriateness of various pharmacotherapies for nicotine addiction led by WFUSM faculty member (project director) • Discussion by WFUSM faculty member (project director) on the role and importance of SPIs in medical student education on tobacco cessation counseling • Discussion by WFUSM faculty member (project director) on the typical SPI-medical student encounter • Discussion by WFUSM faculty member (project director) with review of the internally developed algorithm on tobacco cessation counseling

• •

To identify teachable moments To describe nicotine addiction and pharmacotherapies used to treat it



To describe what a standardized patient is



To discuss the outline of a simulated patient interview with feedback To identify a typical algorithm used in tobacco cessation counseling To address common excuses used by tobacco users in order not to quit To discuss the major categories contained within the TIRFIS • Brief introduction to the items on the TIRFIS by WFUSM faculty member (project director). • Have participants review the TIRFIS at home to further become familiar with the instrument Explore the role of the SPI in training medical students in • As homework, have SPIs start to think about their scenarios and put it into providing culturally appropriate tobacco cessation counseling writing. Have them build their scenarios as they go through the 3-day training

• • •



Session 2 (3.0 hours) Goals: • To explore the impact of culture on tobacco use • To discuss a method of assessing medical student competency in culturally competent tobacco cessation counseling Objective • To explore the participants’ cultural experience with tobacco (eg, work, family, social relationships, religion) and how this experience informs his/her perceptions and values about tobacco use • To discuss “What is culture?” • • •

To discuss how culture influences our behavior To create awareness that there are health disparities among different cultural groups To provide SPIs with the appropriate training to evaluate medical students’ performance on culturally sensitive tobacco cessation counseling

Methods to Meet Objective • Participant discussion on the impact of culture (including racial/ethnic, geographic, and family background) on tobacco use. Display of prompts (tobacco leaf, smokeless tobacco products) to facilitate discussion • PowerPoint presentation by a WFUSM faculty member describing the components’ “culture” • Discussion led by WFUSM faculty member on the influence of culture on behavior and health disparities associated with different cultural groups • Introduction and discussion* to the items on the TBMS-TCCS led by WFUSM faculty member. • View videotapes of a culturally competent SPI-medical student tobacco cessation counseling encounter and a culturally incompetent SPI-medical student tobacco cessation counseling encounter, rate the skills of the student in the culturally competent SPI-medical student encounter by completing the TBMS-TCCS, and discuss, as a large group, the ratings.** Discussion led by WFUSM faculty member

(continued on next page)

Medical Student Education

Vol. 38, No. 1

31

Table 1 (continued) Session 3 (3 hours) Goals: • To practice the SPI role • To practice evaluating and providing feedback to the medical student Objective • To train SPIs to provide specific and constructive feedback to medical students. • To have SPIs focus feedback on behaviors that can be changed, not on personality traits. • To have SPIs act his or her scenario in a confident manner • • • • •

Methods to Meet Objective • Handout and group discussion on ways of providing constructive feedback. Discussion led by WFUSM faculty member

• The SPIs break up into pairs to practice their scenarios as staff members provide assistance as needed To train SPIs to critique another SPI constructively in • The SPIs provide constructive feedback, based on principles of giving feedback his or her role they were taught earlier in the session To train SPIs to evaluate a student with the TIRFIS • View videotape of an SPI-medical student tobacco cessation counseling encounter, To elicit SPI justification for his or her rating of the student rate student skills by completing the TIRFIS, and discuss, as a large group, the on the TIRFIS ratings To help SPIs recognize realistic expectations and limitations • Closing discussion led by project director summarizing expectations regarding of medical students the SPI-medical student encounter and providing feedback To repeat to the group basic points to remember regarding giving feedback

SPI—standardized patient instructor TIRFIS—Tobacco Intervention Risk Factor Interview Scale WFUSM—Wake Forest University School of Medicine TBMS-TCCS—Tobacco Beliefs Management Scale-Tobacco Cultural Concerns Scale * The instrument was modified based on some of the feedback from the SPIs. ** The culturally competent videotape was scripted and filmed by project staff members and a WFUSM faculty member, who played the role of the minority patient. The culturally incompetent video was of an interaction between an SPI-medical student that was clinically sound but did not address cultural issues in situations where appropriate.

Because no current scales existed to assess cultural competency related to tobacco use, the DBMS-CCS was specifically reworded and reviewed by experts of different ethnic groups to ensure that it captured beliefs and cultural concerns of tobacco users. Data reported in this report are the first to assess the scale’s reliability after the rewording of the instrument to be tobacco specific. Both instruments are available from any of the authors. For the first two to three counseling sessions, an experienced SPI was paired with a new SPI to review their ratings on the TIRFIS and TBMS-TCCS as a means of improving interrater reliability. Data for this study were compiled over a 7-month period (November 2003 to May 2004). This study was approved by the Institutional Review Board of Wake Forest University School of Medicine. Analyses SPI Evaluation of Training Program. SPIs evaluated each session of the SPI training program. They completed anonymous surveys that assessed content and organization of the training, presenter qualities, and influence of training on knowledge and ability to become a tobacco-specific SPI. To analyze the data, frequencies for each item for Sessions 1, 2, and 3 were calculated. Analysis of variance was used to assess differences in SPI ratings across the sessions both to

evaluate instructor performance and to assess quality assurance of the overall training curriculum. Student Feedback. Students evaluated all SPI encounters following the SPI training sessions. Immediately following the SPI interaction and feedback session, medical students were requested to complete a survey to assess the effectiveness of encounter and feedback in improving their tobacco cessation counseling techniques. To analyze the data, frequencies and mean scores were calculated for each item. Utility of Measures to Assess Student Performance. The TIRFIS and TBMS-TCCS were assessed for their ability to reliably assess counseling skills and to determine the SPIs’ willingness to utilize the full scales to evaluate student performance. Descriptive statistics, internal reliability (Cronbach’s alpha), interrater reliability (via percent agreement), Cohen’s kappa statistic (a measure of concordance), and a generalizability analysis were computed.23,24 The purpose of the generalizability analysis was to decompose the measurement variance, taking into account potential sources of variation, including the student, the instrument item, and the SPIs. This methodology was important for considering the effect of using many SPIs (who may have had variations in the way they scored students)

32

January 2006

Family Medicine

on establishing interrater reliability. The underlying and the use of the TIRFIS and TBMS. They also enstatistical methodology of a generalizability analysis is joyed the group discussion about culture and its influence on tobacco use. At the end of the first session, a random effects models and variance component analysis. DeVellis provides a succinct description of generalfew SPIs mentioned that they felt overwhelmed about izability theory in his text on scale development.24 their role. At the conclusion of the training, however, Descriptive statistics and internal reliability was based on 55 SPI-stuTable 2 dent encounters during the 2003–2004 academic SPI Rating of Sessions 1, 2, and 3 of the SPI Training Program year. The interrater reliability was assessed using SESSION2 40 videotaped SPI-student encounters. The 40 Overall 13 23 3 Score (n=6) (n=11) (n=9) videotapes were viewed Item1 by two independent SPIs CONTENT (in private sessions) and Reading materials helped me understand the importance of 4.5 4.8 4.3 4.5 scored using the TIRFIS culturally competent tobacco cessation counseling. and TBMS-TCCS. Cost of Implementing the SPI Training Program. The cost of implementing the SPI training program was also calculated. Cost measures include recruitment of SPIs, materials development, training costs (eg, refreshments), and faculty and staff time spent preparing for and delivering the training program. Results SPI Evaluation of Training Program Twelve SPIs provided feedback regarding their experience of the SPI training program. They were 51.6 years of age, on average (SD =20.5, range 24–83), with the majority being female (n=8) and white (n=7). Persons who self-identified as black (n=2) or Hispanic (n=3) comprised the remainder of the SPIs. The SPIs highly rated the content, organization, and presenter characteristics of the training program (Table 2). The SPIs stated that they especially appreciated the opportunities for practicing their scenarios

Handouts were useful.

4.5

4.8

4.4

4.3

Slides/overheads were useful.

4.6

4.8

4.4

4.8

Practicing after viewing the videos was useful.

5.0

NA

5.0

5.0

The small group exercise(s) were useful.

5.0

NA

NA

5.0

Topics were covered in a logical sequence.

4.6

4.5

4.7

4.7

Amount of information covered in today’s training was reasonable.

4.7

4.3

4.7

4.8

Presenter(s) were well organized.

4.8

5.0

4.7

4.9

Material was presented in a clear and concise manner.

4.7

4.8

4.5

4.8

Presenter(s) were enthusiastic/stimulating/engaging.

4.7

4.8

4.5

4.9

Presenter(s) were well-informed on the topics.

4.8

5.0

4.7

4.9

Presenter(s) actively involved participants.

4.7

4.5

4.7

4.9

Presenter(s) encouraged questions on the topic.

4.8

5.0

4.6

4.9

Presenter(s) established rapport.

4.8

4.8

4.7

4.9

Presenter(s) set clear/reasonable expectations.

4.8

4.8

4.8

4.8

I am confident that I can provide feedback to students on their tobacco cessation counseling skills.

4.3

3.8

4.4

4.5

ORGANIZATION

PRESENTER(S)

1 2 3

Scoring: 1 (not at all) to 5 (very much) There were no statistically significant differences across sessions using analysis of variance (ANOVA). Session 1 only included the newly recruited SPIs since the original SPIs had attended this session previously. One SPI did not attend Training Session #2, and three SPIs did not attend Training Session #3.

SPI—standardized patient instructor

Medical Student Education

Vol. 38, No. 1

33

Table 3 SPI Rating of Confidence in Knowledge and Ability to Be an SPI at the End of Session 3 of the Training Program (n=9)* Compared to before you came to this 3-day training, how confident do you feel now about the following: My knowledge regarding the adverse influence tobacco use has on health My knowledge regarding the importance of teaching medical students to appropriately counsel their patients on tobacco cessation My knowledge on tobacco dependence treatment My knowledge on the health disparities between racial/ethnic and demographic groups My knowledge on the tobacco use disparities between racial/ethnic and demographic groups My knowledge about the role culture plays in tobacco use My ability to play the part of a tobacco-using standardized patient My ability to use the Tobacco Intervention Risk Factor Interview Scale (TIRFIS) to evaluate medical students on their counseling skills My ability to use the Tobacco Beliefs and Management Scale and Tobacco Cultural Concerns Scale (TBMS-TCCS) to evaluate medical students on their counseling skills My ability to provide good feedback to the medical student

Average Score 4.9 4.8 4.4 4.1 4.2 4.3 4.4 4.3 4.2 4.3

* n=9 Scoring: 1 (not confident at all) to 5 (very confident) SPI—standardized patient instructor

the SPIs rated that they were highly confident about their knowledge and skill gained through the training (Table 3). Student Feedback Fifty-five students provided feedback regarding the SPI-student interaction. The average student was 26.4 years of age (SD 3.0, range 23–36), with the majority being male (66%) and white (48%). Black (10%), Asian or Pacific Islander (26%), Hispanic (6%), and other (10%) racial/ethnic groups were also well represented among students providing feedback. In general, medical students highly rated the SPI encounter using a structured evaluation form (Table 4). Twenty-one students also responded to an open-ended question that elicited additional comments regarding their interview experience; 18 of the responses were positive. Specific comments included: “Although somewhat contrived, the experience is still a beneficial teaching tool. In receiving feedback I was able to better assess my own strengths and weaknesses,” “[I] very much enjoyed this experience…[it] gave me great confidence to do this in the future,” and “A good format, a good learning experience…helpful feedback.” Of the three students who gave negative feedback, two were specifically directed at the encounter. “I felt I did a fairly good job of explaining reasons and counseling my patient, but after my feedback was given I feel the patient mistook a lot of what I said and held it against me, quoting things I said out of the correct context.” Another student stated “One must remember

Table 4 Medical Student Feedback About the SPI Encounter* Item** To what extent did the patient you interviewed behave like a real patient you might see in a genuine clinical setting? How realistic was this general scenario involving a doctor-patient interaction (ie, a setting involving a relatively healthy patient being seen by his/her doctor under similar circumstances?) (n=54) By the conclusion of the interview, were you able to forget you were acting out a contrived role and get into the role of this patient’s primary care doctor? How helpful do you feel it is to receive feedback about your interview from the patient immediately following the interview? Were the feedback and discussion following the interview informative, instructional, and constructive? (n=54) In future smoking cessation counseling, are you likely to use the skills or techniques that were reviewed after today’s interview? How effective do you feel this exercise is in teaching patient counseling skills? How much did you learn today as a result of this exercise?

Average Score 3.0

* n=55 ** Scoring: 0 (not at all), 1 (a little), 2 (somewhat), 3 (much), 4 (very much) SPI—standardized patient instructor

3.2

2.5

3.4

3.4 3.6

3.4 3.2

34

January 2006

that actual interviews are a completely different experience, with real patients being much more resistant to stopping smoking. In real life, much of what we were taught simply does not work.”

Family Medicine same SPI-student encounter. Table 6 shows the percent of agreement between two raters on each item of both instruments and the Cohen’s Kappa statistic (a measure of concordance).

TIRFIS Perfect agreement ranged from 17.5% to 47.5 % on the individual items. Yet, an average of 78% (range 61%–93%) of paired SPI ratings were within one point of agreement across the total TIRFIS scale. Kappa values were considered slight to fair using Landis and Koch’s guide on measurement agreement.23 The low kappa values were at least partially due to the fact that the majority of SPI ratings fell within categories 4 and 5, making the values of these categories rather high in their marginal distributions. As a result, a small deviation from perfect agreement resulted in a large drop in concordance. The generalizability analysis, a method to decompose the measurement variance taking into account Table 5 student, item, and SPI, demonstrated that the Medical Student Performance on the TIRFIS, the TBMS, and the TCCS* measure is dependable TIRFIS1 and that SPI variability is Subscale2 Mean (SD) Range relatively mild compared Relationship 11.91 (2.4) 4.5–15 0.75 to overall variability of Positive focus 10.98 (2.6) 4–15 0.73 the scores (overall G Instigating behavior change 11.96 (2.3) 5–15 0.67 coefficient=0.51)25 (Table General interview techniques 12.15 (2.1) 7–15 0.60 7).

Utility of Measure to Assess Student Performance The TIRFIS and TBMS-TCCS subscales demonstrated good internal reliability (Table 5). There was also a wide range of scores attesting to the sensitivity of the instruments to assess variations in student performance. This suggests that the sessions were effective in appropriately training SPIs to use the instruments to assess students and pick up variations in student performance. Analysis of interrater reliability (n=40 videotape reviews by two independent SPIs) also suggests that the SPIs provide comparable performance evaluations on the TIRFIS and the TBMS-TCCS when viewing the

Scale total

TBMS2 Item Discussed why I should quit using tobacco Explored my fears about quitting Explored my beliefs about different treatments such as nicotine gum, patch, or Zyban to help me quit. Scale total TCCS3 Item Explored my beliefs about the importance of tobacco use in my culture Acknowledged the importance of my cultural preferences Worked out a quit plan that is sensitive to my cultural beliefs Scale total

47.00 (7.7)

32–60

0.87

Mean (SD) 3.64 (1.0) 2.96 (1.2) 3.49 (1.3)

Range 2–5 1–5 1–5

10.09 (3.29)

4–15

0.63

Mean (SD) 2.05 (1.3) 2.31 (1.2) 2.36 (1.2) 6.73 (3.29)

Range 1–5 1–5 1–5 3–15

0.89

* n=55 TIRFIS—Tobacco Intervention Risk Factor Interview Scale TBMS—Tobacco Beliefs Management Scale TCCS—Tobacco Cultural Competency Scale 1 2

3

Scoring on TIRFIS items from 5 to 1, with a higher score reflecting better performance Items of the TIRFIS Subscales: relationship—confidence, joining, empathy; positive focus—reinforce effort, highlight positive consequences, reframing failure; instigating behavior change—stages of change, nicotine dependence, behavior change technique; general interview techniques—organization, rapport/eye contact/body language, closure, and follow-up. Scoring from 5 to 1, with a higher score reflecting better performance

TBMS-TCCS Per fect ag reement ranged from 30%–35% on the TBMS and 25%– 30% on the TCCS. Similar to the TIRFIS, the majority of paired ratings fell within 1 point of agreement. The generalizability coefficient was quite low (G=0.07) on the TBMS. The effect is mainly due to a low variance in student scores after factoring out other sources of variation (primarily SPI and item). Additionally, student-SPI interaction is high compared to either student or SPI variation alone. This suggests that the pairing of the SPI and student may, in some way, influence the ratings on the

Medical Student Education

Vol. 38, No. 1

35

Table 6

Table 7

Interrater Reliability of the TIRFIS and TBMSTCCS: Paired Ratings of SPI-Student Encounters*

Decomposition of the Measuring Variance of the TIRFIS and TBMS-TCCS, Taking Into Account the Variability in Students, SPIs, and Items (Generalizability Analysis)*

TIRFIS Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Item 7 Item 8 Item 9 Item 10 Item 11 Item 12 Overall TBMS Item 1 Item 2 Item 3 Overall TBMS

Perfectly Agree % 47.5 37.5 27.5 17.5 22.5 30.0 35.0 42.5 35.0 27.5 42.5 25.0 32.5

Paired Ratings Differ by 1 Point % 22.5 45.0 65.0 43.5 45.0 42.5 47.5 37.5 52.5 47.5 50.0 42.5 45.1

30.0 35.0 35.0 33.3

42.5 27.5 30.0 33.3

Kappa** 0.01 0.39 0.36 0.21 0.24 0.03 0.23 0.30 0.47 0.32 0.13 0.03 0.27

0.23 0.25 0.18 0.25

Source of Variation Student SPI Item Student-SPI Interaction Item-SPI Interaction Student-Item Interaction Residual

Variance Component (Degrees of Freedom) TIRFIS TBMS TCCS 0.16 (39) 0.03 (39) 0.46 (39) 0.08 (1) 0.08 (1) 0.39 (1) 0.08 (11) 0.06 (2) 0.00 (2) 0.14 (39) 0.42 (39) 0.20 (39) 0.10 (11) 0.14 (2) 0.01 (2) 0.11 (429) 0.30 (78) 0.00 (78) 0.52 (429) 0.43 (78) 0.44 (78)

Generalizability coefficient (for absolute decision)

0.51

0.07

0.55

* The purpose of this type of analysis is to partition the total variance observed on the TIRFIS, TBMS, and TCCS scores to variances among students, SPIs, and items on the survey. The underlying statistical methodology of a generalizability analysis is random effects models and variance component analysis. TIRFIS—Tobacco Intervention Risk Factor Interview Scale TBMS—Tobacco Beliefs Management Scale TCCS—Tobacco Cultural Competency Scale SPI—standardized patient instructor

TCCS Item 1 Item 2 Item 3 Overall TCCS

30.0 25.0 27.5 27.5

45.0 40.0 47.5 44.2

0.29 0.18 0.37 0.28

* n=40 ** The kappa values are considered slight to fair using Landis and Koch’s23 recommended review criteria. The low kappa values are partially due to the fact that the majority of observations fall into categories 4 and 5, making the values of these categories rather high in their marginal distributions. As a result, a small deviation from perfect agreement leads to a large drop in concordance. TIRFIS—Tobacco Intervention Risk Factor Interview Scale TBMS—Tobacco Beliefs Management Scale TCCS—Tobacco Cultural Competency Scale

TBMS. The TCCS, however, was not similarly affected. The generalizability coefficient (G= 0.55) suggests that the measure is dependable and that the SPI variance component was lower than the student variability or other factors. Cost of Implementing the SPI Training Program Costs associated with the training session included SPI payment ($1,400), refreshments ($90), producing printed and bound material handouts ($90), mailing

($60), and copying and printing ($30). Faculty and staff time to prepare for and conduct the training were estimated at 130 hours. Salary cost for time spent by faculty and staff was approximately $3,730 ($2,560 for faculty and $1,170 for staff). Following completion of the training, each SPI was paid $20 per SPI-medical student encounter, resulting in a cost of $1,100 to date (55 SPI-medical student encounters times $20 per SPImedical student encounter). The total cost to implement the SPI training program was estimated to be $5,400 for 12 SPIs, resulting in a cost of $450 per SPI for recruitment and training ($5,400 total training cost/12 SPIs) and $98 per student encounter to date ($5,400 total training cost + $1,100 SPI-medical student encounter costs to date)/55 SPI-medical student encounters). Additional costs over time can be incurred should SPIs leave the program (requiring additional training) and/or retraining needs are identified. Discussion The SPI-medical student tobacco cessation counseling encounter and SPI feedback to students on their tobacco cessation counseling skills have been successful, both from the point of view of the students19 and

36

January 2006

the SPIs. This paper provides initial evidence of the success of the program in training SPIs to be effective standardized patients as well as instructors who can provide constructive and reliable feedback to students. Essential to developing a cadre of tobacco-specific SPIs is a good training program that trains lay individuals to play the role of a tobacco-using patient as well as an instructor who can critique student tobacco cessation counseling techniques. Further, as SPIs become more integral “teachers” of medical education, it is essential that their ratings of student performance be consistent. Students will certainly insist on fair grading practices. Medical education administrators also have a responsibility to their students to ensure that they are evaluated in an equitable manner. There are limitations to this study, however, that deserve attention. The relatively low interrater reliability scores on the two instruments indicate cause for concern. The low kappa values are partially due to the fact that the majority of observations fall into categories 4 and 5, making the values of these categories rather high in their marginal distributions. As a result, a small deviation from perfect agreement leads to a large drop in concordance. This could be improved by several mechanisms. First, the sample size could be increased, thereby minimizing the drop in kappa values when disagreement occurs among only a small number of SPIs. Second, more training can be done to enhance the distinctions between scores 4 and 5, thus improving perfect agreement between raters. Finally, scores 4 and 5 could be combined if the difference between 4 and 5 is not clinically meaningful. If the ultimate goal of instruments is to provide feedback to the student, the SPIs must be able to articulate specific improvements that could raise a student’s score from a 4 to a 5; otherwise, the items should be combined without jeopardizing the utility of the instruments. A second limitation to this study is that the TBMSTCCS has not been previously validated since the wording was revised from diabetes to tobacco. Although our study provides initial evidence of the reliability of the TBMS-TCCS, the limited sample size weakens any conclusions drawn about its measurement utility. Of some concern is the relatively low generalizability coefficient observed on the TBMS. This appears to be primarily related to low variance in student scores, which may be improved once students in this curriculum are trained on cultural issues related to tobacco. Further, there are no published validated scales that assess overall tobacco cultural competency. This represents a deficit in the published literature, especially for persons interested in understanding the cultural nuances of tobacco use and teaching culturally competent counseling skills specific to tobacco. Despite these limitations, we believe that the SPI training program described in this paper can be incor-

Family Medicine porated into a medical education curriculum to develop a cohort of knowledgeable, effective SPIs who can articulate ways to improve the students’ tobacco intervention counseling skills. We also feel that the basic training may be adapted to other health behaviors or medical concerns. In fact, family medicine faculty and staff at WFUSM have recently proposed to create a SPI-obesity curriculum that builds on the work presented in this paper. The challenges in adapting the program include identifying counseling instruments that are appropriate to other health behaviors, the time-consuming task of practicing scenarios and ensuring interrater reliability, and the initial costs. Yet, the groundwork for establishing such a curriculum could save administrators time in the long run by limiting (or eliminating) retraining of SPIs (a time-intensive and costly endeavor) and managing student complaints that result from perceived unfair grading practices of lay individuals. Costs for implementing our training program were initially high. As the program progresses and SPIs are engaged in more encounters, the average cost per encounter will subsequently decline. Since additional costs may be incurred as SPI are retrained or leave the program, it is imperative that medical schools integrate a retention program into their SPI-led medical education program to ensure continuity and cost-efficiency. With the emphasis on both smoked and smokeless tobacco, integration within the existing medical school curriculum, and explicit evaluation of cultural competency, we offer this model as a useful resource to medical educators who wish to institute a program using SPIs to train medical students in tobacco intervention. Acknowledgments: This project was supported by a grant from the National Cancer Institute R25CA096562-01. Corresponding Author: Address correspondence to Dr Foley, Wake Forest University, School of Medicine, Department of Public Health Sciences, Medical Center Boulevard, Winston-Salem, NC 27157. 336-716-9881. Fax: 336-716-7554. [email protected]. REFERENCES 1. Annual smoking-attributable mortality, years of potential life lost, and economic costs—United States, 1995–1999. MMWR Morb Mortal Wkly Rep 2002;51(14):300-3. 2. Cigarette smoking among adults—United States, 2000. MMWR Morb Mortal Wkly Rep 2002;51(29):642-5. 3. Tobacco use among high school students—United States, 1997. MMWR Morb Mortal Wkly Rep 1998;47(12):229-33. 4. Smith SS, Fiore MC. The epidemiology of tobacco use, dependence, and cessation in the United States. Prim Care 1999;26(3):433-61. 5. Tobacco use among US racial/ethnic minority groups—African Americans, American Indians and Alaska Natives, Asian Americans and Pacific Islanders, Hispanics. A Report of the Surgeon General. Executive summary. MMWR Recomm Rep 1998;47(RR-18):v-xv, 1-16. 6. Welty TK, Lee ET, Yeh J, et al. Cardiovascular disease risk factors among American Indians. The Strong Heart Study. Am J Epidemiol 1995;142(3):269-87. 7. Use of smokeless tobacco among adults—United States, 1991. MMWR Morb Mortal Wkly Rep 1993;42(14):263-6. 8. Spit tobacco and youth. Washington, DC: Office of Evaluations and Inspections, US Department of Health and Human Services, 1992.

Medical Student Education 9. Ferry LH. Overcoming barriers to nicotine dependence treatment. Prim Care 1999;26(3):707-46. 10. Spangler JG, George G, Foley KL, Crandall SJ. Tobacco intervention training: current efforts and gaps in US medical schools. JAMA 2002;288(9):1102-9. 11. Orleans CT. Understanding and promoting smoking cessation: overview and guidelines for physician intervention. Annu Rev Med 1985;36:5161. 12. Gilpin E, Pierce J, Goodman J, Giovino G, Berry C, Burns D. Trends in physicians giving advice to stop smoking, United States, 1974–1987. Tob Control 1992;1:31-6. 13. Fiore MC, Bailey WC, Cohen SJ, et al. Treating tobacco use and dependence. Washington, DC: US Department of Health and Human Services, Public Health Service, 2000. 14. Levenkron JC, Greenland P. Validation of the University of Rochester Risk Factor Interview Scale. Am J Prev Med 1987;3(3):152-6. 15. Levenkron JC, Greenland P, Bowley N. Using patient instructors to teach behavioral counseling skills. J Med Educ 1987;62(8):665-72. 16. Levenkron JC, Greenland P, Bowley N. Teaching risk factor counseling skills: a comparison of two instructional methods. Am J Prev Med 1990;6(2 suppl):S29-S34. 17. Eyler AE, Dicken LL, Fitzgerald JT, Oh MS, Wolf FM, Zweifler AJ. Teaching smoking cessation counseling to medical students using simulated patients. Am J Prev Med 1997;13(3):153-8.

Vol. 38, No. 1

37

18. Bland E, Oppenheimer L, Brisson-Carroll G, Morel C, Holmes P, Gruslin A. Influence of an educational program on medical students’ attitudes to substance use disorders in pregnancy. Am J Drug Alcohol Abuse 2001;27(3):483-90. 19. Spangler JG, Enarson C, Eldridge C. An integrated approach to a tobacco-dependence curriculum. Acad Med 2001;76(5):521-2. 20. Prochaska JO, DiClemente CC. Stages and processes of self-change of smoking: toward an integrative model of change. Consult Clin Psychol 1983;51:390-5. 21. Foley KL, Crandall SJ, George G, Roman M, Spangler JG. Reliability of a Smoking Cessation Risk Factor Interview Scale (SCRFIS) for use with standardized patient instructors. J Cancer Educ 2003;18(3):13441. 22. Robins LS, White CB, Alexander GL, Gruppen LD, Grum CM. Assessing medical students’ awareness of and sensitivity to diverse health beliefs using a standardized patient station. Acad Med 2001;76(1):7680. 23. Landis JR, Koch GG. Agreement measures for categorical data. Biometrics 1977;33:159-74. 24. DeVellis RF. Scale development. Theory and applications. Applied Social Research Methods Series Vol. 26. Newbury Park, Calif: Sage Publications, 2003:39-41. 25. Shavelson RJ, Rowley GL, Webb NM. Generalizability theory. Am Psychol 1989;44:922-32.