A structured communication adolescent guide ... - Wiley Online Library

communication

A structured communication adolescent guide (SCAG): assessment of reliability and validity Kim Blake,1 Nicolle Vincent,2 Susan Wakefield,2 Joseph Murphy,3 Karen Mann2 & Matthew Kutcher2

PURPOSE To assess the reliability and validity of a Structured Communication Adolescent Guide (SCAG) in an undergraduate medical education setting using trained adolescent raters. METHOD The SCAG is a 49-item, 6-section (A–F), protocol derived from the Calgary Cambridge Observation Guide that incorporates adolescent psychosocial data collection with the physician’s communication skills. Four trained female adolescents scored 42 videotaped adolescent clinical encounters using the SCAG; a trained psychologist’s rating for each videotape was used as the gold standard. RESULTS Agreement among adolescent raters was determined by calculating intraclass correlation coefficients (ICC). The individual SCAG item scores, combined with the global ratings for each section, resulted in an overall ICC value of 0.93, indicating very strong agreement among the 4 raters. The global rating scores for the sections (Ôinitiating the sessionÕ, Ôinitiating separationÕ, Ôonce adolescent is alone ) lifestyleÕ and ÔclosureÕ) produced an ICC range of 0.58–0.93. However, the ICC values for the 2 remaining sections (Ôhow was information collectedÕ and Ôgathering informationÕ) global rating scores were below 0.30, signifying low agreement. Overall agreement between the adolescent raters and the gold standard resulted in an ICC value of 0.78. This is evidence of the SCAG’s criterion validity. CONCLUSION The SCAG is a reasonably valid tool for use in guiding an encounter with an adolescent patient. However, 2 sections require modifications

to improve their reliability and thus the SCAG’s overall performance. Our results suggest that the SCAG shows promise as a potentially useful teaching resource in undergraduate medical education in adolescent medical interviewing. KEYWORDS education, medical, undergraduate ⁄ *methods; *communication; physician–patient relations; data collection; videotape recording ⁄ methods; reproducibility of results. Medical Education 2005; 39: 482–491 doi:10.1111/j.1365-2929.2005.02123.x

INTRODUCTION While a visit to the doctor’s office is probably uncomfortable for many adolescents, the doctor too may not be completely certain that the adolescent’s underlying reason for the visit has been addressed. For example, a need for contraception may be difficult for an adolescent to articulate, making it essential that the doctor be attuned to the nuances of adolescent communication in these and similar instances. Communication skills training for doctors has become a routine component of medical education worldwide,1 but a focus on the particular skills of communicating with adolescents has only recently emerged as a distinct part of the undergraduate and postgraduate curricula.2–6 Finding ways of exposing our medical students, postgraduates and faculty to learning these skills is necessary to ensure appropriate levels of confidence and competence.4,7–10

1

Department of Pediatrics, IWK Health Centre, Halifax, Canada Dalhousie University, Canada 3 Dental Clinical Sciences, Dalhousie University, Canada 2

Correspondence : Dr Kim Blake, Director of Pediatric Undergraduate Education, IWK Health Centre, 5850 University Avenue, Halifax NS B3J 3G9, Canada. Tel: (902) 470 6499; Fax: (902) 470 7216; Cell: (902) 488 0128; E-mail: [email protected]

482

One way to approach such learners is through the use of standardised patients (SPs). There is an evolving body of knowledge to support adult SPs as reliable scorers in OSCE-type settings, in longer more in-depth encounters, in undercover visits to family

Blackwell Publishing Ltd MEDICAL EDUCATION 2005; 39: 482–491

483

Overview What is already known on this subject Structured communication guides for adult interviewing are widely used in medical education. There are unique challenges involved in adolescent interviewing and this is an area of training which has been highlighted as inadequate in many medical schools. What this study adds The structured communication adolescent guide (SCAG) is a reasonably reliable and valid tool that can be used by: (1) medical students when interviewing adolescent patients and (2) adolescent patients to offer feedback to the medical student. Suggestions for future research Modify the SCAG and reassess reliability and validity using trained and untrained adolescents Test reliability and validity of the SCAG with physicians.

practitioners and in general medical educational ⁄ teaching settings.11–15 Employment of adolescents as SPs has gained presence in the literature, especially when involving risk-taking scenarios and issues around recruitment.16–18 This study is a continuation of work undertaken over the past several years which studied the improvement of psychosocial interviewing skills by senior medical students after they had interviewed an adolescent standardised patient and received structured feedback from the SP and mother. These interviews were all videotaped and in the present study these videotaped interviews were watched by a group of trained adolescents who independently rated the interviewers using the Structured Communication Adolescent Guide (SCAG). In the original study, feedback from adolescent SPs was found to be an effective teaching and learning method. This previous study, however, had highlighted the need for a structured guide to facilitate such

feedback, as the SPs felt that having such a guide would permit them to provide more valuable feedback to students.4 The study described in this paper was conducted to evaluate the reliability and validity of the SCAG that was developed following the outcome of the previous work. Development of the SCAG The SCAG was based on the Calgary Cambridge Observation Guide19 and modified to incorporate specific adolescent psychosocial data-gathering and communication skills that all medical professionals are expected to use when eliciting psychosocial data from an adolescent and his or her parent. The Calgary Cambridge Observation Guide (CCOG)19 is a well-established communication assessment tool for which there is considerable evidence concerning its validity, reliability and utility. The CCOG has been used at Dalhousie University for several years and has been found to be a valuable tool in teaching general communication skills. However, the CCOG does not address several aspects which are of considerable importance to an adolescent interview, including confidentiality, having a non-judgemental approach, creating a triangle of communication between the adolescent, the parent and the interviewer, and separation of the adolescent from the parent. To address these areas, a modified version of the original CCOG was created. It became the ÔStructured Communication Adolescent GuideÕ (SCAG) [Appendix 1]. The first version of the SCAG included 49 items and 6 global ratings. It was divided into 6 sections as follows: initiating the session, how was the information collected, gathering information, initiating separation, once the adolescent is alone (including adolescent lifestyles) and closure. The adolescent encounter also requires the physician to ensure that potential risk-taking behaviours, other than the presenting complaint, are addressed. For example, an adolescent may visit a doctor with a stomach ache or twisted ankle. If the doctor deals only with the immediate issue, they may neglect to uncover the risk-taking behaviour that leads to the stomach ache or twisted ankle. Therefore, a major component of the SCAG was the incorporation of an adolescent lifestyle section that included the ÔHEADDSSÕ mnemonic to ensure that risk-taking behaviours were addressed during the interview: Home, education, alcohol, drugs, diet, sex and suicide.20


484

communication The objective of the study we report here was to document the validity and reliability of the SCAG by exploring the agreement within and among trained adolescent raters in their scoring of videotaped fourth-year medical student–adolescent encounters, and to compare adolescents’ ratings with a Ôgold standardÕ established by a psychologist.

tape. For an elaboration of the study methods see Fig. 1. Data analysis Intraclass correlation coefficients (ICC) were used to estimate inter-rater reliability (agreement), intrarater reliability and criterion validity.21 Readability analyses were also conducted.22

METHODS Subjects The study participants were 4 female adolescents (mean age ¼ 14.2 years) who were trained to score the SCAG. Selected participants were average or above in academic standing and committed to the project for a year. Full informed consent was obtained from all participants who were provided compensation for their time and work. Procedure

Inter-rater reliability was calculated using the 4 adolescent females’ scores on each of the 42 videos. Intrarater reliability was calculated by comparing 2 SCAG scorings of the same videotape rated by a single adolescent at 2 different points in time (at least 1 month interval). ICC was analysed using the scores from 5 videotapes rated by the adolescent at 2 different time points. Criterion Validity was calculated by establishing interclass correlations between the average score of the 4 adolescent raters on each videotape with the gold standard.

Training Readability analysis The 4 adolescents watched a set of 4 30–45minute videotaped mother-and-daughter SP interviews conducted by a senior medical student. The 4 adolescents completed the 49-item SCAG by marking: Ôdid notÕ, ÔdidÕ or Ôdid wellÕ for each item. A psychologist who had scored the videotapes using the SCAG for a previous study had established a Ôgold standardÕ score. The training goal was that the adolescents reach an 80% scoring concordance, both among themselves and with the gold standard. Four videotaped interviews were used and reviewed with the SP trainer until the 80% agreement was reached. Task Once training was completed, each adolescent used the SCAG to score a set of 42 videotapes obtained from a previous study of improving senior medical students’ adolescent communication skills.4 Consent from individuals involved in the videos was obtained. Each videotaped clinical encounter was approximately 30–45 minutes in duration and was based on 1 of 4 possible issues: difficult clinical control of either asthma, epilepsy, attention deficit disorder or diabetes. The adolescents signed out these videotapes and scored them independently. Each adolescent completed approximately 2 videotapes per week and returned the SCAG forms upon completion of each

Readability analysis was performed to determine the reading level of the SCAG. Both Flesch Reading Ease and Flesch-Kincaid Grade Level were calculated.22

RESULTS Inter-rater reliability A total score that included both the global ratings and the individual items was derived for each completed SCAG. Inter-rater reliability for the total scores was 0.93. When global ratings were excluded from the total score, the inter-rater reliability of item scores was 0.94. Global ratings alone had an interrater reliability of 0.85. Inter-rater reliability was calculated for each section of the SCAG (A–F). Table 1 presents ICC for the global ratings and median item scores of each section of the SCAG. Aside from Ôhow was the information collectedÕ and Ôgathering informationÕ (sections B and C) all sections produced coefficients from 0.58 to 0.95 for both global ratings and individual items. The inter-rater reliability for the individual items in sections B and C were 0.24 and 0.50, respectively. Similarly, global ratings for these 2 categories had interrater reliabilities below 0.30 (Fig. 2).


485

To assess the reliability and validity of the Structured Adolescent Communication Guide (SCAG) with trained adolescent raters

Participants: 4 adolescent females (average age 14.2)

Training: 4 adolescents used SCAG while watching video interviews until there was 80% agreeability

Videos: Videos from a previous study* (n=42) were scored separately using the SCAG by each of the 4 adolescents. They reviewed 1 video at a time and passed in their SCAG forms after each viewing

Retest: A randomly selected video was scored by each adolescent at 2 different occasions (with at least 1 month from first viewing)

Gold Standard Scores*

Criterion Validity

Interrater reliability

Intrarater reliability

Figure 1 Illustration of methods: purpose, subjects, procedure and analysis. This study documented the improvement of medical students’ psychosocial interviewing skills upon receiving feedback, from a mother–daughter standardised patient team, in an adolescent interview. All interviews were videotaped and scored by a gold standard. Source: Blake et al.4

Intrarater reliability

Criterion validity

Intrarater reliability for the total SCAG scores was 0.93. For individual item scores (global ratings excluded) the intrarater reliability ICC was 0.95, and global ratings alone produced an intrarater reliability of 0.90. The coefficients for all global ratings exceeded 0.66 (Fig. 3). Median item coefficients were above 0.87 for all categories except E (lifestyles), which produced a coefficient of 0.55.

The 4 adolescents’ total SCAG scores were averaged and compared to the total SCAG scores of the gold standard. The resulting intraclass correlation was 0.78. For individual item scores (global ratings excluded) the ICC was 0.66, while for the 6 global ratings the coefficient was 0.76. For all global rating sections, coefficients were greater than 0.52, with the exception of sections B (how the information was collected) and


communication

Table 1 Intraclass coefficient (ICC) for the global ratings (global) and the median item scores (items) for each section of the SCAG SCAG sections Initiating (A) How was information collected (B) Gathering information (C) Separation (D) Adolescent alone* Lifestyles (E) Closure (F)

Inter-rater reliability Items Global

Intrarater reliability Items Global

Criterion validity Items Global

0.87 0.24

0.89 ) 0.15

1.00 1.00

0.79 0.67

0.76 0.00

0.85 ) 0.11

0.50 0.92 0.67 0.95 0.80

0.29 0.94 N⁄A 0.77 0.58

0.95 0.89 0.55 1.00 0.88

0.69 0.99 N⁄A 0.67 0.80

0.09 0.72 0.33 0.80 0.33

) 0.12 0.86 N⁄A 0.73 0.52

*The adolescent alone does not have a global rating.

Global rating

Global rating 1.00

0.80

0.80

0.60

0.60 ICC

ICC

1.00

0.40 0.20

–0.20

0.40 0.20

0.00

0.00 A

B

C

D

E

F

–0.20

A

B

–0.40

C

D

E

F

SCAG categories SCAG categories

Figure 2 Inter-rater reliability. A ¼ initiating the session; B ¼ how was information collected; C ¼ gathering information; D ¼ initiating separation; E ¼ once adolescent is alone ) lifestyles; F ¼ closure. ICC ¼ inrtraclass correlation coefficient.

Figure 4 Criterion validity. A ¼ initiating the session; B ¼ how was information collected; C ¼ gathering information; D ¼ initiating separation; E ¼ once adolescent is alone ) lifestyles; F ¼ closure. ICC ¼ inrtraclass correlation coefficient.

C (information gathering). Item scores for these 2 sections revealed coefficients less than 0.10 (Fig. 4). Global rating 1.00

Figure 5 provides a visual comparison of the interrater ⁄ intrarater reliability and the criterion validity as determined by global rating scores.

0.90 0.80 0.70 0.60 ICC

486

Readability analysis

0.50 0.40 0.30

The Flesch Reading Ease of the SCAG, on a 100-point scale, was 52.7. The higher the score, the easier the document is to understand. The SCAG produced a Flesch-Kincaid Grade Level of 7.7.

0.20 0.10 0.00 A

B

C

D

E

F

SCAG categories

Figure 3 Intrarater reliability. A ¼ initiating the session; B ¼ how was information collected; C ¼ gathering information; D ¼ initiating separation; E ¼ once adolescent is alone ) lifestyles; F ¼ closure. ICC ¼ intraclass correlation coefficient.

DISCUSSION Our study’s purpose was to assess the reliability and validity of the SCAG. The results indicate that


487

Interrater Reliability

1.00

Intrarater Reliability

0.80

Criterion Validity

ICC

0.60 0.40 0.20 0.00 –0.20

A

B

C

D

E

F

–0.40 SCAG categories

14-year-old adolescent girls (Grade 9) can reliably score a clinical encounter using the SCAG. High levels of agreement between the gold standard and adolescents also support the SCAG’s validity. In clinical practice at Dalhousie University, the SCAG has been used in teaching and evaluating the impact of undergraduate, postgraduate and CME teaching in adolescent communication skills. The SCAG is also currently in use at other teaching centres both nationally and internationally. The SCAG has received positive feedback at educational and medical conferences.9,23 With such widespread interest in the SCAG, it is important that the psychometric properties support its effectiveness. Despite the very high levels of agreement with the SCAG in this study, 2 areas of the Guide require attention. These are section B, Ôhow was information collectedÕ (with 5 items concerning such topics as body language) and section C, Ôgathering informationÕ (with 10 items concerning areas such as listening skills and rapport). Low levels of agreement were found, both among adolescents and between the adolescents compared to the gold standard. One plausible explanation is that the items in sections B and C are more subjective to the rater (e.g. was there appropriate body language?) than other items in the SCAG that are more specific (e.g. in section D: Ôwas confidentiality discussed?Õ), and therefore more difficult to rate. It is possible that the addition of further details as to what constitutes, Ôappropriate body languageÕ would result in the adolescent rater being able to score more reliably. Another potential source of difficulty may be the raters’ ability to differentiate between ÔdidÕ and Ôdid wellÕ options on the SCAG. Closer analysis of the scores suggests that the majority of problematic items arose from raters’ failure to reliably score whether the subjective items were ÔdoneÕ or Ôdone wellÕ. The raters did not appear to have difficulty differentiating whether or not the checklist item was performed, but whether it was done well.

Figure 5 Global rating ICC scores for the 6 SCAG sections (A–F); a comparison of the inter-rater ⁄ intrarater reliability and the criterion validity. A ¼ initiating the session; B ¼ how was information collected; C ¼ gathering information; D ¼ initiating separation; E ¼ once adolescent is alone ) lifestyles; F ¼ closure. ICC ¼ intraclass correlation coefficient.

The inter-rater reliability scores for the SCAG also support its reliability with median item coefficients above 0.87 for all categories except E (lifestyles), which produced a coefficient of 0.55. This section (E) deals with personal and behavioural questions. The 4 adolescents may have had different personal views of how these questions should be asked. This speculation is supported through feedback received from 20 adolescents in a recent focus group. Readability analysis revealed that the reading level of the SCAG is relatively high. This implies that word and sentence length will need to be shortened. Also, the grade level score indicated the SCAG is appropriate for eighth graders and above. This was suitable for our intended population but may prove difficult for adolescents with a lower reading level. From these results, it was apparent that some revisions to the SCAG were needed. Revisions are continuing and we are currently conducting additional studies to determine the results of the modifications. A modified version of the SCAG has been used in the undergraduate programme at Dalhousie Medical School, CME Workshops and for formative and summative OSCE stations. With these modifications, we expect the SCAG to show increased reliability of scores from adolescents when they rate encounters with medical personnel. As well, the feedback that the adolescents provide while using the SCAG will be increasingly helpful to the receiver. Having such a tool that can be used to assess and provide feedback to medical students and practising physicians on their performance on clinical interviews with adolescents will be useful in further research studies. The development of the SCAG has obvious implications for both education and research. As this study utilized only female adolescents raters it will be important to validate this revised instrument with


488

communication male adolescents to observe if there are any gender differences in interpretation of the SCAG. If the SCAG proves to be valid and reliable not only among female adolescents but male raters as well, this will offer a valuable tool for future research and assessment in this important field of medical education.

9

10

11

Contributions:

all authors have been integrally involved in the research. KB, SW and KM were involved in the preliminary research for which the Structured Adolescent Guide was originally developed. NV and MK were research assistants for the present study and oversaw the project as well as participating in data analysis and writing. All authors met regularly to discuss the project. This area of research is KB’s main educational research interest and is unique to this field of adolescent medical education. Funding: Dalhousie University Division of Medical Education Research Grants. Conflicts of interest: none. Ethical approval: ethical approval for this research was sought and obtained from the Dalhousie University office of Human Research Ethics and Integrity.

12

13

14

15

REFERENCES 1

2

3 4

5

6

7

8

Makoul G, Schofield T. Communication teaching and assessment in medical education: an international consensus statement. Patient Educ Couns 1999;37 (2):191–5. Djuricich AM. Teaching medical residents about teenagers: an introductory curriculum in adolescent medicine. Acad Med 2002;77 (7):745–6. Goldbloom RB. Pediatric Clinical Skills, 3rd edn. Philadelphia, PA: Saunders 2002. Blake K, Mann KV, Kaufman DM, Kappleman M. Learning adolescent psycosocial interviewing using simulated patients. Acad Med Suppl 2000;75 (10): S56–8. Ford CA, Reif C, Rosen DS, Emans SJ, Lipa-glaysher B, Fleming M, Wilson T. The AMA Residency Training in Adolescent Preventive Services Project: report from the working group. Am Med Assoc J Adolesc Health 2001;29 (1):50–8. Emans SJ, Bravender T, Knight J, Frazer C, Luoni M, Berkowitz C, Armstrong E, Goodman E. Adolescent medicine training in pediatric residency programs: are we doing a good job? Pediatrics., 1998;102 (3 Part 1):588–95. Schuster MA, Bell RM, Petersen LP, Kanouse DE. Communication between adolescents and physicians about sexual behavior and risk prevention. Arch Pediatr Adolesc Med 1996;150 (9): 906–913. Clower SM. Urban female teenagers’ perceptions of medical communication. Adolescence 2002;35: 571–85.

16 17

18

19 20

21 22 23

Blake K, Massie-Clarke M, Kaufman DM, Mann K. Improving medical student’s confidence regarding adolescent interviewing. RIME 41st Annual Conference on Research in Medical Education Abstracts. Improving the Nation’s Health, 113th AAMC Annual Meeting, San Francisco, CA, USA, November 8–13, 2002: 15. Joffe A, Radius S, Gall M. Health counseling for adolescents: what they want, what they get, and who gives it. Pediatrics 1998;82:481–5. Dull P, Haines DJ. Methods for teaching physical examination skills to medical students. Fam Med 2003;35 (5):343–8. Stillman PL, Regan MB, Philbin M, Haley HL. Results of a survey on the use of standardized patients to teach and evaluate clinical skills. Acad Med 1990;65:288–92. Tamblyn R, Schnabl G, Klass D, Kopelow M. The relationship between the accuracy of standardized patient presentation and their reliability as raters on student performance. In: Hart I, Harden RM, Desmarchais J, eds. Current Developments in Assessing Clinical Competence. Montreal: Can-Heal 1992:405. Tamblyn RM, Klass DJ, Scnabl GK, Kopelow ML. Sources of reliability and bias in standardized patient rating. Teach Learn Med 1991;3 (2):74–85. Vu NM, Marcy MM, Colliver JA, Verhulst SJ, Travis TA, Barros HS. Standardized (simulated) patients’ accuracy in recording clinical performance check-list items. Med Educ 1992;26:99–104. Blake K, Greaven S. Recruiting and following adolescent standardized patients. Acad Med 1999;74:584. Hanson M, Tiberius R, Hodges B, Mackay S, McNaughton N, Dickens S, Regehr G. Evaluation methods. What do we know? Implications of suicide contagion for the selection of adolescent standardized patients. Acad Med 2002;77 (10):S100–2. Woodard CA, Gliva-Mcconvey G. Children as standardized patients: initial assessment of effects. Teach Learn Med 1995;7:188–91. Kurtz SM, Silverman JD. The Calgary–Cambridge Observation Guides. Med Educ 1996;30:83–9. Van Amstel LL, Lafleur DL, Blake KD. Raising our HEADSS. Adolescent psychosocial documentation in the emergency department. Acad Emerg Med 2004;11 (6): 648–55. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull 1979;86:420–8. Flesch RF. A new readability yardstick. J Appl Psychol 1948;32:221–33. Blake K, Vincent N, Wakefield S, Mann K, Murphy J. A Structured Communication Adolescent Guide. Assessment of reliability and validity. The 10th International Ottawa Conference on Medical Education, Programs and Abstracts, Ottawa, Canada. Proceedings of the 10th Annual Ottawa Conference, July, 2002;(3G4):39.

Received 5 January 2004; editorial comments to authors 3 March 2004; accepted for publication 23 June 2004


489

APPENDIX 1


490

communication


491
