Original article
The readability of paediatric patient information materials: Are families satisfied with our handouts and brochures? Erik Nathan Swartz BSc MD MSc FAAP FRCPC EN Swartz. The readability of paediatric patient information materials: Are families satisfied with our handouts and brochures? Paediatr Child Health 2010;15(8):509-513. ObjECTivES: To determine the grade reading level of eight easily
available patient handouts for parents of children with otitis media, to calculate the internal reliability and consistency of seven readability tests, and to explore the relationship between the grade reading level of a handout and parent satisfaction. METHOdS: Eight patient handouts developed for the parents of children with otitis media were collected, stripped of all formatting and analyzed using seven different readability formulas. Cronbach’s alpha and intraclass correlation coefficients (ICC) were calculated to determine the internal reliability and consistency of the seven formulas. Parents were surveyed on their satisfaction with three of the handouts, and their responses were compared with the handouts’ reading grade levels using ANOVA. RESulTS: Only four of the eight handouts had a mean grade 8 or less reading level. None of the handouts had a grade 5 or less reading level. Cronbach’s alpha was calculated to be 0.990, single measures ICC 0.931 and average class ICC 0.990, indicating extremely high internal reliability/consistency among the different readability tests. One-way ANOVA showed no evidence of a significant difference in parental satisfaction with the three handouts tested, despite their different grade reading levels (grades 7, 10 and 14) (P=0.24). CONCluSiONS: While many readability formulas are available, the high internal reliability/consistency among them indicates that only one formula needs to be used to assess readability (eg, the FleschKincaid Grade Level formula). Because there was no difference in parental satisfaction among three patient handouts that had widely different grade reading levels, studies should explore whether writing patient information materials explicitly to achieve low grade reading level scores is a worthwhile strategy. Key Words: Comprehension; Informatics; Literacy; Patient education
handout; Readability
F
amily-centred medicine is a cornerstone of 21st century paediatrics. Paediatricians diagnose and treat young patients while attempting to educate parents about their children’s diseases. As a profession, paediatricians also support the concept of anticipatory guidance, hoping to give parents direction even before problems occur. As much as possible, this education is given in person at office visits. However, it is very difficult to elicit a full history, perform a thorough physical examination, make a diagnosis,
la lisibilité des documents d’information pour les parents : les familles sont-elles satisfaites de notre documentation et de nos dépliants? ObjECTiFS : Déterminer le niveau de langue de huit documents faciles à obtenir destinés aux parents d’enfants atteints d’otite moyenne, calculer la fiabilité interne et l’uniformité de sept tests de lisibilité et explorer le lien entre le niveau de langue d’un document et la satisfaction des parents. MÉTHOdOlOGiE : Les chercheurs ont accumulé huit documents élaborés pour les parents d’enfants atteints d’otite moyenne, dénués de toute mise en page, et les ont analysés au moyen de sept formules de lisibilité différentes. Ils ont calculé le coefficient alpha de Cronbach et le coefficient de corrélation intraclasse (CCI) pour déterminer la fiabilité interne et l’uniformité des sept formules. Les parents ont été sondés au sujet de leur satisfaction à l’égard de trois des documents, et les chercheurs ont comparé leurs réponses avec le niveau de langue des documents, au moyen du test ANOVA. RÉSulTATS : Seulement quatre des huit documents avaient un niveau de langue moyen de 8e année ou moins. Aucun n’avait un niveau de langue de 5e année ou moins. Les chercheurs ont obtenu un coefficient alpha de Cronbach de 0,990, des mesures uniques de CCI de 0,931 et une classe moyenne de CCI de 0,990, indiquant une fiabilité interne et une uniformité extrêmement élevées entre les divers tests de lisibilité. Le test ANOVA unidirectionnel ne démontre aucune manifestation de différence significative quant à la satisfaction des parents à l’égard des trois documents vérifiés, malgré leurs divers niveaux de langue (7e, 10e et 14e année) (P=0,24). CONCluSiON : Il existe de nombreuses formules de lisibilité. D’après la fiabilité interne et l’uniformité entre elles, une seule formule est nécessaire pour évaluer la lisibilité (p. ex., le test de lisibilité Flesch-Kincaid). Puisqu’il n’y avait pas de différence dans la satisfaction des parents entre trois documents aux niveaux de langue très différents, les études devraient viser à évaluer si la rédaction d’information pour les parents, conçue explicitement pour réduire le niveau de langue, est une stratégie valable.
develop a treatment plan and then provide parents with adequate time to ask questions. Furthermore, it is not uncommon for parents to think of new, important questions after leaving the doctor’s office. Patient handouts are frequently used as an educational aid to provide parents with information about their children’s diagnoses and treatments, and to answer their most frequently asked questions. Readability formulas provide an easy method to assess the reading difficulty of these aids by calculating the
Department of Pediatrics, University of British Columbia, Vancouver; Department of Pediatrics, Richmond Hospital, Richmond, British Columbia Correspondence: Dr Erik Nathan Swartz, Department of Pediatrics, Richmond Hospital, 7000 Westminster Highway, Richmond, British Columbia V6X 1A2. Telephone 604-233-3222, fax 604-233-5620, e-mail
[email protected] Accepted for publication October 9, 2009 Paediatr Child Health Vol 15 No 8 October 2010
©2010 Pulsus Group Inc. All rights reserved
509
Swartz
number of formal years of schooling that a reader would need to understand the tested reading sample on the first reading. The aims of the present study were to determine the grade reading level of eight easily available patient handouts for parents of children with otitis media, to calculate the internal reliability and consistency of seven readability tests, and to explore the relationship between the grade reading level of a handout and parent satisfaction. METHOdS Handout topic choice Acute otitis media is the medical term for inflammation of the middle ear. In North America, it is the most common condition for which antibiotics are prescribed (1). Approximately US$3 billion is spent annually in the United States on direct and indirect costs related to acute otitis media (2). For these reasons, the topic of otitis media is addressed in many readily available handouts for distribution to the parents of children suffering with this infection, and was chosen to be the subject of the test materials used in the present study. Sources of test materials • Canadian Paediatric Society (CPS) • American Academy of Pediatrics • Micromedex CareNotes System (Thomson Reuters [Healthcare] Inc, USA) • American Academy of Family Physicians (familydoctor.org) • MD Consult • Nemours Foundation • National Institutes of Health • UpToDate All of the handouts used are freely available on the Internet, with the exception of Micromedex CareNotes System, MD Consult and UpToDate, which require paid subscriptions. Readability formulas • Gunning-Fog Index • Flesch-Kincaid Grade Level • Simplified Measure of Gobbledygook • Coleman-Liau Index • Automated Readability Index • Fry Readability Formula • Dale-Chall Readability Formula In different ways, all of these formulas calculate the number of formal years of schooling that a reader would need to understand the tested reading sample on the first reading. Readability formula testing The eight patient handouts were stripped of all formatting and graphics and saved as text (.txt) files. These files were 510
then run through an online tool (3) that reported the Gunning-Fog Index, Flesch-Kincaid Grade Level, Simplified Measure of Gobbledygook, Coleman-Liau Index and Automated Readability Index. The text files were also sent to the Lister Hill National Center for Biomedical Communications at the National Library of Medicine (Bethesda, USA), where they were analyzed using an in-house readability analyzer (4) that determined their Fry readability grade levels and Dale-Chall scores. The grade reading level scores calculated by the seven different formulas for each of the eight tested source materials were analyzed using SPSS version 16.0 (SPSS Inc, USA) for Windows (Microsoft Corporation, USA). Reliability analysis Reliability was measured in two ways: • Internal consistency: Cronbach’s alpha was calculated using SPSS (version 16.0) for Windows. • Inter-rater reliability: Single measures intraclass correlation coefficients (ICC) give the reliability for a typical single test’s rating, while average measures ICC give the reliability for the mean of the ratings for all tests. These ICC were calculated using SPSS (version 16.0) for Windows. Sources of test materials used for parent satisfaction determination Once readability testing of the eight source materials was completed using the aforementioned seven readability formulas, three were chosen for the next phase of the present study in which parents were surveyed on their satisfaction with particular materials. Micromedex CareNotes System was ranked as the most readable source material by six of seven readability formulas, and the second most readable by the remaining formula (Automated Reading Index). UpToDate was ranked as the most difficult source material by all seven readability formulas. The CPS material scored in between the two. Study eligibility All caregivers of children attending the Pediatric Ambulatory Clinic at the Stollery Children’s Hospital (Edmonton, Alberta) were eligible to participate, except for the following: • caregivers attending with a child whom they believed had an ear infection; • caregivers who had previously participated in the study; and • caregivers of children who had previously had another caregiver participate. Between August 7 and August 28, 2007, caregivers of consecutive patients attending the Pediatric Ambulatory Clinic were asked to participate by a receptionist if they met eligibility criteria. Those who wished to take part in the study were given an opaque, sealed envelope containing an information sheet/consent form, a data collection sheet, and one of three randomly selected test materials from Paediatr Child Health Vol 15 No 8 October 2010
The readability of paediatric patient information materials
Reading Grade Level
15
100
Readability Formula ARI Coleman Liau Dale Chall Flesch Kincaid Fry Gunning Fog SMOG
80
Usefulness (mm)
20
10
60
40
5 20
AAP
CareNotes
CPS
Familydoctor.org MD Consult
Nemours
NIH
UpToDate
0
Patient Information Handout Canadian Paediatric Society Micromedex CareNotes
Figure 1) Cluster bar graph of reading grade level versus patient
information handout. Each bar represents a different readability test. AAP American Academy of Pediatrics; ARI Automated Readability Index; CPS Canadian Paediatric Society; NIH National Institutes of Health; SMOG Simplified Measure of Gobbledygook either Micromedex CareNotes, UpToDate or the CPS. Caregivers read the information sheet and test material and then completed the data collection sheet in the clinic waiting room. The data collection sheet asked for demographic information including age, sex, first language spoken, level of schooling, and country where the majority of their schooling was undertaken. The top of the sheet asked, “How useful would this handout be to the caregiver of a child with an earache?” The study participant was asked to answer this question by making a large mark on a 100 mm visual analogue scale (VAS). Sample size Before the start of the study, sample size calculations were completed using the statistical software package PASS 2003 (NCSS, USA). Equal sample sizes and equal SDs in each of the three groups were assumed for one-sided, two independent samples Student’s t tests with Bonferroni correction. A sample size of 345 (n=115 for each of the three groups) would be needed, given a significance value of P
