Factsheets were downloaded in *.pdf format from the MSA Trust website ... format for analysis. The file conversion was made with the Adobe online file converter.
FACULTY OF ARTS, HUMANITIES AND SOCIAL SCIENCES
The Readability of Multiple System Atrophy Trust Factsheets. A content analysis based upon two measurements of readability.
TW5221 Research Project Assignment
Submitted by Dermot Keenan
A research report of interest to authors of healthcare information and those for whom the subject of readability is important.
December 2017
Summary Multiple System Atrophy (MSA) is a rare neurodegenerative disease of uncertain cause and no known cure. The MSA Trust places Factsheets on its website to guide patients and their carers in the management of the disease. This research assessed the difference in readability of the most downloaded Factsheets using the Flesch-Kincaid grade level score and Propositional Idea Density. The correlation of these two measurements was also assessed. Flesch-Kincaid and Propositional Idea Density scores were selected as their readout is based upon completely different elements of the analysed text. The list of most downloaded Factsheets was provided by the MSA Trust. Factsheets were downloaded from the MSA Trust website and converted to Microsoft (MS) Word for analysis with the Flesch-Kincaid grade score. Propositional Idea Density (PID) was calculated using the Computerized Propositional Idea Density Rater (CPIDR 3) from the University of Georgia. Measurements were made for each section of each Factsheet and for each complete Factsheet. The eight most downloaded patient Factsheets vary in their readability, both between and within each document. Flesch-Kincaid readability scores vary between 6 and 11, which is from easy to quite difficult to read. Propositional Idea Density scores between 6 and 10 propositions per sentence, suggesting all Factsheets require prior knowledge to be understood. There is high degree of correlation between the Flesch-Kincaid and Propositional Density scores at the level of complete document scores. There is a much lower level of correlation when scores at document section level are compared. The most difficult Factsheet scored by Flesch-Kincaid is that outlining Basic Benefits and Entitlements, whereas Propositional Idea Density scores find the Bowel Management Factsheet most difficult.
ii
Acknowledgements Thanks to Dr Yvonne Cleary has been an excellent tutor and has given invaluable feedback. Thanks also to Emma Rushton, Information and Services Manager at the MSA Trust, for guidance regarding the most downloaded Factsheets for analysis. Thanks especially to my wife Lynne for her tolerance, humour and constructive comments.
iii
Table of Contents Summary............................................................................................................................................... ii Acknowledgements ...............................................................................................................................iii 1
INTRODUCTION ............................................................................................................................. 1
2
LITERATURE REVIEW ..................................................................................................................... 2 2.1 Readability and its measurement ............................................................................................. 2 2.2 The importance of readability. .................................................................................................. 2 2.3 Flesch-Kincaid formula .............................................................................................................. 3 2.4 Propositional idea density ......................................................................................................... 3 2.5 Readout of readability indices................................................................................................... 4 2.6 Readability of healthcare documentation ................................................................................. 4 2.7 Summary ................................................................................................................................... 5
3
METHODOLOGY ............................................................................................................................ 6 3.1 Method type ............................................................................................................................. 6 3.2 Data collection instrument........................................................................................................ 6 3.3 Sample ...................................................................................................................................... 6 3.4 Data analysis techniques ........................................................................................................... 6
4
3.4.1
Flesch-Kincaid................................................................................................................ 7
3.4.2
Propositional Idea Density ............................................................................................ 7
3.4.3
Statistics ........................................................................................................................ 7
RESULTS and DISCUSSION ............................................................................................................. 8 4.1 Comparison of Propositional Idea Density and Flesch-Kinkaid grade level scores .................... 8 4.2 Factsheet readability ............................................................................................................... 10 4.3 Consistency of readability within a Factsheet. ........................................................................ 11
5
CONCLUSIONS and RECOMMENDATIONS .................................................................................. 13
6
REFERENCES ................................................................................................................................ 14 Appendix A: The data collection instrument. .................................................................................. 16 Appendix B: Results. ....................................................................................................................... 17 Appendix C: Variability between Factsheets. .................................................................................. 18 Appendix D: CPIDR3 software. ........................................................................................................ 19 Appendix E: Flesch-Kincaid grade scale formula. ............................................................................ 20 Appendix F: Factsheet example, PEG Feeding, the easiest to read. ................................................ 21 Appendix G: Factsheet example, Basic Benefits and Entitlements, the least easy to read, with the most difficult Section highlighted. .................................................................................................. 25
iv
1 INTRODUCTION This is a report based on research into the readability of Factsheets produced by the MSA Trust. McLaughlin (1969) described readability as “the degree to which a given class of people find certain reading matter compelling and comprehensible.” Readability was assessed using two methodologies; the Flesch-Kincaid grade level score (Dubay 2006) and the Density of Propositional Ideas (Kintsch and van Dijk 1978). These two measurements are based upon analysis of completely different elements of text and were selected for that reason. The Flesch-Kincaid grade level score (FK) is based on the premise that long sentences of words comprising many syllables are difficult to read. Propositional Idea Density (PID) seeks to quantify semantic structure (Kintsch and van Dijk 1978). A content analysis was conducted on the eight most downloaded Factsheets, designed to answer the following research question: Are the most downloaded MSA Trust Factsheets consistently easy to read? This report will present the methods, results and conclusions of the research with a review of the relevant literature. Readability results are presented which were generated using two different software tools. These tools were not validated as part of this study. There is no information regarding the populations of people who downloaded the Factsheets, or their motives for doing so. There are many definitions of readability and many approaches to measuring it. The two approaches here were selected because they assess readability very differently. The report does not include any analysis of which readability assessments may be best or worst. Statistics generated using MS Excel are reported. The validation of Excel is not within the scope of this report. The FK readout corresponds clearly to a grade level, whereas the PID readout does not. Analysis of PID scores follows the lead of DeFrancesco and Perkins. They noted (DeFrancesco and Perkins 2012, p.23) that in the absence of norm-referenced data on Propositional Idea Density, “sentences with extremely dense propositional load” might “create cognitive overload for some readers.” The report is laid out below in four sections, commencing with Section 2, a review of current literature on readability, its importance in healthcare and the evolution of readability measurement. Section 3 describes the way in which this research was conducted and the results collected and analysed. Section 4 shows the research findings and discusses the main themes which they exhibit. Section 5 is about the conclusions which may be drawn from the research findings and includes suggestions for future research. The next section of this report is a review of literature on readability, its measurement and importance in on-line healthcare documentation.
1
2 LITERATURE REVIEW This section discusses the literature on readability, its importance and its specific relevance in healthcare documentation. This research is based upon the Flesch-Kincaid grade level score and Propositional Idea Density, therefore there is additional information on these two metrics.
2.1 Readability and its measurement As more people access healthcare information through the internet (Fox 2006, Tustin 2010), source credibility and trust become more important. According to Fox (2006, p 1), “Very few people check the source or age of the information they find.” The high level of healthcare web usage coupled with concerns regarding the quality of online healthcare information prompted the European Commission in 2002 to develop a code of conduct intended to improve the quality of on-line healthcare information (McInnes 2011). Readability is one quality metric. Readability of content is also important as levels of literacy are declining (Kuczera 2016). Healthcare content destined for a wide audience must take into account the level of literacy of that audience if it is to be accessed, read and understood. This literature review will examine readability, its importance and measurement in healthcare documentation. L.A Sherman in 1893 became a pioneer of readability study when he published his manual Analytics of Literature (Dubay 2006). He became the first person on record to have statistically analysed the written word and drawn conclusions regarding the clarity of text. He also realised the importance of taking into account the audience of the text. Sherman even introduced the concept of communicating notions, which may be regarded as a precursor to the ideas of Kintsch (Kintsch 1978). Dubay (2004, p3.) suggests that “Dale and Chall’s (1949) definition of readability may be the most comprehensive.” He quotes Dale and Chall: “The sum total (including all the interactions) of all those elements within a given piece of printed material that affect the success a group of readers have with it. The success is the extent to which they understand it, read it at an optimal speed, and find it interesting.” Klare (1984) states that the term readable may refer to one of three characteristics of reading material; to legibility of the reading material, to ease of reading due to the interest the material creates or to ease of understanding owing to the style of writing. Dubay (2004, p3.) quotes the creator of the SMOG readability formula, G. Harry McLaughlin, who in 1969 defined readability as: “the degree to which a given class of people find certain reading matter compelling and comprehensible.” This definition stresses the interaction between the text and a class of readers, of known characteristics such as reading skill, prior knowledge, and motivation. Consideration of these definitions leads to the conclusion that readability is dependent upon both the text and the reader of that text, and upon technical construction of the text and the interest which it can generate.
2.2 The importance of readability. Klare (1984) notes that, in the 1970’s the US government produced one million words of new regulations every day. He further cites reading skills in the US having declined during the preceding two decades. Grey and Leary (1935, p.3) also remarked that “about 50 per cent of our population cannot read with ease.” Klare (1984) postulates that making writing easier to understand may offer a partial solution to the issue of increasing reading volume and literacy 2
decrease. Growth in the volume of stored information between 1986 and 2007 is estimated at 23% per year (Hilbert & Lopez 2011), whilst Kuczera et al (2016) show that one quarter of adults between 16 and 65 years have low basic literacy skills. Thus, the issue which Klare (1984) highlighted is current and the importance of readability in documents has grown in consequence. In a different but related slant on literacy, Badarudeen and Sabharwal (2010, p.2572) refer to health literacy as “capacity to obtain, interpret, and understand basic health information and services and the competence to use such information and services to enhance health.” They go on to state that “poor health literacy is estimated to contribute more than 73 billion dollars of additional burden to the US healthcare system.” Readability is key to the processes of interpreting and understanding healthcare information. In this instance, its weakness attracts high cost. However, before it is possible to know how readable a particular text is likely to be for a particular audience, Klare (1974) suggests that it must either be measured through exposure to readers through experiment or its readability predicted using counts of language elements in the writing. Badarudeen and Sabharwal (2010) cite the following readability assessment tools as being commonly used in healthcare. Flesch Reading Ease scale 1944, the Flesch-Kincaid Grade 1951, Fry Readability Graph 1977, McLaughlin’s SMOG grading 1969, the Gunning Fog Index 1948, the DaleChall Readability formula 1948, and Suitability Assessment of Materials.
2.3 Flesch-Kincaid formula Flesch is probably the “one most responsible for publicizing the need for readability” (Dubay 2006). His formula had a huge impact on journalism at that time. His best-selling book, The Art of Plain Talk, compared the Flesch Readability Scores of different magazines and was responsible for drawing attention to the levels of literacy required to understand each magazine. Through his readability research, he is also credited (Dubay 2006) with stimulating editors to make the front pages of New York papers easier to read. Flesch separated the assessment of ease of reading, from that of the interest of a piece. Ease of reading was assessed through average counts of syllables per word and average sentence length in a 100-word sample, whilst interest was measured through counts of personal words and personal sentences. In spite of its popularity, Flesch’s original readability formula had several shortcomings. Six minutes to score a hundredword sample was felt impractical; conversational text could be mis-scored; weighting of scores tended towards the easiest to measure parameters such as sentence length and some observers were uncomfortable with the scoring scale as well as score differentiation between harder-toread texts. The test was simplified by Farr, Jenkins and Paterson in 1951(Dubay 2004) and then modified to give a read-out which corresponds to the US school grade system. This is the “FleschKincaid formula, the Flesch Grade-Scale formula or the Kincaid formula” (Dubay 2004, p21).
2.4 Propositional idea density All of the above systems assess the readability of text based upon the validated correlation of counts of word or sentence length with text complexity. Kintsch (Kintsch and van Dijk 1978) researched the retention and understanding of text. He maintains that his “model’s predictions are relevant not only for recall but also for readability of texts” (Kintsch and van Dijk 1978, p.372). Kintsch assumes the existence of an “underlying semantic structure” (Kintsch and van Dijk 1978, p.364), which can be characterised in terms of what he describes as a proposition. He maintains that the structure of discourse can be represented by a set of propositions and that we communicate through the establishment of “a linear or hierarchical sequence of propositions” (Kintsch and van Dijk 1978, p.365). The more complex a given text is, the greater number of 3
propositions are communicated within it. Snowden (1996) went on to state that an estimation of the number of propositions in a text may be based upon the number of verbs, adjectives, adverbs, prepositions, and conjunctions in that text. The more propositions which are included in a given text length, the higher the density of propositions (or the higher the propositional idea density) and therefore the higher the text complexity.
2.5 Readout of readability indices Dale-Chall, Flesch and Gunning Fog approaches all assess the readability of text based upon the validated correlation of counts of word or sentence length with text complexity. They all transform these counts into a score based upon different statistical treatment. Often the score relates to expected reading standard of US school and college levels, which may arguably not be transferable to adults. There is no similar calibrated output from measurement of propositional idea density. Propositional idea density does not correlate with results from a derivative of the Flesch Reading Ease Test, Flesch-Kincaid Readability. (Covington 2008). Covington quotes Miller and Kintsch when he suggests that “the idea density of a text determines the amount of work a reader must do to understand it” (Covington 2008, p.3). He continues that readers familiar with a subject may easily read texts with a higher idea density than those unfamiliar with the subject. Idea density may therefore be “a measure of the amount of pre-supposed knowledge” (Covington 2008, p.3). Thus Kintsch’s approach acknowledges that the reader him/herself is important is the assessment of readability. Kintsch states “a reader's knowledge determines to a large extent the meaning that he or she derives from a text” (Kintsch and van Dijk 1978, p. 371). DeFrancesco and Perkins (2012) also note the absence of norm-referenced data with which to interpret propositional density scores, but cite research regarding the capacity of short term memory as being able to hold perhaps four pieces of unrelated information. On this basis, sentences with more than this number of propositions should be difficult to read.
2.6 Readability of healthcare documentation Readability assessment of healthcare materials has become increasingly important. This is due to understanding of the proportion of the population who have low levels of literacy (Kuczera 2017) along with confirmation of the frequency with which the internet is used to source healthcare information (Sillence 2006). Sillence (2006) also comments on the issue of access to credible high quality on-line health advice. McInnes and Haglund (2011) refer to European Commission code of conduct for websites in their review assessment of readability of different healthcare website content. The number of returns from a Google Scholar search for “readability of healthcare materials” shows an increase of more than 5% over the past two years, reflecting the importance of this subject. This research project seeks to establish readability scores of most accessed Patient Factsheets posted on-line by The Multiple System Atrophy Trust. Measurements of the Flesch Readability Ease Test and of Proposition Idea Density will be made. The objective is to highlight those Factsheets, or sections of Factsheets whose readability demands a higher level of literacy.
4
2.7 Summary The level of literacy is declining. Academic interest in readability is increasing. The demand for online healthcare documentation is increasing. The readability of this material must take account of the range of literacy, if it is to make materials accessible to the widest range of literacy possible. The following section of this report presents the methodology used in this research.
5
3 METHODOLOGY This content analysis research study analysed the readability the most downloaded MSA Trust Factsheets. Two metrics were applied to the most downloaded MSA Trust Factsheets, the FleschKincaid grade level and Propositional Idea Density scores. Each whole document was analysed, followed by analysis of each document section. (A document section was defined as the text under a section header). Analysis was conducted in MS Word for the Flesch-Kincaid grade level score (Microsoft Office 2012). Propositional Idea Density was calculated using CPIDR3 software (see Appendix D: CPIDR3 software, page 22). From the application of these two metrics and the analysis of the results obtained, the following research question will be answered: Are the most downloaded MSA Trust Factsheets consistently easy to read?
3.1 Method type A quantitative content analysis was conducted based upon metrics from the Flesch-Kincaid grade level score and Propositional Idea Density Analysis.
3.2 Data collection instrument Data was collected and analysed in an MS Excel spreadsheet. (For a copy of the data collection instrument template including sample results, see Appendix A: The data collection instrument, page 19).
3.3 Sample The following (most-downloaded) Factsheets were analysed.
3.4
A guide to multiple system atrophy. Bowel management in MSA. Living with postural hypotension. Continence in MSA. Saliva control. Monitoring blood pressure. PEG feeding. Basic benefits and entitlements.
Data analysis techniques
Factsheets were downloaded in *.pdf format from the MSA Trust website (http://www.msatrust.org.uk/support-for-you/factsheets/ ). They were converted to MS Word *.docx file format for analysis. The file conversion was made with the Adobe online file converter (https://cloud.acrobat.com/exportpdf ).
6
3.4.1 Flesch-Kincaid The option to enable readability statistics was enabled in MS Word 2013. Each Factsheet file in *.docx format was opened in Word. The spelling and grammar check was run, ignoring any requests for edits of grammar or spelling. On completion of this, readability statistics were displayed and the following recorded.
Number of sentences. Number of passive sentences. Flesch Reading Ease. Flesch-Kincaid grade level.
The operation was repeated for each section and subsection of the Factsheets and the results were stored in MS Excel for analysis.
3.4.2 Propositional Idea Density CPIDR3 software (Covington 2012) and the associated manual were downloaded from http://www.ai.uga.edu/caspr and installed on a PC running MS Windows 10 (64-bit). CPIDR3 automatically determines Propositional Idea Density on the basis that “although it is roughly correct to equate every verb, adjective, adverb, conjunction, and preposition with an idea (proposition), numerous readjustment rules are needed to get an accurate count.” Covington (Covington 2012, p7)) continues “CPIDR 3 does not understand every sentence in full and therefore does not produce perfect proposition counts, but it has been shown to be more reliable than most if not all human raters.” On this basis the software was selected. Text was selected from each *.docx Word file and copy-pasted into the analytical window of CPIDR3. On selecting analyse from the menu, the following metrics were output and recorded.
Ideas (corresponding to Propositions). Words. Density.
The Idea Density per sentence was calculated by dividing Ideas measured by CPIDR3, by sentences as measured by MS Word.
3.4.3 Statistics The MS Excel add-in for data analysis was enabled. The statistical functions for correlation and for regression analysis were used, as were the built-in formulas for mean and standard deviation. The next section of the report summarises the results and their significance.
7
4 RESULTS and DISCUSSION In this section, a summary and commentary on the results are presented. (The full set of results is presented in Appendix B: Results, page 20). Firstly in 4.1, examination of the two metrics, Propositional Idea Density and the Flesch-Kinkaid grade level scores and how they compare. Next in 4.2, the readability of individual Factsheets is considered; which appear easiest to read and why. Finally in 4.3, to answer the research question regarding consistency of readability, variability of readability scores vary within a Factsheet is presented.
4.1 Comparison of Propositional Idea Density and Flesch-Kinkaid grade level scores In this section the scores for FK and PID are shown, along with regression analysis between FK and PID. The regression is performed on the PID scores for each complete Factsheet (a total of 8 observations). It is also performed upon the dataset of scores for all the sections of the 8 factsheets (a total of 60 observations). Figure 1: The PID and FK scores for the most-downloaded Factsheets.
PID and FK scores measure different elements of text, therefore their readouts may not be directly comparable. Covington (Covington 2008, p.3) states “idea density is a promising tool” and goes on “it is distinct from lexical diversity and Flesch-Kincaid readability index.” However, the manner in which FK and PID scores change depending upon the complexity of the text, is measurable through regression analysis.
8
Firstly the regression analysis of FK and PID scores for the complete Factsheets. Figure 2: Regression analysis of FK and PID scores for whole Factsheets.
The correlation coefficient (Multiple R) of 0.94 is high and demonstrates a high degree of correlation between the two scores. The P value of substantially less than 0.05 suggests that the result is statistically significant. The R square value shows how many data points lie upon the regression line, 89%. This is in contrast to Covington’s findings (Covington 2008).
However, when the regression of FK and PID scores for all the Factsheet sections is examined, the results are completely different and the correlation between FK and PID scores is almost nonexistent. See Figure 3 below. Now the Multiple R (correlation coefficient) suggests little relationship between FK and PID scores, very few points lie upon the regression line and the P value gives little statistical relevance to the correlation. Figure 3: Regression analysis of FK and PID score for all Factsheet sections.
There data point circled in red appears to be an outlier. This result is generated from the “Symptoms” section in the Postural Hypotension Factsheet. This list generates a large number of ideas when scored with CPIDR3, however the descriptions use quite small clear words. This appears inconsistent with the rest of the data set. If it is removed and the regression recalculated, the result is shown in figure 4 below. 9
Figure 4: Regression analysis of FK and PID score for all Factsheet sections with outlier removed.
With the outlier removed, the correlation between FK and PID scores is much higher and the P value suggests that there is now a real relationship between FK and PID.
4.2 Factsheet readability The Flesch-Kincaid Grade scale and Propositional Idea Density per sentence are shown in Table 1. For each Factsheet. Scores are shown in column “index scores.” In column “In order of Readability”, number “1” represents the easiest Factsheet to read, scored by FK and PID. The combined score is a simple way of placing them in order of their readability, the lowest score being the easiest to read. PEG Feeding is therefore easiest to read and Basic Benefits and Entitlement the most challenging. Figure 5: Readability scores by Factsheet Index Scores Document FK Grade PEG Feeding Monitoring Blood Pressure Postural Hypotension Saliva Control A Guide to MSA Continence in MSA Basic benefits and entitlements Bowel Management
6.2 7.7 8.3 8.7 10.2 10.6 11.1 10.2
Ideas / sent 6.7 8.1 8.4 8.8 9.7 9.7 10.6 11.0
In order of Readability Combined Readability Order Order FK Order PID 1 2 3 4 5 7 8 6
1 2 3 4 5 6 7 8
2 4 6 8 10 13 15 14
The Flesch-Kincaid grade scale score assesses readability and presents the result as a US grade level. The range of FK readouts is between 6.2 and 11.1. A sixth grade student would be aged 11 to 12 years. The grade readout would be the typical reading level expected at that age. Eleventh grade would represent a 16-17 year old’s reading level. This is a large difference in readability. According to Dubay (Dubay 2004), Flesch suggested that only 50% of adults could comfortably read text rated as fairly difficult and corresponding to the capabilities of 10th to 12th grade. Half of the most-downloaded Factsheets have scores of 10 or greater and are fairly difficult. 10
Unlike the Flesch-Kincaid grade scale result, the Propositional Idea Density readout cannot be directly related to a school grade. Its readability-related interpretation is less clear-cut. Covington (2008, p.3) proposes that “the idea density of a text determines the amount of work a reader must do to understand it.” He continues explaining that a familiar idea to a reader will be more easily understood and that this reader can manage texts with a higher idea density comfortably. He therefore suggests “idea density is an indirect measure of the amount of pre-supposed knowledge.” The results above suggest that those Factsheets with the highest FK grade scores also require the highest amount of background knowledge, if they are to be understood. A further interpretation of propositional density analysis scores is provided by DeFrancesco and Perkins (DeFrancesco and Perkins 2012, p.23) who cite Millers notion “that a person could hold from five to nine pieces of unrelated information in the short-term memory for processing, but more recent research indicates that the estimate should be lowered to as few as four.” They also cite the mechanism for transferring information from short, to long short memory and suggest that higher density material can overload this and cause difficulty in learning the content. All the Factsheets have Propositional Densities per sentence greater than 4.
4.3 Consistency of readability within a Factsheet. Coefficient of variability, (%CV) is a measure of the variation within a dataset. The standard deviation is divide by the mean of the dataset and expressed as a percentage. FK and PID scores were calculated for each Factsheet section. The %CV’s were then calculated for the scores of all the sections within a Factsheet. A high %CV shows that there is high variation between the readability scores for the various sections within a Factsheet. %CV was calculated separately for FK and PID scores. The results are shown in Figure 6 below. Figure 6: Coefficients of variability (%CV’s) of FK and PID scores between Factsheet sections.
11
Figure 7 below shows the FK and PID mean scores for each Factsheet and the range of scores which are found for the individual sections. There is variation in both the readability scores and the range of scores within each Factsheet. Figure 7: Range and mean of FK and PID scores across Factsheet sections
The following section of this report presents the conclusions and recommendations.
12
5 CONCLUSIONS and RECOMMENDATIONS The research question for this study was: Are the most downloaded MSA Trust Factsheets consistently easy to read? This content analysis study has been based upon readouts from the Flesch-Kincaid grade scale and Propositional Density Analysis per sentence. Their agreement is very high when the entire block of text within a Factsheet is analysed, yet there is little agreement when the analysis is of the smaller sections. The correlation apparent in Figure 2 is not consistent with the literature (Covington 2008, DeFrancesco and Perkins 2012). It may therefore be the case that analysis of a larger text sample reduces the scope for readout differences between FK and PID. A further study would be needed to validate this conclusion. Figure 1 shows that there is considerable difference in the readability scores across the most downloaded MSA Trust Factsheets, with Figure 2 demonstrating the agreement between FK and PID in this regard. Four of the eight Factsheets have FK scores which would correspond from easy to read through plain English easily understood by 13 to 15 year olds. The remainder would be scored as fairly difficult to read (Wikipedia 2017). The PID scores exceed a density of 4 propositions per sentence for all the Factsheets analysed. DeFrancesco and Perkins ((DeFrancesco and Perkins 2012) would suggest that this density is likely to cause difficulties for some readers. Covington’s work (Covington 2008) would suggest some previous experience would be needed to fully understand the Factsheets. Thus they could not be described as easy to read. There is variation in the level of readability within all of the Factsheets. The variation in readability is different by Factsheet as shown in Figure 6. The Factsheets are not consistently easy to read. Further work is needed to understand the meaning of the Propositional Idea Density scores and their significance when applied to problems of readability such as this content analysis study. The results regarding correlation between FK and PID scores are intriguing and demand more experimental work to confirm. Supporting this work with a usability study would reinforce its credibility. A group of users with different levels of background knowledge regarding the disease would give very useful feedback. This would be entirely appropriate given McLaughlin’s readability definition (McLaughlin 1969) and its reference to the reader. Examination of those sections of Factsheets where readability appears particularly difficult could be fruitful. Consideration of edits might offer improvement in both overall readability and consistency of readability within a Factsheet.
13
6 REFERENCES Badarudeen, S., Sabharwal, S., (2010) ‘Assessing Readability of Patient Education Materials,’ Clinical Orthopaedics and Related Research, available: DOI 10.1007/s11999-010-1380-y [accessed 10 Oct 2017]. Covington, M., (2008) ‘Idea Density-A Potentially Informative Characteristic of Retrieved Documents’, IEEE Southeastcon 2009, Mar 2009, available: http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=5165454 [accessed 15 Nov 2017]. Covington, M. (2012) CPIDR 5.1 User Manual [online], available: https://pdfs.semanticscholar.org/a381/783cfefb1ddb4f6e8a1883a26567e45ec71d.pdf [accessed 5 Nov 2017]. De DeFrancesco, C., & Perkins, K. (2012) ‘An analysis of the proposition density, sentence and clause types, and non-finite verbal usage in two college textbooks’, Proceedings of the 11th Annual College of Education & GSN Research Conference, p. 20-25. available: http://digitalcommons.fiu.edu/cgi/viewcontent.cgi?article=1199&context=sferc [accessed 15 Nov 2017]. Doak, L., Doak, C., Root, J., (1996) SAM Suitability Assessment of Materials for Evaluation of Health-Related Information for Adults, available: www.BeginningsGuides.net [accessed 16 Oct 2017] Dubay, W. (2004) The Principles of Readability, available: http://www.impactinformatio.com/impactinfo/readability02.pdf [accessed 12 Oct 2017]. Dubay, W. (2006) The Classic Readability Studies, available: http://www.ecy.wa.gov/quality/plaintalk/resources/classics.pdf [accessed 12 Oct 2017]. Fox, S. Pew (2006) Online Health Search 2006, available: http://www.pewinternet.org/2006/10/29/online-health-search-2006/ [Accessed 10 Oct 2017]. Fry, E., (1977), ‘Fry’s Readability Graph, Clarification, Validity and Extension to Level 17’, The Journal of Reading, 21 (3), 242-252, available: http://www.jstor.org/stable/40018802, [accessed 17 Oct 2017]. Grey, W., Leary, B., What Makes a Book Readable, available: https://archive.org/stream/whatmakesabookre028092mbp/whatmakesabookre028092mbp_ djvu.txt [accessed 15 Oct 2017]. Hilbert, M., Lopez, P., (2011) The World’s Technological Capacity to Store, Communicate and Compute Information, Science 332, 60-64. Kintsch, W., van Dijk, T., (1978) ‘Toward a Model of Text Comprehension and Production’, Psychological Review, 85 (5), 363-394. Klare, G., (1984) ‘Readability’, in Pearson, Handbook of Reading Research Volume 1, [online], Psychology Press, available: https://books.google.fr/books?id=46cyc9Qnx9wC&dq=editions:HG8eManVONcC&source=gbs _navlinks_s [accessed 13 Oct 2017]. 14
Klare, G., (1974) ‘Assessing Readability’, Reading Research Quarterly, available: http://www.jstor.org/stable/747086 [accessed 12 Oct 2017] Kuczera, M., Field, S., Windisch, C., (2016) Building Skills for All, A Review of England, available: https://www.oecd.org/unitedkingdom/building-skills-for-all-review-of-england.pdf [accessed 10 Oct 2017] McInnes, N., Haglund, B., (2011) ‘Readability of Online Health Information: Implications for health literacy’, Informatics for Health and Social Care,36 (4), 173-189. McLaughlin, G. H. (1969) ‘SMOG grading - a new readability formula’, Journal of Reading 22, 639-646. Microsoft Office, (2017) Test your document’s readability [online], available: https://support.office.com/en-us/article/Test-your-document-s-readability-85b4969e-e80a4777-8dd3-f7fc3c8b3fd2 [accessed 5 Nov 2017]. Sillence, E., (2007) ‘Going on line for health advice: Changes in usage and trust practices over the last five years’, Interacting with Computers, 19 (3), available: https://doi.org/10.1016/j.intcom.2006.10.002 [accessed 7 Oct 2017]. Snowdon, D. A., Kemper, S. J., Mortimer, J. A., Greiner, L. H.,Wekstein, D. R., & Markesbery, W. R. (1996) ‘Linguistic ability in early life and cognitive function and Alzheimer’s disease in late life: Findings from the Nun Study.’ JAMA, 275, 528-532. Tustin, T. (2010) The Role of Patient Satisfaction in Online Health Information Seeking, The Journal of Health Communication, 15 (1), available: DOI: 10.1080/10810730903465491[accessed 11 Oct 2017] Wikipedia (2017) Flesch-Kincaid readability tests [wiki], available: https://en.wikipedia.org/wiki/Flesch–Kincaid_readability_tests#cite_note-8 [accessed 5 Nov 2017].
15
Appendix A: The data collection instrument.
16
Appendix B: Results. Ideas per Sentences Passive Fl Ease FK Grade Ideas Words Density sentence 3 12 54.40 10.10 30.00 47.00 0.638 10.00 7 12 54.10 12.20 63.00 137.00 0.460 9.00 9 33 55.80 9.00 65.00 129.00 0.504 7.22 2 0 38.30 12.60 15.00 31.00 0.484 7.50 1 0 28.70 13.70 10.00 19.00 0.526 10.00 2 0 62.80 9.20 19.00 39.00 0.487 9.50 3 0 71.10 8.20 38.00 80.00 0.475 12.67 2 0 63.50 10.10 39.00 71.00 0.549 19.50 2 0 50.10 13.00 48.00 82.00 0.585 24.00 2 50 71.70 8.70 31.00 68.00 0.456 15.50 1 0 52.30 13.40 14.00 31.00 0.453 14.00 2 0 40.00 15.80 34.00 67.00 0.507 17.00 4 25 50.10 11.60 60.00 109.00 0.550 15.00 19 10 57.70 10.10 302.00 585.00 0.516 15.89 10 10 51.40 10.40 123.00 231.00 0.532 12.30 17 5 46.50 11.80 181.00 366.00 0.495 10.65 9 11 56.00 10.90 122.00 232.00 0.526 13.56
Doc Bowel Management Bowel Management Bowel Management Bowel Management Bowel Management Bowel Management Bowel Management Bowel Management Bowel Management Bowel Management Bowel Management Bowel Management Bowel Management Bowel Management Bowel Management Bowel Management Bowel Management
Section Intro How Dig Works Autonomic N Sys Bowel Probs with MSA Why does Cons Occur (Ssection) Why does Cons Occur (Ssection) Why does Cons Occur (Ssection) Why does Cons Occur (Ssection) Why does Cons Occur (Ssection) Why does Cons Occur (Ssection) Why does Cons Occur (Ssection) Why does Cons Occur (Ssection) Why does Cons Occur (Ssection) Why does Cons Occur (Ssection) Other Problems Diaorrhea Other factors
Sub section Section Tot Section Tot Section Tot Section Tot 1 2 3 4 5 6 7 8 9 Section Tot Section Tot Section Tot Section Tot
Bowel Management
Total Doc
Total Doc
10.20 1480.00 3063.00
0.483
11.04
A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA A Guide to MSA
What is MSA What is MSA What is MSA What is MSA What is MSA What is MSA What is MSA What is MSA Symptoms Symptoms Symptoms Symptoms Treatment & management Treatment & management Treatment & management Treatment & management Treatment & management Treatment & management Treatment & management Treatment & management Treatment & management Treatment & management Treatment & management Treatment & management Treatment & management Treatment & management Treatment & management Treatment & management Treatment & management Common questions Common questions Common questions Common questions Common questions Common questions Common questions Common questions Common questions Common questions Common questions Common questions Research
Intro 2 0 37.10 How Common is MSA 4 25 37.20 Who gets MSA 4 0 41.60 What does MSA mean 3 33 2.90 How are the nerve cells damaaged 4 75 48.20 Why do nerves cells become 4 damaged 0 56.20 How is MSA diagnosed 6 16 47.50 Section Tot 28 28 46.70 Intro 2 0 58.00 First signs 7 42 44.20 What happens next 6 0 20.40 Section Tot 17 17 53.00 Intro 2 0 33.50 Treatment 5 40 61.20 What medication can I take 3 0 51.40 Movement problems 5 20 66.10 Balance difficulties 3 0 59.90 Difficulties with bladder12cpontrol 8 45.80 Constipation 8 25 47.90 Erectile dydfunction 4 0 40.20 Dizzy with movement 9 11 59.70 Antibiotics 2 50 16.10 Swallowing 3 0 61.00 Sleep disorders 5 0 58.30 Emotional reactions 4 25 43.70 Aids and equipment 4 0 58.10 Complementary therapies 8 0 35.50 Counseling 2 0 65.10 Section Tot 80 11 52.90 Why has none heard of MSA 5 0 59.40 How is MSA different from 3 Parkinsons 0 52.30 Will my children get MSA2 50 41.40 Dementia 2 0 57.10 Can I travel abroad on holiday 8 25 55.10 What diet 3 0 55.50 I am worried about partner 3 looking 0 after 56.90 me Who can help at home 3 0 36.10 Worried about finances 6 16 44.40 What about the future 4 25 77.00 Why has palliative care been 6 mentioned 16 40.60 Section Tot 55 12 56.00 Section Tot 18 22 48.80
12.60 12.30 11.90 19.80 12.40 9.40 12.30 11.30 10.20 10.80 12.20 10.20 13.90 10.30 9.20 8.00 9.90 11.30 9.20 11.80 8.00 14.90 9.70 10.50 9.70 7.90 11.70 8.80 9.60 10.00 12.00 13.40 8.20 9.70 9.50 10.60 14.20 11.30 7.00 11.20 9.50 10.00
24.00 54.00 40.00 69.00 50.00 93.00 52.00 98.00 48.00 105.00 39.00 65.00 69.00 139.00 339.00 653.00 32.00 58.00 64.00 20.00 72.00 145.00 175.00 333.00 21.00 43.00 55.00 114.00 27.00 48.00 39.00 87.00 32.00 61.00 102.00 214.00 48.00 99.00 34.00 68.00 74.00 152.00 16.00 32.00 28.00 60.00 54.00 110.00 36.00 67.00 44.00 81.00 64.00 126.00 21.00 38.00 685.00 1394.00 93.00 165.00 36.00 73.00 41.00 72.00 11.00 24.00 64.00 133.00 35.00 71.00 29.00 64.00 31.00 73.00 50.00 107.00 40.00 73.00 42.00 88.00 517.00 1022.00 154.00 327.00
0.444 0.580 0.538 0.531 0.457 0.600 0.496 0.519 0.552 0.533 0.497 0.526 0.488 0.482 0.563 0.448 0.525 0.477 0.485 0.500 0.487 0.500 0.467 0.491 0.537 0.543 0.508 0.553 0.491 0.564 0.493 0.569 0.458 0.481 0.493 0.453 0.425 0.467 0.548 0.477 0.506 0.471
12.00 10.00 12.50 17.33 12.00 9.75 11.50 12.11 16.00 9.14 12.00 10.29 10.50 11.00 9.00 7.80 10.67 8.50 6.00 8.50 8.22 8.00 9.33 10.80 9.00 11.00 8.00 10.50 8.56 18.60 12.00 20.50 5.50 8.00 11.67 9.67 10.33 8.33 10.00 7.00 9.40 8.56
A Guide to MSA
Total Doc
Total Doc
10.20 2446.00 5075.00
0.482
9.67
17
134
253
11 54.10
13 49.90
Appendix C: Variability between Factsheets. Table showing the % coefficient of variation between FK and PID scores per section of each Factsheet.
Table showing the mean of the section scores for FK and PID within each Factsheet.
18
Appendix D: CPIDR3 software. The software used was Version 3.2.2785.24603 compiled 17-Aug-07 13:40:06 PM EST. The following description of software function is taken from the on-line CPIDR3 software manual (Covington 2012, p7). How CPIDR works
“The premise of CPIDR is that although it is roughly correct to equate every verb, adjective, adverb, conjunction, and preposition with an idea (proposition), numerous readjustment rules are needed to get an accurate count. CPIDR 3 does not understand every sentence in full and therefore does not produce perfect proposition counts, but it has been shown to be more reliable than most if not all human raters. The part-of-speech tags are those of the Penn Treebank (Santorini 1995; not later versions). The most important ones are: . Sentence-ending punctuation CC coordinating conjunction CD cardinal number DT determiner IN preposition, except to JJ, JJR, JJS adjective (positive, comparative, superlative) MD modal verb NN, NNS noun (singular, plural) RB, RBR, RBS adverb (positive, comparative, superlative) TO to (preposition or infinitive) VB, VBZ, VBD, VBN, VBG verb (various forms)
19
Appendix E: Flesch-Kincaid grade scale formula. The following description of Flesch-Kincaid grade level score is from Microsoft Office (Microsoft Office 2017).
Flesch-Kincaid Grade Level test This test rates text on a U.S. school grade level. For example, a score of 8.0 means that an eighth grader can understand the document. For most documents, aim for a score of approximately 7.0 to 8.0. The formula for the Flesch-Kincaid Grade Level score is: (.39 x ASL) + (11.8 x ASW) – 15.59 where: ASL = average sentence length (the number of words divided by the number of sentences) ASW = average number of syllables per word (the number of syllables divided by the number of words).
Table from Wikipedia, after Flesch, showing grade level and associated reading capability.
20
Appendix F: Factsheet example, PEG Feeding, the easiest to read.
21
22
23
24
Appendix G: Factsheet example, Basic Benefits and Entitlements, the least easy to read, with the most difficult Section highlighted.
25
26
27
28
