Understanding the Internal and External Validity of ...

Journal of Health Communication International Perspectives

ISSN: 1081-0730 (Print) 1087-0415 (Online) Journal homepage: http://www.tandfonline.com/loi/uhcm20

Understanding the Internal and External Validity of Health Literacy Interventions: A Systematic Literature Review Using the RE-AIM Framework Kacie Allen , Jamie Zoellner , Monica Motley & Paul A. Estabrooks To cite this article: Kacie Allen , Jamie Zoellner , Monica Motley & Paul A. Estabrooks (2011) Understanding the Internal and External Validity of Health Literacy Interventions: A Systematic Literature Review Using the RE-AIM Framework, Journal of Health Communication, 16:sup3, 55-72, DOI: 10.1080/10810730.2011.604381 To link to this article: http://dx.doi.org/10.1080/10810730.2011.604381

Copyright Taylor and Francis Group, LLC

Published online: 27 Sep 2011.

Submit your article to this journal

Article views: 2060

View related articles

Full Terms & Conditions of access and use can be found at http://www.tandfonline.com/action/journalInformation?journalCode=uhcm20 Download by: [173.190.228.162]

Date: 29 January 2016, At: 22:50

Journal of Health Communication, 16:55–72, 2011 Copyright # Taylor & Francis Group, LLC ISSN: 1081-0730 print=1087-0415 online DOI: 10.1080/10810730.2011.604381

Understanding the Internal and External Validity of Health Literacy Interventions: A Systematic Literature Review Using the RE-AIM Framework KACIE ALLEN, JAMIE ZOELLNER, MONICA MOTLEY, AND PAUL A. ESTABROOKS

Downloaded by [173.190.228.162] at 22:50 29 January 2016

Department of Human Nutrition, Foods and Exercise, Virginia Polytechnic Institute and State University, Roanoke, Virginia, USA We conducted a systematic literature review, using the RE-AIM framework, with the goal of determining what information is available to inform research to practice translation of health promotion interventions developed to address health literacy. Thirty-one articles reflecting 25 trials published between 2000 and 2010 met inclusion criteria. Two researchers coded each article, using a validated RE-AIM (reach, effectiveness=efficacy, adoption, implementation, maintenance) data extraction tool, and group meetings were used to gain consensus on discrepancies. Across all studies (14 randomized controlled trials, 11 quasi-experimental; 24 clinic-based, 1 community-based), the mean level of reporting RE-AIM indicators varied by dimension (reach ¼ 69%; efficacy= effectiveness ¼ 58%; adoption ¼ 36%; %; implementation ¼ 35%; maintenance ¼ 11%). Among participants enrolled in the 25 interventions, approximately 38% were identified as low health literate. Only eight of the studies examined health literacy status as a moderator of intervention effectiveness. This review suggests that the current research on health promotion for participants with low health literacy provides insufficient information to conclude whether interventions for health literacy can attract the target population, achieve an effect that is sustainable, or be generalized outside of clinical settings. Recommendations for enhancing the design and reporting of these trials are provided.

There is a large body of literature that indicates that low health literacy and health are related (Berkman et al., 2004, Berkman et al., 2011). This relationship exists across health care utilization (e.g., related to decreased use of various screening procedures and increased use of emergency services), health knowledge (e.g., related to less awareness of health consequences of chronic disease), lifestyle behaviors (e.g., more likely to smoke and less likely to adhere to medication protocols), and health outcomes (e.g., lower global health status). Indeed, health literacy, defined as ‘‘the degree to which individuals have the capacity to obtain, process, and understand basic health information’’ is a better predictor of health status than age, income, employment, ethnicity, or education level (American Medical Association, 2004).

This research was funded in part by a grant from the National Institutes of Health-National Cancer Institute (1R03CA136457-01A2). Address correspondence to Kacie Allen, Department of Human Nutrition, Foods and Exercise, Virginia Polytechnic Institute and State University, VT Riverside, 1 Riverside Circle SW, Suite #104, Roanoke, VA 24016, USA. E-mail: [email protected]

55

Downloaded by [173.190.228.162] at 22:50 29 January 2016

56

K. Allen et al.

Further, it is estimated that 87 million (36%) of adults in the United States are at basic or below basic health literacy levels and that the annual costs of low health literacy range from $106 billion to $238 billion (Vernon, Trujillo, Rosenbaum, & DeBuono, 2007). In response to the prevalence, health outcomes, and costs associated with low health literacy, the issue has been on the national health care agenda for well over a decade and has generated a vast amount of research. A number of systematic reviews of the literature have been completed, including a seminal report published by the Agency for Healthcare Research and Quality in 2004. That review concluded that there was a strong link established between health literacy and health outcome, but that the 29 intervention studies that attempted to mitigate the effects of low health literacy on health were of only fair quality and were equivocal on intervention effectiveness. Reviews of literature that appeared shortly after the AHRQ report highlighted similarly mixed findings and concluded that the body of literature needed stronger research designs, more comprehensive interventions, and consistency in measurement and analyses of the influence of health literacy on health outcome (Berkman et al., 2004; Santo, Laizner, & Shohet, 2005). More recent reviews of health literacy-related intervention research focused on experimental research designs and reported on the effectiveness of interventions to change knowledge, social cognitions, patient and provider satisfaction, and clinical outcomes. Although there were notably positive trials related to improving clinical outcomes (DeWalt et al., 2004; Weiss, Francis, Senf, & Heist, 2006), the primary outcomes that demonstrated improvement included health knowledge, some health behavior change, and social cognitions (Clement, Ibrahim, Crichton, Wolf, & Rowlands, 2009). Recommendations from these reviews include examining potential best practices, improving the internal validity of trials, using theoretical approaches for intervention development, and comparing simple versus more complex interventions (Berkman et al., 2011; Santo et al., 2005; Schaefer, 2008). Each of these previous reviews has primarily focused on the internal validity of these studies. Specifically, they have attempted to provide sufficient evidence of cause and effect between the independent and dependent variable while ruling out plausible alternative explanations (Shadish, Cook, & Campbell, 2002). However, these review articles have also commented on the potential lack of generalizability of health literacy interventions when considering different populations, settings, or even nations. However, to date no review of health literacy literature has systematically addressed nor reported on issues related to external validity. External validity is defined as the generalizability of results to or across target populations, settings, times, and so forth (Shadish et al., 2002). In other behavioral intervention domains (e.g., physical activity, nutrition, tobacco use, weight management), there have been a number of reviews of literature using the RE-AIM framework (Glasgow, Vogt, & Boles, 1999)—a set of criteria to assess the reporting on outcomes related to both internal and external validity—to demonstrate the degree to which researchers were reporting on external validity issues. (Akers, Estabrooks, & Davy, 2010; Bull, Gillette, Glasgow, & Estabrooks, 2003; Dzewaltowski, Estabrooks, Klesges, Bull, & Glasgow, 2004; Estabrooks, Dzewaltowski, Glasgow, & Klesges, 2002; Glasgow, 2007; Glasgow, Klesges, Dzewaltowski, Bull, & Estabrooks, 2004; Glasgow, Nelson, Strycker, & King, 2006; Klesges, Dzewaltowski, & Glasgow, 2008; White, McAuley, Estabrooks, & Courneya, 2009). The RE-AIM framework includes the assessment of:

Understanding Internal and External Validity

57

Downloaded by [173.190.228.162] at 22:50 29 January 2016

Reach (i.e., the number, proportion, and representativeness of study participants), Effectiveness (i.e., changes in primary study outcomes, quality of life, and potential negative effects), Adoption (i.e., the number, proportion, and representativeness of settings and staff who agree to deliver an intervention), Implementation (i.e., the degree to which an intervention was delivered as intended and the cost of implementation), and Maintenance (i.e., extent to which an intervention and its effects are sustained over time). The purpose of this paper is to present the findings of a RE-AIM review in order to determine the degree to which health literacy interventions targeting disease self-management and health promotion reported on internal and external validity indicators that could inform research to practice translation and to identify methodological gaps related to research design, evaluation, and reporting.

Methods Selection of Studies for Review Articles that were published between 2000 and February 2010 and that met the inclusion criteria illustrated in Table 1 were identified. This review focused on health Table 1. Inclusion criteria for articles Data type Participants Study design Control condition Intervention

Measurement

Setting Primary outcome(s) (at least one of these outcomes)

Inclusion criteria >17 years of age Experimental and quasi-experimental . Any comparator including active control, inactive control, or pre- and postmeasure . Mention of literacy, health literacy, or numeracy in description of the intervention . Targets disease self-management and=or health promotion . Assesses literacy, health literacy, or numeracy directly among participants . Follow-up period one week or greater after the intervention was delivered . Any . Clinical outcomes . Health literacy or numeracy . Health behavior . Proposed theoretical mediators . Self-reported health status or quality of life . Utilization of health care . Satisfaction with interventions . Health professional behavior or communication skills . .

Downloaded by [173.190.228.162] at 22:50 29 January 2016

58

K. Allen et al.

literacy and behavioral interventions targeting disease self-management and health promotion. Therefore, as illustrated in Table 1, we focused on experimental and quasi-experimental designs with the key criteria being some comparator, whether pre-post or a comparison group. Cross-sectional, descriptive, and purely methodological articles were excluded, as well as single-session studies that focused on comprehension and knowledge gains assessed in a same day follow-up. Finally, because the potential for health interventions to initiate behavior change is a key focus of this review, articles that reported on brief interventions with a follow-up one week or sooner after the intervention were excluded. The literature search was conducted from September 2009–February 2010. Three primary search strategies were used to identify potentially relevant articles including: (1) queries with search engines, (2) reverse citation of Rapid Estimate Assessment of Literacy in Medicine (REALM) and Test of Functional Health Literacy Assessment (TOFHLA) validation articles (Davis et al., 1991; Parker, Baker, Williams, & Nurss, 1995) and (3) hand searches from the reference lists of five health literacy reviews (Berkman et al., 2004; Berkman et al., 2011; Clement et al., 2009; Santo et al., 2005; Schafer, 2008). The PubMed and Web of Science search engines that were used included the databases Science Citation Index Expanded, Social Sciences Citation Index, Arts & Humanities Citation Index, and MEDLINE. The search terms used were health literacy, numeracy, and intervention(s). Since REALM and TOFHLA are two of the earliest developed and most widely used metrics to assess health literacy, reverse citation of these validation articles was an important element of the search strategies (Davis et al., 1991; Parker et al., 1995). Finally, to thoroughly evaluate the degree of RE-AIM reporting across a series of related papers reporting on a single trial, for any article meeting inclusion criteria we also extracted data from companion papers describing different aspects of the same study (e.g., methods paper, cost-effectiveness paper). The initial search yielded 587 articles that potentially met inclusion criteria (see Figure 1). The titles and abstracts of the articles were screened by two researchers (JZ and KA) and 519 were excluded for not meeting inclusion criteria (e.g., cross-sectional design; no measure of health literacy among participants). The full-texts of 72 papers were independently assessed against the inclusion criteria and an additional 41 papers were excluded for reasons detailed in Figure 1. Of note, we considered all validated and direct measures of health literacy and found that studies included in our review used REALM (Bosworth et al., 2005, 2008; Bosworth, Olsen, Dudley, et al., 2009; Bosworth, Olsen, Grubber, et al., 2009; Brock & Smith, 2007; Cavanaugh et al., 2009; Coleman et al., 2003; Davis et al., 2008; DeWalt et al., 2004; Ferreira et al., 2005; Gazmararian, Jacobson, Pan, Schmotzer, & Kripalani, 2010; Rothman, DeWalt, et al., 2004; Rothman, Malone, et al., 2004; Rothman et al., 2005), modified REALM (van Servellen et al., 2003; van Servellen et al., 2005), TOFHLA (Cordasco et al., 2009; Yin et al., 2008), S-TOFHLA (DeWalt et al., 2004; 2006; 2009; Gerber et al., 2005; Handley, Shumway, & Schillinger, 2008; Kim, Love, Quistberg, & Shea, 2004; Murray et al., 2007; Ntiri & Stewart, 2009; Paasche-Orlow et al., 2005; Schillinger et al., 2008; Schillinger, Handley, Wang, & Hammer, 2009; Seligman et al., 2005; Wallace et al., 2009), and A-TOFHLA (Rudd et al., 2009). In total, 31 articles representing 25 trials were included in this systematic review.

Downloaded by [173.190.228.162] at 22:50 29 January 2016

Understanding Internal and External Validity

59

Figure 1. Selection of health literacy intervention studies for systematic review. (1Contact lead author for complete citations of 41 reviewed, but excluded articles.)

RE-AIM Criteria A 21-item validated data extraction tool based on the RE-AIM Framework was used to code eligible articles on the degree to which indicators of internal and external validity were reported (Akers et al., 2010; Dzewaltowski et al., 2004; Estabrooks et al., 2002; Glasgow et al., 2004). In addition, as described below, we added 10 exploratory indicators across the reach, effectiveness, adoption, and implementation dimensions. Reach Articles were coded for information on the method used to identify the target population, inclusion criteria, exclusion criteria, participation rate (i.e., sample size=the

60

K. Allen et al.

Downloaded by [173.190.228.162] at 22:50 29 January 2016

number of eligible participants exposed to recruitment activities), and representativeness (i.e., comparison of participants and eligible nonparticipants on demographics and key outcomes). In addition to the items included in the original validation tool, we also coded articles for including information on presence or absence of a health literacy-based inclusion criteria and the proportion of participants with low health literacy. Efficacy=Effectiveness Effectiveness is defined as a test of whether a ‘‘program does more good than harm delivered under real-world conditions’’ while efficacy is defined as a test of whether a ‘‘program does more good than harm delivered under optimum conditions’’ in a controlled setting (Flay, 1986). Indicators for this dimension included the assessment of changes in the primary outcome, the completion of analyses on those present at follow-up versus an intention-to-treat approach, the presence of a quality-of-life measure, and the reporting of percentage attrition. Including quality-of-life measures can provide a critical participant-centered check on the impact of delivery practices and an opportunity to better understand standardized effects across a diverse range of studies. Again, we included additional criteria related to the purpose of this review, such as whether the researchers reported trial type (i.e., efficacy versus effectiveness; single or multileveled). Finally, we also coded whether an article assessed changes in health literacy scores or completed any mediation or moderation analyses that would help identify the robustness and mechanisms of intervention effects. As defined by Baron and Kenny (1986) a moderator variable is one that alters the strength of the casual relationship, whereas a mediator variable is on the causal pathway and alters the effect of the independent variable on the dependent variable. Using health literacy as an example, moderation examines the robustness of effects across different levels of health literacy (does the intervention work in the same way for high and low health literate clients). Mediation implies a causal sequence where an intervention may change health literacy, and this change in health literacy results in improved health outcomes. Adoption Indicators for the reporting of adoption were assessed at both the setting and staff levels. Specifically, articles were reviewed to identify the extent to which they provided a description of intervention location, the staff who delivered intervention, the methods used to identify and target the staff who delivered the intervention (delivery agent), and the level of expertise of those who delivered the intervention. Further, the degree to which an article provided information on setting or staff inclusion and exclusion criteria as well as rate of delivery agent or setting adoption was coded. We also added an indicator to the coding tool that reflected whether or not the authors reported on the cost of adoption (i.e., start-up costs) of the intervention. Implementation Articles were assessed to determine if they reported on the duration and frequency of the intervention, the extent the protocol was delivered as intended, and the cost of delivery. To determine the degree to which other important implementation factors were reported across articles, we added indicators for reporting the theoretical basis of the intervention and the methods used to address low literacy. Further, we coded

Understanding Internal and External Validity

61

whether or not authors reported on the degree to which the participants received intervention components.

Downloaded by [173.190.228.162] at 22:50 29 January 2016

Maintenance Reporting of individual-level maintenance was coded as present if the study included assessments of intervention outcomes at six or more months after the completion of individual participation in the interventions. Six months is widely accepted as the demonstrated sustainability time frame in the health behavior change field (Glasgow et al., 1999; Prochaska & DiClemente, 1983; Prochaska & Velicer, 1997). At the organizational level, studies were coded on whether or not they reported that the intervention was sustained after research activities were completed and on any costs associated with maintenance. Coding Protocol and Scoring All studies that met inclusion criteria were abstracted and coded independently by two members of the research team. Specifically, each reviewer coded a ‘‘yes’’ or ‘‘no,’’ reflecting the presence or absence of the RE-AIM indicators outlined above. In addition, appropriate data was extracted from the articles when a given indicator was reported. Following the individual coding of each article, the research team met to discuss articles, resolve uncertainty, and gain consensus in coding. Analyses included providing count and percentage data across RE-AIM indicators. To determine the overall quality of RE-AIM reporting, we also examined the degree to which articles reported across the 21 items from the validated data extraction tool (see Table 2). Specifically, quality was assessed based on tertile cut-points of reporting 15–21, 8–14, or 0–7 dimensions reflecting high, moderate, and low quality, respectively.

Results Our search yielded 31 articles, from which merging papers yielded 25 unique studies. Of those, 14 were randomized controlled trials (Bosworth et al., 2005, 2008; Bosworth, Olsen, Dudley, et al., 2009; Bosworth, Olsen, Grubber, et al., 2009; Cavanaugh et al., 2009; Cordasco et al., 2009; DeWalt et al., 2006; Ferreira et al., 2005; Gerber et al., 2005; Handley, Shumway, & Schillinger, 2008; Murray et al., 2007; Rothman, DeWalt, et al., 2004; Rothman et al., 2005; Rudd et al., 2009; Schillinger et al., 2008; Schillinger et al., 2009; Seligman et al., 2005; van Servellen et al., 2003; van Servellen et al., 2005; Yin et al., 2008) and 11 were quasi-experimental studies (Brock & Smith, 2007; Coleman et al., 2003; Davis et al., 2008; DeWalt et al., 2004; 2009; Gazmararian, Jacobson, Pan, Schmotzer, & Kripalani, 2010; Kim et al., 2004; Ntiri & Stewart, 2009; Paasche-Orlow et al., 2005; Rothman, Malone, et al., 2004; Wallace et al., 2009). The average number of RE-AIM indicators reported was 9.0 (
1.8) out of 21, with a range of 4 to 17 indicators reported. One trial was rated as high quality (Handley et al., 2008; Schillinger et al., 2008; Schillinger et al., 2009) while 18 and six, respectively, were rated as moderate and low quality. The majority of the studies were conducted in a single site, and most interventions were delivered in clinical settings, with the exception of one study delivered in a community setting (Ntiri & Stewart, 2009).

62

K. Allen et al.

Table 2. Proportion of health literacy interventions reporting RE-AIM dimensions and components (n ¼ 25 trials)

Downloaded by [173.190.228.162] at 22:50 29 January 2016

RE-AIM dimensions and components

Proportion reporting

Reach Method to identify target population Inclusion criteria Exclusion criteria Participation rate Representativeness Average across Reach Components Efficacy=effectiveness Measures=results for at least one follow-up Intent-to-treat analysis utilized Quality-of-life or potential negative outcomes Percent attrition Average across Efficacy=Effectiveness Components Adoption Description of intervention location Description of staff who delivered intervention Method to identify staff who delivered intervention (target delivery agent) Level of expertise of delivery agent Inclusion=exclusion criteria of delivery agent or setting Adoption rate of delivery agent or Setting Average across Adoption Components Implementation Intervention duration and frequency Extent protocol delivered as intended (%) Measures of cost of implementation Average across Implementation Components Maintenance Assessed outcomes 6 months post intervention Indicators of program-level maintenance Measures of cost of maintenance Average across Maintenance Components 

100.0 96.0 76.0 44.0 28.0 68.8 100.0 32.0 22.0 76.0 57.5 100.0 12.0 4.0 72.0 18.0 8.0 36.0 72.0 20.0 12.0 35.0 20.0 8.0 4.0 11.0

Based on denominator of 25 intervention trials, reported across 31 articles.

Reach The overall proportion of reported indicators of RE-AIM dimensions was highest for reach at 68.8% (Table 2). Typical information relative to the internal validity of the data, such as reporting of specific inclusion and exclusion criteria, was quite high, as was the proportion that reported on the methods used to identify the target population. However, the description of the methods varied greatly across studies from single-sentence descriptors to thorough reporting of the protocol used. All studies provided a sample size and these ranged from 20 to over 600 participants with a

Understanding Internal and External Validity

63

Downloaded by [173.190.228.162] at 22:50 29 January 2016

median of 217. When considering components that align with external validity and impact generalizabilty, there was a substantial drop in the proportion of articles reporting participation rate and representativeness. Providing a range and median for participation rate is problematic as the operational definitions used to determine a denominator varied widely (e.g., those who responded to recruitment activities versus those who were exposed to recruitment). The indicators added specifically for this review indicated that only two of the 25 studies applied an inclusion criterion to ensure that participants had a low health literacy status (DeWalt et al., 2004; Seligman et al., 2005). Furthermore, estimates revealed that only about 38% of participants across these 25 interventions had low health literacy. Efficacy=Effectiveness The overall proportion reporting for efficacy=effectiveness was 57.5% (Table 2). Similar to the reach dimension, articles included strong reporting of factors associated with internal validity, such as changes in the primary outcome and percent attrition. The average attrition rate was around 14% and ranged from 2% to 39%. Of the studies that reported participants with high attrition (25% or higher), only one out of five reported using an intent-to-treat analysis. Seventeen (68%) trials reported on present-at-follow-up data, three reported on intent-to-treat analysis, and five studies reported on intent-to-treat and present-at-follow-up data. Of the studies that reported attrition rates, eight compared characteristics of dropouts with those who remained in the study. From those eight studies, five studies reported significantly different characteristics. Coding showed that only about one in five studies reported whether the focus was on efficacy or effectiveness. Of trials that reported, two indicated that they were efficacy trials (Davis et al., 2008; R. L. Rothman et al., 2004; R. L. Rothman et al., 2005) and four indicated that they were effectiveness trials (DeWalt et al., 2004; Handley et al., 2008; Kim et al., 2004; Schillinger et al., 2008; Schillinger et al., 2009; van Servellen et al., 2003; van Servellen et al., 2005). Only two studies reported on multileveled interventions that included strategies for both the health professional and patient (Davis et al., 2008; Gazmararian et al., 2010), one focused solely on the health professional (Seligman et al., 2005), and the remainder were individual patient-level interventions. The types of control conditions included 16 trials that reported using usual care (Bosworth et al., 2005, 2008; Bosworth, Olsen, Dudley, et al., 2009; Bosworth, Olsen, Grubber, et al., 2009; Cavanaugh et al., 2009; Cordasco et al., 2009; DeWalt et al., 2006; Ferreira et al., 2005; Gazmararian et al., 2010; Handley, et al., 2008; Murray et al., 2007; Rothman, DeWalt, et al., 2004, Rothman et al., 2005; Rudd et al., 2009; Schillinger et al., 2008; Schillinger et al., 2009; Seligman et al., 2005; van Servellen et al., 2003; van Servellen et al., 2005; Yin et al., 2008) and three used a minimal intervention control (Bosworth, Olsen, Grubber et al., 2009; Gerber et al., 2005; Rudd et al., 2009). Only two trials examined changes in health literacy as a result of their interventions, and both were successful in improving health literacy (Nitiri & Stewart, 2009; van Servellen et al., 2003; van Servellen et al., 2005). Only one study reported analyses to determine mediators of intervention effects (Paasche-Orlow et al., 2005), while eight studies reported the influence of health literacy as moderator on outcomes (DeWalt et al., 2006; Ferreira et al., 2005, Gerber, 2005; Kim et al., 2004; Paasche-Orlow et al., 2005; Rothman, DeWalt, et al., 2004; Rothman, Malone, et al., 2004; Rothman et al., 2005; Wallace et al., 2009). Results were mixed

64

K. Allen et al.

with regard to the degree to which health literacy status influenced intervention effectiveness.

Downloaded by [173.190.228.162] at 22:50 29 January 2016

Adoption The average percentage of reporting for the adoption elements across all studies was 36%. All studies provided a description of the intervention location. The next most commonly reported adoption element was the level of expertise of the delivery agent (72%). In contrast, few studies reported on the description of intervention staff (12%), method to identify target delivery agent (4%), inclusion=exclusion criteria of delivery agent (18%), adoption rate of delivery agent or setting (8%), and information on intervention start-up costs (4%). Implementation The mean level of reporting across implementation components was 35%. The most commonly reported implementation element was intervention duration and frequency (72%). Only five of 27 studies reported on the extent protocol delivered as intended, and even fewer studies (n ¼ 3) reported on measures of implementation costs. Approximately a third of the studies reported the theoretical basis of the intervention and the majority of these used social cognitive theory. All of the studies reported on the methods used to address health literacy in their intervention. The most frequent methods cited were an emphasis on verbal communication, the use of concrete, simplified explanations, pictorial depictions, and the teach-back method. Finally, more than half of the studies included a description of the degree to which participants received the intervention. Methods to assess participant receipt typically included monitoring class attendance or the completion of intervention telephone calls. Maintenance Among the RE-AIM dimensions, maintenance indicators were reported the least (11%). Only one study reported on measures related to cost of maintenance, and rarely did studies report on outcomes assessed at longer than or equal to 6 months following the intervention (n ¼ 5) or indicators of program-level maintenance (n ¼ 2).

Discussion The overarching goal of this paper was to provide an initial understanding of the state of research on health literacy interventions targeting disease self-management and health promotion from both an internal and external validity perspective. Within the context of the RE-AIM framework, the health literacy intervention studies included in this review share a number of similarities with, and a few distinctions from, the intervention literature that targets smoking, physical activity, nutrition, and weight-loss maintenance (Akers et al., 2010; Glasgow et al., 2004; White et al., 2009). When considered within the context of health literacy-related intervention research, our review also filled a number of gaps identified in previous systematic reviews by identifying the degree to which information on patient, provider, and setting characteristics were generalizable to different populations and settings (Santo et al., 2005; Schaefer, 2008). Our systematic review supports the consistent finding that the current body of literature on health promotion interventions regularly reports on internal validity

Downloaded by [173.190.228.162] at 22:50 29 January 2016

Understanding Internal and External Validity

65

indicators of reach, such as the method to identify the target population and inclusion=exclusion criteria (Akers et al., 2010; White et al., 2009). However, reporting of participation rate (44%) was considerably lower in health literacy intervention studies when compared with the studies targeting changes in physical activity, nutrition, and smoking behaviors (e.g., 76% reporting; Glasgow et al., 2004). Still, our systematic review suggests that health literacy intervention researchers report representativeness (30%) to a greater degree than in other behavioral intervention domains (range 0–14%; Akers et al., 2010; Glasgow et al., 2002; White et al., 2009). Although this finding might be promising in light of other reviews of literature, we identified that few studies specifically targeted participants who could benefit most—those with low health literacy. Further, as the proportion of participants with low health literacy was modest across studies (i.e., 38%), and few studies examined health literacy as a moderator, it is unclear what conclusions can truly be made about the applicability of the body of literature in typical community or clinical settings. Of the eight studies included in our review that performed a moderation analysis by health literacy category, four studies found no moderating effects across all outcomes measured (DeWalt et al., 2006; Paasche-Orlow et al., 2005; Rothman, Malone, et al., 2004; Wallace et al., 2009), three found mixed effects across outcomes (Gerber et al., 2005; Kim et al., 2004; Rothman, DeWalt, et al., 2004; Rothman et al., 2005) (RCT, N ¼ 217), and only one found that the intervention was more effective for low health literate participants compared with their higher health literate counterparts (Ferreira et al., 2005). Assessing changes in health literacy status might be viewed as controversial, because many scholars view health literacy and the associated metrics as static. For this reason, very few studies assess longitudinal changes in health literacy status. This divisive assumption creates a dilemma because there is little available evidence to suggest that health literacy is indeed static. Advancing research related to the sensitivity and specificity of health literacy metrics, as well as understanding changes in health literacy, could help advance the health literacy field. Furthermore, those studies that did examine the potential of interventions to change health literacy demonstrated that improvements in health literacy were achievable (Nitiri & Stewart, 2009; van Servellen et al., 2003; van Servellen et al., 2005). The reporting of other factors that could facilitate, or impede, the translation of health literacy interventions into practice varied in a similar way to reach. For example, our review indicated that studies on health literacy were more likely to provide a description of the location where the intervention was delivered (100%) when compared with previous reviews (range 48–60%; Akers et al., 2010; White et al., 2009). Unfortunately, it appears that information on the methods used to engage those who would deliver the intervention (or a description of their characteristics), intervention costs, and individual and program-level maintenance were all rarely reported (