efficacy. A systematic review of randomized trials of computer-based patient education found that interactive computer interventions were effective in increasing ...
Usability Testing And Outcomes Of An Interactive Computer Program To Promote Smoking Cessation In Low Income Women Anna M. McDaniel, DNS RN"2; Sondra Hutchison, MSN'; Gail R. Casper, RN PhD1; Raymond T. Ford2; Renee Stratton2, and Mary Rembusch3 'School of Nursing, Indiana University, Indianapolis, IN 2School of Informatics, Indiana University, Indianapolis, IN 3Indiana University Medical Group Research Network, Indianapolis, IN method to deliver smoking cessation information and may increase motivation to quit in the primary care setting.
ABSTRACT
The purpose of this study was to develop and test an interactive computer-mediated smoking cessation program for inner-city women. A non-probability sample of 100 women who receive care at an innercity community health center in Indianapolis participated in the usability study. Women completed the computer program in the clinicfollowing baseline data collection. Next, participants completed a brief satisfaction instrument. Data on cognitive and behavioral outcomes of the program were obtained by telephone interview one week later. Satisfaction with the program was high (mean satisfaction score was 60.2 with 70 indicating highest possible satisfaction). Average time for completing the computer program was 13.6 minutes. Overall, 79% of the participants reported at least one behavioral change related to smoking. The results indicate that interactive computer technology may be useful for promoting smoking cessation in low-income women.
Computer technology has been widely used in education for health care professionals7 but computerized patient education is less common due, in part, to concerns about patient acceptance and efficacy. A systematic review of randomized trials of computer-based patient education found that interactive computer interventions were effective in increasing knowledge, especially for patients with chronic disease8'9. Patient receptivity to computerized education is reported to be high across diverse medical conditions and age groups 0-12.
Although computerized patient education has been shown to increase patient knowledge level, few studies have addressed the use of interactive computer technology for influencing behavior change. Tate and colleagues'3 used Internet technology to deliver structured behavioral weight loss therapy. However, the participants in that study were recruited from employees of a large hospital network who regularly used computer technology in the work setting. Little is known about the acceptability and efficacy of interactive computer technology for influencing behavior change in lowincome populations who have less experience with and access to interactive technology.
INTRODUCTION
The health effects of smoking among women in the United States are devastating. More than 165,000 women die each year due to smoking'. Lung cancer is currently the most common cause of cancer death in women in the United States2. Nearly 20 million women have quit smoking, although the "quit ratio" (percentage of persons who have ever smoked who have quit smoking) remains lower for women (46.2%) than for men (50.1%)'.
The purpose of this study was to assess the usability and impact of an interactive computer-mediated smoking cessation program for inner-city women. Usability testing is a widely used technique to assess the utility of a system or product to accomplish a goal from the perspective of the end-user'4 5. Usability testing, an important step in usability engineering, is a standard process in the information technology industry, but may be overlooked in non-commercial software development for consumer health informatics.
Routine smoking cessation intervention by health care providers as brief as three minutes can significantly increase quit rates3. However, studies have shown that as few as one in four smokers is advised to quit smoking by their health care providers4'5. The most common reason for omission of smoking cessation intervention in primary care is the restricted time available for preventive health teaching during patient visits6. Multimedia instructional technology is an innovative and efficient
AMIA 2002 Annual Symposium Proceedings
509
PROGRAM DEVELOPMENT
The women who participated in the focus groups expressed a preference for learning about diagnosis and treatment options from a health care professional, but wanted information about instrumental behavior change (e.g., going on a diet or starting an exercise program) from "real people". Salient beliefs about smoking included skepticism of messages perceived as "scare tactics". Personal knowledge of smokingattributable disease, such as a relative with cancer, was more believable and trustworthy and increased the feeling of personal vulnerability to the deleterious health effects of smoking. This information was used in the development of the smoking cessation program. For example, content that was focused on knowledge of the health risks of smoking were presented in a video clip of a health care professional or a simple graphic. Message content that was intended to influence the user's affective domain (e.g., attitudes toward smoking or motivation to quit) were presented in vignettes of former smokers who were similar in age and ethnicity to the intended audience.
Framework The program interface and content were developed using principles of user-centered design and the Persuasive Health Message (PHM) framework'6. According to this framework, messages must address two factors to effectively motivate health behavior change. Constant factors should be present in any health message and consist of four components: (1) threat; (2) efficacy; (3) cues to action; and (4) audience profile. An implied or expressed threat will motivate behavior change only if it is balanced with an efficacy component' . The efficacy component of a health message should include information about both response efficacy (i.e., there are specific actions that will decrease the threat) and self-efficacy (i.e., the recipient of the message has the ability to carry out the recommended action). Cues to action include the structure and organization of the message itself as well as the appeal of the message source. The final constant component of health messages consists of the demographic and cultural profile of the targeted audience.
Program Navigation and Interface The program was created using Macromedia Director 7.0 () software. This program, called A New Beginning, was designed to deliver tailored smoking cessation messages in a format that would be relevant and acceptable to the users. All textual information that appeared on screen also was simultaneously presented in audio to decrease difficulty for low literacy participants. The program was navigated by using a touch-screen monitor to input data. Based on data the user entered, algorithms derived from the scientific literature on smoking cessation and behavior change, determined program content and progression through a series of customized output screens.
In addition to these constant components, transient factors are essential to designing health messages that are acceptable to the consumer. Transient factors must be identified so that message content can be modified to appeal to the specific target population. Salient beliefs about the perceived threat and efficacy of the response should be incorporated into the message. For example, if smokers are unaware of the availability of effective treatment for nicotine dependence, a persuasive message should incorporate information about efficacy of nicotine replacement therapy. Knowledge of salient beliefs is important for building effective yet acceptable counter-arguments. In addition, beliefs of salient referents in the target audience's environment should be included in the threat and efficacy components of the message.
The initial screen included a video segment that welcomed the user to the program. This screen was followed by several dynamic tutorial views that instructed the woman on how to enter information using the touch screen monitor in response to computer-generated questions and prompts. Next, screens that solicited specific demographic and smoking information from the woman were presented. The remaining content was tailored for each individual, based on information she provided about her smoking patterns and readiness to quit smoking. As the woman navigated through the program, she reached a final screen that included an encouraging message about her stage of readiness and offered options for further actions, such as talking to
Message Content To design the content of the program messages, we conducted a series of focus groups with women who were similar to the target audience. Fifteen women who had attended a smoking cessation program at a public-supported, inner-city health care system participated in three focus group interviews. All participants had made at least one quit attempt in the past, but only five were abstinent at the time of the interview. The semi-structured interview schedule was designed to elicit information about motivations to quit smoking, perceived barriers to smoking cessation, and preferences for source and content of information about quitting.
510
her health care provider about quitting, or enrolling in a smoking cessation program.
Satisfaction with the interactive program was assessed using an investigator-developed tool to measure user satisfaction with salient aspects of the program (e.g., ease of use, acceptability of interactive computer format). The instrument contained 14 items In addition, two open-ended questions were included to obtain individual opinions on user preferences. The Cronbach alpha reliability coefficient of the satisfaction instrument was 0.76. An important component of usability testing is assessing the impact of the application. In this case, we examined effects on cognitive and behavioral outcomes in women who used in the program. Cognitive outcomes were measured using two well established instruments based on the precepts of the Transtheoretical Model of Behavior Change °.
Figure 1. Touch screen user interface METHODS
RESULTS
Both formative and sunmmative evaluation strategies were used in the usability test design. Formative evaluation of program usability was conducted throughout the development process. Experts in smoking cessation and women's health reviewed program content. The design team created a prototype design and layout of the program for analysis by health care professionals familiar with intended users. Feedback from the formative evaluation was incorporated into the design of the final program.
Sample A non-probability sample of women smokers who were patients at an inner-city community health center affiliated with a public-supported hospital at a large academic medical center were recruited from the waiting area while awaiting their clinic appointment. Of 228 eligible women smokers approached in the clinic, a total of 110 agreed to participate (52% refusal rate), but 10 subjects either left the clinic before completing the computer program (n=2) or were lost to follow-up (n=8). The final sample consisted of 100 (91% of those enrolled) women. The women completed the computer program after their visit to avoid disruption of clinic workflow. Time constraints were the most frequent cited reason for refusal to participate.
The summative evaluation was designed to test usability in a naturalistic environment. Users were recruited from women at a neighborhood community health center affiliated with an inner-city health care network while awaiting their clinic appointment. Inclusion criteria for the study included: currently smoke cigarettes, over age 18, physically and mentally capable of participation, able to read and speak English, and have a telephone in the home. Women who agreed to participate completed selfreport instruments used to measure cognitive factors related to cuitting smoking'7"8, level of nicotine dependence' and smoking history, as well as relevant demographic characteristics. After obtaining these baseline measures, participants were instructed on how to launch the interactive computer program. At the conclusion of the program, participants completed a brief satisfaction survey and follow-up interview to elicit feedback about the program. Participants were contacted by telephone after one week to assess changes in cognitive measures and short-term behavioral outcomes (e.g., quit attempts) in women who had used the program.
The sample included women between the ages of 18 and 71, with a mean of 41.5 years (s.d. = 12.4). The majority of the women participating were Caucasian (68%) and 23% of the sample were AfricanAmerican, which is similar to the racial distribution of the clinic population. A majority (57%) of participants rated their current health status as fair or poor. Educational level of the participants was typical of an inner-city population with 35 subjects reporting no high school diploma. On average, participants reported a long history of smoking (M=22.8 years, s.d.=12.5) with an average age of onset of regular smoking of 16.7 years of age (range from 9 to 36 years). Most of the participants (86%) had made at least one quit attempt (>24 hours without smoking), with 54 participants reporting multiple quit attempts in the past. Forty percent reported smoking more than a pack of cigarettes per
511
positive association between satisfaction with the program and the number of quitting-related behaviors reported. Table 1. Summary of self-reported behavioral outcomes
day. Nicotine dependence level was relatively high in the group, with 48% identified as highly dependent on the Fagerstron Test of Nicotine Dependence'9. Although 60% of participants reported that they wanted to quit smoking "a lot" or "very much", only 23% were "seriously thinking about quitting smoking in the next 30 days".
Behavior Cut down on number of cigarettes Discussed quitting with friends/family Read information about quitting Talked to doctor about quitting Quit smoking for 24 hours Called a smoking cessation program
Usability Initially, we encountered several problems with the interactive prograrm We discovered that occasionally some information inappropriate to the user was being delivered onscreen. Coding errors in "button" assignment were detected and this problem was corrected. Rarely, a user received no tailored smoking cessation information. Content was determined by algorithms based on scoring of user input. If user response to all questions was below the "threshold" determined by the algorithm, no information was displayed. Re-specification of the decision rules eliminated this problenm
% 52 40 24 15 15 6
Cognitive Outcomes Repeated measures analysis of variance was used to detect changes in cognitive factors associated with smoking from baseline to follow-up. Overall, participants reported a significant decrease in favorable attitudes toward smoking regardless of readiness to quit (p=.01). A complete description of these findings are reported elsewhere21.
Average time to complete the program was 13.9 minutes (range of 7.2 to 88.8 minutes). No one failed to complete the program.
DISCUSSION The findings of this study were useful for evaluating the design and usability of an interactive smoking cessation program. The use of focus groups as a usercentered design strategy resulted in important information for targeting content and delivery (i.e., information source) of the smoking cessation message. Incorporating this information is essential for user acceptability and to maximize impact.
Satisfaction Participant satisfaction with the program was high. The mean score on the satisfaction measure was 60.2 (s.d. = 6.3) with a possible range of 14-70. There were no relationships between satisfaction and age, health status, or past computer experience. Satisfaction was related to ethnicity (p=.03), education level (p=.02), and readiness to quit smoking (p
