Examining Ways To Assess Consumer Satisfaction

Examining Ways To Assess Consumer Satisfaction In Ohio’s PASSPORT Program

Suzanne R. Kunkel Latona D. Murdoch Jane K. Straker Robert A. Applebaum

Scripps Gerontology Center

September, 2001

SGC0067

Acknowledgments We could not have completed this study without the assistance of a number of individuals. Our partners at the PASSPORT Administrative agencies in Cincinnati and Rio Grande including Ken Wilson, Erin Ghartley, Kim Clark, Bonnie Dingess, and Jeff Hunter, were invaluable. PASSPORT administrators from around the State also contributed to the data collection component of the study. At the Ohio Department of Aging, Judy Patterson, Roland Hornbostel, Craig Martin, and Mark Molea contributed to the success of the study. At Scripps we thank Betty Williamson for ably producing the report. Most importantly we would like to thank the case managers who participated and the older consumers who let us into their homes to complete the project. We hope that this study can contribute to current efforts designed to make PASSPORT the best program of its kind.

Background Over the past decade there has been a growing interest in consumer satisfaction. From health care to fast food to manufactured goods, we want to know what consumers think about the services or products provided. Home care, a critical element of the long-term care delivery system, is another area in which consumer satisfaction is of increasing interest. Medicaid spends more than $17 billion a year on home care, and the private sector spends twice that amount (Burwell, 2001). Making sure that consumers are satisfied with these services is critical in efforts to ensure the quality of long-term care services. Ohio’s PASSPORT Medicaid home and community-based service program, administered by the Department of Aging, serves over 24,000 chronically disabled older people each year. Older Ohioans are served through a regional network of 13 PASSPORT administrative agencies, termed PAA’s, that provide a case management service directly for PASSPORT clients; the PAA’s also contract with a range of in-home providers to deliver services such as personal care, home delivered meals, homemaking, and adult day care. The job of the case management agencies is to assess consumer needs and, in conjunction with the client and his/her family, develop a service plan that meets the consumer’s needs and preferences. Case managers arrange and monitor the quality of these services. For the case management agencies and the Ohio Department of Aging to evaluate the quality of the PASSPORT program, assessing the satisfaction of consumers with their services is an essential piece of information. Recognizing the importance of consumer input, this study is designed to address two research questions: (1) Can we develop a reliable and cost effective approach to collecting

1

satisfaction data from consumers currently enrolled in the PASSPORT program? (2) What can we learn about the quality of the program from individuals who terminate from PASSPORT? To test the feasibility of a cost-effective consumer satisfaction approach, we used the Home Care Satisfaction Measure (HCSM), a consumer satisfaction measure that has been developed by researchers at Boston University in conjunction with consumers (Geron, 2000). The measure covers home health aide, homemaker, home delivered meals, and case management services. The instrument has been tested across the U.S. with over 6,000 older consumers of inhome care. The HCSM has consistently relied on research interviewers, rather than agency staff, to collect data from consumers. Because agencies need to hear from a large sample (and possibly all) of their consumers, paying researchers to conduct these interviews is costly. An alternative approach would use case managers to collect satisfaction data during the in-person client visit completed on a routine basis. The possibility that case managers could collect consumer satisfaction data was supported by findings from a pilot project conducted by the Council on Aging of Southwest Ohio. Their study found that case managers were able to administer the survey as part of their ongoing work with consumers. However, this pilot project did not have the resources to test the reliability of the instrument and did not study the differences between case manager and research interviewer data collection results. For this component of the research, we started with an instrument that had been widely used by researchers in home care. We were interested in whether case managers, if well trained, would collect data comparable to research interviewers. The fact that case managers arrange and

2

monitor services might affect what they can learn from consumers about their satisfaction with these services. The first phase of this research was designed to look at these issues. Our second question focused on consumers who had left the PASSPORT program. PASSPORT agencies report about a 3% turnover rate each month, indicating that about one-third of clients enrolled in the program leave over the course of one year. Understanding more about the circumstances and satisfaction of these consumers has important implications for quality strategies. There are numerous reasons that individuals leave PASSPORT. The major reasons are: death (32%), entering a nursing home (47%), withdrawal (6%), or entering a hospice or requiring long-term hospitalization (7%). About 5% of terminated clients were classified in a category called “other reason for leaving.” We did not try to contact the families of deceased clients, but instead focused our attention on those consumers who left PASSPORT for nursing homes, who chose to withdraw from the program, and who left for other reasons. The methods and findings from these two phases will be organized in this paper around the two major research questions. We first present the results of the case manager collected data, followed by the findings from those individuals who left the PASSPORT program.

3

CAN CASE MANAGERS COLLECT CONSUMER SATISFACTION DATA? Methods Case managers were recruited from PAA’s in Regions 1 and 7 to administer a consumer satisfaction instrument to current PASSPORT clients. In Region 1 (Cincinnati area) both PASSPORT and ESP1 case managers were solicited; in Region 7 (Rio Grande area) only PASSPORT case managers participated. A training session was held in each area. The session emphasized the different role case managers were being asked to play in gathering consumer satisfaction information, and reviewed interviewing techniques for collecting reliable research data. Once the training was completed, data collection began as case managers conducted their six-month in-home visits. The satisfaction instrument was administered at the end of the home visit after the case manager explained the purpose of the study and obtained the client's written consent. When an interview was completed, the case manager turned in the interview and client contact information to the supervisor who then forwarded the information to Scripps. A subset of clients who participated in the case manager interviews was randomly selected for a re-interview by researchers. Research interviewers were hired by the Scripps Gerontology Center and trained using materials identical to those used to train case managers. Re-interviews were typically done within two to three weeks of the original interview.

1

ESP is the tax-levy-fund ed progra m in P SA 1 . Clients in this program receive services similar to

PASSPO RT, but do not meet income or level-of-care eligibility for that program .

4

Participants All participants were PASSPORT or ESP clients in PAA 1 and PASSPORT clients in PAA 7. Region 1 is comprised of five counties in southwest Ohio and covers the greater Cincinnati area. In contrast, Region 7 includes ten primarily rural counties across southern Ohio with a few large towns, usually county seats. The response rate on re-interviews was 67% in PAA 1, and 90% in PAA 7 for an overall rate of 77%. Measures The Home Care Satisfaction Measure (HCSM) consists of separate scales for a range of home-based services, including: homemaker, home health aide, home delivered meals, grocery service and case management. Case managers administered only those scales that matched the services each client was receiving, with the exception of the case management scale. Case managers did not administer the case management scale because of possible conflict of interest. During the re-interview, scales for the client's services and the case management scale were administered. Scales varied in length from 11 to 13 questions. The response set was the same for all scales (1 = Yes, definitely; 2 = Yes, I think so; 3 = Maybe yes, maybe no; 4 = No, I don't think so; and 5 = No, definitely not). Results We were interested in whether case managers would collect data that is as reliable as that collected by research interviewers. Beyond the stability of the scores, we were also interested in whether there were any significant differences in the responses case managers received from their clients, compared to those given to “outside” interviewers. If the responses given to case managers are stable (not likely to vary significantly each time the survey is given), but those 5

responses are different from what consumers tell the research interviewers, the advisability and feasibility of this data collection strategy are called into question. Tables 1 and 2 present the findings from this phase of the study. Table 1 shows that the internal scale reliability (alpha) scores for the case-managergathered satisfaction data were all acceptably high. While the re-interview reliability score for homemaker services was noticeably higher than the case-manager reliability, the latter was still high enough to have confidence in the stability of the scores. We concluded that case managers who have been trained in the administration of the HCSM could, indeed, gather reliable consumer satisfaction data.

Table 1 Comparing Case Manager and Re-interview Scores and Reliability on Selected HCSM Scales Service

Published Mean/sda

CM Mean/sd

Re-interview Mean/sd

CM alpha score

Re-interview alpha score

Homemaker

77.39 (17.56)

84.62 (10.78)

84.78 (12.98)

.7314

.8379

Home Health Aide

80.12 (16.63)

86.60 (11.38)

85.66 (11.95)

.7833

.7954

Home Delivered Meal

66.39 (20.49)

77.92* (14.68)

74.45* (15.67)

.7808

.7601

Case Manager

73.04 (19.05)

84.78 (12.91)

Homemaker N=47; Home Health Aide N=51; Home Delivered Meals N=44; Case Manager N=94 * p=.030 a These scale means were reported in Geron and Chassler, 1998.

6

.8236

Assuming that scores will be stable and reliable if case managers gather the data, the next concern is whether there are any differences between how consumers answer questions asked by their case manager and what they tell a research interviewer. Table 1 shows that the casemanager-collected and re-interview scores on the Homemaker and Home Health Aide HCSM scales were quite similar. For example, the homemaker mean for case manager collected data was 84.62 compared to 84.78 for the re-interview. There was a however, a statistically significant difference on the home-delivered meal scale (77.92 vs. 74.45); the data gathered by the case managers yielded a significantly higher score (respondents more satisfied) than that gathered during the re-interview. To understand the variation in scores on home-delivered meals, we looked at each item on the scale. Table 2 presents a comparison of the mean scores on each item, and gives the percentage of people who did not change significantly in their responses between the case manager interview and the researcher interview. A great majority (80% or higher) of the respondents did not change significantly on most items, and there was little difference in mean scores on six of the eleven items. Those items which showed some variability are marked with an asterisk on Table 2. These were the items on which 25 to 40% of the consumers had a considerable change in their responses. On each of these items, respondents were more negative when they talked with the “outside” interviewers. It is noteworthy that these items appear to address the quality of the food itself rather than the meal service overall or the people who deliver the meals. When we discussed these findings with the PAA Site Directors, they expressed a distinct lack of surprise that the most negative responses were about the quality of meals. 7

Table 2 Descriptive Statistics for the Home Delivered Meals Items Question

CM Mean/sd

Reinterview Mean/sd

No or Little Change

1.

Happy w/# of Meals

89.58 (19.86)

88.33 (21.05)

86.7%

2.

Meals arrive late

78.72 (25.53)

77.22 (28.11)

80.0%

3.

Like time meals arrive

86.98 (21.41)

85.56 (21.64)

88.9%

4.

Meals fail to be delivered

87.23 (21.41)

87.50 (21.90)

86.7%

5.

Meals of the kind I like

67.93 (29.18)

65.56 (28.85)

62.2%

6.

Need more meals

88.54 (17.07)

85.56 (23.53)

88.9%

7.

Meals are great

64.06 (30.90)

65.56 (30.29)

75.6%

8.

Meals arrive too early

86.46 (16.27)

84.44 (27.84)

86.7%

9.

Cooked the way I like

63.83 (31.63)

53.33 (33.54)

68.9%

10. Poor choice of meals

66.85 (31.65)

61.67 (35.19)

57.8%

11. So bad I can’t eat it

69.15 (35.82)

66.11 (35.82)

62.2%

N=45

Everyone has an “expert” opinion about the food they consume. In addition, clients may feel more comfortable complaining about food since it is not a negative comment about a person or about a service on which the individual depends. The fact that the responses about food quality were consistently more negative when the researchers were asking the questions rules out the possibility that respondents were just reacting to a particularly bad meal; if this were the case, the negative responses would not be systematically more common in the research interviews, but

8

rather would have been equally likely to occur with case managers as with outside interviewers. This pattern that emerged regarding the food questions emphasizes the importance of training care managers so that they help consumers feel comfortable in giving an honest opinion, even when it is negative. The PAA Directors also raised the issue of whether there is a difference in ratings of frozen meals vs. hot delivered meals, suggesting that when consumers control which meal they will have at a particular time, they may have more positive responses about the quality of the meals themselves. Since this question is related to the important issues of consumer choice and control, it would benefit from further empirical attention. The impact of consumer self-direction on ratings of service quality is a significant question beyond the scope of this project. However, there is no reason to assume that consumer choice and control would negatively affect the reliability of satisfaction data gathered by case managers. Summary and Conclusion Our major finding is that case managers can collect reliable satisfaction data from consumers. Data collected by case managers produced high levels of reliability, and, in most instances, results were comparable to data collected by research interviewers. Using case managers is cost-effective; such a strategy allows area agencies to collect data on all enrollees, allowing for the possibility that meaningful comparisons could be made across providers. We recommend that the Ohio Department of Aging and the Area Agencies on Aging implement this data collection strategy statewide. Using this approach does require effort and resources from both the area agencies and ODA. Specific recommendations are presented below:

9

Training Training is essential to the success of this effort. To minimize the amount of difference between case manager and research interviewers, we recommend that all case managers receive formal interview training through a standardized training package. Because case managers arrange and monitor the services received by PASSPORT clients, collecting data adds a new dimension to the interaction with clients. Adding to the complexity of the case manager role mandates good training. Our study found that case managers could collect reliable data. Our training and implementation experience highlighted the importance of thorough training to clarify the purpose of the survey, support case managers in taking on a new role, and to assure that data are collected using standard research techniques. A train-the-trainer model is recommended. Researchers will provide the content, materials, and initial training. ODA and site training staff will participate in the centralized training and would then assume the ongoing training function. To support this model, we recommend a standardized training package that includes both classroom and computer-based learning. The computer-based part of the training should utilize technological innovations now available, such as video imaging and interactive media. Training materials should be designed to train new case managers on an ongoing basis. This training model assumes that turnover by case managers will continue. A designated trainer at each site can supplement the training materials. We also recommend that case managers not receive this training as part of their initial clinical training package. We also recommend that case managers be briefed on satisfaction results, and have opportunities to ask questions about the survey as part of an ongoing training function.

Further Research Additional research should be undertaken to examine some questions related to statewide implementation. While the pilot study found that data could be collected

10

reliably, statewide implementation creates two major changes. First, rather than being trained directly by research trainers, case managers will be trained by agency staff. The use of the train-the-trainer model and the development of the standardized training materials is designed to address this issue, but using a different training model could have an affect on the process. A second area of concern involves the use of all case managers. The pilot used a handful of case managers; these volunteers may have been different than the typical case manager. Does changing the approach to training case managers affect the reliability of the data collected? Does having all case managers collect these data, rather than a select group of volunteers, affect the data collection? To assure that these changes do not affect data reliability, we recommend that, after state-wide implementation, the initial scale work on the reliability be repeated. Further work can improve the HCSM instrument. The HCSM is a valid and robust instrument that has been carefully developed. However, as agencies gain wide experience with implementation, it makes sense to continually improve the scale. We recommend that some changes be explored, including the addition, modification, and deletion of items. For instance, one problematic HCSM question asks: “Is the worker thorough?” Many consumers appear to be confused by the meaning of this item. We recommend that ODA also explore the development of satisfaction measures for additional services not currently covered, such as Adult Day Care, an increasingly popular waiver service. Technical assistance on integrating satisfaction data into an overall state and agency quality monitoring strategy is needed. Once these satisfaction data are collected, the final but critical step in the quality process is to use these data to improve services. What is the best way to compare data across PAAs or providers? Are there adjustments that need to be made to the data? For example, are there differences in the amount of missing data by PAA or provider? Do certain PAAs or providers serve a more disabled

11

caseload, necessitating adjustments for comparison purposes? Finally, how can these data be incorporated into an overall quality plan for the PAAs and ODA?

12

WHAT CAN WE LEARN ABOUT THE QUALITY OF PASSPORT FROM THOSE WHO LEAVE THE PROGRAM? Methods Two distinct populations were studied to address this question. The questions asked of each population were very similar. One population consisted of former PASSPORT clients who had disenrolled to enter a skilled nursing facility. The second population was made up of former PASSPORT clients who had disenrolled for “other” reasons. These “other” disenrollments include several categories: “Client does not agree with care plan;” “Client withdrew;” “Other.” Over a four month time period, individuals in each group were identified and a letter of introduction was sent explaining the purpose of the study and advising the former PASSPORT client that a Scripps interviewer would be calling in a few days. The research interviewer made a maximum of six attempts to reach each client. Interviews were conducted within 45 days of disenrollment. Participants We used our pilot sites (PAA 1 and PAA 7) to collect data on PASSPORT clients entering nursing homes. Case managers were asked to identify the primary decision-maker for each client and to supply the contact information for this person. This was typically the client themselves or a family member. When the client was the primary decision maker, the survey was administered in person by a Scripps interviewer. This happened in only 14% of the cases. When the primary decision maker was a family member or close friend, the letter of introduction was sent to them and the survey was conducted by telephone. The response rate for clients was 54% and 61% for non-client decision makers.

13

“Other” disenrollments were collected from all PAAs across Ohio. In this population, most (68%) of the interviews were conducted with former PASSPORT clients themselves. The remaining interviews were conducted with the primary decision maker. The response rate among former clients was 50% and 49% among non-client decision makers. Measures Clients who terminated from the PASSPORT program were surveyed about their experiences.2 The survey began with a list of home care services typically offered through PASSPORT: personal care, home health aide, home delivered meals, transportation, adult day care, and an emergency response system. If the client received personal care or homemaker service, a series of eight questions was asked about satisfaction with each service. The questions included the following domains: amount of service needed and received, service reliability, ease of changing services, and overall performance of the workers. Similar questions were asked about case management services. The survey continued with questions about the decision to either end PASSPORT services or to enter a nursing facility. These included whether the decision was sudden or considered over a period of time, an evaluation of the client's health at the time of the decision, whether a client was also receiving help from family and friends, and whether feelings about the PASSPORT workers and any family or friend caregivers influenced the decision to disenroll. Finally, the participants were asked open-ended questions about the most important factor in their decision to end services and what could have been changed to help the client remain in PASSPORT.

2

Copies of the surveys for term inating clients and the HCSM are attac hed to this rep ort.

14

Results: Nursing Home Disenrollments Almost half of the terminations from PASSPORT were for individuals moving to a nursing home. A review of PASSPORT service utilization data show that these individuals needed a substantial amount of assistance. More than four of five clients in this category received personal care and homemaking service from PASSPORT (See Table 3). Three in four used emergency response systems and more than six in ten received home delivered meals. These individuals received considerable assistance from family and friends, with two-thirds receiving help with shopping and laundry, transportation, and medications. Almost half received family assistance with meals and bathing. Participants were asked whether they received enough hours of their PASSPORT service, and if not, how many more hours were needed. Of those clients receiving personal care services, almost 70% said they were receiving an adequate number of hours of service. The 30% who said they needed more service reported needing about seven additional hours per week. Of those receiving homemaker services, about one-third said they needed, on average, an additional seven hours per week. Questions regarding the quality of personal care, homemaking and case management services were combined into three indices, with a score of “4" indicating highest satisfaction. Respondents reported high satisfaction with the two major services arranged by PASSPORT: personal care and homemaker. Mean scores of 3.4 and 3.5 respectively indicate that most clients

15

Table 3 Description of Nursing Home and “Other” Disenrollments From PASSPORT Disenrolled to Nursing Home

Disenrolled “Other”

Percent Receiving PASSPORT Services

Personal Care Home Delivered Homemaker Transportation Emergency Response System Adult Day Care

85.2 62.3 80.3 31.1 77.0 9.8

65.6 57.8 68.8 23.4 64.1 9.4

Percent Receiving Services Provided by Family and Friends

Shopping, Cleaning, Laundry Bathing, Grooming, Dressing Medication, Health Monitoring Transportation Meals

65.6 44.3 62.3 65.6 49.2

84.8 47.8 65.2 87.0 78.3

Self-Rated Health

Excellent Good Fair Poor

3.3 18.0 16.4 62.3

6.7 33.3 40.0 20.0

Mean Satisfaction Scores

Personal Care Homemaker Case Management

3.44 3.48 3.82

3.32 3.38 3.53

rated their services as either good or excellent. Satisfaction scores with case management of 3.8 indicated that almost all clients rated this service as excellent. When asked whether they would recommend PASSPORT to a friend, the vast majority of respondents (96.7%) answered positively. Respondents were also asked about the decision to enter the nursing home. Most (68.9%) reported facing an event of some type that required the move to a nursing facility. Hospitalization was one major event that triggered a change, with six of ten reporting that the client was in the 16

hospital at the time the decision was made. Self-ratings of the client's health show that over 93% of participants stated that the client's health was very important in the decision to enter a nursing home. We also asked a series of open ended questions about the decision to enter the nursing home. When asked what the most important reason for leaving PASSPORT was, respondents gave three major reasons: a problem with the client's health, the client needed 24-hour care, and issues with their caregivers. Another question asked about what changes were needed in PASSPORT to enable clients to remain in the program. The majority of respondents indicated that nothing could have been changed. The next most likely answer was having 24-hour care. These data suggest that PASSPORT clients who moved to nursing homes were typically quite satisfied with the services received, but, due to health conditions, were no longer able to remain at home. Approximately 30% of the respondents did feel that additional services were needed from PASSPORT. Whether added services could have helped family members maintain a client in the community for a longer period of time is unknown. Examination of plans of care for high risk clients may provide more insight into this question. Results: Other Disenrollments As with the nursing home clients, service utilization by clients who disenrolled for other reasons was examined. More than half of these clients receive personal care, meals, and homemaker service. Family and friends provide a great deal of assistance to these consumers. More than three-fourths received family help with shopping, chores, laundry, transportation, and meals. Of those clients receiving personal care services, eight of ten indicated that they had enough hours of personal care. Those needing more service required an average of just under five 17

additional hours per week. The vast majority of those receiving homemaker services (85%) reported that they received enough care. Those asking for more service estimated needing 7.5 additional hours per week (See Table 3). The mean scores for the personal care, homemaker and case manager satisfaction indices also show high satisfaction with PASSPORT services. Again, most respondents reported either good or excellent for their ratings. The majority (86.9%) of these former PASSPORT clients said they would recommend PASSPORT to a friend. When asked about the decision to end PASSPORT services, this group presented a mixed set of reasons. One group reported that they no longer needed the care and that their health had improved. A second group reported the presence of a new or returning informal caregiver who could now provide the needed assistance. The third major group were those reporting frustrations with their service workers’ job performance. In general, PASSPORT clients who terminated from the program indicated that they received the right amount of service and reported high satisfaction with PASSPORT services. Many left the program because their condition improved. Some clients reported leaving because they were frustrated by the competency and/or reliability of their in-home workers. Summary And Conclusion In this area of study we examined those individuals who had terminated from the program. The primary reasons consumers left the program surrounded their health conditions, family circumstances, and problems with in-home workers. Consumers who left PASSPORT did not appear to be less satisfied with services than the current enrollees.

18

Information about those consumers who terminate from PASSPORT needs to be enhanced. In many cases the reason for leaving is not recorded in a consistent manner by each of the PASSPORT agencies, and thus an accurate portrayal of why individuals leave the program has not been available. We do know that termination rates for PASSPORT clients vary across sites and using common definitions in a standard way could give ODA and sites a better understanding of termination patterns. We recommend that ODA build in a strategy for periodically assessing service satisfaction and reasons for leaving for consumers who terminate from PASSPORT. A cost-effective sampling frame and data collection approach need to be developed. While those clients terminating do not appear to be less satisfied, ongoing monitoring of this question is important.

19

Reference Burwell, B. (2001). Medicaid Long-Term Care Expenditure in FY 2000. Cambridge, MA: Medstat Group. Geron, S. M., & Chassler, D. (1998). The Home Care Satisfaction Measure (HCSM): Instrument design and results of psychometric analysis. Boston: Boston University School of Social Work.

20

APPENDIX MATERIALS Home Care Satisfaction Survey Assessor Checklist For Non-participants Interview Schedule for Non-participants Interview Schedule for “Client Withdrew” Disenrollments Interview Schedule for “Nursing Home” Disenrollment

21