MEDINFO 2001 V. Patel et al. (Eds) Amsterdam: IOS Press © 2001 IMIA. All rights reserved
Technology as an Aid to Coping with Caring: A Usability Evaluation of a Telematics Intervention Mary Chambers and Samantha Connor University of Ulster at Coleraine, Northern Ireland
frequently take on the role of caregiver, bringing a sense of achievement and satisfaction. Not infrequently, however, the positive aspects of family care giving are countered by those of worry and stress [3]. Stress is an inherent and inevitable part of being a carer, and stress and burden have provided the background for many studies of family caregiving [4, 5, 6, 7, 8]. Ways in which this stress can be ameliorated is an important area of research [9] and exploration of the stress process [10] has shown that coping is an influential resource on the extent to which a stressful situation affects psychological wellbeing.
Abstract This paper presents evaluation data for an interactive software program designed to provide family carers with information, advice and psychological support by way of feedback of their coping capacity. The multimedia program consists of an information-based package that provides carers with advice on health promotion and relaxation and offers them a range of coping strategies (for example, positive self-talk, assertiveness training and relaxation tapes and videos). The program also includes a carer’s self-assessment instrument, designed to provide both family and professional carers with information to assess how family carers are coping with their care-giving role. As part of the usability evaluation, casual users (family carers, professional carers and older people) were invited to test the program and were administered a program evaluation questionnaire measuring quality and efficiency in utility and usability. Quantitative data were analysed by using the Statistical Package for the Social Sciences (SPSS) and qualitative data were analysed by content analysis. Findings indicated that the program is visually pleasant, easily understood, responds quickly and corresponds with user expectations. A number of recommendations are made for improvement of the navigation of the program.
ACTION (Assisting Carers using Telematics Interventions to meet Older persons Needs) is a European project where technology has the capability to act as an important tool for family carers, providing information and education, as well as being a medium for increased communication and support. The primary aim of the project to maintain or enhance the autonomy, independence and quality of life of frail older and disabled people and their family carers by the application of telematic technology [11]. This technology, along with the carers’ own television and a remote control device, allows carers to communicate directly with professional carers and other family carers, as well as provide access to educational and support material. A number of information packages were developed specifically for carers, addressing needs highlighted by them earlier in the project [12]. These packages were then validated by each of the countries involved and the results of this evaluation were fed back to the development teams and resulted in a further prototype of the ACTION system. This paper presents evaluation results for the second stage of the validation process. While the ACTION system is multi-faceted, results are presented for formal and informal carers use of one part, an interactive software program designed to provide family carers with information, advice and psychological support to aid coping with caring.
Keywords: Coping, psychological stress, health care technology, program evaluation, caregivers, disabled, elderly.
Introduction Changing demography across the Western world is resulting in an increasingly older population requiring support and assistance in order to remain actively engaged in family and community life. At the same time, there is a changing philosophy with regard to health and social services provision particularly across Europe and manifested in the shift from institutional to community care. A key objective of this movement is helping older people to live in their own homes for as long as possible [1,2]. To achieve this and to ensure that the older person receives the necessary care and support, family members
Whilst many authors [13, 14, 15] have endeavoured to articulate a comprehensive and meaningful definition of coping it has yet to be conquered. How carers cope is very individual and reflects the interplay of a number of factors including personal, environmental and cultural. In order to provide structure and guidance social-cognitive learning
1130
Chapter 13: Evaluation
An important facet of any assistive technology is how easy users find it to operate in their daily life and this becomes even more crucial when the users are older and/or disabled. The user interface to a software application is the way in which the user and the product communicate with one another, the way users indicate to the application what they want to do next through interactions with display elements via input devices such as a mouse, keyboard, or in this case a remote control [22]. Usability is therefore a measurable and essential characteristic of a product user interface. Nielsen’s model [23] of system/program acceptability was adopted within the study as it explores the key dimensions of utility and usability and how they relate to the broader and more long term concept of program acceptability.
theory [16] was used as the theoretical framework to underpin both the instrument development and the coping strategies. The coping strategies section of the program is aimed at helping carers cope with the emotional demands upon them, is accessible to anyone and as far as can be ascertained this is the first time such a program had been developed for family carers. Contained within the programme is information and education regarding physical and mental health, the experience of caring, responding to difficult situations and ways to manage. Practical advice is offered on achieving good health and includes relaxation tapes and relaxation videos. From a mental health perspective a range of coping strategies are included based on the work of Gerrard [17] and Meichenbaum [18], for example, positive self-talk, cognitive re-structuring and cognitive reframing, assertiveness and social skills training [19]. A key objective of the program is the emphasis on normalisation reassuring the carers that their feelings and reactions are normal and experienced by others.
According to Nielsen, the term user acceptability is a broader concept than usability as it is concerned with an understanding of the likelihood that the system/program will be regarded as an appropriate system with functionality that the user feels comfortable with to use again, over a long period. Thus, it includes elements of social acceptability and practical acceptability. The question of social acceptability is in many ways a question of people’s values and beliefs and whether they find it desirable. Practical acceptability includes costs, compatibility, reliability and usefulness. Usefulness can be split up in two factors, namely, utility and usability.
The literature suggested that whilst a range of instruments are available to assess carers coping ability, they have as their primary objective the collection of information to assist professional carers in decision-making [20, 21]. The concept of carers being able to carry out a self-assessment in a systematic manner, obtain feedback and be able to select adjuncts to coping from a range of options has not yet been explored, hence the importance of this intervention. The carers self-assessment instrument is designed to provide both family and professional carers with information to assess how family carers are coping with their care-giving role. Initially family carers are required to complete a biography, which gathers some personal data as a baseline. The self-assessment program is divided into 4 weekly question sets of 25 questions each. The questions address five areas of caring that have been identified by carers as being important. Access is available only by the use of a PIN number to protect confidentiality. The family carer completes the questionnaire using the remote control. Questions are of a Likert type – Strongly Agree to Strongly Disagree, using a 1-5 scale. The carer has the opportunity to choose whether to not to send the questionnaire results to the professional carer. Regardless of whether the results are sent to the professional or not, the carer will receive feedback in text format as to their coping capacity. The program design is such that depending on the carer’s questionnaire scores the carer will be directed to different coping strategies including contacting a professional carer if necessary. The identified health professional, for example the practice nurse or district nurse, will also be able to access the carers’ data, again using a PIN number. The program enables the professional carer to examine individual question responses, section responses and provides an aggregated graphical representation of carer details over time.
In Nielsen’s model, utility is concerned with whether the users consider that they can benefit from the system/program and if they find it useful for their needs. This leads to the following considerations regarding content: • is it appropriate • is it right, i.e. does it include all necessary parts and yet at the same time does not involve too many components • is it developed to a sufficient level of complexity and amount of details • is it delivered with a language, including text and illustrations that is possible for the users to understand Usability, is a multi-dimensional concept relying on five usability attributes: • Learnability means that the program should be easy to learn to use, so that after a short instruction the user should be able to use the program. • Efficiency means that the program should be efficient to use, so that a high level of productivity is possible to reach as soon as the user has learnt to use the system. • Memorability so the user can remember how to use the program and if they have not used the program for a long time it can easily be used again. • Errors are to be avoided and if an error occurs it should easily be solved by the user. • Satisfaction is an important challenge and focuses on the pleasant aspects of the program. It is of
1131
Chapter 13: Evaluation
an overall usability score, general rating of the program, a more detailed usability profile in terms of attractiveness, controllability, efficiency, helpfulness and learnability, and a detailed listing of program aspects that are especially good and/or problematic. The report also relates the above results to specific additional variables, namely, countries, user group, gender, age, and educational level of the users.
significant importance that the user likes the program and is subjectively satisfied when using it.
Objectives It is evident that the system will only be successful if it is acceptable to family carers and the persons they care for. The objectives of the study described in this paper were to evaluate if the interactive software program and carers selfassessment instrument for coping program is of high quality and is efficient in relation to utility and usability.
Qualitative data from the questionnaire was analysed using content analysis, a method of organising data into main themes which illustrate the issues raised by respondents during the evaluation process [25, 26]. These responses provided more detailed feedback concerning the identification and prioritisation of problems and necessary changes to the programs as well as contextual information relating to the local adaptation of the program.
Materials and Methods The Website Analysis and MeasureMent Inventory usability instrument version 2.5 [24] was used under license to evaluate formal and informal carers use of the coping progam and the instrument to assess coping skills. In consultation with the author, minor changes were agreed to the wording, a front sheet was added to collect demographic information, and a third section was added containing openended items in order to provide essential contextual data concerning the program. Version 1 of the instrument achieved Cronbach’s Alpha reliability coefficient measures of between 0.697 and 0.960 for the five subscales (attractiveness, control, efficiency, helpfulness and learnability). Version 2.5 is shorter and uses the most reliable items from version 1, while retaining the five subscales.
The research team followed the ethical guidelines developed in ACTION [27] during the evaluation. In addition, project partners received approval from their appropriate ethical committee or equivalent. The principle of informed consent was carefully followed in the evaluation of the programs. Participants were sufficiently informed about all aspects of the research. They were told that they have the right to withdraw from the research at any time and were also promised confidentiality and security of their own personal data.
Results and discussion In total 242 casual users completed ACTION-WAMMI questionnaires to evaluate formal and informal carer’s use of the instrument to assess coping skills and the instrument to assess carer/telematics interface and professional/telematics interface. A breakdown of these users by participating country is as follows: Sweden 22%, England 20%, N. Ireland 23%, Portugal 21% and Republic of Ireland 14%. The program was evaluated by cared for people (n=26) family carers (n=103) and professional carers (n=113) in all partner countries. The majority of respondents (73%) were female.
The WAMMI consists of 20 usability items and takes approximately 10-15 minutes to complete. The questionnaire has been used in different European countries and is able to provide an overall usability score as well as provide more detailed usability profile in terms of five usability scales which have been developed out of analysis of large numbers of responses by subjects. These scales are Attractiveness, Control, Efficiency, Helpfulness, and Learnability. Thus the questionnaire is able to discern how well each program rates on each of these scales as well as providing a detailed listing of those aspects of the program respondents have found to be particularly good or particularly problematic.
In the interpretation of usability, a score of 50 is average, therefore below 50 is lower than average and above 50 is above average. Usability for the program generated a Global Usability score for the program of 67, well above average. This is a weighted composite of items from each of the other five WAMMI scales of Attractiveness, Controllability, Efficiency, Helpfulness and Learnability. Global Usability centres around the concept that it must be easy for users to access what they need or want from the program, that there is a good, understandable level of organisation, and that the program ‘speaks the users language’.
Casual users were involved in the testing of individual programs in this phase of testing. Professional carers in the health care sites and further education settings were encouraged to view the program, complete a questionnaire, and place it in a brown envelope. These envelopes were placed in boxes for the researchers to collect. Only a code number identified questionnaires and respondents were assured that their answers were anonymous. Family carers and cared for people were also targeted in the health care sites as well as in home settings with the procedure being the same as above.
The Attractiveness score was highest at 76 indicating that it the program is visually pleasant and offers much of direct interest to the intended users, both in functionality and information. The program scored 60 on the scale of Controllability. This score showed that users feel that they
Questionnaire data were checked, coded and inputted into the Statistical Package for the Social Sciences (SPSS for Windows Version 9.0). The WAMMI data report includes
1132
Chapter 13: Evaluation
felt the program provides reassurance and emotional support and enables them to assess their own coping capacity and provides useful information regarding how to enhance their coping skills. Highlighted throughout was the need for this type of information and support to be more readily available to family carers. The potential for expanding the program was also addressed and the feedback supplied by all participants will be used to further improve the nature and quality of the program.
can navigate around the program with ease. The program scored 71 on Efficiency signifying that users feel that they can quickly locate what is of interest to them and they feel that the program responds at a reasonable speed. The program scored 74 on the scale of Helpfulness. This score demonstrated that the program corresponds with the users expectations about its content and structure i.e. is not misleading. The score on the Learnability scale was 67.5 showing that users are able to start using the site with the minimum of introduction and that everything is easy to understand from the start.
Acknowledgements
Analysis of each of the 20 statements in the ACTIONWAMMI showed the amount of agreement or disagreement users expressed with each. This identified the strongest features of the program and those that require any improvement. Users rated the program very favourably. Statements such as ‘The pages are very attractive’, ‘The program seems logical to me’, ‘Everything in this program is easy to understand’, and ‘The program helps me find what I am looking for’, received a very high level of agreement. Only a couple of areas received ratings that showed room for improvement. These were ‘Learning to find my way around this programme is a problem’ and ‘Remembering where I am in the program is difficult’. Qualitative data from the questionnaire in general supported the quantitative findings. Family carers felt the program to accurately address emotional issues pertinent to caring, illustrated by the following quote:
We would like to take this opportunity to thank everyone who contributed to the development, testing and evaluation of this program. We would like to acknowledge the good will and support of the family carers who contributed to this and other aspects of the ACTION project. Without the commitment and dedication of family carers the project would not have been possible. Thanks are also extended to the European Union DGX111 Telematics Application Programme, Disabled and Elderly for the financial support given to the project, as well as all the professional carers who contributed in a variety of ways.
References [1] Davies A. Disability, homecare and the caretaking role in family life. J Adv Nurs 1995: 5: pp. 475-84. [2] Evers A. The future of elderly care in Europe: limits and aspirations. In Scharf F and Wenger G, eds. International Perspectives on Community Care for Older People. Avebury: Aldershot, 1995, pp. 3-20.
“Realisation that you are not the only one in this very trying role. Carer’s feelings pretty well detailed here as they actually are! Some home truths. I’m over worked, stressed and overweight. Sound, good advice. Excellent program. Could be applied to many areas of life in general. Whoever wrote this had real feeling and understanding of the carer under strain. Brilliant!”
[3] Nolan M, Grant G, and Keady J. Understanding family care: A Multidimensional model of caring and coping. Buckingham: Open University Press, 1996. [4] Twigg J and Atkin K. Carers perceived: policy and practice in informal care. Buckingham: Open University Press, 1994.
Findings indicate that the program is useful to carers and of high quality, and efficient in relation to utility and usability. The program was highly rated in terms of Global Usability and it’s five component scales of Attractiveness, Controllability, Efficiency, Helpfulness, and Learnability. This illustrates that the program is visually pleasant, easily understood, responds quickly and corresponds with user expectations. Users felt there was room for improvement in the navigation of the program.
[5] Liston R, Mann L, and Banerjee A. Stress in informal carers of hospitalised elderly patients. J R Coll Physicians Lond 1995: 29(5), pp. 388-391. [6] Schneider J, Murray J, Banerjee S, and Mann A. EUROCARE: A cross-national study of co-resident spouse carers for people with Alzheimer’s disease: I – Factors associated with carer burden. International Journal of Geriatric Psychiatry 1999: 14(8), pp. 651661.
The ACTION-WAMMI was used in evaluation of the whole ACTION system. A consideration for future research is that the questionnaire be tailored to be more specific to the coping program.
[7] Schofield H, Bloch S, Nankervis J, Murphy B, Singh B, and Herrman H. Health and well-being of women family carers: a comparative study with a generic focus. Australian and New Zealand Journal of Public Health 1999: 23 (6), pp. 585-589.
Concluding Remarks The evaluation data obtained suggests that the program prototype is easy to use and informative. From a family carer perspective it is pitched at the right level. It is evident from the data the program has the power to enhance carers quality of life, autonomy and independence. Family carers
[8] Millan-Calenti J, Gandoy-Crego M, Antelo-Martelo M, Lopez-Martinez M, Riviero-Lopez M, and MayanSantos J. Helping the family carers of Alzheimer’s patients: from theory to practice. A preliminary study.
1133
Chapter 13: Evaluation
[10] Pearlin L. The sociological study of stress. J Health Soc Behav 1989: 30, pp. 241-256.
Archives of Gerontology and Geriatrics 2000: 30(2), pp. 131-138. [9] Nolan M, Grant G, and Keady J. Understanding family care: A Multidimensional model of caring and coping. Buckingham: Open University Press, 1996. [11] Magnusson L, Berthold H, Brito L, Chambers M, Emery D, and Daly T. Using Telematics with older people: the ACTION project. Nursing Standard 1998: 13(5), pp.36-40.
[21] Nolan M and Grant G. Helping new carers of the frail elderly patient: the challenge for nurses in acute care settings. Journal of Clinical Nursing 1992: (1), pp. 303-307.
[12] Berthold H. Users needs and priorities. Deliverable D04.1. Action Project DE3001. Brussels: EU Commission, 1997.
[22] Mayhew D. The Usability Engineering Lifecycle. California: Morgan Kaufmann Publishers Inc., 1999. [23] Nielsen J. Usability engineering. Boston: Academic Press, 1993.
[13] Pearlin L and Schooler C. The structure of coping. J Health Soc Behav 1978: 19, pp.1-21.
[24] WAMMI - Q - 2.5 EN (c) Copyright HFRG Ireland & Nomos Management AB Sweden, 1998.
[14] Lazarus R and Folkman S. Stress, appraisal and coping. New York: Springer, 1984.
[25] Strong J, Ashton R., Chant D, and Cramond T. An investigation of the dimensions of chronic low back pain: the patients’ perspectives. British Journal of Occupational Therapy 1994: 57, pp. 204-208.
[15] Turnbull A and Turnbull H. Participatory research in cognitive coping: from concepts to research planning. In Turnbull A, Patterson J, Behr S, Murphy D, Marquis J, and Blue-Banning M, eds. Cognitive, Coping, Families and Disability. Baltimore: Paul H. Brooks, 1993.
[26] Reutter L and Ford J. Perceptions of public health nursing: views from the field. J Adv Nurs 1996: 24, pp. 7-15.
[16] Bandura A. Social learning theory. Englewood Cliffs, New Jersey: Prentice Hall, 1977.
[27] Magnusson L and Barbosa da Silva A. Ethical guidelines for ACTION (DE3001). Brussels: EU Commission, 1998.
[17] Gerrard B, Boniface W, and Love B. Interpersonal skills for health professionals, Virginia: Preston Publishing, 1980.
Address for correspondence
[18] Meichenbaum D and Novaco R. Stress innoculation: a preventative approach. Issues in Mental Health Nursing 1985: 7(1-4), pp. 419-435.
Professor Mary Chambers Room B035 School of Health Sciences University of Ulster at Coleraine Cromore Road County Londonderry Northern Ireland BT52 1SA. Telephone: 028 70324286. Fax: 028 70324287. E-mail:
[email protected].
[19] McCammon C and Chambers M. Report on a theoretical framework to influence instrument development. Deliverable D08.1. Action Project DE3001. Brussels: EU Commission, 1998. [20] Quayhagen M and Quayhagen M. Coping with conflict: measurement of age-related patterns. Research on Ageing 1982: 4, pp.364-377.
1134
