Usability evaluation of a webbased patient ...

JAN

JOURNAL OF ADVANCED NURSING

ORIGINAL RESEARCH

Usability evaluation of a web-based patient information system for individuals with severe mental health problems Lauri Kuosmanen, Tiina Jakobsson, Jari Hyttinen, Marita Koivunen & Maritta Va¨lima¨ki Accepted for publication 11 June 2010

Correspondence to L. Kuosmanen: e-mail: [email protected] Lauri Kuosmanen PhD RN Project Manager Department of Nursing Science, University of Turku, Finland, and Primary Health Care Organisation of City of Vantaa, Finland Tiina Jakobsson MNSc RN Project Manager Department of Nursing Science, University of Turku, Finland, and Omnia, The Joint Authority of Education in Espoo Region, Finland Jari Hyttinen PhD Professor Department of Biomedical Engineering, Tampere University of Technology, Finland Marita Koivunen PhD Nursing Manager Department of Nursing Science, University of Turku, Finland, and Satakunta Hospital District, Pori, Finland Maritta Va¨lima¨ki PhD RN Professor, Nursing Director Department of Nursing Science, University of Turku, Finland, and Hospital District of Southwest Finland, Turku, Finland

¨ LIMA ¨ KI KUOSMANEN L., JAKOBSSON T., HYTTINEN J., KOIVUNEN M. & VA

(2010) Usability evaluation of a web-based patient information system for individuals with severe mental health problems. Journal of Advanced Nursing 00(0), 000–000. doi: 10.1111/j.1365-2648.2010.05411.x

M.

Abstract Aim. This paper is a report of a study conducted to compare service users’, nursing students’ and Registered Nurses’ evaluations of the usability of a patient education website intended for individuals with severe mental health problems. Background. There is an obvious need for reliable mental health information on the Internet. When evaluating the usability of Internet-based patient education methods, the opinions of all parties need to be ascertained. Methods. An explorative descriptive design was used. Twenty-one service users, 20 nursing students and 35 Registered Nurses were recruited for the study in 2003 and 2004. Data were collected using a self-developed questionnaire on the content, structure and visual appearance of the website. Results. Service users had positive attitudes towards computer and Internet use but they needed support when using the computer and Internet. According to the evaluations, the content, structure and visual appearance of the website were good and it could be adopted for clinical practise after minor revisions. There were some differences in the evaluations between participant groups. Nurses were the most critical group, and the service user group was the most satisfied, although they were less experienced with using both computers and the Internet. Conclusion. It is especially important to include service users’ evaluations at the early stages of the development process of Web-based patient education systems. It is possible to produce an information technology-based patient education system for individuals with severe mental health problems. Nurses working in psychiatric services need to pay more attention to supporting service users in computer and Internet use. Keywords: nursing, service user involvement, severe mental health problems, usability evaluation, web-based patient information system

2010 The Authors. Journal compilation 2010 Blackwell Publishing Ltd

1

L. Kuosmanen et al.

Introduction Improving insight, compliance and capacity for independent living is a major goal in the treatment of individuals with severe health problems (Merinder 2000). Patient education is one way to improve individuals’ understanding of the illness and increase their willingness to persevere with treatment (Pekkala & Merinder 2002, Haynes et al. 2008). Education has a positive impact on patients’ compliance in taking medication (Sleath et al. 2003) and thus on reducing readmissions (Hodgson et al. 2001). However, information as such does not automatically increase knowledge levels (Forster et al. 2006), while educational needs may vary between different patient groups (Chieng et al. 2001). Therefore, a series of educational solutions (Rotondi et al. 2007), specific curricula (Chieng et al. 2001) and nontraditional learning methods are needed (Haynes et al. 2008) to achieve learning. The Internet has shown promise in providing information for patients (Lockyer et al. 1999, Suslow et al. 2001, Wantland et al. 2004). Multimedia programmes may support the intelligibility of the written word by combining text, pictures, photos and colours (Wofford et al. 2005). Recent studies have shown that increased use of Internet healthcare information has a positive influence on patients’ health behaviour (Gimenez-Perez et al. 2002, Wantland et al. 2004, Wofford et al. 2005), knowledge levels (Christensen et al. 2004), self-efficacy and self-care abilities (Wyndra 2001, Tate et al. 2003). A number of support systems or self-help programmes to be used via Internet have been developed (Mazzi & Kidd 2002, Guendelman et al. 2002, Wu et al. 2005), and computer and Internet use have been tested with various groups (Nahm & Resnik 2001). However, these applications have been less systematically developed and used among individuals with severe mental health problems, who may easily be excluded from the information society (Eysenbach & Jadad 2001). In addition, the usability of patient information application systems has seldom been tested among mental health service users (Marks et al. 2007).

Background Due to their illnesses, people with severe mental health problems may have difficulties in processing information, learning new ideas and recalling matters, which may further impede forming new concepts, thinking logically and with sustained concentration (Sadock & Sadock 2005). Inability to concentrate may also culminate in problems in the selection of relevant information (Morss et al. 1993). 2

Patients, especially those with schizophrenia spectrum psychosis, for example, may have problems understanding their illness and treatment, because the relationship between knowing, learning and psychiatric symptoms is complex (Macpherson et al. 1996). Information technology has the potential to enable people with severe mental health problems to acquire relevant information and to support them in coping independently by giving them an opportunity to access personalized services at any time (Rotondi et al. 2005). Mental health problems as such have not been a barrier to computer use, and patients themselves have positive attitudes toward web use (Spinhoven et al. 1993, Podil et al. 2000). The Internet may be particularly useful to those who are isolated or lonely, because information received via the Internet has decreased loneliness and depression (Shaw & Grant 2002). Earlier studies on the usability of these methods indicate that computer-based cognitive-behavioural therapy (Marks et al. 2007) and different psycho-educational programmes (Rotondi et al. 2005) have demonstrated their usefulness in providing psychosocial support and treatment (Rotondi et al. 2007). These applications have proved easier to use when they have entailed immediate benefits for users. If the technical usability of application is also felt to be good, IT will increase user satisfaction and benefits (Ammenwerth et al. 2006). Although computer use and Internet-based education are still not often used in mental health services (Wantland et al. 2004) and preliminary trials have not shown them superior to other methods (Jones et al. 2001), there is an obvious need for reliable mental health information on the Internet. In addition, high quality and effective web design in terms of content, structure and visual appearance are needed. If the usability, quality and reliability of Internet-based services are ensured, the use of technologies as part of daily practise may attract users in mental health care (Marsico & Levialdi 2003). All parties–service users, nursing students and nurses– must have an opportunity to evaluate the website before their final dissemination to clinical settings. To evaluate the effectiveness of a Web-based educational tool for individuals with severe mental health problems, we designed a website application named Mieli.Net (in English Mental.Net) (see Koivunen et al. 2007, Va¨lima¨ki et al. 2008). The website is intended for people with schizophrenia spectrum psychosis (F20-29, ICD-10) (WHO 2007), and can be used both in inpatient and outpatient settings. The language used on the website is Finnish, which is the mother tongue of 92% of Finnish people. The quality of all material included was evaluated by a multi-professional group, including service user representatives. Links on the website


JAN: ORIGINAL RESEARCH

are all to official and nonprofit organizations. The website is password protected. At its most basic level it includes a website offering evidence-based and easily understandable and organized information for patients on their illness, its treatment, the social support available, and information about patients’ rights. At the more advanced level, it includes: (1) patientcentred, tailored information using multimedia material (text, voices, figures, photos); (2) a channel for peer support by patients (discussion room, chat room and open Internet diary) and (3) an eSupport tool for counselling and support between service users and staff by means of a question-andanswer column. Nurses can also use the website to update their knowledge of schizophrenia spectrum psychoses. In this paper we report a usability evaluation of the website from the perspective of service users, nursing students and nurses.

The study Aim The aim of the study was to compare service users’, nursing students’ and registered nurses’ evaluations of the usability of the website addressed for individuals with severe mental health problems.

Design An exploratory descriptive design was used. Usability testing can be done at different stages of the development phase (Eslami et al. 2007) and with a variety of technologies and patient populations (Eysenbach & Ko¨hler 2002). The most useful usability results are obtained by testing a few users with many small tests. Keeping in mind these recommendations, we used methods of data collection that are commonly used in studies of human–computer interaction (Nielsen 1993). This small-scale pilot study was followed by a larger randomized controlled trial evaluating the effects of the website (Kuosmanen et al. 2009).

Participants Convenience sampling was used to recruit the participants for the study. A total of 76 people from three different groups were invited to participate in the study: 21 service users in two local mental health service user associations, 20 students at one university of applied sciences and 35 Registered Nurses working in one psychiatric hospital. The general inclusion criteria were as follows: age 18–65 years, ability to speak Finnish and voluntary participation with written

Usability evaluation of a web-based patient information system

informed consent. The nurses participating in this evaluation were different from those who participated in the randomized controlled trial (see Kuosmanen et al. 2009).

Data collection The data were collected between spring 2003 and autumn 2004 in two different environments. Service user data were collected at mental health service user associations. Students’ and nurses’ data were collected in a laboratory environment where, in addition to the usability evaluations, some physiological measurements were made. These will be reported elsewhere. It was assumed that physiological measurements might have a negative effect on service users’ evaluations. Before collecting service users’ evaluations, we first informed them about the study and the representatives of the associations gave them a description of the website both orally and in writing. Those who had attended a presentation arranged an appropriate time and place for the evaluation, and were given a private facility to use the website. To facilitate responding to the evaluation questionnaire, a paper copy of the website was also made available. After familiarizing themselves with the website, respondents completed the questionnaire independently. When the nursing students’ and nurses’ data were collected, they were given oral information about the website development project. Next, they were shown the test laboratory environment. They then received written information about the study and their informed consent was requested. If they were willing to give their consent, they signed a consent form. A research assistant then explained the protocol of the usability evaluation, and introduced the website to be evaluated. Finally, participants familiarized themselves independently with the website, following which they were asked to complete the questionnaire evaluating its usability. Respondents’ age, gender, marital status and education were elicited. Their previous computer and Internet use was elicited on a scale from 1 to 5 (1 = every day, 2 = every week, 3 = monthly, 4 = less than monthly, 5 = not at all). Computer and Internet skills were also elicited with a 5-point scale (1 = very good, 2 = quite good, 3 = neither good nor poor, 4 = quite poor, 5 = very poor), and attitudes toward computers and Internet use (1 = very positive, 2 = quite positive, 3 = neither positive nor negative, 4 = quite negative, 5 = very negative). The data were collected using a questionnaire developed for the study. The content, structure and visual appearance of the website was explored. The content of the questionnaire was based on the quality criteria for health-related websites (Nielsen 1993, eEurope 2002). Altogether 35 items on a


3

L. Kuosmanen et al.

five-point scale (1 = totally agree, 2 = almost agree, 3 = no opinions, 4 = almost disagree, 5 = totally disagree) focused on the usability of the websites. The usability evaluation of the websites focused on three areas. Cronbach’s alpha values for the three different areas of the questionnaire were calculated and were 0Æ68 for content, 0Æ90 for structure and 0Æ85 for visual appearance, showing satisfactory internal consistency of the questionnaire (see Burns & Grove 2005). First, the content of the webpage was evaluated by focusing on its comprehensibility (10 items) (eEurope 2002). The items related to the content concerned, for example, the reliability of the information and whether the source of information, purpose of the website, publisher, time of updating or option to give feedback were included in the pages. Second, opinions of the website structure were elicited with 11 items. These included usability, navigation, colouring to facilitate navigation between pages, and general overview of the pages. Last, the visual appearance of the website was focused on. This was evaluated with 14 items (eEurope 2002). The goal was to assess the success of the visual elaboration, such as text type, colour, clarity of language used, etc. A more detailed description of the item content is given in Tables 2–4.

Ethical considerations Approval for the study was obtained from the appropriate ethics committees.

Data analysis The statistical software package SPSS 12Æ0 was used to analyse the data. Descriptive statistics were computed. When analysing questions concerning the content, structure and appearance of the website, the number of cells in cross-tabulation was reduced by grouping the answers into a smaller number of categories, and the data were re-categorized as follows: ‘totally agree’ and ‘almost agree’ to ‘agree’ and ‘almost disagree’ and ‘totally disagree’ to ‘do not agree’. Differences between categorical variables were tested using chi-square tests. Fisher’s exact test was used when sample sizes and expressed frequencies were small. Cronbach’s alpha values were used to assess the homogeneity of items in each evaluation area in each group. In all tests, a P-value less than 0Æ05 were evaluated as statistically significant. Items where more than 75% of respondents in all groups agreed were considered to be at a satisfactory level, with no special need for improvement. In items where this level was 4

not reached, it was assumed that there was a need for improvement.

Results There were some statistically significant differences in the respondent demographics. In the service user group respondents were older, more often divorced or widowed and had less education. Further, service users evaluated their Internet skills as poorer than respondents in the other groups. In addition, they reported less frequent computer and Internet use. There were no statistically significant differences in computer skills and in attitudes towards computers and Internet (Table 1).

Website Content Out of 10 items evaluating the content of the website, there were five where more than 75% of respondents in all groups agreed with the statement. Participants agreed that on the website the target group was defined, content was reliable, the purpose of the website was stated, there was an opportunity to give feedback and the information on the website was relevant. Areas where respondents found a need for improvement were marking the references, naming the publisher of the website, giving information about the updating date, pictures and new information. When differences between groups were examined, it was found that service users more often disagreed with the statement that the purpose of the website was stated, and more often agreed that the updating had been noted (Table 2).

Website Structure Out of 11 items evaluating the structure of the website, there were six items where more than 75% of respondents in all groups agreed with the statement. According to their evaluations, the website was easy to use, the front page enabled easy navigation to the next page, it was easy and quick to move from one page to another, the general overview of the website was clear and each page contained only one issue. Participants did not agree totally that all relevant information could be found on the front page, that navigation between pages was consistent, that the colouring helped to move from one page to another, that the structure between pages was consistent, and that the creator of the website was mentioned on every page. When examining differences between groups, it was found that service users more often agreed that navigation was consistent, that the colouring helped to move from one page to another and that the creator of the website was mentioned on every page (Table 3).


JAN: ORIGINAL RESEARCH

Usability evaluation of a web-based patient information system

Table 1 Students’, service users’ and nurses’ demographics

Background

Students Service users Nurses (N = 35), (N = 20), (N = 21), n (%) P-value n (%) n (%)

Age (years) 20–30 15 (75) 31–40 4 (20) 41–50 1 (5) 51–60 0 (0) Gender Male 4 (20) Female 16 (80) Marital status Unmarried 7 (35) Married 13 (65) Divorced 0 (0) Widowed 0 (0) Education Comprehensive 0 (0) school Secondary school 12 (60) graduate Vocational degree 7 (35) Academic degree 1 (5) Computer use Every day 15 (75) Every week 5 (25) Monthly 0 (0) Less than monthly 0 (0) Not at all 0 (0) Computer skills Very good 3 (15) Quite good 9 (45) Neither good 8 (40) nor poor Quite poor 0 (0) Very poor 0 (0) Internet use Every day 8 (40) Every week 12 (60) Monthly 0 (0) Less than monthly 0 (0) Not at all 0 (0) Internet skills Very good 3 (15) Quite good 10 (50) Neither good 7 (35) nor poor Quite poor 0 (0) Very poor 0 (0) Attitude towards computer Very positive 10 (50) Quite positive 7 (35) Neither positive 2 (10) or negative Quite negative 1 (5) Very negative 0 (0)

1 3 11 6

(5) (14) (52) (29)

10 (48) 11 (52) 8 4 8 1

(37) (26) (20) (17)

15 (43) 20 (57)

0Æ146