Evaluating Consumer Health Information Websites: The Importance of Collecting Observational, User-driven Data Maaike van den Haak VU University, Faculty of Arts
[email protected]
Abstract The internet is an increasingly popular source of consumer health information (CHI). Worldwide, millions of people regularly visit CHI websites to seek answers to medical issues, either for themselves or their loved ones. As CHI websites play such a vital role in people’s health behaviour, the evaluation of these sites has often been addressed in scholarly studies. The vast majority of these studies have focused on evaluation methods for assessing the quality of CHI websites (expert testing). While this emphasis on quality is understandable – health information needs to be accurate and actual – calls are increasingly made to not only ensure the quality of information on CHI websites, but to also focus on the usability of these sites. This paper offers a review of the usability studies that have so far been conducted regarding CHI websites. We will discuss the potential of, in particular, behavioural user research for CHI websites, by discussing the methodological benefits and drawbacks of the existing usability studies. In addition, we will examine the role of user characteristics in the evaluation of CHI web sites. The results of our review indicate that behavioural user research, when conducted properly, may offer valuable insights that complement those gained by means of expert testing. Keywords: Consumer health information (CHI) websites; usability; observational user research; think aloud method; user characteristics.
Introduction The popularity of consumer health information (CHI) websites has grown enormously over the last decade [1]. Each day, numerous people visit these sites in search of health information to help them decide, for instance, whether to see a doctor, which medicines to take or how to prevent certain illnesses [2, 3, 4, 5]. As such, CHI websites potentially have a major impact on people’s health behavior. Therefore it is generally deemed essential, by medical experts and users alike, that these
Charlotte van Hooijdonk VU University, Faculty of Arts
[email protected]
sites should meet high standards regarding the quality of information, i.e. regarding the extent to which the information on these sites is, e.g., accurate, complete and up-to-date. As a result of this emphasis on quality, most evaluation studies in a CHI context have focused on methods for assessing the quality of CHI websites. These methods often involve (medical) experts who rate information on CHI websites according to predefined quality criteria (i.e., heuristics). Criteria commonly listed in studies involving expert heuristics include accuracy, completeness, readability, design, disclosures and references provided [6, 7, 8, 9, 10]. While the quality of CHI websites has received considerable attention – a focus which is understandable as unreliable or incomplete information could have minor or more serious consequences for its users – the usability of CHI websites seems to have been largely ignored. As such, there is as yet little insight into the extent to which these sites offer their users information that is easily found, well understood and relevant to the users’ needs. This lack of attention for usability is remarkable, as usability testing has long been established as an invaluable part of the design process of other (non-profit) websites, including municipal websites [11, 12], academic library web sites [13, 14, 15], and commercial web sites [16, 17]. Even within the broader medical field, usability testing has come to play a role in the design of (online) health-related products such as brochures [18], self-care applications [19, 20] and interactive health information systems [21, 22]. The general consensus of studies concerned with usability testing is that this form of evaluation offers valuable insights into how users interact with a particular site or product and what may be improved in order to enhance users’ experience. As such, it is not surprising that calls have been made to incorporate usability testing into CHI website design, and that an increasing number of CHI evaluation studies no longer merely involve experts but also include potential users in the design process. The contribution of these users often concerns their opinions through focus groups or interviews and not their actual interaction with a CHI website through observational
research. However, the latter is seen as the standard research method when conducting usability tests. There are different types methods when conducting observational usability testing. One of them is the thinking aloud method (concurrent and retrospective) which involves users performing a number of tasks on a website while verbalizing their thoughts concurrently or retrospectively [23]. Another method is eye tracking in which users also perform several tasks on a website while their eye movements are registered [24]. This paper reviews what behavioral usability studies have so far been conducted with respect to CHI websites. We will discuss the methodological benefits and drawbacks of these usability studies, and examine the role of user characteristics in the evaluation of CHI web sites.
Research method We conducted a literature review of behavioral usability studies conducted on CHI websites. Below, we describe the applied including and excluding criteria and the employed search strategies.
Including and excluding criteria Including and excluding criteria were established in advance. Studies were included when (1) they involved the evaluation of health information websites intended for consumers; (2) they evaluated CHI websites using actual users; (3) they evaluated CHI websites using the think aloud method or other observational measures. Studies were excluded when (1) they did not dealt with content other than health information; (2) they did not evaluate health information that was intended for consumers; (3) they evaluated CHI websites using a predefined list of quality criteria; and (4) they were published before 2000.
Search strategies The first author systematically searched the following databases between December 2009 and January 2010: PUBMED, EMBASE, Cochrane Library and Information Science & Technology Abstracts (LISTA). Searching was based on a combination of the following key words: consumer health information, websites, internet, computer, evaluation studies, user testing and usability. The second author systematically searched the following databases: CINAHL, Web of Science, and ComAbstracts, in the same period as the first author, with the same combination of key words. Table 1 shows the number of retrieved citations per database. In total, we retrieved 685 citations. After removing all duplicate references, titles and abstracts were analyzed according to our inclusion and criteria (see section above), which were met by 21 retrieved articles [25-45].
Table 1. Number of retrieved citations per database Database PUBMED EMBASE Cochrane Library CINAHL Web of Science ComAbstracts LISTA Total
Citations 280 203 10 57 102 17 16 685
Results Methodological issues Although we can conclude that observational research has been done in the evaluation of the usability of CHI websites, the research has not always been appropriately conducted. Some studies mention that they have conducted observational research, but do not go into the specific details, like where and how the data was collected, and type of data that was collected [29, 37, 40]. Also, there are differences in how the behavioral data was collected. Some studies used screen capture and videoanalysis software [30, 32, 33, 34], while others recorded the data using free-text [35, 40, 45]. A number of studies used the thinking aloud method, but the collected verbalizations are not analyzed or it is not clear how the verbal protocols were analyzed [35, 45]. Beside, the think aloud method was not always effectively used as the specific participants, i.e., elderly and lowliterates, stopped thinking out loud while they were performing tasks [25, 38, 39].
User variety The results showed that the evaluation of the usability of CHI websites has been done with different users. For example, the information seeking behavior of adolescents for online health information has been investigated by [29, 32]. The results of [32] showed that the search strategies used by adolescents were similar to those used by adults, i.e., using search engines, and formulating general search strings. However, adolescents had more problems with spelling of queries, browsing of websites, and understanding the content. Also, elderly have been involved in usability research of CHI websites. The usability problems found by [39, 41] indicate that elderly have difficulty with scrolling, font size, use of multimedia, and navigation. Beside age also the level of literacy has been taken into account in usability studies of CHI websites. Studies conducted by [25, 38] investigated how low-literacy adults accessed and evaluated online health information. The results showed that this user group had most difficulties in reading and comprehending health
information, and that their search strategies were ineffective. Finally, both lay persons and medical professionals are involved in the evaluation of the usability of CHI websites. In the report of the results of these usability studies no difference is made between the usability problems of these different users groups [28, 35]. In general, lay users have difficulties with comprehending the information presented in CHI websites due to use of medical terminology. They also formulate ineffective queries as they do not have clear label for the medical concepts they are looking for [28, 34, 42].
Conclusion We conducted a literature review to search for behavioral usability studies conducted on CHI websites. In total we found 21 articles that met our inclusion criteria. All the studies concluded with the notion that usability research involving actual users provided insights on the user perspective on online health information. Also, the results of the usability test identified improvements to the CHI websites. However, in order for usability testing to be effective, the research has to be properly conducted. Not all studies found reported how the usability was conducted. There were also differences in how the data was recorded and analyzed afterwards. Moreover, several studies did not take specific user characteristics into account when using the think aloud method. We also found that different users groups have been involved in the evaluation of the usability of CHI websites. Beside age and literacy, it would be interesting take other user characteristics, like health status and health literacy, into account in usability testing on CHI websites.
References [1] Fox, S. Online Health Search 2006 (Pew Internet & American Life Project), URL (consulted November 2008): http://ppp.pewinternet.org/pdfs/PIP_Online_Health_2006.pdf [2] Cline, R.J.W. and K.M. Haynes. Consumer health information seeking on the internet: the state of the art. Health Education Research. 16: 671-692, 2001 [3] Anderson, J.G. Consumers of e-health: Patterns of Use and Barriers. Social Science Computer Review. 22: 242-248, 2004 [4] Rice, R.E. Influences, usage, and outcomes of Internet health information searching: Multivariate results from the Pew surveys. International Journal of Medical Informatics. 75: 8-28, 2006
[5] Andreassen, H.K., M.M. Bujnowska-Fedak, C.E Chronaki, R.C. Dumitru, I. Pedule, S. Santana, H. Voss, and R. Wynn. European citizens’ use of E-health services: A study of seven countries. BMC Public Health. 7:e7, 2007 [6] Kim, P., T.R. Eng, M.J. Deering, and A. Maxfield. Published criteria for evaluating health related web sites: review. Br Med J. 318: 647-649, 1999 [7] Eysenbach, G., J. Powell, and O. Kuss, O. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA. 20: 2691-2700, 2002 [8] Harland, J. and P. Bath, P. Assessing the quality of websites providing information on multiple sclerosis: evaluating tools and comparing sites. Health Informatics Journal. 13:207-221, 2007 [9] Bernstam, E.V., D.M. Shelton, M. Walji, and F. MericBernstam. Instruments to assess the quality of health information on the World Wide Web: what can our patients actually use? International Journal of Medical Informatics. 74: 13-19, 2005 [10] Ademiluyi, G., C.E. Rees, and C.E. Sheard. Evaluating the reliability and validity of three tools to assess the quality of health information on the Internet. Patient Education and Counseling. 50: 151-155, 2003 [11] Cullen, R., and C. Houghton, C. Democracy online: An assessment of New Zealand Government Web Sites. Government Information Quarterly. 17: 243-267, 2000 [12] Haak, van den, M.J., M. Jong, de, and P.J. Schellens. Evaluating municipal websites: A methodological comparison of three think-aloud variants. Government Information Quarterly. 26: 193-202, 2009 [13] Battleson, B., A. Booth, and J. Weintrop. Usability testing of an academic library web site: a case study. Journal of Academic Librarianship. 237: 188-189, 2001 [14] Campbell, N. Usability Assessment of Library-Related Web Sites: Methods and Case studies, LITA, Chicago, 2001 [15] Haak, M.J., van den, M. Jong, de, and P.J. Schellens. Retrospective versus concurrent think-aloud protocols: Testing the usability of an online library catalogue. Behaviour & Information Technology. 22: 339-351, 2003 [16] Sienot, M. Pretesting web sites: a comparison between the plus-minus method and the think-aloud method for the World Wide Web. Journal of Business and Technical Communication. 11: 469-482, 1997 [17] Benbunan-Fich, R. Using protocol analysis to evaluate the usability of a commercial Web site. Information and Management. 39: 151-163, 2001 [18] Kools, M., R.A.C Ruiter, M.W.J. Wiel, van de, and G. Kok. Testing the usability of access structures in a health education
brochure. British Journal of Health Psychology, 12: 525-541, 2007
online information from the National Cancer Institute. J Med Internet Res. 7: e25, 2005
[19] Nijland, N., J. Gemert-Pijnen, van, H. Boer, M.F. Steehouder, and E.R. Seydel, E.R. Evaluation of internet-based technology for supporting self-care: problems encountered by patients and caregivers when using self-care applications. J. Med Internet Research.10: e13, 2008
[32] Hansen, D.L., H.A. Derry, P.J. Resnick,, and C.R. Richardson. Adolescents Searching for Health Information on the Internet: An Observational Study. Journal of Medical Internet Research. 5: e25, 2003
[20] Johnston, S.K., H.Q. Nguyen, and S. Wolpin. Designing and Testing a Web-Based interface for Self-monitoring of Exercise and Symptoms for Older Adults with Chronic Obstructive Pulmonary Disease. Computers, Informatics, Nursing. 27: 166-174, 2009 [21] Peute, L.W.P., R. Spithoven, P.J.M. Bakker, and M.W.M Jaspers. Usability Studies on Interactive Health Information Systems; Where do we stand? (327-332). In: S.K. Andersen (Ed.). eHealth Beyond the Horizon – Get IT There, IOS Press, 2008 [22] Jaspers, M.W.M. A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. International Journal of Medical Informatics. 78:340-353, 2009 [23] Haak, M.J., van den. A penny for your thoughts – investigating the validity and reliability of think-aloud protocols for usability testing. 2008. [24] Goldberg, J.H., M.J. Stimson, M. Lewenstein, N. Scott, and A.M. Wichansky. Eye tracking in web search tasks: Design implications. Proceedings of the Eye Tracking Research & Applications Symposium 2002. ACM, New York, 2002. [25] Birru, M.S., V.M. Monaco, L. Charles, H. Drew, V. Njie, T. Bierria, et al. Internet usage by low-literacy adults seeking health information: an observational analysis. Journal of Medical Internet Research. 6: e25, 2004 [26] Crystal, A. and J. Greenberg. Relevance criteria identified by health information users during web searches. Journal of the American Society for Information Science and Technology. 57: 1368-1382, 2006 [27] Czaja, S.J., J. Sharit, and S.N. Nair. Usability of the Medicare Health Web Site, JAMA. 300:790-792, 2008 [28] Eysenbach, G., and C. & Köhler. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and indepth interviews. BMJ. 3247: 573–577, 2002 [29] Gilbert, L.K., J. Temby, and S. Rogers. Evaluating a teen STD prevention Web site. Journal of Adolescent Health. 37: 236–242, 2005 [30] Graham, L., T. Tse, and A. Keselman. Exploring user navigation during online health information seeking. AMIA Annual Symposium Proceedings. 299-303, 2006 [31] Grama, L,M., M. Beckwith, W. Bittinger, D. Blais, C. Lollar, A. Middleswarth, et al. The role of user input in shaping
[33] Hinchliffe, A., and W.K. Mummery. Applying usability testing techniques to improve a health promotion website. Health Promotion Journal of Australia. 19: 29-35, 2008 [34] Keselman, A., A.C. Browne, and D.R. Kaufman. Consumer Health Information Seeking as Hypothesis Testing. Journal of the American Medical Informatics Association. 15: 484-495, 2008 [35] Lewis, D., S. Gundwardena, and G. El Saadawi. Caring Connection: Developing an Internet Resource for Family Caregivers of Children With Cancer. CIN: Computers, Informatics, Nursing. 23: 265-274, 2005 [36] McCray A.T., E. Dorfman, A. Ripple, N.C. Ide, M, Jha, D.G. Katz, et al. Usability issues in developing a Web-based consumer health site. AMIA Annual Symposium Proceedings. 556-560, 2000 [37] McKemmish, S., R. Manaszewicz, F. Burstein, and J. Fisher, J. Consumer empowerment through metadata-based information quality reporting: The Breast Cancer Knowledge Online Portal. Journal of the American Society for Information Science and Technology. 60: 1792-1807, 2009 [38] Moore, M., R.G., Bias, K. Prentice, R. Fletcher, and T. Vaughn. Web usability testing with a Hispanic medically underserved population. J Med Libr Assoc. 97: 114-121, 2009 [39] Nahm, E.S., J. Preece, B. Resnick, and M.E. Mills. Usability of Health Web Sites for Older Adults. Computers, Informatics, Nursing. 22: 326-334, 2004 [40] Ostergren, M.J and B.T. Karras. ActiveOptions: leveraging existing knowledge and usability testing to develop a physical activity program website for older adults. AMIA Annual Symposium Proceedings. 578-582, 2007 [41] Priest, L., L. Nayak, and I. Stuart-Hamilton, I. Website task performance by older adults. Behaviour & Information Technology. 26: 285-295, 2007 [42] Sillence, E., P. Briggs, P.R. Harris, and L. Fishwick. How do patients evaluate and make use of online health information? Social Science & Medicine. 64: 1853–1862, 2007 [43] Sillence, E., P. Briggs, P.R. Harris, and L. Fishwick. A framework for understanding trust factors in web-based health advice. International Journal of Human-Computer Studies. 64: 697-713, 2006 [44] Williams, P., D. Nicholas, P. Huntington, and F. McLean, F. Surfing for health: user evaluation of a health information
website. Part two: fieldwork. Health Info Libr J. 19: 214-225, 2002 [45] Zeng, Q.T., S. Kogan, R.M. Plovnick, J. Crowell, E.M. Lacroix, and R.A. Greenes, R. A. Positive attitudes and failed queries: an exploration of the conundrums of consumer health information retrieval. Int J Med Inform. 73: 45-55, 2004
About the Authors Maaike van den Haak (Ph.D. 2008) is currently affiliated with the Language and Communication of the Faculty of Arts at VU University. Her research interests include the methodology of usability evaluation methods, with particular emphasis on think-aloud protocols, in the medical domain. Charlotte van Hooijdonk (Ph.D. 2008) is currently affiliated with the Language and Communication of the Faculty of Arts at VU University. Her research focuses on the design and evaluation of multimodal information presentation in the medical domain. She is a member of IEEE-PCS.
