Designing for Safety and Usability: User-Centered Techniques in Medical Device Design Practice Chris Vincent & Ann Blandford UCLIC, University College London, Gower Street London, WC1E 6BT, UK
[email protected],
[email protected] The design of systems affects the likelihood and nature of errors that people might make with them, and the ease of error recovery. If developers are to design systems that are less prone to errors propagating, they need to consider the users and user contexts. There are many techniques and resources available to support developers in this. In this paper we report on an interview study involving professionals from major manufacturers of medical devices, to better understand their development practices and the external forces that shape those practices. This identified barriers to user-centered design and corresponding opportunities for support. Results are divided into four themes. These are: collaborative working practices; understanding the user and their situation; providing adequate justification for the adoption of a user-centered approach; and the provision of clear guidance and support. Our findings highlight the importance of ensuring that techniques are adequately justified, applied at the correct time, aligned with the development lifecycle and easy to adopt. INTRODUCTION Within healthcare, there is a growing recognition that “human error” is not independent of the broader system within which individuals act and interact; United Kingdom Department of Health (DH, 2001). Under this view, errors are not attributed in entirety to the individual involved; rather, they are a consequence of a failure of one or several parts of the system. This approach is reflected in regulatory controls; for example, in the European Union (EU), the medical device industry is subject to a framework requiring those who market a device to be: “reducing, as far as possible, the risk of use error due to the ergonomic features of the device and the environment in which the device is intended to be used.” (EU, 1993) In the US, the Food and Drug Administration (FDA) may require developers to demonstrate how human factors considerations were applied during product development, and has recently announced an initiative to improve the safety and effectiveness of infusion pumps. This is in light of concerns regarding the user interaction such as: “confusing or unclear on-screen user instructions, which may lead to improper programming of medication doses or infusion rates.” (FDA, 2010b) More generally, there have been several reported examples of infusion devices where the design has compromised use. Examples include unintentional rebooting of the pump, key bounce or numeric entry errors resulting in over-infusion, and pumps that impose a programming sequence that does not match the hospital workflow; U.S. Association for the Advancement of Medical Instrumentation (AAMI/FDA, 2010; FDA, 2010a). The premise of our work is that interventions applied during the design process can reduce the likelihood of such interaction difficulties arising. The aim of the study reported here was to better understand how, in practice, developers of interactive medical devices such as infusion pumps build a user-centered
perspective into their development processes, how external forces shape those processes, and what tools and resources might be useful to them in future. Beyond the formal documentation such as the Medical Devices Directive, there are informal resources such as the Design for Patient Safety guide to electronic infusion devices, produced by the UK National Patient Safety Agency (NPSA, 2010) and human error analysis methods such as the Technique for Human Error Assessment (THEA: Pocock, Fields, & Harrison, 2001). Previous work has asked related questions regarding the design and delivery of home use devices (Gupta, 2007). In that study, developers suggested that Human Factors Engineering (HFE) helped identify issues and problems, but that there was a lack of science-based support. Designers reported that they placed minimal reliance on usability guidance and tended to rely upon experience and user trials. The study reported here takes a similar approach, but focuses on the design of devices that are typically used in hospital settings. METHOD In this work we used grounded theory to explore the issues regarding the development of infusion devices. We interviewed practitioners involved in development, training or marketing in order to build an understanding of current techniques and identify opportunities for support. Grounded theory is a method designed to support the building of theory through qualitative analysis of data. The practicalities and suitability of this method are described elsewhere (Furniss, Blandford, & Curzon, 2011). We chose the method as it suited the diverse and complex setting of the healthcare industry. Participants We interviewed a range of professionals who have an interest in the interactive properties of infusion devices. Table 2 describes the background of the participants. 10 participants
were chosen based upon their industrial experience or participation in UK National Health Service (NHS) safety initiatives. Where participants held senior positions, they maintained awareness of relevant tools and techniques. Procedure An approach email was sent to participants inviting them to become involved in research aiming to transform the design and use of interactive medical devices (www.chi-med.ac.uk). Contacts were established in March 2010 and interviews were conducted between April 2010 and October 2010. We used semi-structured interviews based upon a series of core questions (Table 1). Where possible, interviews were audio recorded and transcribed for analysis. In cases where this was not possible, extensive notes were taken. Nine interviews were face to face and two over the phone. One participant was interviewed over the phone and face to face in order to gain additional data (MDC-01-01 and MDC-01-02). Data were transcribed and loaded into ATLAS Ti (Scientific Software Development GmbH). Interviews were analyzed in sequence. We made the final report available to participants to verify that their views were accurately represented. Quotations used to illustrate themes are taken from transcripts, and consequently from three participants, but the analysis underpinning the results is from all 11 datasets. Analysis and Conceptual Development The first author conducted a process of open coding. As successive transcripts were analyzed, the population of codes Table 1 Interview Topics: User Centered Design (UCD) Approach Topic Description T1: Personal Practitioner role and responsibility, Background, internal and external relationships Organizational and dependencies. Structure T2: Fit in Landscape Known stakeholders. T3: Example Product Example product including interactive properties. T4: Awareness of Awareness, interpretation, utility and Standards and relevance of design guidelines and Support standards. T5: Interface Design Awareness, interpretation, utility and Methods relevance of UCD tools, details of development process. T6: Interface Design Mechanisms to prevent input error, Challenges interface design drivers / trade offs. Fit within development process. T7: Interface Design Application of user testing, Assessment evaluative techniques, verification and validation. Fit within development process. T8: Post Marketing Training, user documentation, Activities monitoring of device alerts and recalls, opportunities for support, constraints and dependencies.
Table 2 Description of Participants Ref Company Position Profile HCINA Director of 01-01 Research Lab, Usability Consultancy MDC- Global Patient Safety 01-01 healthcare Advocate provider MDC- See MDCSee MDC-01-01 01-02 01-01 MDC- Global Business 02-01 healthcare Development provider Manager MDC- Global Training and 03-01 healthcare Marketing provider Professionals MDC- Software Team Lead 04-01 consultancy MDC04-02 MDC05-01
Software consultancy
Software Development and Usability Human Factors Program Manager Business Manager
Recording Method Notes
Notes Transcript Notes Notes Notes Transcript
Global Notes healthcare provider MDC- Global Notes 06-01 healthcare provider MDC- Global Vice President Notes 02-02 healthcare Marketing provider MDC- Local Chief Executive Transcript 07-01 healthcare provider Note. MDC = Medical Device Company, HCI = HCI Consultant, MDC-XX-YY: XX = Medical Device Company serial, YY = Interview number for company.
Figure 1. Identified themes and meta-themes.
grew to 132. Codes were abstracted to determine themes and meta-themes. 6 themes were identified (Figure 1), namely: Collaborative Working Practices; Understanding the User and their Situation; Adequate Justification of User Centered Design; Clear Guidance and Support; Communication of Mandatory Controls and Industry Wide Standardization. The majority of codes related to a single theme, in a minority of cases a single code related to multiple themes. Themes were grouped under two meta-themes, namely Regulation and Design and Development (Figure 1). In this paper, we focus on the Design and Development themes. RESULTS Overall, participants were familiar with a range of HFE and UCD techniques and reported that they performed over and above the mandatory requirements. One company had recently revised the development process to include additional usability tests and documentation; in another case, the output of user tests and evaluation were stored on a database and shared across the company. Collaborative Working Practices Several participants reported examples of poor communication compromising device design. A usability practitioner spoke of isolated team members and geographical constraints leading to breakdowns in communication: “the designer is completely isolated, he is out in [location].” (MDC-04-02) Some manufacturers subcontracted to external design consultancies and found establishing common processes or toolsets could be difficult. When multiple organizations worked together, introduction of new processes required justification, and organizations had: “problems understanding why it should be different and why somebody from outside should come and apply new process.” (MDC-04-02) Usability practitioners highlighted potential conflicts relating to working alongside marketing professionals: “it is becoming better, but in the beginning I have a feeling that the marketing department which is the major contact point… usually they were the ones that defined what the pumps should look like.” (MDC-04-02) Other interviews provided positive examples regarding good communication speeding the development timeline: “they were very responsive to design change suggestions, and were able to come up with prototypes pretty quickly, because there wasn’t a huge structure between us and them…. …in a large corporation it’s probably multiple levels of approval before something gets back to the engineers and someone says right, okay, design it.” (MDC-07-01) In summary, good communication practices and the development of a common perspective were seen as essential. Difficulties included team members not being co-located or organizational structure or excessive bureaucracy slowing decision making. Suggested solutions included embedding usability practitioners in engineering teams, coordinating the
timing of development activities and collecting and confirming requirements across multiple stakeholders. Understanding the User and their Situation All participants were aware of the need to include user tests; however, there were reports of relatively uninvolved sessions, where participants were restricted to commenting on the aesthetic aspects of the device. Participants reported difficulties in assessing how many “users” should be involved in user tests and maintained that justifying the resource required was difficult. Participants emphasized difficulties in establishing the level to which the user had been trained: “This is real life, you know, there are people who are untrained, who were not there at the day of the training or who were hired only a couple of days after the training and then they receive training by a colleague who may not have understood the product fully.” (MDC-04-02) Participants also spoke of situations where users have been trained on a legacy product type and may experience difficulties in transferring to a new product: “You know, when new technology comes in they’re actually quite intimidated by it and it’s like… it’s new I’ve got to retrain and everything else.” (MDC-07-01) In summary, participants found it hard to understand how user tests could be designed to be indicative of situated use. Clarification of best practice and guidance regarding the range and type of users and testing techniques was required. Adequate Justification of User Centered Design Justification of resource challenged practitioners across a range of UCD and HFE techniques. Participants raised a need for methods to establish the cost and benefit of techniques: “the understanding of the customer when it comes to usability methods is why do we need to do that and why would it be useful, I have to explain it every time.” (MDC04-02) There were also concerns that those responsible for purchasing or evaluating devices may be too detached from the development team or too late in the development process to ensure UCD techniques are adopted. Participants reported significant penalties for not adopting a UCD approach, including potential delays to market, product recalls and litigation, as one participant put it: “what people don’t realize is that, lets say they can’t afford to, the argument should come back, you can't afford not to, because of litigation.” (MDC-01-02) Companies worked alongside notified bodies (providers of advice on regulatory compliance) to help determine the appropriate level of documentation and process. A close relationship and frequent dialogue with the notified body was seen as beneficial. Participants had difficulties in justifying guidance that was ambiguous, open ended or conflicting. Clear Guidance and Support Given concerns regarding the clarity of support, we asked developers about the utility of the various resources.
Participants stated they needed prohibitively large amounts of time to search, assimilate and implement guidance, and complained about the relevance and specificity of material: “I can only talk about the usability standard.... I think this one is not really well done… because it is too general, the process as described is so fuzzy. […] I look at MAUDE [refers to FDA Manufacturer and User Facility Device Experience database] but I can’t get useful information concerning the products I am working on.” (MDC-04-02) Participants suggested that any developer intending to use a systematic approach is faced with a bewildering array of documents containing complex interdependencies and lengthy annexes, and they reported guidance that was device specific, graphical and practical as preferential: “For instance such a nice guide like your NHS one here [refers to NPSA booklet (NPSA, 2010)] where you just show three case studies and also list the process they were doing along with the pitfalls, this would be much more beneficial than 8 pages of standard and 60 pages of annex.” (MDC-04-02) Participants also reported a lack of available guidance: “there is no guidance when it comes to usability, it is a bit like operating in an empty space.” (MDC-04-02)
communications between those with UCD and technical focuses (Blandford, Keith, Butterworth, Fields, & Furniss, 2007) – an important challenge in medical device design. Work is needed to develop and test a comprehensive and useful set of personas and scenarios to support development teams, and to explore other mediating representations that support communication across different cultures within teams. Understanding the User and their Situation Practitioners reported difficulties in anticipating the level of training or experience possessed by the user. It is unrealistic to expect the manufacturer to imagine every possible combination of device, user and usage scenario; however, a reasonable coverage of likely eventualities is necessary. Usability standard IEC 62366:2007 calls for developers to collect a user profile that considers age, gender, linguistic and cultural background along with the level of education, professional competence, potential disabilities and intended conditions for use. Consequently, there needs to be a concept of the user from the outset (Wiklund, 2006).Well-defined and complete user requirements are therefore beneficial, although challenges remain relating these to functional requirements. (Martin, Norris, Murphy, & Crowe, 2008).
DISCUSSION Adequate Justification of User Centered Design Having identified these four key themes, we now propose ways in which these challenges could be addressed through changes in development processes, and identify future research challenges in this area. Collaborative Working Practices Participants highlighted the challenges of poor communication and conflicting agendas within and across organizations involved in development. Elsewhere in the industry, Samaras and Horst (2005) report on the adoption of a common development framework such as systems engineering, which was found to improve the integration of ergonomic and engineering practice. Some of our participants took an agile approach to development, which poses particular challenges to integrating a human factors perspective as it is difficult to keep usercentered requirements gathering, user testing and product development synchronized (Kollmann, Sharp, & Blandford, 2009). Agile processes need to involve usability practitioners, with a good grasp of the technical aspects of development, within engineering teams to help developers to avoid falling into the trap of designing for themselves. However, to help achieve the rapid turn-around that is common in agile processes, there is a need for user representations such as personas (Cooper, 1999) and scenarios (Carroll, 1995) that represent the users and their needs. Within more traditional development processes, personas and scenarios have been applied to medical design across a number of contexts (Shah, Robinson, & AlShawi, 2009; Sutcliffe et al., 2010; Wiklund, 2006). A particular value of personas and scenarios is that they have been found to be effective mediating representations that support
Participants reported that securing the necessary resource for UCD was difficult and had to be constantly justified. There are methods available to support; however, they often need to be adapted for a given domain and set of circumstances. For example, cost savings achieved through more usable devices leading to reductions in training needs or staff numbers are specific to a domain of application and context of use. Karat, 1997 states that for every dollar of investment spent on ease of use, there is a two to one hundred dollar return on investment. Ensuring UCD techniques are applied at the correct point in time and matched to the domain of application has the potential to maximize the benefit. Recording and managing the resource assigned in this area allows organizations to compare and contrast across cases. As well as cost-justifying UCD, it is important to recognize that there are several overarching principles that allow organizations to realize the maximum benefit from UCD or HFE. These include problem prioritization and severity assessment (methods to help assign development resource) (Hertzum, 2006), ensuring usability principles are applied proactively to architectural design (Bass & John, 2003), choosing appropriate methods (Jaspers, 2009) and tailoring the methods to the phase of design (Kamper, 2002). Clear Guidance and Support Developers were often unaware of guidance or felt it to be inappropriate or inadequate. There are opportunities for improvement, including identifying and addressing gaps in provision, signposting appropriate support, and considering suitable ways to present and distribute information.
Developers stated a preference for concise, graphically illustrated and design orientated material such as the guidelines produced as part of a multidisciplinary “Design for Patient Safety” initiative (NPSA, 2010). The rationale behind device developers, ergonomists and design professionals producing guidance together is that content will stipulate necessary design requirements and be communicated in a way that is accessible to a wide audience (Buckle, Clarkson, Coleman, Ward, & Anderson, 2006). Future work could suggest practical, easily adopted resources that allow developers to predict error and integrate appropriate mitigations. Approaches could include provision of scenarios of use (based upon observational study in hospitals and patients homes), checklists of design features, interactive tools to analyze proposed systems and examples and novel designs that have been proven to reduce error. Conclusion Our participants, who represent most of the major manufacturers of medical devices, were all aware of the need for UCD or HFE techniques; however, they had difficulties in implementation. Through this study, we have identified key areas where developers need tools, techniques and evidence to inform the requirements for new designs and to evaluate the degree to which candidate design options are likely to lead to safer systems that are less susceptible to human error. ACKNOWLEDGMENTS This work forms part of the UK EPSRC funded CHI+MED project (EP/G059063). We would like to thank Chris Quinn and the reviewers for their input. The study obtained approval from the UCL research ethics committee. REFERENCES U.S. Association for the Advancement of Medical Instrumentation / Food and Drug Administration. (2010). Infusing patients safely: Priority issues from the AAMI/FDA infusion device summit. Retrieved from http://tinyurl.com/4ru69sg Bass, L., & John, B. E. (2003). Linking usability to software architecture patterns through general scenarios. Journal of Systems and Software, 66(3), 187-197. Blandford, A., Keith, S., Butterworth, R., Fields, B., & Furniss, D. (2007). Disrupting digital library development with scenario informed design. Interacting with Computers, 19(1), 70-82. Buckle, P., Clarkson, P. J., Coleman, R., Ward, J., & Anderson, J. (2006). Patient safety, systems design and ergonomics. Applied Ergonomics, 37(4), 491-500. Carroll, J. M. (1995). Scenario-based design : envisioning work and technology in system development. NY: Wiley. Cooper, A. (1999). The inmates are running the asylum. Indianapolis: Sams: Prentice Hall. UK Department of Health, (2001). Building a safer NHS for patients: Implementing an organisation with a memory: Great Britain, Department of Health.
European Union, (1993). Council directive 93/42/EEC concerning medical devices. Retreived from http://tinyurl.com/688f4o7 U.S. Food and Drug Administration, (2010a). Infusion pump workshop: Transcript for May 25th 2010. Retrieved from http://tinyurl.com/4mrleam U.S. Food and Drug Administration, (2010b). White paper: Infusion pump improvement initiative. Retrieved from http://tinyurl.com/2u6q6oy Furniss, D., Blandford, A., & Curzon, P. (2011). Confessions from a grounded theory PhD: Experiences and lessons learnt. Paper presented at the ACM SIGCHI conference, Vancouver. Gupta, S. P. (2007). Design and delivery of medical devices for home-use: Drivers and challenges. PhD, The University of Cambridge, Cambridge. Hertzum, M. (2006). Problem prioritization in usability evaluation: From severity assessments toward impact on design. International Journal of Human-Computer Interaction, 21(2), 125-146. Jaspers, M. W. M. (2009). A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. International Journal of Medical Informatics, 78(5), 340-353. Kamper, R. J. (2002). Extending the usability of heuristics for design and evaluation: Lead, follow get out of the way. International Journal of Human-Computer Interaction, 14(3-4), 447-462. Karat, C. M. (1997). Cost-justifying usability engineering in the software life cycle. In T. K. Helander, Landauer & P. Prabhu (Eds.), Handbook of Human-Computer Interaction (pp. 689-704). New York: Elsevier Press. Kollmann, J., Sharp, H., & Blandford, A. (2009). The importance of identity and vision to user experience designers on agile projects. Paper presented at the Agile Conference, Chicago. Martin, J. L., Norris, B. J., Murphy, E., & Crowe, J. A. (2008). Medical device development: the challenge for ergonomics. Applied Ergonomics, 39(3), 271-283. UK National Pateint Safety Agency, (2010). Design for patient safety: A guide to the design of electronic infusion devices. London: National Patient Safety Agency. Pocock, S., Fields, B., & Harrison, M. (2001). THEA: A reference guide: University of York, Department of Computer Science. Shah, S. G. S., Robinson, I., & AlShawi, S. (2009). Developing medical device technologies from users' perspectives: A theoretical framework for involving users in the development process. International Journal of Technology Assessment in Health Care, 25(4), 514-521. Sutcliffe, A., Thew, S., de Bruijn, O., Buchan, I., Jarvis, P., McNaught, J., & Procter, R. (2010). User engagement by user-centred design in e-Health. Philosophical Transactions of the Royal Society a-Mathematical Physical and Engineering Sciences, 368(1926), 4209-4224. Wiklund, M. (2006). Defining and designing for worst-case users. Medical Device and Diagnostic Industry, 28(7), 5259.
