Usability Assessment of Pacemaker Programmers CHRISTINE C. CHIU, KIM J. VICENTE,* ILAN BUFFO-SEQUEIRA, ROBERT M. HAMILTON, and BRIAN W. MCCRINDLE From the Division of Cardiology, The Hospital for Sick Children, and the *Cognitive Engineering Laboratory, Department of Mechanical and Industrial Engineering, University of Toronto, Toronto, Ontario, Canada
CHIU, C.C., ET AL.: Usability Assessment of Pacemaker Programmers. There is a perception among clinicians of usability differences in the user interface of pacemaker programmers, but there is an absence of literature in this area. The purpose of this study was to describe usability differences in pacemaker programmers. Forty-two programmer users completed self-administered questionnaires and two usability experts independently performed heuristic evaluation to identify features that violated general usability principles. Programmers from seven manufacturers (coded A–G) were evaluated. There was a balanced representation of users: nurses (58%) versus technologists (40%) who are employed in community (50%) versus academic (45%) hospitals, novice versus expert users based on the median users’ programming experience of 60 months (range 1–300 months). Significant differences between programmers were found in overall user satisfaction and ease of programmer use (P < 0.0001) in the display, controls, operation, and physical dimension of the programmers (P < 0.05). Heuristic evaluations showed frequent violations of usability principles in all programmers. Problematic areas include reliance on user recall, inconsistency in operation of critical controls, poor readability, and not anticipating user wants or action. Programmer interface designs do not consistently meet user needs or general usability principles. This impacts on the safe and effective use of programmers. Guidelines in programmer design should be established, particularly with respect to labeling, location, and operation of critical controls. (PACE 2004; 27:1388–1398) pacemaker, implantable defibrillator, programmer, usability, user interface, heuristic evaluation, medical device design Introduction Implantable pacemakers and defibrillators are used by clinicians for management of patients with heart rhythm disorders.1−3 The follow-up assessment of these implanted devices are conducted through special laptop computers called “pacemaker programmers.” A pacemaker programmer is made up of computer hardware called the platform and a software interface from which users operate the controls and perform their follow-up tasks. Broadly speaking, one may conceptualize the programmer user interface as having input, output, and processing components within the human operator element and the computer element (Fig. 1) The relationship between the design of any user interface and ease of use for the human operator is critical to the overall usability. Usability is generally defined as the effectiveness, efficiency, and user satisfaction in the design of a product for a specific use.4,5 Poor user interface designs can
Supported in part by the Cardiology Advisory Council at the Hospital for Sick Children, Toronto, Canada. Address for reprints: Christine Chiu, BSc, Div. of Cardiology, The Hospital for Sick Children, 555 University Ave., Toronto, Ontario, Canada M5G-1X8. Fax: 416-813-7547; e-mail:
[email protected] Received February 3, 2004; revised June 9, 2004; accepted July 8, 2004.
1388
lead to errors in operation, inefficiencies, and user frustrations.6 Usability testing plays an important role in unmasking design flaws leading to improvements to meet user needs.7 There are no guidelines in the user interface design of pacemaker programmers. General human to computer interface guidelines are probably adopted to varying degrees in most designs.6,8 Personal communication with engineers from pacemaker manufacturers has shown varying involvement of human factors specialists in the design of programmers. These issues may be responsible for the perception amongst clinicians of usability differences between programmers from different manufacturers, although there is an absence of literature in this regard. The purpose of this study was to describe the usability differences and identify problematic areas in the user interface of pacemaker programmers. Methods Data were collected from users and usability experts to ensure accuracy and confidence in interpreting the findings of this study.9 Programmer users were surveyed by anonymous completion of a self-administered questionnaire. Two undergraduate engineering students who specialized in human factors and usability research on medical equipment independently performed heuristic evaluation of the programmers. This study was
October 2004
PACE, Vol. 27
USABILITY ASSESSMENT OF PACEMAKER PROGRAMMERS
approved by the Hospital for Sick Children Research Ethics Board. Pacemaker Programmers Manufacturers of pacemaker programmers were coded by letters A–G . The platforms assessed in this study are those used in device clinics as of November 2002. Participants Volunteers were recruited from the allied professional participants of a regional conference on electrophysiology and device management in November 2002. It was estimated that about 50 of 90 participants were involved in pacemaker follow-up from Canadian provinces of Ontario, Quebec, and Nova Scotia. Observers with no hands-on programming experience or employees of pacemaker manufacturers were excluded. Additional volunteers were recruited by telephoning ten pacemaker clinic sites whose staff did not attend the November 2002 conference. User Questionnaire The questionnaire is comprised of three sections: (1) cover sheet explaining the study, (2) user information data form, and (3) user opinion data form to complete for each programmer type used in their practice. On the user opinion form, there were 20 Likert scale items with 4–6 discrete response options (e.g., ranked from 1–6, very easy to very difficult) and open-ended items to allow respondents to write comments in their own words: what they like the most and dislike the most about the programmer, what they would like to change about the programmer, and to write down any problems they have encountered. Heuristic Evaluation This is a methodology in human factors engineering whereby clinical or human factors expert(s) evaluates user interface system to determine how well it complies with established usability principles like the 15-point checklist8,10 (Appendix 1). This technique is useful in identifying problematic areas and allows comparison of interface designs. Two usability experts independently worked with the principle investigator to evaluate six pacemaker programmers by a walk-through of a typical operator task list:11 • Communication or interrogation of the implanted device with the programmer • Obtain parameter settings, lead impedance, battery data, histogram counter data PACE, Vol. 27
• Obtain other diagnostic data like trends, arrhythmia events, etc. • Perform stimulation and sensing threshold tests • Print a copy of the electrogram (EGM) /electrocardium (ECG) • Identify lead or battery failure if present • Optimize any parameter settings by reprogramming • Obtain final interrogation and printout of parameters The usability experts commented on violations of the user interface to established heuristic principles and made suggestions for improvements. Usability experts did not evaluate programmer E since there were only six user evaluations received and their devices were no longer implanted. Data Analysis Questionnaire responses were coded using the ranking scheme established for the response options of the Likert scale items on the data forms (e.g., from 1, very easy, to 6, very difficult). Missing data was not imputed but reported as missing for that category. Programmer performances were reported by comparison of their features rather than comparison of manufacturers. In view of the small number of evaluations (< 10) received for programmers E–G, these were not included in the quantitative analysis but their differences were described. Descriptive statistics (median and range) were used to summarize the characteristics of the users and the questionnaire items. The Kruskal-Wallis test, a nonparametric test, was used to test for differences in the rank of median responses for three or more groups, in this case, between programmers A–D. A P value < 0.05 was defined as statistically significant. The mean responses were ranked from the questionnaire items to discriminate the differences between these four programmers. Textual comments/keywords from users and usability experts were sorted into the components of the user interface model depicted in Figure 1. From this, the authors identified and summarized major patterns in which differences exist between the user interface of programmers. Results Forty-two users (38 recruited from a regional conference, 4 from telephone solicitation) completed the questionnaires and provided assessment on seven different pacemaker programmers. There was a balanced representation of users in terms of user education, place of employment, and programming experience (Table I). The profile was skewed in only two areas: (1) Users were potentially frequent users of pacemaker
October 2004
1389
CHIU, ET AL.
Figure 1. Conceptual model of the clinician-programmer elements as adapted from a model of humancomputer system.4 This model helps us to conceptualize the relationship that exists between the clinician operator and the programmer. Designs in user interface of the programmer must adopt a systems approach. It must not only address the programmer aspect but also the limitations posed by the clinician aspect (human factor) and the work environment in which the interaction takes place. There is an analogous relationship between the programmer and the clinician in terms of input (controls, perception), processing (operation, cognition), and output (display, action).
programmers in terms of clinic frequency (≥1 clinic per week) and programming volume (conduct programming >20 times per month). (2) Users have other clinical responsibilities that are nondevice related, like heart failure clinic, vascular clinic, catheter laboratory intensive care unit or chemotherapy nursing, noninvasive ECG testing (stress, Holter, ECG, tilt test), research, billing, administration, and triage. The physical attributes
of these seven programmers are listed in Table II along with the number of users who provided assessment. Significant differences between programmers were found in overall user satisfaction, ease of programmer use, and in the various components of the user interface model (Table III). Some of the user comments were listed per verbatim in Appendix 2 as they provided some insight into the user’s
Table I. Profile of Users who Completed the Questionnaire Characteristic
N
Category
n
%
Education
40
Workplace
38
Responsibilities
36
Device clinic volume
39
Programming volume
39
Nurse Tech Fellow Community hospital Academic hospital Private clinic Performs other duties Dedicated to devices Weekly Biweekly Monthly or less >21/month 11–20/month
