Exploring the Usability of Mobile Apps Supporting

ORIGINAL ARTICLE

LEADERSHIP

Exploring the Usability of Mobile Apps Supporting Radiologists’ Training in Diagnostic Decision Making Min Soon Kim, PhD a,b,c, Michael R. Aro, MD d, Kraig J. Lage, MD d, Kevin L. Ingalls, MD d, Vivek Sindhwani, MD d, Mia K. Markey, PhD e,f Abstract Purpose: The objective of this study was to conduct a usability evaluation of mobile apps for supporting education and training in radiologic diagnostic decision-making processes. Methods: Of 381 mobile apps available at two major stores (Google Play and iTunes), eight iOS apps were selected for laboratory-based usability tests. Six staff radiologists completed eight app-specific task sets, using a think-aloud strategy. The triangular methods approach included quantitative performance measures, System Usability Scale (SUS), and qualitative thematic analysis using heuristic usability principles of usability issues. Results: Overall, radiologists achieved higher than 70% success, with favorable SUS scores, in completing the tasks for seven of the eight apps. However, task success rate and SUS score had a weak relation (r ¼ 0.23), indicating that the perceived usability may not reflect the holistic usability of the app. Task analysis and self-report revealed 108 usability issues, which were condensed to 55 unique issues and categorized by nine usability themes and mapped to ten usability heuristics. Nonintuitive functionality (eg, nonintuitive or misleading labels) was the most frequent theme observed, leading to inefficient navigation. These usability findings were consistent with the 13 improvements the radiologists suggested. Conclusions: This study demonstrates the feasibility of usability evaluation of radiology mobile apps and suggests potential improvements in the development of radiology mobile apps. This study also suggests that proficiency with mobile devices may not be equivalent to being an expert user, proficient in using the apps. Key Words: Usability, mobile app, decision making, education J Am Coll Radiol 2016;13:335-343. Copyright ! 2016 American College of Radiology

INTRODUCTION Mobile Apps in Radiology Education and Training More than 85% of physicians in the United States use smart phones and 53% use tablets daily in their practice areas [1]. There are four major app stores (iTunes, Google Play, Windows, and BlackBerry), but the majority of apps are offered through the iTunes and Google Play stores. In February 2015, the iTunes App Store contained a

Department of Health Management and Informatics, University of Missouri School of Medicine, Columbia, Missouri. b University of Missouri Informatics Institute, Columbia, Missouri. c Department of Emergency Medicine, Icahn School of Medicine at Mount Sinai, New York, New York. d Department of Radiology, University of Missouri, Columbia, Missouri. e Department of Biomedical Engineering, University of Texas, Austin, Texas.

ª 2016 American College of Radiology 1546-1440/15/$36.00 n http://dx.doi.org/10.1016/j.jacr.2015.07.021

approximately 32,000 medical mobile apps, whereas Google Play’s app store had about 23,000 medical apps [2,3]. Medical apps fall under many different categories, including reference apps, such as the Physician’s Desk Reference app, medical calculators, and apps designed to access electronic health records or personal health information [4]. There are numerous potential and realized benefits of mobile device use in health care settings. In radiology, f

Department of Imaging Physics, The University of Texas MD Anderson Cancer Center, Houston, Texas. Corresponding author and reprints: Min Soon Kim, PhD, University of Missouri School of Medicine, Department of Health Management and Informatics, CE728 Clinical Support & Education, DC006.00, 5 Hospital Drive, Columbia, MO 65212; e-mail: [email protected]. The authors have no conflicts of interest related to the material discussed in this article.

335

research studies have shown that mobile apps may have positive effects by providing facilities for diagnostic reading [5], decision support [6], medical books [7], interactive encyclopedias [8], and journal reading [9].

Lack of Usability in Radiology Apps Recent increases in the functionality of apps have come at the expense of their “usability.” Usability is defined as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction, in a specified context of use of the system” [10]. Although functional evaluations of mobile apps receive much attention, few usability evaluation studies have been conducted, especially for health care mobile apps [11]. Although many health care-related mobile apps are available for consumers, only 28% of smart phone users and 18% of tablet users report being “very satisfied” with the quality of these apps [1]. In other words, most apps are hard to learn and inefficient and unsatisfying to use. As such, it is estimated that 95% of downloaded mobile apps are abandoned within a month [12], and 26% of apps are used only once, possibly because of the lack of attention to usability [13]. Poor health care system design may lead to increased cognitive load [14], medical errors [15], and decreased quality of patient care [16]. The objective of this study was to evaluate the usability issues radiologists encounter when they use mobile apps that are designed to support training in diagnostic decision making. METHOD Study Setting University of Missouri Health Care is a tertiary care academic medical center. The Department of Radiology includes more than 28 highly trained clinicians and researchers and successful training programs of more than 25 resident physicians. This study incorporated the strengths of an interdisciplinary team of experts in health informatics (M.S.K. and M.K.M.) and clinical radiology (M.R.A., K.J.L., K.L.I., and V.S.). This study was approved by the institutional review board. Selection of Radiology Apps Two online stores with major market share (Google Play [Google, Mountain View, California] and iTunes [Apple, Cupertino, California]) were searched on July 10, 2014, by screening apps’ titles and descriptions. Apps were excluded if they (1) only provided access to reference material, (2) were designed solely for trivial medical calculations, (3) were designed for specific commercial vendor products, 336

(4) were designed for use by a specific hospital or clinic only, or (5) were written in a language other than English. Of 381 apps identified as eligible, 21 were selected for potential usability evaluation. Finally, the team selected 8 iOS (Apple) apps that are available for the iPad device (Apple) for this study (Table 1). The iOS platform and iPad device were chosen because of wide acceptance in the radiology community in the United States [17,18]. Links to the apps studied are provided in Table 1.

Usability Evaluation Participants and Sampling Frame. Estimates of the sample size for a usability test are based on how many of a fixed number of usability issues can be uncovered by users (problem discovery rate) [19]. On the basis of a review of Table 1. Names of the apps for usability test, seller, price, and store links 1. Case Review for iPhone and iPad (Elsevier, iOS free) https://itunes.apple.com/us/app/case-reviews-for-iphone-ipad/ id522176594?mt¼8 2. AART Ultrasound Cards Lite (Virtual Flashcards, iOS, Android free) https://play.google.com/store/apps/details?id¼com.virtualflash cards.aartsonolite https://itunes.apple.com/us/app/arrt-sonography-ultrasound/ id749310888?mt¼8 3. Dexnote (DexNote, iOS free) https://itunes.apple.com/us/app/dexnote/id536419130 4. Brain MRI Sectional Walker (Ryo Matsuda, iOS $9.99) https://itunes.apple.com/us/app/brain-mri-sectional-walker/ id755407893?mt¼8 5. Diagnostic Radiology - Dynamic Approach to Abdominal Radiology (BestApps, $44.99) https://itunes.apple.com/us/app/diagnostic-radiology-dynamic/ id523516157?mt¼8 6. MED Imaging Case (Olivier Decrock, iOS $1.99, Android free) https://play.google.com/store/apps/details?id¼com.medimaging case&hl¼en https://itunes.apple.com/us/app/medimaging-case/id531101 017?mt¼8 7. Radiology Assistant (Wouter Veldhuis, iOS $5.99, Android $7.32) https://itunes.apple.com/us/app/radiology-assistant-medical/ id383464901?mt¼8 https://play.google.com/store/apps/details?id¼nl.radiology assistant.android&hl¼en 8. Radiology Head (3D4Medical.com, iOS $2.99) https://itunes.apple.com/us/app/radiology-head/id692751041? mt¼8

Journal of the American College of Radiology Volume 13 n Number 3 n March 2016

the literature, 5 to 10 representative users can uncover 82% to 95% (95% confidence interval) of usability issues [20]. A convenience sampling method was used to select radiologists for the study. Physician demographics including age, gender, race, and history of previous experience with the radiology apps used in this study were gathered in a paperbased questionnaire and aggregated for analysis. Tasks. Eight different app-specific task sets were created. The tasks were designed to evaluate as many features and functionalities as possible. Each of the tasks had a clear objective and avoided excessive cognitive challenges or ambiguity [21,22]. For example, the tasks used for Brain MRI Sectional Walker were as follows: Task 1: Open the app Task 2: Unlock the app Task 3: Review MRI section 16 Task 4: Zoom in and out the left-side cerebellum and the temporal lobe of section 16 Task 5: Indicate the left optic nerve of the T1-weighted image of section 16 Task 6: Turn on selectable areas of section 16 Task 7: Review the MRI sections from 16 to 1 Task 8: Lock the app Task 9: Close the app

Data Collection Usability data were collected between November 12, 2014, and December 19, 2014. Six staff radiologists performed the tasks using a think aloud strategy. Each session lasted approximately 30 min. The sessions were recorded by connecting the iPad to a MacBook laptop (Apple) using QuickTime software (Apple) on the MacBook. Above the iPad device, a separate mobile observation device was connected to a PC laptop to record screen activities and hand gestures. This twofold recording allowed comprehensive capture of the human computer interaction. After physicians completed the tasks, they were asked to complete the demographic survey and System Usability Scale (SUS), a 10-item Likert-type scale that is a subjective assessment of a system [23]. We concluded each session with a debriefing session during which physicians were asked to comment on the overall ease of use, specific tasks they found difficult, and suggested improvements. Interesting observations detected by the facilitator were discussed as well. Data Analysis We confirmed that there were no app interface changes during the data collection period that may have

influenced the study and tasks. Morae Manager (TechSmith, Okemos, Michigan) was used for video analysis to code where the physicians had difficulties and errors. Video analysis took approximately 1.5 hours for each 20min recorded session, excluding the debriefing session. Performance Measures. Usability was evaluated using three quantitative performance measures: mean time to complete the tasks (seconds), task success rate (percentage), and number of issues identified during the test [24]. Task success rate was the percentage of subtasks that physicians successfully completed without significant errors and was the most objective performance measure. Time on task calculated how long it took each physician to complete each task. A geometric mean was calculated for the performance measures, with a confidence interval (CI) at 95%. Performance measures have a strong tendency to be positively skewed, so a geometric mean was used because it provides the most accurate measure for sample sizes less than 25 [25]. Usability Issues by Themes and Heuristics. Two steps of qualitative usability analyses were conducted. First, the physicians were asked a set of semistructured questions after each group of tasks to obtain their immediate interpretations of given tasks and system design and to facilitate the elaboration of usability issues they reported from the user’s perspective. Second, through video analysis, granular usability issues were coded using themes that were derived by thematic analysis and mapped to usability heuristics suggested by the mHIMSS App Usability Work Group for mobile devices to note any violations of usability principles [26]. The usability heuristics mHIMSS proposed were (1) simplicity, (2) naturalness, (3) consistency, (4) forgiveness and feedback, (5) effective use of language, (6) efficient interaction, (7) effective information presentation, (8) preservation of context, and (9) minimization of cognitive overload.

RESULTS Participant Characteristics Five faculty radiologists and one fellow in clinical radiology volunteered for the study and were deemed to have equivalent clinical roles and duties. All six physicians were men. Four physicians self-identified their race as white, one self-identified as Asian/Pacific Islander, and one selfidentified as black/African American. The physicians’ ages ranged from 31 to 47 years, and the mean age was 38 years. Three physicians reported brief prior experience with four of the apps on a web platform years previously,

Journal of the American College of Radiology Leadership n Kim et al n Mobile Apps Usability in Diagnostic Decision Making

337

and none had experience using the mobile apps used in the study; therefore, we did not attempt to control for app experience. Because of the small sample size, we did not attempt to control for age.

Quantitative Results Using the video analysis, the task completion times, percentage task success rates, and number of usability issues of the eight apps were computed (Table 2). One physician failed all the tasks from apps 2 through 8, and his data were not included when computing the task completion times but were used when computing the task success rates and number of issues. Because different sets of tasks were used to evaluate the tasks, direct comparison of the performance measures are not meaningful, but they are reported to show approximate quantitative measures taken for a physician user to complete a full course of app use. Physicians spent the least amount of time with app 3 (80 seconds; 95% CI, 61-105 seconds) and the most amount of time with app 2 (193 seconds; 95% CI, 142-264 seconds). Overall, physicians achieved higher than a 70% success rate in completing the tasks for all the apps except app 8. Physicians completed the tasks with the highest success rate for app 1 (100%) and the lowest success rate for app 8 (50%). During a debriefing session, physicians were asked to comment on usability issues they experienced when they completed the tasks. The physicians reported 28 usability issues. In addition to these self-reported usability issues, posttest video analysis revealed 80 usability issues. The frequency of usability issues observed showed that physicians encountered most usability issues with apps 2 and 4 (19 issues) and the smallest number of usability issues with app 6 (8 issues). Note that the numbers indicate the raw counts of issues; more granular analysis (eg, types and severity) of usability issues is described in the next section. Table 2. Quantitative results: mean time to complete the tasks, percentage task success rate, and number of issues identified during the test Task Completion Time Task Success App (Tasks) (sec) (95% CI) Rate (%) Issues App 1 (7) 106 (78e144) 100 11 App 2 (6) 193 (142e264) 70 19 App 3 (4) 80 (61e105) 76 11 App 4 (8) 109 (76e157) 79 19 App 5 (7) 92 (72e118) 87 10 App 6 (5) 109 (82e145) 83 8 App 7 (7) 92 (68e124) 73 15 App 8 (6) 165 (118e231) 58 15 Note: Six physicians tested the eight apps (listed in Table 1).

338

All six physicians completed the SUS for the eight apps (Table 3). The mean SUS score (range, 0-100) among the physicians ranged from 41 (not acceptable) for app 2 to 76 (acceptable) for app 5 [26]. Physicians rated five apps as being acceptable (apps 1, 3, 5, 6, and 7), one app as being high marginal (app 8), one app as being low marginal (app 4), and one app as being not acceptable (app 2). Although most of the apps were rated favorably, the standard deviations suggest that there was substantial variation among physicians. This result indicates that physicians had a variable experience in using the apps. The Pearson correlation coefficient between individual task success rate, the most objective performance measure, and physicians’ SUS score suggests that physicians’ task success had weak positive relation to how user friendly physicians perceived the app to be (r ¼ 0.23; 95% CI, "0.07 to 0.49; P ¼ .12). However, caution is warranted given the statistical insignificance and low power of the study (45%).

Usability Issues by Usability Heuristics All of the usability issues reported by the physicians and identified by video analysis were coded by nine themes that were derived by thematic analysis and were mapped to usability heuristics suggested by mHIMSS (Table 4). The usability themes used were based on the characteristics of the usability issues. Multiple similar issues were grouped as single unique issue type. This process reduced the selfreported 28 usability issues and 80 issues from posttest task analysis to 55 issues. The nine usability themes revealed according to the frequency of unique issues were unintuitive functionality (23), unclear navigation pathway (6), unintuitive information presentation (6), illogical location of information (5), inconsistent information display (4), lack of information (4), lack of functionality (3), waste of information resources (2), and unclear status (2). The unique issues were mapped to related usability Table 3. Overall SUS scores for each apps evaluated (n ¼ 6) App App 1 App 2 App 3 App 4 App 5 App 6 App 7 App 8

SUS Score, Mean # SD 75 # 19 41 # 23 70 # 4 60 # 25 76 # 17 72 # 14 75 # 17 66 # 28

Implication Acceptable Not acceptable Acceptable Low marginal Acceptable Acceptable Acceptable High marginal

Note: A score of 0 to 50 was not acceptable, a score of 50 to 62 was low marginal, a score of 63 to 70 was high marginal, and a score of 70 to 100 was acceptable. SUS ¼ System Usability Scale.

Journal of the American College of Radiology Volume 13 n Number 3 n March 2016

Table 4. Usability issues identified by the task analysis and by the self-report Quotations From Self-Reported Usability App Issues Representative Usability Issues Identified by Task Analysis App 1 Unintuitive functionality: “finding the case note Unintuitive functionality: (eg, users spent time testing inactive buttons and texts that look active) [UH: naturalness; efficient interaction; page was hard because there was no minimize cognitive overload] indication of whether there were more images Inconsistent information display (eg, users were forced to switch the in a stack (eg, CT scan)” [UH: efficient orientation to complete the task without giving an option to choose interaction; effective use of language; the orientation) [UH: naturalness; efficient interaction; consistency] minimize cognitive overload] Unintuitive functionality: “I was not very familiar Unclear navigation pathway (eg, users had to move back to home page or higher level to find questions, submit the answer) [UH: with the different icons. Icons need to be naturalness; efficient interaction; consistency; preservation of labeled like diagnostics, questions, notes etc.” context] [UH: naturalness; minimize cognitive overload] Illogical location of information (eg, users were unable to access the instruction manual/help because it was located at the bottom of “About” tab, which typically contains only information on app development information such as version or release) [UH: naturalness; efficient interaction; consistency] Unintuitive information presentation (eg, instructions were not grouped or listed by themes or priority) [UH: naturalness; minimize cognitive overload] App 2 Unintuitive functionality: “not intuitive; difficult Unintuitive functionality (eg, swipe up/down for grading was not intuitive due to the lack of indication if it was marked; took time to knowing what to do/where to move to locate the statics) [UH: efficient interaction; minimize cognitive perform tasks” [UH: naturalness; efficient overload; naturalness] interaction; minimize cognitive overload] Unclear status (eg, when card was played, it was hard to know whether Unintuitive functionality: “Not intuitive so it it was actually played) [UH: naturalness; forgiveness and feedback; would take start up time” [UH: naturalness; minimize cognitive overload] minimize cognitive overload] Unclear navigation pathway (eg, users had trouble figuring out what Unclear status: “not entirely clear if I actually functionality to use to move between layers within the app) [UH: completed the task” [UH: naturalness; naturalness; efficient interaction] forgiveness and feedback; minimize cognitive Illogical location of information (eg, users were unable to locate the overload] instruction manual/help because it was located a level below that it Illogical location of information: “no instructions as to see the answers or how to go forward” did not appear in the main page of the app) [UH: naturalness; [UH: naturalness; minimize cognitive overload] efficient interaction; consistency; preservation of context] Unintuitive information presentation (eg, instruction in the help section was written in inconsistent typeface, size that to search for the answer to the question was time demanding) [UH: naturalness; efficient interaction; consistency; minimize cognitive overload] App 3 Unintuitive functionality: “took me several tries Unintuitive functionality (eg, users had to click and hold on the link to open the media but most users just tapped the link several times and to open links to the article” [UH: naturalness; gave up) [UH: naturalness; efficient interaction; minimize cognitive efficient interaction; minimize cognitive overload] overload] Lack of functionality (eg, settings section doesn’t have any option to Unintuitive functionality: “took time change) [UH: effective information presentation; efficient interaction] understanding that the text menu items are also buttons” [UH: naturalness; minimize cognitive overload] Unintuitive information presentation: “The icons were little small” [UH: efficient interaction] Lack of information: “can’t see any images” [UH: efficient interaction] App 4 Unintuitive functionality: “All the buttons Unintuitive functionality (eg, “lock” button function was hard to have it on the bottom and top don’t work” worked) [UH: naturalness; efficient interaction; minimized cognitive (eg, “unlock/lock doesn’t work”) [UH: overload] (continued)

Journal of the American College of Radiology Leadership n Kim et al n Mobile Apps Usability in Diagnostic Decision Making

339

Table 4. Continued Quotations From Self-Reported Usability Issues efficient interaction; minimize cognitive overload] Unintuitive functionality: “scroll images were cumbersome” [UH: efficient interaction; minimize cognitive overload] Unintuitive functionality: “very cumbersome app. Hard to follow/test” [UH: efficient interaction; minimize cognitive overload] App 5 Unintuitive functionality: “No idea of how to select an image (eg, single or double tap)” [UH: naturalness; efficient interaction; minimize cognitive overload] Unintuitive functionality: “unclear indications of how to scroll through stack of images” [UH: naturalness; efficient interaction; minimize cognitive overload] Unclear navigation pathway “hard to follow and move around” [UH: naturalness; efficient interaction; minimize cognitive overload] Waste of information resource: “too many icons” [UH: simplicity; efficient interaction; naturalness] App

App 6 Unintuitive information presentation: “trouble finding menu items” [UH: efficient interaction; minimize cognitive overload] Lack of information: “couldn’t locate images for submission” [UH: efficient interaction; minimize cognitive overload] Lack of information: “no theoretical information that it was not easy to access information” [UH: efficient interaction; minimize cognitive overload] App 7 Unintuitive functionality: “no content under interactive cases which should be,” “interactive case wasn’t interactive; couldn’t get the interactive case to work” [UH: efficient interaction; minimize cognitive overload; naturalness] Unintuitive functionality: “hard to figure out where to tap for menu item (ie, text or image)” [UH: efficient interaction; minimize cognitive overload; naturalness] Unclear navigation pathway: “No easy way to return to home screen” [UH: efficient

Representative Usability Issues Identified by Task Analysis Unintuitive functionality (eg, goal of dotted scale bar with “- and þ” signs at the both ends were not clear that users kept tapping it trying to zoom the images; it turned out that there was another zoom function available located at the bottom of the app) [UH: naturalness; efficient interaction; minimized cognitive overload] Lack of information (eg, one page graphic instruction was convenient; users kept tapping for further information but not available) [UH: efficient interaction; effective information presentation] Unclear navigation pathway (eg, users had to keep scrolling up and down to locate the information due to the lack of the search box) [UH: naturalness; efficient interaction; minimize cognitive overload] Inconsistent information display (eg, during the use of the app, the orientation changes without warning and there was no way switching back to the original orientation) [UH: forgiveness and feedback, efficient interaction; naturalness; consistency] Illogical location of information (eg, users had to move back and forth to reach the help that was located a level below it and was not visible on the main page of the app; eg, help was located within the “index” and users had to make extra interaction to access the help) [UH: naturalness; efficient interaction; consistency; preservation of context] Illogical location of information (eg, search function was available a level down from the main page within the “index” section, so users had to make extra interaction to access the help) [UH: naturalness; efficient interaction; consistency; preservation of context] Waste of information resource (app logo in the main page shrinks when tapped without giving any further information or response; many users tried in vain) [UH: naturalness; simplicity; minimize cognitive overload] Unintuitive information presentation (eg, content list was not logically organized) [UH: efficient interaction; minimize cognitive overload; naturalness] Lack of the functionality (eg, unable to zoom the image using fingers unlike images in other apps where the users could) [UH: efficient interaction; naturalness] Lack of functionality (eg, submitting a case was very cumbersome due to the lack of a search function that users had to scroll down entire list to find the item they desire to enter) [UH: efficient interaction; naturalness] Unintuitive functionality (eg, interactive case was not functional that users kept tapping to see if the image was interactive) [UH: efficient interaction; minimize cognitive overload; naturalness] Unintuitive functionality (eg, search box does not function properly; once the search was made the term was highlighted in the app, but wouldn’t go away when reset button was clicked; had to return to the home page to reset) [UH: naturalness; efficient interaction; consistency; preservation of context] Unintuitive information presentation (eg, inconsistent coloring of the text made the interface confusing; inactive text were highlighted with blue color that users kept tapping the text to see if they were hyperlinks) [UH: naturalness; efficient interaction; consistency] (continued)

340

Journal of the American College of Radiology Volume 13 n Number 3 n March 2016

Table 4. Continued Quotations From Self-Reported Usability Issues Representative Usability Issues Identified by Task Analysis Illogical location of information (eg, instruction to use the app is located interaction; minimize cognitive overload; outside the app [on the web] and it takes extra step to access naturalness] information) [UH: naturalness; efficient interaction; consistency; Unclear navigation pathway/unintuitive preservation of context] functionality: “difficult to follow; cumbersome” [UH: efficient interaction; minimize cognitive overload; naturalness] App 8 Unintuitive functionality: “finding how to scroll Unintuitive functionality (use of 3-D head model to scroll the images was too sensitive that users kept passing the desired image of the images; ending drawing function” [UH: task) [UH: efficient interaction; minimize cognitive overload efficient interaction; minimize cognitive Inconsistent information display (eg, both active and inactive functions overload; naturalness] have white text that users kept pushing the text, buttons to see if Unintuitive functionality: “couldn’t find out how they function) [UH: efficient interaction; minimize cognitive overload] to annotate slides” [UH: efficient interaction; minimize cognitive overload App

Note: The issues are coded by themes and mapped to the related usability heuristics (UHs).

heuristics. Note that there are overlapping usability heuristics across the usability issues because usability issues are explained by multiple usability attributes. Because of space limitations and the self-explanatory nature of Table 4, we select one example usability issue for discussion in the text. A usability issue that was frequently observed was inefficient interaction due to nonintuitive or misleading labels (text, buttons, icons, or images). For example, when assessing app 5, a physician struggled to find the information needed to complete the tasks (finding the right case). He scrolled or tapped the inactive buttons hoping to move up to the higher level but gave up after several unsuccessful trials.

n

Three physicians commented on consistency in the operation of the platform device (ie, iPad) to help reduce the learning curve because understanding how one system works helps users understand how other systems work. To maintain consistency, information elements such as terminology, data entry fields, and information layout such as graphic design and screen elements must have the similar look and feel. n

n

Suggested Usability Improvement During the debriefing session, physicians suggested usability improvements, which were well aligned with the usability issues identified from the self-report and video analysis. Four physicians emphasized the importance of efficient interaction of the apps. App functionality should be intuitive; cognitive burden is reduced when users interact with the content and task without thinking about the interface. Users should be able to achieve the intended goal with the minimum number of action steps. n

n

n

“Clear indication of buttons or areas to tap to enter a submenu; keep original menu item visible when opening submenus; arrow keys for scrolling in a stack in addition to finger drag.” “Things which require long clicks/double clicks are less intuitive.” “Intuitive touching is a big plus.”

“I prefer apps with dark backgrounds to have images stand out.”

n

“Stick to standardized iPad multi-touch gestures. Scrolling by dragging a separate grid is not intuitive.” “Universally understood signs help for navigating through the app.” “I always want to touch the image to scroll on it.”

Simplicity of an app was another usability attribute noted by three physicians, whereas another physician mentioned the role of redundant functionality. This suggests that a balance should be struck between simplicity and complexity of information use. Only information, visual elements, and functionality necessary to core tasks and decisions should be included and must be easily noticeable. On the other hand, an alternative way of completing a task should be available. n n n

“I like apps that are simple to use, no complexity.” “Simplicity is good.” “Redundancy in ways to perform the same tasks is helpful.”

Physicians always wanted more (interactive) information contents, such as case images and instructive

Journal of the American College of Radiology Leadership n Kim et al n Mobile Apps Usability in Diagnostic Decision Making

341

materials, because the goal of the mobile apps evaluated in this study was “education and training” for the radiologists. n

n n

“Instruction list would be helpful; while easiest to use but it used only anatomy on that; no pathology or cases.” “I like apps with lots of images.” “Cases that give all imaging modalities are best; T1, TL, STIR, DWI etc.”

DISCUSSION Directions for Future Mobile App Development There are no systematic and standard usability guidelines for mobile medical apps. Many scientific studies recommended that health care apps undergo rigorous testing and evaluation of their design and functionality before their use by targeted end users. For example, the Microsoft Development Center introduced 10 description guidelines to maintain the quality of an app’s description section [27]. Apple Developer also suggested iOS human interface guidelines [28]. However, as their names suggest, these are just guidelines, not mandated or regulated. On the other hand, vendors and markets have been using star ratings to describe overall user experience of the mobile apps. However, these rating scales have not been scientifically evaluated for their effectiveness in measuring the usability of the apps. Recently, the FDA published a report, Mobile Medical Applications Guidance for Industry and Food and Drug Administration Staff, on the safety and effectiveness of medical devices, including mobile medical apps [29]. Although encouraging, the guidance does not discuss usability issues either. Further usability studies and systematic usability guidelines for medical mobile apps for clinical training will ensure userfriendly design of these apps and increase satisfaction among clinician users. A user-centered design (UCD) process may ultimately lead to the development of medical mobile apps that are well adopted by their target users [30]. To reflect real-life experiences of health care IT, UCD involves stakeholders such as patients and providers from conceptualization to implementation in a systematic and continuous manner. UCD involves validated usability evaluation methods such as expert evaluation (eg, heuristic evaluation) and user testing. Because user engagement is UCD’s top priority, throughout the design process, users’ needs and wants and the limitations of a product are given extensive attention. Even after implementation, extended usability evaluation is warranted to maintain user acceptance and 342

for effective education. The UCD process will enable the development of usable mobile apps in radiology education and training and may ultimately increase the quality of health care services.

Limitations There were limitations in this study. First, because of its investigative nature, this study involved a small sample size, which may have limited representativeness compared with a study in a larger setting. Second, the selection of the iOS platform and iPad device for this study was made on the basis of current market trends, which may change in the future. As such, the evaluation of mobile apps for other platforms, such as Windows or BlackBerry, is warranted. Third, the majority of mobile apps in radiology have been evaluated on tablet platforms because of the larger display size. However, improvement of display technology and mobility will enable growing use of radiology apps on smaller devices such as smart phones.

CONCLUSIONS We explored usability issues and have suggested potential improvements in the development of radiology mobile apps. Mixed methods revealed more than 100 usability issues, suggesting an urgent need for systematic usability evaluation to improve user acceptance and maintaining maximal educational effectiveness. Dissatisfaction resulting from usability issues may cause physicians to resist using mobile apps for their education or training. The results of this study may be communicated to vendors to assist in the design of mobile education and training apps by highlighting the areas of difficulty radiologists are currently facing. These results could also inform the research and clinical community as evidence-based guidelines are created to help select mobile apps that will yield maximal educational and clinical benefits. TAKE-HOME POINTS n

Mixed-methods usability evaluation incorporating quantitative performance measures, the SUS, and qualitative thematic analysis using heuristic usability principles of usability issues was an effective way of measuring users’ experience with radiology mobile apps, finding more than 100 usability issues with just six physician subjects. Journal of the American College of Radiology Volume 13 n Number 3 n March 2016

n

n

As the correlation coefficient suggests, user performance (task success rate) and physicians’ perceived usability (SUS score) had a weak positive relation, which may indicate the importance of considering triangular systematic usability measures rather than relying on single usability measure. Proficiency with a mobile device may not be equivalent to proficiency in using a particular mobile app.

ACKNOWLEDGMENTS The authors thank the subjects for their participation in this study. They also thank Awatef Ben-Ramadan for help in data collection. REFERENCES 1. Sinha S. The advent of digital health. StrategyþBusiness. Available at: http://www.strategy-business.com/blog/The-Advent-of-Digital-Health? gko¼f2f63. Accessed August 17, 2015. 2. PocketGamer.biz. Application category distribution. Available at: http://www.pocketgamer.biz/metrics/app-store/categories/. Accessed February 18, 2015. 3. AppBrain.com. Most popular Google Play categories. Available at: http://www.appbrain.com/stats/android-market-app-categories. Accessed February 18, 2015. 4. Szekely A, Talanow R, Bagyi P. Smartphones, tablets and mobile applications for radiology. Eur J Radiol 2013;82:829-36. 5. MIM Software. Mobile MIM. Available at: Mobile MIM; http://www. mimsoftware.com/products/mobile/. Accessed August 17, 2015. 6. Barrientos M JM. Radiology Ray x. Available at: https://itunes.apple.com/ am/app/radiology-ray-x/id374083450?mt¼8. Accessed August 17, 2015. 7. BestApps BV. Radiology Assistant for iPad. Available at: https://itunes. apple.com/us/app/radiology-assistant-for-ipad/id532906807?mt¼8. Accessed August 17, 2015. 8. iAtros Software. eRoentgen Radiology DX. Available at: http:// download.cnet.com/eRoentgen-Radiology-DX/3000-2129_4-750231 04.html. Accessed August 17, 2015. 9. Radiological Society of North America. Radiology. Available at: https://itunes.apple.com/us/app/radiology-radiological-society/id741 007884?mt¼8. Accessed August 17, 2015. 10. International Organization for Standardization. ISO 9241-11:1998: ergonomic requirements for office work with visual display terminals (VDTs)— part 11: guidance on usability. Available at: http://www.iso.org/iso/ catalogue_detail.htm?csnumber¼16883. Accessed August 17, 2015. 11. Harrison F, Duce. Usability of mobile applications: literature review and rationale for a new usability model. J Interact Sci 2013;1:12. 12. Mocherman A. Why 95% of apps are quickly abandoned—and how to avoid becoming a statistic. Available at: http://whatsnext.nuance. com/customer-experience/95-apps-quickly-abandoned-avoid-becomingstatistic/. Accessed August 17, 2015.

13. Localytics. 26% of apps downloaded in 2010 were used just once. Available at: http://info.localytics.com/blog/first-impressions-26-percentof-apps-downloaded-used-just-once. Accessed August 17, 2015. 14. Tang PC, Patel VL. Major issues in user interface design for health professional workstations: summary and recommendations. Int J Biomed Comput 1994;34:139-48. 15. Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assoc 2004;11: 104-12. 16. Beuscart-Zéphir MC, Elkin P, Pelayo S, et al. The human factors engineering approach to biomedical informatics projects: state of the art, results, benefits and challenges. Yearb Med Inform 2007: 109-27. 17. Liu C, Zhu Q, Holroyd KA, et al. Status and trends of mobile-health applications for iOS devices: a developer’s perspective. J Syst Software 2011;84:2022-33. 18. Ridley EL. Size matters: iPad tops iPhone for evaluating acute stroke. Available at: http://www.auntminnie.com/index.aspx?sec¼ser&sub¼def& pag¼dis&ItemID¼97731. Accessed August 17, 2015. 19. Lewis JR. Evaluation of procedures for adjusting problem-discovery rates estimated from small samples. Int J Hum Comput Interact 1994;13:445-79. 20. Faulkner L. Beyond the five-user assumption: benefits of increased sample sizes in usability testing. Behav Res Methods Instrum Comput 2003;35:379-83. 21. Kim M, Shapiro J, Genes N, et al. A pilot study on usability analysis of emergency department information system by nurses. Appl Clin Inform 2012;3:135-53. 22. Clarke MA, Belden JL, Kim MS. Determining differences in user performance between expert and novice primary care doctors when using an electronic health record (EHR). J Eval Clin Pract 2014;20: 1153-61. 23. Lewis JR, Sauro J. The factor structure of the System Usability Scale. In: Proceedings of the 1st International Conference on Human Centered Design: Held as Part of HCI International 2009; San Diego, CA. New York: Springer-Verlag; 2009:94-103. 24. Albert W, Tullis T. Measuring the user experience: collecting, analyzing, and presenting usability metrics. New York: Newnes; 2013. 25. Cordes RE. The effects of running fewer subjects on time-on-task measures. Int J Hum Comput Interact 1993;5:393-403. 26. mHIMSS App Usability Work Group. Selecting a mobile app: evaluating the usability of medical applications. Available at: http://www.himss.org/ ResourceLibrary/genResourceDetailPDF.aspx?ItemNumber¼28900. Accessed August 17, 2015. 27. Microsoft. Your app’s description. Available at: https://msdn. microsoft.com/en-us/library/windows/apps/Hh694076.aspx. Accessed August 17, 2015. 28. Apple. iOS human interface guidelines: designing for iOS. Available at: https://developer.apple.com/library/ios/documentation/UserExperience/ Conceptual/MobileHIG/. Accessed August 17, 2015. 29. US Food and Drug Administration. Mobile medical applications: guidance for industry and Food and Drug Administration staff. Available at: http://www.fda.gov/downloads/MedicalDevices/.../UCM263366. pdf. Accessed August 17, 2015. 30. Johnson CM, Johnson TR, Zhang J. A user-centered framework for redesigning health care interfaces. J Biomed Inform 2005;38: 75-87.

Journal of the American College of Radiology Leadership n Kim et al n Mobile Apps Usability in Diagnostic Decision Making

343