Integrating computerized clinical decision support ... - Semantic Scholar

International Journal of Medical Informatics 84 (2015) 1009–1018

Contents lists available at ScienceDirect

International Journal of Medical Informatics journal homepage: www.ijmijournal.com

Integrating computerized clinical decision support systems into clinical work: A meta-synthesis of qualitative research Anne Miller a,c,∗ , Brian Moon b , Shilo Anders a , Rachel Walden a , Steven Brown a,c , Diane Montella a,c a

Vanderbilt University Medical Center, United States Perigean Technologies, United States c U.S Department of Veteran’s Affairs, United States b

a r t i c l e

i n f o

Article history: Received 12 January 2015 Received in revised form 4 September 2015 Accepted 11 September 2015 Keywords: Computerized clinical decision support Workflow integration Clinical decision making Meta-synthesis Qualitative research Human–system interaction Human–computer interaction Usability

a b s t r a c t Purpose: Computerized clinical decision support systems (CDSS) are an emerging means for improving healthcare safety, quality and efficiency, but meta-analyses findings are mixed. This meta-synthesis aggregates qualitative research findings as possible explanations for variable quantitative research outcomes. Inclusion criteria: Qualitative studies published between 2000 and 2013 in English, involving physicians, registered and advanced practice nurses’ experience of CDSS use in clinical practice were included. Search strategy: PubMed and CINAHL databases were searched. Study titles and abstracts were screened against inclusion criteria. Retained studies were appraised against quality criteria. Findings were extracted iteratively from studies in the 4th quartile of quality scores. Two reviewers constructed themes inductively. A third reviewer applied the defined themes deductively achieving 92% agreement. Results: 3798 unique records were returned; 56 met inclusion criteria and were reviewed against quality criteria. 9 studies were of sufficiently high quality for synthetic analysis. Five major themes (clinicianpatient-system integration; user interface usability; the need for better ‘algorithms’; system maturity; patient safety) were defined. Conclusions: Despite ongoing development, CDSS remains an emerging technology. Lack of understanding about and lack of consideration for the interaction between human decision makers and CDSS is a major reason for poor system adoption and use. Further high-quality qualitative research is needed to better understand human—system interaction issues. These issues may continue to confound quantitative study results if not addressed. © 2015 Published by Elsevier Ireland Ltd.

1. Introduction The United States’ Health Information Technology and Clinical Health (HITECH) Act of 2009 was designed to increase the adoption of electronic health records (EHR). EHRs better integrate patients’ healthcare information, and provide a means for building regional and national databases that inform policy and evidencebased practices that improve patient safety, care delivery quality and efficiency [1–4]. Integrated into EHRs, computerized clinical decision support systems (CDSS) are intended to influence clinical decisions and improve the quality of care processes such as ensuring that the criteria for ordering medications (dose, route, absence of contra-indications, allergies or drug interactions) are

∗ Corresponding author at: Vanderbilt University Medical Center, United States. Fax: +1 615 936 7373. E-mail address: [email protected] (A. Miller). http://dx.doi.org/10.1016/j.ijmedinf.2015.09.005 1386-5056/© 2015 Published by Elsevier Ireland Ltd.

met. For this reason CDSSs were included as part of the Office of the National Coordinator’s (ONCs) EHR certification requirements under the Meaningful Use 2 phase of HITECH implementation. However, evidence for CDSS role in improving safety, quality and efficiency has been mixed. For example, Garg et al. [5] found that although a majority of systematically reviewed controlled trials showed improved practitioner performance with CDSS, only 13% improved patient outcomes. More recently, Nuckols et al. [6] systematically reviewed 16 pre-post and quasi-experimental studies of computerized physician order entry systems (CPOE, with and without CDSS), and found no differences in the incidence of preventable adverse medication events. Similarly, in an updated version of an earlier meta-analysis of computerized laboratory monitoring alerts [8], Bayoumi et al. [7] found no reductions in adverse drug events or lengths of hospital stay despite behavioral changed in accordance with the alerts. Similar mixed results are evident in systematic reviews and/or meta-analyses addressing

1010

A. Miller et al. / International Journal of Medical Informatics 84 (2015) 1009–1018

drug dosing, [9] diabetes management, [10] and comprehensive general reviews of CDSS effects [11]. Reviewers’ highlight methodological flaws in research protocols as a means for improving research outcomes, [5,12] but few alternative explanations or research directions are provided. Mixed-methods research combines qualitative and quantitative methods [13]. In the early stages of new initiative development, qualitative methods build contextual knowledge that can suggest interventions, inform research directions and elucidate potentially confounding variables. Used during and following a research intervention, qualitative methods can provide possible explanations for intended and unintended effects that may not have been predictable prior to the intervention [13,14]. The purpose of this meta-synthesis was to conduct an interpretive synthesis of highquality qualitative research to provide alternative explanations for the variation evident in quantitative CDSS research findings. 2. Meta-syntheses Meta-synthesis is an emerging approach for interpreting the aggregate findings of qualitative research studies [15,16]. Major and Savin-Baden [15] propose that “synthesists seek to answer a specific research question through combining qualitative studies. .. that are located in broadly the same tradition, in order to make sense of themes and issues.. .” (p.10). In contrast to a comprehensive literature review, whose purpose is to critically locate a research study within a body of knowledge, a meta-synthesis interprets related findings within the synthesist’s research question using a structured and highly systematic approach [16,17]. Our guiding research question was: ‘What are the possible reasons and causes from healthcare clinicians’ perspectives, for difficulties in integrating CDSS into clinical work?’ We also sought to better understand the contexts into which CDSS technology has been introduced, including the nature of affected decisions and the types of CDSS being implemented. We found no evidence in the Joanna Briggs Institute Library of Systematic Reviews, The Cochrane Library including the DARE database, MEDLINE or the PROSPERO databases that a meta-synthesis associated with CDSS integration in clinical work had been undertaken. 3. Method The sections included in this method are those recommended for high quality meta-syntheses by the Joanna Briggs Institute (JBI) [18]. 3.1. Inclusion criteria 3.1.1. Participants and context The participants in the reviewed studies were clinicians who provided direct care to patients in in-patient or out-patient care settings. Thus, CDSS studies were selected for meta-synthesis review if the study participants included licensed physicians, board-registered nurses (RNs) and advanced practice nurses (NPs). CDSS studies involving only patients or allied health workers, such as pharmacists, were not included. 3.1.2. Phenomenon of interest The phenomena of interest to this meta-synthesis were findings of participants’ experiences using CDSS in clinical practice. That is, we were most interested in the direct inspection of clinicians’ use of CDSS in clinical work. Berner and Lande [19] define CDSS as computer modules, functions, features or systems designed to affect clinicians’ decision making about individual patients at the time these decisions are

to be made. Osheroff et al. [20] propose that CDSS systems present knowledge and patient-related facts, relationships, best practices or new knowledge that has been filtered and presented at an appropriate time to enhance patient care. Table 1 lists the types of applications recognized as CDSS in this meta-synthesis. The CDSS types are based on definitions published by Osheroff et al. [20] and typically coexist with EHRs and CPOEs systems. 3.1.3. Types of studies Phenomenological, grounded theory, ethnography, action research and studies that described the experience or effects of CDSS on participants’ practice in actual clinical settings were included in this meta-synthesis. Surveys without open-ended questions or a ‘free-text’ component and simulations including usability studies were excluded. 3.2. Search strategy An experienced librarian (RW), in collaboration with the primary reviewer (AM) conducted a three-phase literature search. In Phase 1, keywords were generated from our research question constructs, (e.g., CDSS in health care; clinicians’ experience of CDSS adoption). The keywords were then extended using peer reviewed journal articles’ titles; abstracts; keywords and textbook subject indices. In Phase 2; PubMed and CINAHL databases were searched to eliminate the return of CDSS studies from non-healthcare industries. Our specific search strategy is given in Appendix A. Only studies published in English language journals were included. The literature searched was limited to the years 2000–2013 to cover the most contemporary CDSS implementations. In Phase 3; we reviewed the bibliographic reference lists of all studies included in the qualitative synthesis to determine whether additional relevant studies had been overlooked. 3.3. Method of review Studies returned from the database search were reviewed in two iterations. Three reviewers (AM, BM, SA) conducted the first iteration independently. This iteration was limited to a title and abstract review to determine whether the study met inclusion criteria. Each reviewer provided a short reason for excluding a study. These reasons were collated inductively as a post-hoc analysis of excluded CDSS records. Studies meeting inclusion criteria were evaluated for methodological quality using the appraisal tool presented in Appendix B. This tool expands the JBI’s Qualitative Assessment and Review Instrument (QARI) [20] using more explicitly defined qualitative research dimensions (credibility, transferability, dependability and confirmability) and quality criteria [15,21]. Using these criteria, each study was given a score out of a total of 40. Studies with scores in the top 25% of study scores (i.e., the 4th quartile) were included for detailed analysis and results synthesis. We believed that this cut-of criterion achieved a sufficient number of studies for meaningful results while maintaining acceptable levels of quality. 3.3.1. Summarizing study methods and extracting findings Two reviewers (BM, AM) independently read and systematically summarized each of the retained studies using the following JBI defined dimensions [20]: 1. Method—the way or ways the data was collected and how it was used; 2. Phenomena of interest—the primary focus of the study; 3. Setting—the specific location or context in which the study was conducted including its geographical and/or cultural context;


1011

Table 1 Types of CDSS application included in this meta-synthesis. Type of CDSS application

Definition

Smart documentation forms Order sets and multi-step protocols Parameter guidance

Computerized forms that support the complete documentation of care.

Critiques and warnings

Relevant data summaries Multi-patient monitors

Predictive and retrospective analytics Filtered reference information and knowledge resources Expert workup and management advisors Event driven alerts and reminders

Computerized orders or protocols that promote adherence to agreed care policies, best practices and/or care pathways. Proactive interventions that help to ensure parameters such as drug doses and frequencies, diagnostic (laboratory test results) and therapeutic constraints are complete and correct. Reactive immediate alert interventions that prompt clinician assessments of possible errors, hazards or quality improvement opportunities related to new data/orders entered into information systems. Concise presentations of important data about a patient’s circumstances that ensure that all pertinent data are noted and considered. These summaries organize complex data and highlight needed actions. Relevant patient summaries for groups, or lists of patients that highlight critical information that assists in decision making about urgency or priority in real time. The presentation of population data and application of models and algorithms to data over time that can be used to risk-stratify patients or raise awareness about clinical hazards or opportunities. Summaries of reference information from textbooks, consensus opinions and other references, or resources such as nomograms, tables, flow-charts or calculators. Applications that use patient data from the EHR to suggest diagnoses, tests or procedures that might clarify diagnoses or treatment options. Provide active alerts to events that either have (e.g., test result is abnormal test, a new admission has arrived) or have not occurred (e.g., preventative screening test has not occurred according to recommended schedule)

4. Participants—including their demographics and how they were recruited and selected; 5. Data analysis—the techniques used to analyze the data. The two reviewers independently extracted each of the included study authors’ main findings together with participant quotations, field observations or other supporting evidence. Findings were themes or conclusions supported by evidence presented in the included studies. The reviewers compared their respective lists of findings. Common findings were identified. Disagreements about whether a finding existed were resolved by re-reading the relevant study to verify that a finding and its evidence was present and not overlooked or misinterpreted. Once findings had been extracted they were rated as: • Unequivocal—the finding is clearly supported by data (e.g., quotations, direct observations) and is thus not open to challenge; • Credible—the finding and presented data lack a clear association and are thus open to challenge, or • Unsupported—the finding is not supported by data [20]. 3.3.2. Data synthesis The reviewers (AM, BM) independently re-read the studies to ensure that they had captured the study authors’ interpretative context. They then inductively grouped the findings based on ‘conceptual similarity’ to achieve the first synthetic iteration. Following this iteration, the reviewers compared their thematic groups and the content within each theme. A finding could be allocated to more than one thematic group. A second synthetic iteration focused on resolving dissimilarities. The reviewers independently regrouped dissimilarly grouped findings. They again compared their outcomes and met to resolve disagreements. Findings that had initially been allocated to multiple thematic groups were either divided to multiple separate constructs, or resolved to one theme based on ‘best fit’ consensus. Group themes and subcategories were defined and refined as part of this process. The original list of ungrouped findings and the theme and category definitions were submitted to a third reviewer (SA), who was

asked to group the findings deductively using the themes and categories defined by the first two reviewers. The third reviewer’s groups were compared to the first reviewers’ groups. Dissimilarities were discussed and the theme and category definitions were refined to better clarify their meaning. The overall findings were submitted to a forth reviewer (DM) for an assessment of their comprehensibility and congruence. 4. Results 4.1. Preliminary search and screening Fig. 1 presents the search results using a PRISMA flow diagram [22] and shows that of 3798 unique records, 30% were excluded because they presented an aspect of CDSS under development, that is, a system not deployed in a clinical setting. Examples included, algorithm, feature or system developments. An additional 24% of excluded studies involved EHRs and CPOE systems that did not include a CDSS component; 12% were patient-centered CDSS, such as treatment adherence reminders, and 10% did not include qualitative data. The remaining 24% of records also failed to meet minimum inclusion criteria for the reasons presented in Fig. 1. Table 2 shows that the number of records (N) returned increased across the study period, however, the proportions of records under each deletion category remain remarkably consistent. For example the records in rows 1 and 2 account for approximately 50% of records across all years and were related to CDSS and/or EHR/CPOE development. The number of records returned for patient-centered CDSS appears to decrease, while the number of quantitative research studies appears to increase. 4.2. Evaluation against quality criteria In total, 56 full-text studies were assessed as eligible for evaluation against quality criteria (Appendix B). Of these, 59% of the CDSS applications were event driven alerts as defined in Table 1; 19.5% were order sets or multi-step protocols; 9% were expert workup; 3.5% of studies were smart-documentation; 3.5% were

1012


Fig. 1. PRISMA flow diagram of search outcomes. Table 2 Percentages of deleted records with deletion reason over the study period. Year Deletion reason N 1. Development 2. Not CDSS system 3. Patient-centered CDSS 4. Not qualitative 5. Opinion 6. Not healthcare related 7. Implementation 8. Education 9. Policy

"

00 95 33 26 19 0 13 6 2 1 0

" 01 104 36 16 25 2 6 13 2 0 0

" 02 144 32 20 17 2 12 11 4 0 0

" 03 172 33 17 23 2 10 9 7 0 0

" 04 200 30 14 21 4 11 9 12 0 0

filtered references; 3.5% involved parameter guidance and 1.7% included predictive analytics or practice guidelines. We note that there is evidence of historical development across the CDSS types; that earlier implementations may seem less sophisticated compared to more contemporary examples. Osheroff et al’s taxonomy is also more representative of current than past implementations. These percentages should therefore be considered as generally indicative rather than definitive. The evaluation criteria assessed four dimensions of qualitative research quality (credibility, transferability, dependability and confirmability) that are broadly analogous to internal validity, external validity, reliability and concurrent validity, respectively. The combined sample (N = 56) quality score statistics were: mean and standard deviation of 22 and 7.2, respectively; median of 21, range of 10–38 and interquartile range of 15.5–27. Studies with scores in the forth quartile (i.e., >27) were included in meta-synthesis. The forth quartile quality score cut-of for inclusions was considered an acceptable balance between a suitable number of studies for meta-synthesis at a sufficiently high level of quality. Table 3 shows the summary statistics and t-test results for the quality scores of excluded and included studies. The included studies had higher mean quality scores compared to the excluded studies. The most significant differences were associated with dependability and confirmability scores. For dependability, excluded studies typically included none or only one of ‘information attesting to inter-coder reliability’, ‘coder independence’, ‘conflict resolution processes’ and ‘information about

" 05 215 33 15 21 6 9 8 7 0 0

" 06 268 30 24 16 3 10 9 7 1 0

" 07 262 25 28 16 4 6 16 6 0 0

" 08 331 30 30 5 11 11 5 4 3 1

" 09 313 29 33 6 11 8 7 4 0 3

" 10 357 27 31 5 17 9 7 4 0 0

" 11 384 28 26 7 14 11 5 5 2 3

" 12 459 29 27 11 14 8 6 2 2 1

" 13 448 30 16 10 18 10 8 5 1 2

how data saturation was determined’. For confirmability, excluded studies did not show attempts to triangulate data collection, and researcher transparency, including researcher reflexivity processes, were not apparent. 4.3. Synthetic analysis Table 4 summarizes the studies included in synthetic analysis. All but one (Dowding et al. [23]), were undertaken in the United States, and in all but one (Feldstein et al. [24]), data collection was undertaken between 2006 and 2010. All but two studies (Campion et al. [25,26]) were undertaken in outpatient care settings. Inpatient environments were under-represented. Six studies included prescribing clinicians (physicians, nurse practitioners) and three studies [23,25,26] included registered nurses. The studies reflect a diverse set of commercial [27–29], government [23,30,31], and in-house [24–26] developed products. The majority (6) of the CDSS applications are event-based alerts (see Table 1) supporting medication prescribing decisions and processes. Other CDSS applications include multi-step protocols involving dosing calculators [23,25,26]. Methodologically, ethnographic non-participant observation (usually accompanied by semi-structured key informant interviews) was the most common data collection approach. One study [29] included observations with focus groups, and a second study [24] included observations with an artifact analysis. The studies’ foci were consistent, with seven studies focused on

1013

A. Miller et al. / International Journal of Medical Informatics 84 (2015) 1009–1018 Table 3 Summary statistics for excluded and included studies based on quality criteria (Appendix B). Criteria scores max score = 40

Credibility max score = 12

Transferability max score = 12

Dependability max score = 10

Confirmability max score = 6

Excluded studies Mean (Std Dev) Median (range)

N = 47 8 (1.7) 8 (4–11)

7 (2.6) 7 (2–12)

2 (2.3) 2 (0–8)

2 (1.0) 2 (0–4)

Included studies Mean (Std Dev) Median (range) T-test

N=9 9 (1.6) 9 (8–12) P < 0.003

11 (1.6) 11.5 (2–12) P < 0002

7 (2.0) 8 (4–10) P< 0.0001

5 (1.6) 6 (2–6) P < 0.0001

the objective use and effects of CDSS applications on medication ordering processes. Only one study focused specifically on clinicians’ decision-making [23] and only one [24] focused on social interactions and emotional responses to CDSS. All but three studies used inductive or grounded theory thematic content analysis. One study [30] used a theory-based deductive analytical framework and two used pre-determined coding frameworks based on the operationalization of primary research questions [28,29]. In total, 81 study findings were extracted. Of these, 54 (67%) findings were rated as unequivocal, 15 (18.5%) were rated as credible, and 12 (15%) were unsupported by evidence. Table 5 summarizes the themes and subthemes emerging from the meta-synthesis together with the number of rated findings in each subtheme. Reviewers achieved 92% agreement about the allocation of findings to the five themes and subthemes that emerged from the synthesis. Each theme is discussed in more detail below. 4.3.1. Usability issues Usability issues were associated with the presentation of information in a user interface, users’ needs for that information, and/or with the interaction between users and user interface elements. The findings were divided into three main subthemes. Nuisance issues included alerts that compelled documentation compliance [26]; poor fit between alerts and actual patient need (i.e., poor prioritization [24], inappropriate signal to noise [26]), repetitive and redundant alerts [25–27,31], inappropriate alert presentation within a process where alerts became distractions that predispose to missing or forgetting information [30,31]. User control and inefficiency issues involved users’ inability to actively manage alerts by delaying or retrieving them at a later more opportune time [30,31], or by turning them off especially when redundant [24]. Data entry associated with alerts was found to be rigid and inflexible often using repetitive data entry modes that were thought to be overengineered or ‘mouse-heavy’ [25,27,29–31]. Alerts also failed to adequately inform or were unintelligible. Some alerts were written in language more appropriate to specialists such as pharmacists than to prescribing providers [30]. Defaults were poorly selected [26]. Risk or priority levels were missing or unclear. Providers preferred the use of categories (high, moderate, low) or ‘traffic light’ indicators rather than statistical probabilities which some found difficult to interpret [31]. Alerts often did not identify the alerting trigger or the means for resolving it, [31] and presented scales or language that was inconsistent with clinical practice [23]. 4.3.2. Clinician-patient-system integration Poor clinician-patient-system integration issues involved CDSS integration within the socio-technical system beyond just the technological system. The ability to extract ‘meaning’ as an emergent outcome of integrating the CDSS advisories, the patient’s clinical situation and the clinician’s own knowledge and experience, was a core construct within this theme. The clinician-patientsystem integration theme was composed of two subthemes: (1) clinician-system integration involved congruence between clinician cognition (judgment, problem solving) and CDSS logic in

real-world contexts, and (2) patient-system integration involved the congruence between CDSS and other patient data including free text comments. Examples of clinician-system integration issues included: reliance on clinicians to integrate CDSS and information across different EHR modules, such as medication prescription and laboratory test ordering [26]; CDSS requirements for immediate action despite the need for further information or consultation [30]; and the interaction between clinician experience/expertise and CDSS effects, for example, on junior clinicians’ trust in system recommendations [27]. Dowding et al. [23] and Campion et al. [26] also found that nurses became highly familiar with CDSS algorithms and modified data entry in anticipation of system responses. Data entry modification may be based on the nurses’ knowledge of the patient, and thus had a customizing or tailoring effect, or may be based on concern that system recommendations were in some way inconsistent with the clinical situation. These nurses also attempted to calibrate their knowledge of algorithm relationships by attempting to predict algorithm outcomes [23,26]. Patient-system integration issues involved lack of processes for information gathering that would more completely represent the patient. For example, patients’ medication histories may be incomplete [28], or CDSS outputs may not be presented with or accommodate other patient-relevant information such as trends [23]. 4.3.3. The need for ‘better algorithms’ CDSS algorithms themselves were found to be lacking in maturity. Algorithms, for example, may not fully accommodate a work task or problem. In one study, insulin/dextrose dose calculators did not appear to accommodate other clinical information that nurses believed to be important in managing emergent hypo- or hyper-glycemic events [26]. Algorithms may not accommodate more complex patient scenarios. Examples included: CDSS failed to recognize that patients can be appropriately prescribed more than one medication in the same class (e.g., asthma inhalers) resulting in confusing and/or multiple alerts [31]; the failure of alerts to distinguish between allergies and side-effects [31]; inadequate algorithm referencing and inadequate differentiation between actual and theoretical risk [31]. Researchers also found that alerts often lack or did not accommodate contextual information. Insurance algorithms, for example, may not accommodate a patient’s actual income or provide different cost options for similar medications [28]. Alerts may also make inappropriate assumptions about provider knowledge of the content domain or of the patient and often needed to be interpreted [23,25,31]. 4.3.4. System immaturity System immaturity issues were associated with broader platform interoperability (e.g., information flow between infusion pumps and EHRs) [26], and between different departments or healthcare agencies [29]. Other issues included system response times, the requirements for multiple logins, and general system ‘crashes’.

1014


Table 4 Profiles of Studies included in synthetic analysis. Study

Sample

Setting

Context

Data collection

Data analysis

Abramson et al. [27]

All 19 physicians working at least 75% and 2 days /week in the setting

United States. Academic, hospital-based ambulatory internal medicine practice. February–November 2009

2–3 h observations of physician-patient interactions focused on work-flow & physician efficiency when ordering. Semi-structured interviews focused on experiences with new system, struggles compared to old system, CDSS features, UI.

Inductive, grounded theory approach to thematic content analysis from electronic transcripts

Crosson et al. [28]

Purposive sampling. 8 practices that included: 18 physicians; 46 practice staff.

United States. Physician owned family medicine, internal medicine, obstetrics & gynecology outpatient clinics. Prior to December 2010

Ethnographic field observations and semi-structured interviews 3-months pre- and post-system implementation Observation and interview guided focus: ambulatory meaningful use stage 1 e-prescribing: drug-drug, drug-allergy checks with maintained active medication & allergy list and drug-formulary checks

Deductive coding using a before-after template focused on formulary and benefit and medication history. Electronic transcripts

Russ et al. [30]

Snowball sampling. 4 prescribers (MDs/NPs) each from 5 primary care clinics and 2 prescribers each from 8 specialty clinics (N = 16).

United States. VA Medical Center clinics: primary care, anticoagulation, infectious diseases, transplant, psychiatry, nephron, endocrine, gastro & oncology. August 2008–2009

Half-day observations focused on prescribers’ interactions with alerts, understanding how alerts were integrated into patient care, and system factors that influence prescriber-alert interactions. Structured interviews focused on how medication alerts help and hinder clinical work and what changes should be made.

Deductive coding using Norman’s Mental Model Framework. Electronic transcripts

Campion et al. [25]

Convenient sample 25 RN volunteers

United States. Surgical and Trauma Intensive Care Units in an academic tertiary care facility February 2010 – March 2010

Retrospective chart review for incidence of over-rides and subsequent effects on patients’ blood sugar levels 2 3-h ethnographic observations with unstructured interviews focused on the interactions between people, processes and technology related to insulin CDSS.

Inductive, grounded theory approach to thematic content analysis. Electronic transcripts

Campion et al. [26]

Convenient sample 37 RN volunteers

United States Surgical and Trauma Intensive Care Units in an academic tertiary care facility February 2010–March 2010

Ethnography 2–3-h ethnographic observations with unstructured interviews focused on the interactions between people, processes and technology related to computerized intensive insulin therapy in critical care units.

Inductive, grounded theory approach to thematic content analysis. Electronic transcripts

Lapane et al. [29]

Purposive sample: 64 ambulatory care practices more than 25% Medicare patients. 177 physicians & 99 office staff

45% of practices were internal medicine; 39% family medicine, and 25% of practices were single practitioner. April-August 2006

9 months following replacement of in-house EHR with commercial EHR with CDSS support for drug–drug & drug-allergy interactions during prescribing. CDSS = event-driven alerts Two commercial e-prescribing applications sponsored by BCBS; both included automated formulary-insurance benefit checks and medication history documentation with drug–drug and drug-allergy alerts. CDSS = event-driven alerts The VA’s EHR system includes CPOE with medication alerts for drug–drug and drug-allergy interactions. Alerts require a response from a prescriber, a rationale for some over-rides including free-text justifications, a 2-tiered severity rating and can present multiple alerts on one screen. CDSS = event-driven alerts An Intensive insulin therapy protocol computerized and implemented in 2004–2005 involved nurses entering a patient’s blood sugar level. CSD calculates either an insulin dose or a dose of 10% dextrose. CDSS = multi-step protocol An Intensive insulin therapy protocol computerized and implemented in 2004-2005 involved nurses entering a patient’s blood sugar level. CSD calculates either an insulin dose or a dose of 10% dextrose. CDSS = multi-step protocol Commercial e-prescribing applications that included automated formulary—authorization checks. CDSS = event-based alerts

Preliminary site visit interviews and observations informed survey design focused on the effects of e-prescribing on work efficiency compared to other methods (e.g., paper, fax etc,). Group interviews focused on e-prescribing usability, implementation barriers, patient safety.

Deductive codes framed and defined by research foci. Electronic transcripts

1015

A. Miller et al. / International Journal of Medical Informatics 84 (2015) 1009–1018 Table 4 (Continued) Study

Sample

Setting

Context

Data collection

Data analysis

Dowding et al. [23]

Purposive sampling. 4 case sites included. Participants included 115 RN/NPs

United Kingdom. From NHS Trust database: 1. Anticoagulation clinic; 2. RN-led spinal assessment; 3. Walk-in clinic triage; 4. COPD home management June 2006–June 2007

Ethnographic observations of nurse only & nurse-patient consults focused on use of CDSSS during RN/NPs, patient interactions. Post-observation interviews’ (N = 37) on the nature of decision tasks, adequacy of information, deficiencies. In-depth interviews (N = 18) focused on use of technology to inform decisions, usefulness, effects on patient interactions

Inductive, thematic content analysis. Electronic transcripts

Russ et al. [31]

Convenient sample. 12 physicians, 4 NPs & 4 pharmacists

United States. 5 VA Medical Center primary care clinics, August 2008–March 2009

Ethnographic observations focused on medication ordering and alert use. Semi-structured interviews focused on factors that help and hinder work and any suggested changes.

Inductive thematic content analysis Electronic transcripts

Feldstein et al. [24]

Stratified, random sample (every 10th physicians, physician assistants, or NPs); in 12 clinics

United States 12 primary care clinics Prior to 2004.

The sites include commercially developed CDSSS implementations: Anticoagulation dose calculator based on INR Smartform and quality of life score calculator Smartform diagnostic algorithm for level of care Home monitoring alerts using patient entered data The VA’s EHR system includes CPOE with automated medication alerts for drug–drug and drug-allergy interactions. CDSS = event-driven alerts Intrusive CPOE alerts including drug cost alerts, safety or side-effects; medication alternatives, laboratory /radiology reminders by diagnosis; over due maintenance procedures. Access to evidence-based guidelines. CDSS = event-driven alerts

Semi-structured interviews focused on experiences with and education about alerts & reminders; response to medication safety alerts, emotional responses, attention to and perceived barriers to use. Artifacts: analyzed helpful & unhelpful alerts, layout and emotional response..

Inductive, thematic content analysis. Electronic transcripts and annotated screenshots of specific alerts.

Abbreviations: BCBS = blue cross, blue shield; VA = veteran’s affairs; RN = registered nurse, NP = nurse practitioners.

4.3.5. Patient safety Campion et al. [25,26] and Feldstein et al. [24] both found that alerts related to patient safety were perceived to be most helpful (and least annoying) when the domain represented by the alert was not the clinician’s area of expertise. Findings were, however, somewhat ambivalent. On the one hand, junior clinicians were thought to inappropriately accept CDSS recommendations without critical evaluation [29], but few providers recommended turning them off completely [24]. 5. Discussion The purpose of this meta-synthesis was to provide alternative explanations for the variation evident in quantitative CDSS research findings. Specifically, we sought to identify possible reasons and causes for difficulties in integrating CDSS into clinical work while also building a more integrated profile of the contexts into which CDSS has been applied and the nature of affected decisions. Foremost, we found that high quality studies of clinicians’ experience using CDSS were relatively rare, especially when compared to the number of quantitative research studies (Table 2). Although qualitative studies appear to be increasing, they still represent a marginal portion of the overall research output. This is concerning, as it suggests that CDSS implementations are being developed and implemented based on limited understanding of actual clinical work or CDSS effects on actual clinical decision making. This may explain some of the workflow integration issues associated with CDSS. The overall quality of qualitative CDSS studies is also lacking. Even some of the studies selected for synthesis had weaker dependability and confirmability scores and no mixed-methods research studies were eligible for synthetic analysis. Thus, there is consid-

erable room for improvement in the quality of qualitative research and its integration with mixed-method research approaches. Without more and better qualitative research that supports a richer understanding of clinical work contexts, the variations observed by meta-analysts [5–12] are likely to continue as contextual variables are omitted from consideration in CDSS development, and in quantitative research designs. Most qualitative CDSS studies involved alerts and reminders associated with medication prescription and administration. These applications tend to ensure that well-defined, criterion-based or transactional processes are completed comprehensively (e.g., drugallergy; drug–drug interactions are not omitted) and correctly (e.g., right dose range, right route). However these objectives often appear to be undermined by poor information and interaction design that affects the usability of alerts and reminders. Few CDSS studies, appear to accommodate naturalistic decision making, such as diagnostic pattern recognition [33] and situation awarness [34,35]. Additionally, most studies involved out-patient care settings; few studies involved in-patient environments. These findings are also concerning, because they suggest that major aspects of clinical decision making, and segments of the healthcare environment that HITECH is intended to improve, are woefully understudied. Our results suggest clear directions for future research. Most lacking is research related to the interaction between CDSS and clinical decision makers, both in terms of the decisions they make, and their adaptations to and unintended uses of CDSS. Especially intriguing are Dowding et al. [23] and Feldstein et al. [24] accounts of nurses’ adaptions to CDSS algorithms. It is likely that physicians engage in similar behaviors but there is little research to attest this. Research related to the extraction of clinical ‘meaning’ from

1016


Table 5 Summary of emergent themes and sub-themes. Theme

Subtheme * = numbers of rated findings

Usability: problems related specifically to the user interface design (its ‘look’) or interaction design (its behavior).

Nuisance: inappropriate or unnecessary interruption or distraction. Unequivocal = 5*; credible = 4; unsupported = 0

N = 35; 43% of total

Clinician-patient-system Integration: the capacity for information to flow freely between providers, systems and patients in ways that support joint problem solving, decision making, and the construction of meaning. N = 27; 33% of total

Algorithm immaturity: algorithms are incomplete, lack comprehensiveness, are not flexible or do not account for patient related contextual variables. N = 12; 15% of total

System immaturity: basic network or platform problems, in particular interoperability. N = 4; 5% of total Patient safety threats: specific threats that may cause harm to patients. N = 3; 4% of total

User control/inefficiency: requires additional work that is only marginally related to the task at hand; inability to move forward at the desired pace or in the desired way. Unequivocal = 7; credible = 3; unsupported = 4 Transparency/failure to inform: displays, processes or information presentations do not provide sufficient information upon which to make a decision, or complete a task. Unequivocal = 10; credible = 1; unsupported = 1 Clinician-system integration: CDSS is not congruent with clinicians’ expectations, problem solving strategies or workflow processes. Unequivocal = 17*; credible = 2; unsupported = 2

Patient-system integration: the CDSS fits congruently with other representations of the patient and the patient’s condition. unequivocal = 5; credible = 0; unsupported = 1 Algorithm maturity: Algorithm itself is not complete with respect to the variables included in total work tasks or relevant clinical problem. Unequivocal = 5; credible = 3; unsupported = 0

Lack of context awareness: the algorithm does not take account of contextualizing or personalizing patient factors. Unequivocal = 3; credible = 1; unsupported = 0 Unequivocal = 2; credible = 0; unsupported = 2

Unequivocal = 1; credible = 1; unsupported = 1

diverse information sources may also lead to fruitful insights about the presentation of CDSS related information. As confirmed in Table 2, the technical knowledge needed to build and deliver CDSS-related technologies continues to evolve [19,20,32]. However, knowledge about the integration of CDSS into real-world, socio-technical environments is barely emerging. This conclusion should not be surprising. Despite that CDSS and other advanced computational capacities have developed since World War 2, the transition of these technologies into complex sociotechnical systems is relatively recent, especially on scales as large as healthcare [19,32]. Thus knowledge about their effects in realworld environments is just beginning to emerge. The results of this synthesis together with the mixed results of many metaanalyses [5–12] suggest that research approaches that encompass human-system interactions in real-world care settings need to go hand-in-hand with basic development if the promises of CDSS are to be fully realized. We believe that the Office of the National Coordinator’s inclusion of CDSS and user-centered design in its EHR certification requirements is an important first step in this direction. In conclusion, the results of this meta-synthesis suggest a need for high-quality research to better understand the interactions between computational and human reasoning and problem solving in real-world clinical settings, including how clinicians respond and adapt to computational interventions over time. Research is also needed to better understand how CDSS interventions could be presented to human decision makers (patients and clinicians) in user interfaces. Research programs that encompass these issues

will require a far greater collaboration between social, behavioral, cognitive and computer scientists, not just in the design of better computer systems but also in the design of better human-computer interactions. 5.1. Limitations This meta-synthesis is limited by our search criteria, which deliberately focused on problems and barriers to CDSS adoption. However, by clearly defining our search question we were better able to consolidate targeted studies in ways that more effectively support conclusions and future directions. Facilitators and enablers of CDSS adoption were not covered in this meta-synthesis and should be addressed in future meta-syntheses. Our findings are also limited by the databases we searched. It is possible that some studies, especially in the grey literature exist that were not included in this analysis and that if included would change our results and conclusions. Meta-synthesis processes are still emerging with few examples in the published literature especially related to health care informatics, and hence there may be flaws in our method or approach that we are as yet unaware of. It is hoped that this example provides an impetus for further qualitative syntheses in healthcare informatics. Conflict of interest The authors confirm that there is no conflict of interest.

1017


Summary points What was already known:

Search terms

Search results

#1

10873

• Evidence for Clinical Decision Support (CDS) systems’ role in improving patient safety, quality and efficiency has been mixed. • Other than improving research protocols, few alternative explanations or research directions have been provided that would explain the variability in CDS outcomes. • Qualitative research can provide possible alternative explanations but qualitative CDS studies have not been systematically evaluated. What this study has added:

#2

• There are few high quality qualitative research studies, which is concerning as CDS may be implemented with incomplete contextual knowledge. • Although CDS technological capabilities are advanced, knowledge about how to integrated CDS into clinical decision making is still emerging. • Usability, human computer interaction design and poor human-systems integration issues may be confounding CDS randomized control trials. • Mixed qualitative and quantitative research approaches may result more effective CDS implementations and outcomes. #3 #4

Author contributions Anne Miller led and coordinated the project. She participated in conceiving the research questions, developed the research design, coordinated other team members’ efforts, collated and analyzed the data, extracted findings, and drafted the manuscript. Brian Moon and Shilo Anders analyzed the data, extracted findings, participated in thematic development and edited the drafted manuscript. Rachel Walden developed the search strategy, undertook the manuscript search and contributed to the drafted manuscript Steven Brown participated in conceiving the research question and developing the research design. He reviewed the projects findings and edited the manuscript. Diane Montella participated in conceiving, refining and focusing the project’s research question. She participated in developing the search strategy, reviewed the projects findings and participated in editing the manuscript.

#5

(“Decision support systems, clinical”[mh] or “medical order entry systems”[mesh] or “CPOE”[tiab] or “reminder systems”[mh] or “clinical alarms”[mh] or (“decision support”[tiab] and (“electronic health records”[Mesh] or “electronic health records”[tiab] or “electronic medical records”[tiab] or “algorithms”[Mesh] or algorithm*[tiab] or computerized[tiab] or clinical[tiab] or “computer-based”[tiab] or “machine-based”[tiab] or system[tiab] OR systems[tiab] or tool[tiab] or tools[tiab] or alert[tiab] or alerts[tiab] or reminders[tiab] or reminder[tiab] or warning[tiab] or warnings[tiab]))) (barrier[tiab] or barriers[tiab] or challenge[tiab] or challenges[tiab] or failure[tiab] or failures[tiab] or problem[tiab] or problems[tiab] or problematic[tiab] or difficulties[tiab] or “user resistance” or resisted[tiab] or “alarm fatigue”[tiab] or “alert fatigue”[tiab] or usability[tiab] or “negative response”[tiab] or “negative impact”[tiab] or attitudes[tiab] or concerns[tiab] or hinders[tiab] orhindering[tiab] or limited[tiab] OR slowed[tiab] or failed[tiab] or failure[tiab] or failures[tiab] or fail[tiab] or fails[tiab] or unsuccessful[tiab] or “lack of motivation”[tiab] or “lessons learned”[tiab] or “user experience”[tiab] or “cooperative behavior”[mh] or “attitude to computers”[mh] or “attitude of health personnel”[Mesh] or “adverse effects” [subheading] or “adverse events”[tiab] or “adverse effects”[tiab]) #1 and #2 #3 not (“case reports”[Publication Type] OR “Review”[Publication Type] or “Editorial”[Publication Type] or “Comment”[Publication Type] or “News”[Publication Type] or “Newspaper Article”[Publication Type] or “Letter” [Publication Type] or “Practice Guideline” [Publication Type] or “Historical Article” [Publication Type]) #4 and English[la] and (“2000/01/01”[PDAT]: “3000/12/31”[PDAT])

Critical appraisal scoring Appraisal category

Category elements

Credibility

Does the study include an explicit aim or research question? Is the method appropriate to the aim or research question? Are the Researchers familiar with the content domain? Does the research have Ethics/IRB committee approval? Does the procedure include Peer scrutiny? Is the data analysis method consistent with the aim or research question? Has the research setting been described? How were participants recruited? Are inclusion and exclusion criteria explicitly stated? Is the number of participants included stated? Are the number and length of data collection sessions documented? Are data collection dates given? Was intercoder reliability addressed? Was data coding independent? Are coding conflict resolution processes described? Are data production and processing processes described? Was data saturation achieved? Was data collection triangulated? Are all procedures (recruitment, data collection and analysis) transparent? Are limitations discussed candidly?

Dependability

Confirmability

Search strategy PubMed search strategies (the same strategies were used to search CINAHL)

2837

Appendix B.

Transferability

Appendix A.

4047 3351

Key: [tiab] appears in the title or abstract; [la] language; [mh] medical subject heading; [pdat] date of publication

Acknowledgements This project was funded by the United States Department of Veteran’s Affairs. It was conceived as part of a broader program of research coordinated by the Knowledge Based System’s Group within the Department of Veteran’s Affair’s Office of Informatics and Analytics.

3747616

Each study was given a score: 0 = not addressed; 1 = addressed weakly; 2 = addressed strongly for each category element, for a total possible score of 40.

1018


References [1] D. Blumenthal, Launching HITECH, New Engl. J. Med. 362 (2010) 382–385. [2] D. Blumenthal, M. Tavenner, The meaningful use regulation for electronic health records, New Engl. J. Med. 363 (2010) 501–504. [3] A. Jha, C.M. DesRoches, E.G. Campbell, D. Donelan, S.R. Rao, T.G. Ferris, A. Shields, S. Rosenbaum, D. Blumenthal, Use of electronic health records in US hospitals, New Engl. J. Med. 360 (2009) 1628–1638. [4] C.M. DesRoches, E.G. Campbell, S.R. Rao, K. Donelan, T.G. Ferris, A. Jha, R. Kaushal, D.E. Levy, S. Rosenbaum, A.E. Shields, D. Blumenthal, Electronic health records in ambulatory care: a national survey of physicians, New Engl. J. Med. 359 (2008) 50–60. [5] A. Garg, N. Adhikari, H. McDonald, M. Rosas-Arellano, P. Devereaux, J. Beyene, J. Sam, R. Haynes, Effects of computerized clinical decision support systems on practitioner performance and patient outcomes, J. Am. Med. Assoc. 293 (10) (2005) 1223–1238. [6] T.K. Nuckols, C. Smith-Spangler, S.C. Morton, S.M. Asch, V.M. Patel, L.J. Anderson, E.L. Deichsel, P.G. Shekelle, The effectiveness of computerized order entry at reducing preventable adverse drub events and medication errors in hospital settings: a systematic review and meta-analysis, Syst. Rev. 3 (2014) 56, http://dx.doi.org/10.1186/2046-4053-3-56. [7] I. Bayoumi, M.A. Balas, S.M. Handler, L. Dolovich, B. Hutchison, A. Holbrook, The effectiveness of computerized drug-lab alerts: a systematic review and meta-analysis, Int. J. Med. Inform. 83 (2014) 406–415. [8] K.A. McKibbon, C. Lokker, S.M. handler, L.R. Dolovich, A.M. Holbrook, D. O’Reilly, R. Tamblyn, B.J. Hemens, B. Runki, S. Troyan, P.S. Roshanov, The effectiveness of integrated health information technologies across the phases of medication management: a systematic review of randomized controlled trials, J. Am. Med. Inform. Assoc. 19 (2012) 22–30. [9] F. Gillaizeau, E. Chan, L. Trinquart, I. Colombet, R.T. Walton, M. Rege-Walther, B. Burnand, P. Durieux, Computerized advice on drug dosage to improve prescribing practice, Cochrane Database Syst. Rev. 2013 (2013) 11, http://dx. doi.org/10.1002/14651858.CD002994.pub3, Art.No.: CD002894. [10] R. Jeffery, E. Iserman, R.B. Haynes, the CDSSS Systematic Review Team, Can computerized clinical decision support systems improve diabetes management? A systematic review and meta-analysis, Diabet. Med. 30 (2012) 739–745. [11] T.J. Bright, A. Wong, R. Dhurjati, E. Bristow, L. Bastian, R.R. Coeytaux, G. Samsa, V. Hasselblad, J.W. Williams, M.D. Musty, L. Wing, A.S. Kendrick, G.D. Sanders, D. Lobach, Effect of clinical decision support systems: a systematic Review, Ann. Intern. Med. 157 (2012) 29–43. [12] E. Ammenwerth, P. Schnell-Inderst, C. Machan, U. Siebert, The effect of electronic prescribing on medication errors and adverse drug events: a systematic review, J. Am. Med. Inform. Assoc. 15 (5) (2008) 586–600. [13] J.W. Creswell, V.L. Plano Clark, Designing and Conducting Mixed Methods Research, 2nd ed., Sage Publications, Thousand Oaks, CA, 2011. [14] A. Steckler, K.R. McLeroy, R.M. Goodman, S.T. Bird, L. McCormick, Integrating qualitative and quantitative methods: an introduction? Health Educ. Q. 19 (1) (1992) 1–8. [15] C.H. Major, M. Savin-Baden, An Introduction to Qualitative Research Synthesis: Managing the Information Explosion in Social Science Research, Routledge, Oxon, UK, 2010. [16] S. Thorne, L. Jensen, M.H. Kearney, G. Noblit, M. Sadelowski, Qualitative metasynthesis: reflections on methodological orientation and ideological agenda, Qual. Health Res. 14 (2004) 1342–1365.

[17] M. Sandelowski, Meta-jeopardy: the crisis of representation in qualitative metasynthesis, Nurs. Outlook 54 (1) (2006) 10–16. [18] Joanna Briggs Institute, Reviewers’ Manual, 2014 Edition, The Joanna Briggs Institute, Adelaide, Australia, 2014. [19] E.S. Berner, T.J. La Lande, Clinical Decision Support Systems: Theory and Practice, 2nd ed., Springer, New York, NY, 2007. [20] J.A. Osheroff, J.M. Teich, D. Levick, L. Saldana, F.T. Velasco, D.F. Sittig, K.M. Rogers, R.A. Jenders, Improving Outcomes with Clinical Decision Support: An Implementer’s Guide, 2nd ed., HIMSS, Chicago, IL, 2012. [21] M.B. Miles, A.M. Huberman, J. Saldana, Qualitative Data analysis: A methods sourcebook, 3rd ed., Sage Publications, Thousand Oaks, CA, 2014. [22] D. Moher, A. Liberati, J. Tetzlaff, D.G. Altman, The PRISMA Group, preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement, Ann. Intern. Med. 151 (2009). [23] D. Dowding, N. Mitchell, R. Randell, R. Foster, V. Lattimer, C. Thompson, Nurses’ use of computerized clinical decision support systems: a case site analysis, J. Clin. Nurs. 18 (2009) 1159–1167. [24] A. Feldstein, S.R. Simon, J. Schneider, M. Krall, D. Laferriere, D.H. Smith, D.F. Sittig, S.B. Soumerai, How to design computerized alerts to ensure safe prescribing practices, Joint Comm. J. Qual. Saf. 30 (11) (2004) 602–613. [25] T.R. Campion, A.K. May, L.R. Waitman, A. Ozdas, N.M. Lorenzi, C.S. Gadd, Characteristics and effects of nurse dosing over-rides on computer-based intensive insulin therapy protocol performance, J. Am. Med. Inform. Assoc. 18 (2011) 251–258. [26] T.R. Campion, L.R. Waitman, N.M. Lorenzi, A.K. May, C.S. Gadd, Barriers and facilitators to the use of computer-based intensive insulin therapy, Int. J. Med. Inform. 80 (2011) 863–871. [27] E.L. Abramson, V. Patel, S. Malhotra, W.R. Poh, S.N. Osorio, A. Cheriff, C.L. Cole, A. Bunce, J. Ash, Physician experiences transitioning between an older vesus newer electronic health record for electronic prescribing, Int. J. Med. Inform. 81 (2012) 539–548. [28] J.C. Crosson, A.J. Schueth, N. Isaacson, D.S. Bell, Early adopters of electronic prescribing struggle to make meaningful use of formulary checks and medication history documentation, J. Am. Board Fam. Med. 25 (2012) 24–32. [29] K.L. Lapane, R.K. Rosen, C. Dube, Perceptions of e-prescribing efficiencies and inefficiencies in ambulatory care, Int. J. Med. Inform. 80 (2011) 39–46. [30] A.L. Russ, A.J. Zillich, M.S. McManus, B.N. Doebbling, Prescribers’ interactions with medication alerts at the point of prescribing: a multi-method, in situ investigation of the human-computer interaction, Int. J. Med. Inform. 81 (2012) 232–243. [31] A.L. Russ, A.J. Zillich, M.S. McManus, B.N. Doebbling, J.J. Saleem, A human factors investigation of medication alerts: barriers to prescriber decision making and clinical workflow, Proceedings of the American Medical informatics Association Symposium (2009) 548–552. [32] M.A. Boden, Mind as Machine: A History of Cognitive Science, vols. 1 & 2, Oxford University Press, Oxford, UK, 2006. [33] P.M. Todd, G. Gigerenzer, the ABC Research Group, Ecological Rationality: Intelligence in the World, Oxford University Press, New York, NY, 2012. [34] G. Klein, Sources of Power: How people make decisions, The MIT Press, Cambridge MA, 1998. [35] M.R. Endsley, B. Bolte, D.G. Jones, Designing for Situation Awareness: An Approach to User-centered Design, Taylor & Francis, New York, NY, 2003.